ads.text_dataset package

Submodules

ads.text_dataset.backends module

class ads.text_dataset.backends.Base

Bases: object

Base class for backends.

convert_to_text(fhandler: OpenFile, dst_path: str, fname: str | None = None, storage_options: Dict | None = None) → str

Convert input file to a text file

Parameters:

• fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

• dst_path (str) – local folder or cloud storage prefix to save converted text files

• fname (str, optional) – filename for converted output, relative to dirname or prefix, by default None

• storage_options (dict, optional) – storage options for cloud storage

Returns:

path to saved output

Return type:

str

get_metadata(fhandler: OpenFile) → Dict

Get metadata of a file.

Parameters:

fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

Returns:

dictionary of metadata

Return type:

dict

read_line(fhandler: OpenFile) → Generator[str | List[str], None, None]

Read lines from a file.

Parameters:

fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

Yields:

Generator – a generator that yields lines

read_text(fhandler: OpenFile) → Generator[str | List[str], None, None]

Read entire file into a string.

Parameters:

fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

Yields:

Generator – a generator that yields text in the file

ads.text_dataset.backends.OITCC()

class ads.text_dataset.backends.PDFPlumber

Bases: Base

convert_to_text(fhandler, dst_path, fname=None, storage_options=None)

Convert input file to a text file

Parameters:

• fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

• dst_path (str) – local folder or cloud storage prefix to save converted text files

• fname (str, optional) – filename for converted output, relative to dirname or prefix, by default None

• storage_options (dict, optional) – storage options for cloud storage

Returns:

path to saved output

Return type:

str

get_metadata(fhandler)

Get metadata of a file.

Parameters:

fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

Returns:

dictionary of metadata

Return type:

dict

read_line(fhandler)

Read lines from a file.

Parameters:

fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

Yields:

Generator – a generator that yields lines

read_text(fhandler)

Read entire file into a string.

Parameters:

fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

Yields:

Generator – a generator that yields text in the file

class ads.text_dataset.backends.Tika

Bases: Base

convert_to_text(fhandler, dst_path, fname=None, storage_options=None)

Convert input file to a text file

Parameters:

• fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

• dst_path (str) – local folder or cloud storage prefix to save converted text files

• fname (str, optional) – filename for converted output, relative to dirname or prefix, by default None

• storage_options (dict, optional) – storage options for cloud storage

Returns:

path to saved output

Return type:

str

detect_encoding(fhandler: OpenFile)

get_metadata(fhandler)

Get metadata of a file.

Parameters:

fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

Returns:

dictionary of metadata

Return type:

dict

read_line(fhandler)

Read lines from a file.

Parameters:

fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

Yields:

Generator – a generator that yields lines

read_text(fhandler)

Read entire file into a string.

Parameters:

fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

Yields:

Generator – a generator that yields text in the file

ads.text_dataset.dataset module

class ads.text_dataset.dataset.DataLoader(engine: str | None = None)

Bases: object

DataLoader binds engine, FileProcessor and File handler(in this case it is fsspec) together to produce a dataframe of parsed text from files.

This class is expected to be used mainly from TextDatasetFactory class.

processor

processor that is used for loading data.

Type:

ads.text_dataset.extractor.FileProcessor

Examples

>>> import oci
>>> from ads.text_dataset.dataset import TextDatasetFactory as textfactory
>>> from ads.text_dataset.options import Options
>>> df = textfactory.format('pdf').engine('pandas').read_line(
...     'oci://<bucket-name>@<namespace>/<path>/*.pdf',
...     storage_options={"config": oci.config.from_file(os.path.join("~/.oci", "config"))},
... )
>>> data_gen = textfactory.format('pdf').option(Options.FILE_NAME).backend('pdfplumber').read_text(
...     'oci://<bucket-name>@<namespace>/<path>/*.pdf',
...     storage_options={"config": oci.config.from_file(os.path.join("~/.oci", "config"))},
... )
>>> textfactory.format('docx').convert_to_text(
...     'oci://<bucket-name>@<namespace>/<path>/*.docx',
...     './extracted',
...     storage_options={"config": oci.config.from_file(os.path.join("~/.oci", "config"))},
... )
>>> textfactory.format('docx').convert_to_text(
...     'oci://<bucket-name>@<namespace>/<path>/*.docx',
...     'oci://<bucket-name>@<namespace>/<out_path>',
...     storage_options={"config": oci.config.from_file(os.path.join("~/.oci", "config"))},
... )
>>> meta_gen = textfactory.format('docx').metadata_schema(
...     'oci://<bucket-name>@<namespace>/papers/*.pdf',
...     storage_options={"config": oci.config.from_file(os.path.join("~/.oci", "config"))},
... )
>>> df = textfactory.format('pdf').engine('pandas').option(Options.FILE_METADATA, {'extract': ['Author']}).read_text(
...     'oci://<bucket-name>@<namespace>/<path>/*.pdf',
...     storage_options={"config": oci.config.from_file(os.path.join("~/.oci", "config"))},
...     total_files=10,
... )
>>> df = textfactory.format('txt').engine('cudf').read_line(
...     'oci://<bucket-name>@<namespace>/<path>/*.log',
...      udf=r'^\[(\S+)\s(\S+)\s(\d+)\s(\d+\:\d+\:\d+)\s(\d+)]\s(\S+)\s(\S+)\s(\S+)\s(\S+)',
...      df_args={"columns":["day", "month", "date", "time", "year", "type", "method", "status", "file"]},
...      n_lines_per_file=10,
... )

Initialize a DataLoader object.

Parameters:

engine (str, optional) – dataframe engine, by default None.

Return type:

None

backend(backend: str | Base) → None

Set backend used for extracting text from files.

Parameters:

backend ((str | ads.text_dataset.backends.Base)) – backend for extracting text from raw files.

Return type:

None

convert_to_text(src_path: str, dst_path: str, encoding: str = 'utf-8', storage_options: Dict | None = None) → None

Convert files to plain text files.

Parameters:

• src_path (str) – path to source data file(s). can use glob pattern

• dst_path (str) – local folder or cloud storage (e.g., OCI object storage) prefix to save converted text files

• encoding (str, optional) – encoding for files, by default utf-8

• storage_options (Dict, optional) – storage options for cloud storage, by default None

Return type:

None

engine(eng: str) → None

Set engine for dataloader. Can be pandas or cudf.

Parameters:

eng (str) – name of engine

Return type:

None

Raises:

NotSupportedError – raises error if engine passed in is not supported.

metadata_all(path: str, storage_options: Dict | None = None, encoding: str = 'utf-8') → Generator[Dict[str, Any], None, None]

Get metadata of all files that matches the given path. Return a generator.

Parameters:

• path (str) – path to data files. can use glob pattern.

• storage_options (Dict, optional) – storage options for cloud storage, by default None

• encoding (str, optional) – encoding of files, by default ‘utf-8’

Returns:

generator of extracted metedata from files.

Return type:

Generator

metadata_schema(path: str, n_files: int = 1, storage_options: Dict | None = None, encoding: str = 'utf-8') → List[str]

Get available fields in metadata by looking at the first n_files that matches the given path.

Parameters:

• path (str) – path to data files. can have glob pattern

• n_files (int, optional) – number of files to look up, default to be 1

• storage_options (dict, optional) – storage options for cloud storage, by default None

• encoding (str, optional) – encoding of files, by default utf-8

Returns:

list of available fields in metadata

Return type:

List[str]

option(opt: Options, spec: Any | None = None) → None

Set extraction options.

Parameters:

• opt (ads.text_dataset.options.Options) – an option defined in ads.text_dataset.options.Options

• spec (Any, optional) – specifications that will be passed to option handler, by default None

Return type:

None

read_line(path: str, udf: str | Callable = None, n_lines_per_file: int = None, total_lines: int = None, df_args: Dict = None, storage_options: Dict = None, encoding: str = 'utf-8') → Generator[str | List[str], None, None] | DataFrame

Read each file into lines. If path matches multiple files, will combine lines from all files.

Parameters:

• path (str) – path to data files. can have glob pattern.

• udf ((callable | str), optional) – user defined function for processing each line, can be a callable or regex, by default None

• n_lines_per_file (int, optional) – max number of lines read from each file, by default None

• total_lines (int, optional) – max number of lines read from all files, by default None

• df_args (dict, optional) – arguments passed to dataframe engine (e.g. pandas), by default None

• storage_options (dict, optional) – storage options for cloud storage, by default None

• encoding (str, optional) – encoding of files, by default ‘utf-8’

Returns:

returns either a data generator or a dataframe.

Return type:

(Generator | DataFrame)

read_text(path: str, udf: str | Callable = None, total_files: int = None, storage_options: Dict = None, df_args: Dict = None, encoding: str = 'utf-8') → Generator[str | List[str], None, None] | DataFrame

Read each file into a text string. If path matches multiple files, each file corresponds to one record.

Parameters:

• path (str) – path to data files. can have glob pattern.

• udf ((callable | str), optional) – user defined function for processing each line, can be a callable or regex, by default None

• total_files (int, optional) – max number of files to read, by default None

• df_args (dict, optional) – arguments passed to dataframe engine (e.g. pandas), by default None

• storage_options (dict, optional) – storage options for cloud storage, by default None

• encoding (str, optional) – encoding of files, by default ‘utf-8’

Returns:

returns either a data generator or a dataframe.

Return type:

(Generator | DataFrame)

with_processor(processor_type: str) → None

Set file processor.

Parameters:

processor_type (str) – type of processor, which corresponds to format of the file.

Return type:

None

class ads.text_dataset.dataset.TextDatasetFactory

Bases: object

A class that generates a dataloader given a file format.

static format(format_name: str) → DataLoader

Instantiates DataLoader class and seeds it with the right kind of FileProcessor. Eg. PDFProcessor for pdf. The FileProcessorFactory returns the processor based on the format Type.

Parameters:

format_name (str) – name of format

Returns:

a DataLoader object.

Return type:

ads.text_dataset.dataset.DataLoader

ads.text_dataset.extractor module

class ads.text_dataset.extractor.FileProcessor(backend: str | Base = 'default')

Bases: object

Base class for all the file processor. Files are opened using fsspec library. The default implementation in the base class assumes text files.

This class is expected to be used inside ads.text_dataset.dataset.DataLoader.

backend(backend: str | Base) → None

Set backend for file processor.

Parameters:

backend (ads.text_dataset.backends.Base) – a backend for file processor

Return type:

None

Raises:

NotSupportedError – when specified backend is not supported.

backend_map = {'default': <class 'ads.text_dataset.backends.Base'>, 'tika': <class 'ads.text_dataset.backends.Tika'>}

convert_to_text(fhandler: OpenFile, dst_path: str, fname: str | None = None, storage_options: Dict | None = None) → str

Convert input file to a text file.

Parameters:

• fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

• dst_path (str) – local folder or cloud storage (e.g. OCI object storage) prefix to save converted text files

• fname (str, optional) – filename for converted output, relative to dirname or prefix, by default None

• storage_options (dict, optional) – storage options for cloud storage, by default None

Returns:

path to saved output

Return type:

str

get_metadata(fhandler: OpenFile) → Dict

Get metadata of a file.

Parameters:

fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

Returns:

dictionary of metadata

Return type:

dict

read_line(fhandler: OpenFile, **format_reader_kwargs: Dict) → Generator[str | List[str], None, None]

Yields lines from a file.

Parameters:

fhandler (fsspec.core.OpenFile) – file handler returned by fsspec

Returns:

a generator that yields lines from a file

Return type:

Generator

read_text(fhandler: OpenFile, **format_reader_kwargs: Dict) → Generator[str | List[str], None, None]

Yield contents from the entire file.

Parameters:

fhandler (fsspec.core.OpenFile) – a file handler returned by fsspec

Returns:

a generator that yield text from a file

Return type:

Generator

class ads.text_dataset.extractor.FileProcessorFactory

Bases: object

Factory that manages all file processors. Provides functionality to get a processor corresponding to a given file type, or register custom processor for a specific file format.

Examples

>>> from ads.text_dataset.extractor import FileProcessor, FileProcessorFactory
>>> FileProcessorFactory.get_processor('pdf')
>>> class CustomProcessor(FileProcessor):
... # custom logic here
... pass
>>> FileProcessorFactory.register('new_format', CustomProcessor)

static get_processor(format)

processor_map = {'doc': <class 'ads.text_dataset.extractor.WordProcessor'>, 'docx': <class 'ads.text_dataset.extractor.WordProcessor'>, 'pdf': <class 'ads.text_dataset.extractor.PDFProcessor'>, 'txt': <class 'ads.text_dataset.extractor.FileProcessor'>}

classmethod register(fmt: str, processor: FileProcessor) → None

Register custom file processor for a file format.

Parameters:

• fmt (str) – file format

• processor (FileProcessor) – custom processor

Raises:

TypeError – raised when processor is not a subclass of FileProcessor.

class ads.text_dataset.extractor.PDFProcessor(backend: str | Base = 'default')

Bases: FileProcessor

Extracts text content from PDF

backend_map = {'default': <class 'ads.text_dataset.backends.Tika'>, 'pdfplumber': <class 'ads.text_dataset.backends.PDFPlumber'>, 'tika': <class 'ads.text_dataset.backends.Tika'>}

class ads.text_dataset.extractor.WordProcessor(backend: str | Base = 'default')

Bases: FileProcessor

Extracts text content from doc or docx format.

backend_map = {'default': <class 'ads.text_dataset.backends.Tika'>, 'tika': <class 'ads.text_dataset.backends.Tika'>}

ads.text_dataset.options module

class ads.text_dataset.options.FileOption(dataloader: ads.text_dataset.dataset.DataLoader)

Bases: OptionHandler

handle(fhandler: OpenFile, spec: Any) → Any

class ads.text_dataset.options.MetadataOption(dataloader: ads.text_dataset.dataset.DataLoader)

Bases: OptionHandler

handle(fhandler: OpenFile, spec: Dict) → List

class ads.text_dataset.options.OptionFactory

Bases: object

static option_handler(option: Options) → OptionHandler

option_handlers = {Options.FILE_METADATA: <class 'ads.text_dataset.options.MetadataOption'>, Options.FILE_NAME: <class 'ads.text_dataset.options.FileOption'>}

classmethod register_option(option: Options, handler) → None

class ads.text_dataset.options.OptionHandler(dataloader: ads.text_dataset.dataset.DataLoader)

Bases: object

handle(fhandler: OpenFile, spec: Any) → Any

class ads.text_dataset.options.Options(value)

Bases: Enum

An enumeration.

FILE_METADATA = 2

FILE_NAME = 1

ads.text_dataset.udfs module

class ads.text_dataset.udfs.UDF

Bases: object

static from_regex(regex: str) → Callable

ads.text_dataset.utils module

exception ads.text_dataset.utils.NotSupportedError

Bases: Exception

class ads.text_dataset.utils.PY4JGateway

Bases: object

ads.text_dataset.utils.experimental(cls)

Module contents