ads.jobs.templates package

Submodules

ads.jobs.templates.container module

ads.jobs.templates.driver_notebook module

This module runs a Jupyter Python notebook with nbconvert and print the outputs. This is a driver script auto-generated by Oracle ADS.

The following environment variables are used: JOB_RUN_NOTEBOOK:

The relative path of the jupyter Python notebook to be executed.

NOTEBOOK_EXCLUDE_TAGS:

Optional, a list of tags serialized to JSON string. Notebook cells with one of the tags will be excluded from running.

NOTEBOOK_ENCODING:

Optional, the encoding for opening the notebook.

OUTPUT_URI:

Optional, object storage URI for saving files from the output directory.

class ads.jobs.templates.driver_notebook.ADSExecutePreprocessor(**kwargs: Any)

Bases: ExecutePreprocessor

Customized Execute Preprocessor for running notebook.

Initialize the preprocessor

Parameters:

exclude_tags (list, optional) – A list of cell tags, notebook cells with any of these cell tag will be skipped. Defaults to None.

preprocess_cell(cell, resources, *args, **kwargs)

Runs the notebook cell and print out the outputs

class ads.jobs.templates.driver_notebook.OCIHelper

Bases: object

static copy_outputs(output_dir: str, output_uri: dict) → None

Copies the output files to object storage bucket.

Parameters:

  • output_dir (str) – Path of the output directory containing files to be copied.

  • output_uri (str) – URI of the object storage path to store the output files.

static init_oci_client(client_class)

Initializes OCI client with API key or Resource Principal.

Parameters:

client_class – The class of OCI client to be initialized.

ads.jobs.templates.driver_notebook.run_driver() → None

Runs the driver to execute a notebook.

ads.jobs.templates.driver_notebook.run_notebook(notebook_path: str, working_dir: Optional[str] = None, exclude_tags: Optional[list] = None) → None

Runs a notebook

Parameters:

  • notebook_path (str) – The path of the notebook

  • working_dir (str, optional) – The working directory for running the notebook, by default None. If this is None, the same directory of the notebook_path will be used.

  • exclude_tags (list, optional) – Tags for excluding cells, by default None

ads.jobs.templates.driver_oci module

This module runs user code from git repository. The following environment variables are used: GIT_URL:

URL to the Git repository.

GIT_BRANCH:

Optional, the Git branch to checkout.

GIT_COMMIT:

Optional, the Git commit to checkout. By default, the most recent git commit will be checked out.

CODE_DIR:

Optional, the directory for saving the user code from Git repository. Defaults to “~/Code”

GIT_ENTRYPOINT:

Relative path to the entry script/module file in the Git repository.

ENTRY_FUNCTION:

Optional, function name in the entry script/module to be invoked. If this is not specified, the entry script will be run as Python script.

PYTHON_PATH:

Optional, additional paths to be added to sys.path for looking up modules and packages. The root of the Git repository will be added by default. Multiple paths can be separated by os.pathsep, which is colon(:) for Linux and Mac, semicolon(;) for Windows.

OUTPUT_DIR:

Optional, output directory to be copied to object storage.

OUTPUT_URI:

Optional, object storage URI for saving files from the output directory.

GIT_SECRET_OCID:

The OCID of the OCI vault secret storing the SSH key for Git commands.

SKIP_METADATA_UPDATE:

If this variable exists, the update metadata step will be skipped.

OCI_IAM_TYPE:

Authentication method for OCI services. OCI API key will be used if this is set to api_key. Otherwise resource principal will be used.

OCI_CONFIG_LOCATION:

The location of OCI API key when OCI_IAM_TYPE is set to api_key. If this is not set, oci.config.DEFAULT_LOCATION will be used.

OCI_CONFIG_PROFILE:

The profile name to be used for API key authentication. If this is not set, oci.config.DEFAULT_PROFILE will be used.

OCI__GIT_SSH_KEY_PATH:

The location to save the SSH Key for accessing the git repository.

JOB_RUN_OCID:

The OCID of the job run. This is set by the job run.

This module requires the following packages: oci requests GitPython git openssh

class ads.jobs.templates.driver_oci.ArgumentParser(argument_list: list)

Bases: object

Contains methods for parsing arguments for entry function.

Initialize the parser with a list of arguments

Parameters:

argument_list (list) – A list of arguments.

static decode_arg(val)

Decodes the value of the argument if it is a JSON payload.

Parameters:

val (str) – The argument value in a string.

Returns:

None, if the val is None. String value, if the val is a string but not a JSON payload. Otherwise, the object after JSON decoded.

Return type:

Any

static join_values(value_list)

Joins the values of a keyword argument.

Parameters:

value_list (list) – Values in a list of strings.

Returns:

The value of the argument as a string.

Return type:

str or None

parse()

Parses the arguments

Returns:

A tuple of positional arguments (list) and keyword arguments (dict).

Return type:

(list, dict)

class ads.jobs.templates.driver_oci.CredentialManager

Bases: object

static init_oci_client(client_class)

Initializes OCI client with API key or Resource Principal.

Parameters:

client_class – The class of OCI client to be initialized.

static read_secret(secret_id)

Reads and decode the value of of a secret from OCI vault.

Parameters:

secret_id (str) – OCID of the secret

Returns:

The value of the secret decoded with ASCII.

Return type:

str

class ads.jobs.templates.driver_oci.GitManager(repo_url: str, code_dir: Optional[str] = None)

Bases: object

Contains methods for fetching code from Git repository

Initialize the GitManager

Parameters:

  • repo_url (str) – The URL of the repository.

  • code_dir (str) – The local directory for storing the code from Git repository.

Raises:

ValueError – URL is not specified.

fetch_repo()

Clones the Git repository.

setup_code(branch: Optional[str] = None, commit: Optional[str] = None)

Checkouts the branch or commit of the Git repository.

If neither branch nor commit is specified, the tip of the default branch will be used. If both branch and commit are specified, the commit will be used.

Parameters:

  • branch (str, optional) – The name of the branch, by default None

  • commit (str, optional) – The commit ID (SHA1 hash), by default None

class ads.jobs.templates.driver_oci.GitSSHKey(secret_id)

Bases: object

class ads.jobs.templates.driver_oci.JobRunner(job_run_ocid: str = '')

Bases: object

Contains methods for running the job.

Initialize the job runner

Parameters:

  • job_run_ocid (str, optional) – Job run OCID, by default “”.

  • purpose (For local testing) –

  • string (job run OCID can be set to empty) –

copy_artifacts(output_dir: str, output_uri: dict) → None

Copies the output files to object storage bucket.

Parameters:

  • bucket_info (dict) – Containing information for the object storage bucket. The dictionary should have the following keys: name: the name of the bucket. namespace: the namespace of the bucket. prefix: the prefix for saving the files. prefix is optional, defaults to Job run ID.

  • output_dir (str) – Path of the output directory containing files to be copied.

fetch_code()

Gets the source code from Git repository.

run(argv=None)

Runs the job

Parameters:

argv (list, optional) – A list of arguments for the entry script/function, by default None

static run_command(command)
save_metadata() → None

Saves the metadata to job run

static setup_python_path(python_path='', code_dir='')

Adds additional python paths.

update_job_run_metadata_with_rest_api(client: DataScienceClient, metadata: dict) → None

Updates the metadata of the job run by call OCI REST API.

Parameters:

  • client (DataScienceClient) – OCI DataScienceClient

  • metadata (dict) – Metadata to be saved as freeform tags.

ads.jobs.templates.driver_oci.check_internet()

Checks the internet connection by sending GET request to oracle.com

ads.jobs.templates.driver_oci.main()

The main function for running the job.

ads.jobs.templates.driver_python module

This is a driver script auto-generated by Oracle ADS to run Python script in OCI Data Science Jobs.

The following environment variables are used: CODE_ENTRYPOINT:

Relative path to the entry script/module file in the user code directory.

ENTRY_FUNCTION:

Optional, function name in the entry script/module to be invoked. If this is not specified, the entry script will be run as Python script.

PYTHON_PATH:

Optional, additional paths to be added to sys.path for looking up modules and packages. The root of the user code will be added by default. Multiple paths can be separated by os.pathsep, which is colon(:) for Linux and Mac, semicolon(;) for Windows.

OUTPUT_DIR:

Optional, output directory to be copied to object storage.

OUTPUT_URI:

Optional, object storage URI for saving files from the output directory.

WORKING_DIR:

Optional, the working directory of the user code. This can be specified as a relative path from /home/datascience/decompressed_artifact/code If this is not specified, the working directory will be /home/datascience/decompressed_artifact/code

This module requires the following package: oci

class ads.jobs.templates.driver_python.ArgumentParser(argument_list: list)

Bases: object

Contains methods for parsing arguments for entry function.

Initialize the parser with a list of arguments

Parameters:

argument_list (list) – A list of arguments.

static decode_arg(val)

Decodes the value of the argument if it is a JSON payload.

Parameters:

val (str) – The argument value in a string.

Returns:

None, if the val is None. String value, if the val is a string but not a JSON payload. Otherwise, the object after JSON decoded.

Return type:

Any

static join_values(value_list)

Joins the values of a keyword argument.

Parameters:

value_list (list) – Values in a list of strings.

Returns:

The value of the argument as a string.

Return type:

str or None

parse()

Parses the arguments

Returns:

A tuple of positional arguments (list) and keyword arguments (dict).

Return type:

(list, dict)

class ads.jobs.templates.driver_python.JobRunner(code_dir: str)

Bases: object

Contains methods for running the job.

Initialize the job runner

Parameters:

code_dir (str) – the path to the directory containing the user code.

run(entrypoint: str, entry_function: Optional[str] = None, argv: Optional[list] = None)

Runs the user code

Parameters:

  • entrypoint (str) – Path to the file serve as the entrypoint.

  • entry_function (str, optional) – Name of the function in the entrypoint, by default None. If this is not specified, the entrypoint will be run as a python script.

  • argv (list, optional) – Arguments to be parsed and passed into the entry_function, by default None. The arguments here are used only when running the entry_function. When entry_function is None, sys.argv[1:] will be used as command line argument.

setup_python_path(python_paths: str)

Adds additional python paths.

Parameters:

python_paths (str) – Additional python paths to be added to sys.path. Multiple paths can be separated by os.pathsep, which is colon(:) for Linux and Mac.

class ads.jobs.templates.driver_python.OCIHelper

Bases: object

static copy_outputs(output_dir: str, output_uri: dict) → None

Copies the output files to object storage bucket.

Parameters:

  • output_dir (str) – Path of the output directory containing files to be copied.

  • output_uri (str) – URI of the object storage path to store the output files.

static init_oci_client(client_class)

Initializes OCI client with API key or Resource Principal.

Parameters:

client_class – The class of OCI client to be initialized.

ads.jobs.templates.driver_python.main()

The main function for running the job.

Module contents