ads.jobs.builders.runtimes package

Submodules

ads.jobs.builders.runtimes.artifact module

class ads.jobs.builders.runtimes.artifact.Artifact(source, runtime=None)

Bases: object

Represents a OCI Data Science Job artifact. The Artifact class is designed to add an additional processing step on runtime/source code. before uploading it as data science job artifact.

A sub-class should implement the build() method to do the additional processing. A sub-class is designed to be used with context manager so that the temporary files are cleaned up properly.

For example, the NotebookArtifact implements the build() method to convert the notebook to python script. with NotebookArtifact(runtime) as artifact:

# The build() method will be called when entering the context manager # The final artifact for the job will be stored in artifact.path upload_artifact(artifact.path) # Files are cleaned up when exit or if there is an exception.

build()

Builds the runtime artifact in the temporary directory. Subclass should implement this method to: 1. Process the runtime 2. Set the self.path to the final artifact path

Raises:

NotImplementedError – When this method is not implemented in the subclass.

static copy_from_uri(uri, to_path, unpack=False, **storage_options)

Copy file(s) to local path

Parameters:

  • uri (URI of the file) – The URI of the source file or directory, which can be local path of OCI object storage URI.

  • to_path (path-like object) – The local destination path. If this is a directory, the source file/directory will be placed under it.

  • unpack (bool) – Indicate if zip or tar.gz file specified by the uri should be unpacked. This option has no effect on other files.

  • storage_options – Storage options for fsspec. For OCI object storage, the default_signer from ads.common.auth will be used if storage option is not specified.

Returns:

The actual path of file/directory at destination. For copying a single file and to_path is a filename, this will be the same as to_path. For copying a single file and to_path is a directory, this will be to_path + filename. For copying a directory, this will be to_path + directory name.

Return type:

str or path-like object

class ads.jobs.builders.runtimes.artifact.NotebookArtifact(source, runtime)

Bases: Artifact

Represents a NotebookRuntime job artifact

CONST_DRIVER_SCRIPT = 'driver_notebook.py'
build()

Prepares job artifact for notebook runtime

class ads.jobs.builders.runtimes.artifact.PythonArtifact(source, runtime=None)

Bases: Artifact

Represents a PythonRuntime job artifact

ARCHIVE_DIR = 'archive'
CONST_DRIVER_SCRIPT = 'driver_python.py'
USER_CODE_DIR = 'code'
build()

Prepares job artifact for script runtime. If the source is a file, it will be returned as is. If the source is a directory, it will be compressed as a zip file.

class ads.jobs.builders.runtimes.artifact.ScriptArtifact(source, runtime=None)

Bases: Artifact

Represents a ScriptRuntime job artifact

build()

Prepares job artifact for script runtime. If the source is a file, it will be returned as is. If the source is a directory, it will be compressed as a zip file.

ads.jobs.builders.runtimes.base module

class ads.jobs.builders.runtimes.base.Runtime(spec: Optional[Dict] = None, **kwargs)

Bases: Builder

Base class for job runtime

Initialize the object with specifications.

User can either pass in the specification as a dictionary or through keyword arguments.

Parameters:

  • spec (dict, optional) – Object specification, by default None

  • kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.

CONST_ARGS = 'args'
CONST_ENV_VAR = 'env'
CONST_MAXIMUM_RUNTIME_IN_MINUTES = 'maximumRuntimeInMinutes'
CONST_TAG = 'freeformTags'
property args: list

Command line arguments

attribute_map = {'env': 'env', 'freeformTags': 'freeform_tags'}
property environment_variables: dict

Environment variables

Returns:

The runtime environment variables. The returned dictionary is a copy.

Return type:

dict

property envs: dict

Environment variables

property freeform_tags: dict
property kind

Kind of the object to be stored in YAML. All runtimes will have “runtime” as kind. Subclass will have different types.

property maximum_runtime_in_minutes: int

Maximum runtime in minutes

property type: str

The type of the object as showing in YAML

with_argument(*args, **kwargs)

Adds command line arguments to the runtime. Existing arguments will be preserved. This method can be called (chained) multiple times to add various arguments. For example, runtime.with_argument(key=”val”).with_argument(“path/to/file”) will result in: “–key val path/to/file”

Parameters:

  • args – Positional arguments. In a single method call, positional arguments are always added before keyword arguments. You can call with_argument() to add positional arguments after keyword arguments.

  • kwargs – Keyword arguments. To add a keyword argument without value, set the value to None.

Returns:

This method returns self to support chaining methods.

Return type:

Runtime

Raises:

ValueError – Keyword arguments with space in a key.

with_environment_variable(**kwargs)

Sets environment variables

Returns:

This method returns self to support chaining methods.

Return type:

Runtime

with_freeform_tag(**kwargs)

Sets freeform tag

Returns:

This method returns self to support chaining methods.

Return type:

Runtime

with_maximum_runtime_in_minutes(maximum_runtime_in_minutes: int)

Sets maximum runtime in minutes

Returns:

This method returns self to support chaining methods.

Return type:

Runtime

ads.jobs.builders.runtimes.container_runtime module

class ads.jobs.builders.runtimes.container_runtime.ContainerRuntime(spec: Optional[Dict] = None, **kwargs)

Bases: Runtime

Represents a container job runtime

To define container runtime: >>> ContainerRuntime() >>> .with_image( >>> “iad.ocir.io/<your_tenancy>/<your_image>”, >>> entrypoint=[“/bin/sh”, -c], >>> cmd=”sleep 5 && echo Hello World”, >>> ) >>> .with_environment_variable(MY_ENV=”MY_VALUE”) or >>> ContainerRuntime() >>> .with_image(“iad.ocir.io/<your_tenancy>/<your_image>”) >>> .with_cmd(“sleep 5 && echo Hello World”) >>> .with_entrypoint([“/bin/sh”, “-c”]) >>> .with_environment_variable(MY_ENV=”MY_VALUE”)

Docker ENTRYPOINT and CMD can be either “exec form” or “shell form” (See references). The exec form is used when a list is passed in. The shell form is used when a space separated string is passed in.

When using the ContainerRuntime with OCI Data Science Job, the exec form is recommended. For most images, when the entrypoint is set to [“/bin/sh”, “-c”],

cmd can be a string as if you are running shell command.

Reference: https://docs.docker.com/engine/reference/builder/#entrypoint https://docs.docker.com/engine/reference/builder/#cmd

Initialize the object with specifications.

User can either pass in the specification as a dictionary or through keyword arguments.

Parameters:

  • spec (dict, optional) – Object specification, by default None

  • kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.

CONST_CMD = 'cmd'
CONST_ENTRYPOINT = 'entrypoint'
CONST_IMAGE = 'image'
attribute_map = {'cmd': 'cmd', 'entrypoint': 'entrypoint', 'env': 'env', 'freeformTags': 'freeform_tags', 'image': 'image'}
property cmd: str

Command of the container job

property entrypoint: str

Entrypoint of the container job

property image: str

The container image

with_cmd(cmd: str)

Specifies the command for the container job.

Parameters:

cmd (str) – Command for the container job

Returns:

The runtime instance.

Return type:

self

with_entrypoint(entrypoint: Union[str, list])

Specifies the entrypoint for the container job.

Parameters:

entrypoint (str or list) – Entrypoint for the container job

Returns:

The runtime instance.

Return type:

self

with_image(image: str, entrypoint: Optional[Union[str, list]] = None, cmd: Optional[str] = None)

Specify the image for the container job.

Parameters:

  • image (str) – The container image, e.g. iad.ocir.io/<your_tenancy>/<your_image>:<your_tag>

  • entrypoint (str or list, optional) – Entrypoint for the job, by default None (the entrypoint defined in the image will be used).

  • cmd (str, optional) – Command for the job, by default None.

Returns:

The runtime instance.

Return type:

self

ads.jobs.builders.runtimes.python_runtime module

class ads.jobs.builders.runtimes.python_runtime.CondaRuntime(spec: Optional[Dict] = None, **kwargs)

Bases: Runtime

Represents a job runtime with conda pack

Initialize the object with specifications.

User can either pass in the specification as a dictionary or through keyword arguments.

Parameters:

  • spec (dict, optional) – Object specification, by default None

  • kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.

CONST_CONDA = 'conda'
CONST_CONDA_REGION = 'region'
CONST_CONDA_SLUG = 'slug'
CONST_CONDA_TYPE = 'type'
CONST_CONDA_TYPE_CUSTOM = 'published'
CONST_CONDA_TYPE_SERVICE = 'service'
CONST_CONDA_URI = 'uri'
attribute_map = {'conda': 'conda', 'env': 'env', 'freeformTags': 'freeform_tags'}
property conda: dict

The conda pack specification

Returns:

A dictionary with “type” and “slug” as keys.

Return type:

dict

with_custom_conda(uri: str, region: Optional[str] = None)

Specifies the custom conda pack for running the job

Parameters:

  • uri (str) – The OCI object storage URI for the conda pack, e.g. “oci://your_bucket@namespace/object_name.” In the Environment Explorer of an OCI notebook session, this is shown as the “source” of the conda pack.

  • region (str, optional) –

    The region of the bucket storing the custom conda pack, by default None. If region is not specified, ADS will use the region from your authentication credentials, * For API Key, config[“region”] is used. * For Resource Principal, signer.region is used.

    This is required if the conda pack is stored in a different region.

Returns:

The runtime instance.

Return type:

self

See also

https

//docs.oracle.com/en-us/iaas/data-science/using/conda_publishs_object.htm

with_service_conda(slug: str)

Specifies the service conda pack for running the job

Parameters:

slug (str) – The slug name of the service conda pack

Returns:

The runtime instance.

Return type:

self

class ads.jobs.builders.runtimes.python_runtime.DataFlowNotebookRuntime(spec: Optional[Dict] = None, **kwargs)

Bases: DataFlowRuntime, NotebookRuntime

Initialize the object with specifications.

User can either pass in the specification as a dictionary or through keyword arguments.

Parameters:

  • spec (dict, optional) – Object specification, by default None

  • kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.

convert(overwrite=False)
class ads.jobs.builders.runtimes.python_runtime.DataFlowRuntime(spec: Optional[Dict] = None, **kwargs)

Bases: CondaRuntime

Initialize the object with specifications.

User can either pass in the specification as a dictionary or through keyword arguments.

Parameters:

  • spec (dict, optional) – Object specification, by default None

  • kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.

CONST_ARCHIVE_BUCKET = 'archiveBucket'
CONST_ARCHIVE_URI = 'archiveUri'
CONST_CONDA_AUTH_TYPE = 'condaAuthType'
CONST_CONFIGURATION = 'configuration'
CONST_SCRIPT_BUCKET = 'scriptBucket'
CONST_SCRIPT_PATH = 'scriptPathURI'
property archive_bucket: str

Bucket to save archive zip

property archive_uri

The Uri of archive zip

attribute_map = {'archiveUri': 'archive_uri', 'condaAuthType': 'conda_auth_type', 'configuration': 'configuration', 'env': 'env', 'freeformTags': 'freeform_tags', 'scriptBucket': 'script_bucket', 'scriptPathURI': 'script_path_uri'}
property configuration: dict

Configuration for Spark

convert(**kwargs)
property script_bucket: str

Bucket to save script

property script_uri: str

The URI of the source code

with_archive_bucket(bucket) → DataFlowRuntime

Set object storage bucket to save the archive zip, in case archive uri given is local.

Parameters:

bucket (str) – name of the bucket

Returns:

runtime instance itself

Return type:

DataFlowRuntime

with_archive_uri(uri: str) → DataFlowRuntime

Set archive uri (which is a zip file containing dependencies).

Parameters:

uri (str) – uri to the archive zip

Returns:

runtime instance itself

Return type:

DataFlowRuntime

with_conda(conda_spec: Optional[dict] = None)
with_configuration(config: dict) → DataFlowRuntime

Set Configuration for Spark.

Parameters:

config (dict) – dictionary of configuration details https://spark.apache.org/docs/latest/configuration.html#available-properties. Example: { “spark.app.name” : “My App Name”, “spark.shuffle.io.maxRetries” : “4” }

Returns:

runtime instance itself

Return type:

DataFlowRuntime

with_custom_conda(uri: str, region: Optional[str] = None, auth_type: Optional[str] = None)

Specifies the custom conda pack for running the job

Parameters:

  • uri (str) – The OCI object storage URI for the conda pack, e.g. “oci://your_bucket@namespace/object_name.” In the Environment Explorer of an OCI notebook session, this is shown as the “source” of the conda pack.

  • region (str, optional) – The region of the bucket storing the custom conda pack, by default None. If region is not specified, ADS will use the region from your authentication credentials, * For API Key, config[“region”] is used. * For Resource Principal, signer.region is used. This is required if the conda pack is stored in a different region.

  • auth_type (str, (="resource_principal")) – One of “resource_principal”, “api_keys”, “instance_principal”, etc. Auth mechanism used to read the conda back uri provided.

Returns:

The runtime instance.

Return type:

self

See also

https

//docs.oracle.com/en-us/iaas/data-science/using/conda_publishs_object.htm

with_script_bucket(bucket) → DataFlowRuntime

Set object storage bucket to save the script, in case script uri given is local.

Parameters:

bucket (str) – name of the bucket

Returns:

runtime instance itself

Return type:

DataFlowRuntime

with_script_uri(path) → DataFlowRuntime

Set script uri.

Parameters:

uri (str) – uri to the script

Returns:

runtime instance itself

Return type:

DataFlowRuntime

with_service_conda(slug: str)

Specifies the service conda pack for running the job

Parameters:

slug (str) – The slug name of the service conda pack

Returns:

The runtime instance.

Return type:

self

class ads.jobs.builders.runtimes.python_runtime.GitPythonRuntime(spec: Optional[Dict] = None, **kwargs)

Bases: CondaRuntime, _PythonRuntimeMixin

Represents a job runtime with source code from git repository

Initialize the object with specifications.

User can either pass in the specification as a dictionary or through keyword arguments.

Parameters:

  • spec (dict, optional) – Object specification, by default None

  • kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.

CONST_BRANCH = 'branch'
CONST_COMMIT = 'commit'
CONST_GIT_SSH_SECRET_ID = 'gitSecretId'
CONST_GIT_URL = 'url'
CONST_SKIP_METADATA = 'skipMetadataUpdate'
attribute_map = {'branch': 'branch', 'commit': 'commit', 'conda': 'conda', 'entryFunction': 'entry_function', 'entrypoint': 'entrypoint', 'env': 'env', 'freeformTags': 'freeform_tags', 'gitSecretId': 'git_secret_id', 'outputDir': 'output_dir', 'outputUri': 'output_uri', 'pythonPath': 'python_path', 'skipMetadataUpdate': 'skip_metadata_update', 'url': 'url'}
property branch: str

Git branch name.

property commit: str

Git commit ID (SHA1 hash)

property skip_metadata_update

Indicate if the metadata update should be skipped after the job run

By default, the job run metadata will be updated with the following freeform tags: * repo: The URL of the Git repository * commit: The Git commit ID * module: The entry script/module * method: The entry function/method * outputs. The prefix of the output files in object storage.

This update step also requires resource principals to have the permission to update the job run.

Returns:

True if the metadata update will be skipped. Otherwise False.

Return type:

bool

property ssh_secret_ocid

The OCID of the OCI Vault secret storing the Git SSH key.

property url: str

URL of the Git repository.

with_argument(*args, **kwargs)

Specifies the arguments for running the script/function.

When running a python script, the arguments will be the command line arguments. For example, with_argument(“arg1”, “arg2”, key1=”val1”, key2=”val2”) will generate the command line arguments: “arg1 arg2 –key1 val1 –key2 val2”

When running a function, the arguments will be passed into the function. Arguments can also be list, dict or any JSON serializable object. For example, with_argument(“arg1”, “arg2”, key1=[“val1a”, “val1b”], key2=”val2”) will be passed in as “your_function(“arg1”, “arg2”, key1=[“val1a”, “val1b”], key2=”val2”)

Returns:

The runtime instance.

Return type:

self

with_source(url: str, branch: Optional[str] = None, commit: Optional[str] = None, secret_ocid: Optional[str] = None)

Specifies the Git repository and branch/commit for the job source code.

Parameters:

  • url (str) – URL of the Git repository.

  • branch (str, optional) – Git branch name, by default None, the default branch will be used.

  • commit (str, optional) – Git commit ID (SHA1 hash), by default None, the most recent commit will be used.

  • secret_ocid (str) – The secret OCID storing the SSH key content for checking out the Git repository.

Returns:

The runtime instance.

Return type:

self

class ads.jobs.builders.runtimes.python_runtime.NotebookRuntime(spec: Optional[Dict] = None, **kwargs)

Bases: CondaRuntime

Represents a job runtime with Jupyter notebook

Initialize the object with specifications.

User can either pass in the specification as a dictionary or through keyword arguments.

Parameters:

  • spec (dict, optional) – Object specification, by default None

  • kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.

CONST_EXCLUDE_TAG = 'excludeTags'
CONST_NOTEBOOK_ENCODING = 'notebookEncoding'
CONST_NOTEBOOK_PATH = 'notebookPathURI'
CONST_OUTPUT_URI = 'outputURI'
attribute_map = {'conda': 'conda', 'env': 'env', 'excludeTags': 'exclude_tags', 'freeformTags': 'freeform_tags', 'notebookEncoding': 'notebook_encoding', 'notebookPathURI': 'notebook_path_uri', 'outputURI': 'output_uri'}
property exclude_tag: list

A list of cell tags indicating cells to be excluded from the job

property notebook_encoding: str

The encoding of the notebook

property notebook_uri: str

The URI of the notebook

property output_uri: list

URI for storing the output notebook and files

with_exclude_tag(*tags)

Specifies the cell tags in the notebook to exclude cells from the job script.

Parameters:

*tags (list) – A list of tags (strings).

Returns:

The runtime instance.

Return type:

self

with_notebook(path: str, encoding='utf-8')

Specifies the notebook to be converted to python script and run as a job.

Parameters:

path (str) – The path of the Jupyter notebook

Returns:

The runtime instance.

Return type:

self

with_output(output_uri: str)

Specifies the output URI for storing the output notebook and files.

Parameters:

output_uri (str) – URI for storing the output notebook and files. For example, oci://bucket@namespace/path/to/dir

Returns:

The runtime instance.

Return type:

self

class ads.jobs.builders.runtimes.python_runtime.PythonRuntime(spec: Optional[Dict] = None, **kwargs)

Bases: ScriptRuntime, _PythonRuntimeMixin

Represents a job runtime using ADS driver script to run Python code

Initialize the object with specifications.

User can either pass in the specification as a dictionary or through keyword arguments.

Parameters:

  • spec (dict, optional) – Object specification, by default None

  • kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.

CONST_WORKING_DIR = 'workingDir'
attribute_map = {'conda': 'conda', 'entryFunction': 'entry_function', 'entrypoint': 'entrypoint', 'env': 'env', 'freeformTags': 'freeform_tags', 'outputDir': 'output_dir', 'outputUri': 'output_uri', 'pythonPath': 'python_path', 'scriptPathURI': 'script_path_uri', 'workingDir': 'working_dir'}
with_working_dir(working_dir: str)

Specifies the working directory in the job run. By default, the working directory will the directory containing the user code (job artifact directory). This can be changed by specifying a relative path to the job artifact directory.

Parameters:

working_dir (str) – The path of the working directory. This can be a relative path from the job artifact directory.

Returns:

The runtime instance.

Return type:

self

property working_dir: str

The working directory for the job run.

class ads.jobs.builders.runtimes.python_runtime.ScriptRuntime(spec: Optional[Dict] = None, **kwargs)

Bases: CondaRuntime

Represents job runtime with scripts and conda pack

Initialize the object with specifications.

User can either pass in the specification as a dictionary or through keyword arguments.

Parameters:

  • spec (dict, optional) – Object specification, by default None

  • kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.

CONST_ENTRYPOINT = 'entrypoint'
CONST_SCRIPT_PATH = 'scriptPathURI'
attribute_map = {'conda': 'conda', 'entrypoint': 'entrypoint', 'env': 'env', 'freeformTags': 'freeform_tags', 'scriptPathURI': 'script_path_uri'}
property entrypoint: str

The relative path of the script to be set as entrypoint when source is a zip/tar/directory.

property script_uri: str

The URI of the source code

property source_uri: str

The URI of the source code

with_entrypoint(entrypoint: str)

Specify the entrypoint for the job

Parameters:

entrypoint (str) – The relative path of the script to be set as entrypoint when source is a zip/tar/directory.

Returns:

The runtime instance.

Return type:

self

with_script(uri: str)

Specifies the source code script for the job

Parameters:

uri (str) – URI to the Python or Shell script, which can be any URI supported by fsspec, including http://, https:// and OCI object storage. For example: oci://your_bucket@your_namespace/path/to/script.py

Returns:

The runtime instance.

Return type:

self

with_source(uri: str, entrypoint: Optional[str] = None)

Specifies the source code for the job

Parameters:

  • uri (str) – URI to the source code, which can be a (.py/.sh) script, a zip/tar file or directory containing the scripts/modules If the source code is a single file, URI can be any URI supported by fsspec, including http://, https:// and OCI object storage. For example: oci://your_bucket@your_namespace/path/to/script.py If the source code is a directory, only local directory is supported.

  • entrypoint (str, optional) – The relative path of the script to be set as entrypoint when source is a zip/tar/directory. By default None. This is not needed when the source is a single script.

Returns:

The runtime instance.

Return type:

self

Module contents