ads.jobs.builders.runtimes package
Submodules
ads.jobs.builders.runtimes.artifact module
- class ads.jobs.builders.runtimes.artifact.Artifact(source, runtime=None)
Bases:
object
Represents a OCI Data Science Job artifact. The Artifact class is designed to add an additional processing step on runtime/source code. before uploading it as data science job artifact.
A sub-class should implement the build() method to do the additional processing. A sub-class is designed to be used with context manager so that the temporary files are cleaned up properly.
For example, the NotebookArtifact implements the build() method to convert the notebook to python script. with NotebookArtifact(runtime) as artifact:
The build() method will be called when entering the context manager
The final artifact for the job will be stored in artifact.path upload_artifact(artifact.path)
Files are cleaned up when exit or if there is an exception.
- CONST_DRIVER_NOTEBOOK = 'driver_notebook.py'
- CONST_DRIVER_UTILS = 'driver_utils.py'
- build()
Builds the runtime artifact in the temporary directory. Subclass should implement this method to: 1. Process the runtime 2. Set the self.path to the final artifact path
- Raises:
NotImplementedError – When this method is not implemented in the subclass.
- static copy_from_uri(uri, to_path, unpack=False, **storage_options)
Copy file(s) to local path
- Parameters:
uri (str) – The URI of the source file or directory, which can be local path of OCI object storage URI.
to_path (path-like object) – The local destination path. If this is a directory, the source file/directory will be placed under it.
unpack (bool) – Indicate if zip or tar.gz file specified by the uri should be unpacked. This option has no effect on other files.
storage_options – Storage options for fsspec. For OCI object storage, the default_signer from ads.common.auth will be used if storage option is not specified.
- Returns:
The actual path of file/directory at destination.
For copying a single file and to_path is a filename, this will be the same as to_path.
For copying a single file and to_path is a directory, this will be to_path + filename.
For copying a directory, this will be to_path + directory name.
- Return type:
str or path-like object
- class ads.jobs.builders.runtimes.artifact.GitPythonArtifact
Bases:
Artifact
- CONST_DRIVER_SCRIPT = 'driver_oci.py'
- build()
Prepares job artifact for GitPythonRuntime.
- class ads.jobs.builders.runtimes.artifact.NotebookArtifact(source, runtime=None)
Bases:
PythonArtifact
Represents a NotebookRuntime job artifact
- CONST_DRIVER_SCRIPT = 'driver_notebook.py'
- build()
Prepares job artifact for notebook runtime
- class ads.jobs.builders.runtimes.artifact.PythonArtifact(source, runtime=None)
Bases:
Artifact
Represents a PythonRuntime job artifact
- CONST_DRIVER_SCRIPT = 'driver_python.py'
- USER_CODE_DIR = 'code'
- build()
Prepares job artifact for PythonRuntime.
- class ads.jobs.builders.runtimes.artifact.ScriptArtifact(source, runtime=None)
Bases:
Artifact
Represents a ScriptRuntime job artifact
- build()
Prepares job artifact for script runtime. If the source is a file, it will be returned as is. If the source is a directory, it will be compressed as a zip file.
ads.jobs.builders.runtimes.base module
- class ads.jobs.builders.runtimes.base.Runtime(spec: Optional[Dict] = None, **kwargs)
Bases:
Builder
Base class for job runtime
To initialize the object, user can either pass in the specification as a dictionary or through keyword arguments.
- Parameters:
spec (dict, optional) – Object specification, by default None
kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.
- CONST_ARGS = 'args'
- CONST_ENV_VAR = 'env'
- CONST_MAXIMUM_RUNTIME_IN_MINUTES = 'maximumRuntimeInMinutes'
- CONST_TAG = 'freeformTags'
- property args: list
Command line arguments
- attribute_map = {'env': 'env', 'freeformTags': 'freeform_tags'}
- property environment_variables: dict
Environment variables
- Returns:
The runtime environment variables. The returned dictionary is a copy.
- Return type:
dict
- property envs: dict
Environment variables
- property freeform_tags: dict
- property kind: str
Kind of the object to be stored in YAML. All runtime implementations will have “runtime” as kind. Subclass will have different types.
- property maximum_runtime_in_minutes: int
Maximum runtime in minutes
- property type: str
The type of the object as showing in YAML
- with_argument(*args, **kwargs) Self
Adds command line arguments to the runtime.
This method can be called (chained) multiple times to add various arguments.
- Parameters:
args – Positional arguments. In a single method call, positional arguments are always added before keyword arguments. You can call with_argument() to add positional arguments after keyword arguments.
kwargs – Keyword arguments. To add a keyword argument without value, set the value to None.
- Returns:
This method returns self to support chaining methods.
- Return type:
Self
- Raises:
ValueError – Keyword arguments with space in a key.
Examples
>>> runtime = Runtime().with_argument(key1="val1", key2="val2").with_argument("pos1") >>> print(runtime.args) ["--key1", "val1", "--key2", "val2", "pos1"]
>>> runtime = Runtime() >>> runtime.with_argument("pos1") >>> runtime.with_argument(key1="val1", key2="val2.1 val2.2") >>> runtime.with_argument("pos2") >>> print(runtime.args) ['pos1', '--key1', 'val1', '--key2', 'val2.1 val2.2', 'pos2']
>>> runtime = Runtime() >>> runtime.with_argument("pos1") >>> runtime.with_argument(key1=None, key2="val2") >>> runtime.with_argument("pos2") >>> print(runtime.args) ["pos1", "--key1", "--key2", "val2", "pos2"]
- with_environment_variable(**kwargs) Self
Sets environment variables
Environment variables enclosed by
${...}
will be substituted.You can use
$$
to escape the substitution.Undefined variable enclosed by
${}
will be ignored.Double dollar signs
$$
will be substituted by a single one$
.
- Returns:
This method returns self to support chaining methods.
- Return type:
Self
Examples
>>> runtime = ( ... PythonRuntime() ... .with_environment_variable( ... HOST="10.0.0.1", ... PORT="443", ... URL="http://${HOST}:${PORT}/path/", ... ESCAPED_URL="http://$${HOST}:$${PORT}/path/", ... MISSING_VAR="This is ${UNDEFINED}", ... VAR_WITH_DOLLAR="$10", ... DOUBLE_DOLLAR="$$10" ... ) ... ) >>> for k, v in runtime.environment_variables.items(): ... print(f"{k}: {v}") HOST: 10.0.0.1 PORT: 443 URL: http://10.0.0.1:443/path/ ESCAPED_URL: http://${HOST}:${PORT}/path/ MISSING_VAR: This is ${UNDEFINED} VAR_WITH_DOLLAR: $10 DOUBLE_DOLLAR: $10
- with_freeform_tag(**kwargs) Self
Sets freeform tag
- Returns:
This method returns self to support chaining methods.
- Return type:
Self
- with_maximum_runtime_in_minutes(maximum_runtime_in_minutes: int) Self
Sets maximum runtime in minutes
- Returns:
This method returns self to support chaining methods.
- Return type:
Self
ads.jobs.builders.runtimes.container_runtime module
- class ads.jobs.builders.runtimes.container_runtime.ContainerRuntime(spec: Optional[Dict] = None, **kwargs)
Bases:
Runtime
Represents a container job runtime
To define container runtime:
>>> ContainerRuntime() >>> .with_image("iad.ocir.io/<your_tenancy>/<your_image>") >>> .with_cmd("sleep 5 && echo Hello World") >>> .with_entrypoint(["/bin/sh", "-c"]) >>> .with_environment_variable(MY_ENV="MY_VALUE")
Alternatively, you can define the
entrypoint
andcmd
along with the image.>>> ContainerRuntime() >>> .with_image( >>> "iad.ocir.io/<your_tenancy>/<your_image>", >>> entrypoint=["/bin/sh", -c], >>> cmd="sleep 5 && echo Hello World", >>> ) >>> .with_environment_variable(MY_ENV="MY_VALUE")
The entrypoint and cmd can be either “exec form” or “shell form” (See references). The exec form is used when a list is passed in. The shell form is used when a space separated string is passed in.
When using the ContainerRuntime with OCI Data Science Job, the exec form is recommended. For most images, when the entrypoint is set to
["/bin/sh", "-c"]
,cmd
can be a string as if you are running shell command.References
https://docs.docker.com/engine/reference/builder/#entrypoint https://docs.docker.com/engine/reference/builder/#cmd
To initialize the object, user can either pass in the specification as a dictionary or through keyword arguments.
- Parameters:
spec (dict, optional) – Object specification, by default None
kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.
- CONST_CMD = 'cmd'
- CONST_ENTRYPOINT = 'entrypoint'
- CONST_IMAGE = 'image'
- attribute_map = {'cmd': 'cmd', 'entrypoint': 'entrypoint', 'env': 'env', 'freeformTags': 'freeform_tags', 'image': 'image'}
- property cmd: str
Command of the container job
- property entrypoint: str
Entrypoint of the container job
- property image: str
The container image
- with_cmd(cmd: str) ContainerRuntime
Specifies the command for the container job.
- Parameters:
cmd (str) – Command for the container job
- Returns:
The runtime instance.
- Return type:
- with_entrypoint(entrypoint: Union[str, list]) ContainerRuntime
Specifies the entrypoint for the container job.
- Parameters:
entrypoint (str or list) – Entrypoint for the container job
- Returns:
The runtime instance.
- Return type:
- with_image(image: str, entrypoint: Optional[Union[str, list]] = None, cmd: Optional[str] = None) ContainerRuntime
Specify the image for the container job.
- Parameters:
image (str) – The container image, e.g. iad.ocir.io/<your_tenancy>/<your_image>:<your_tag>
entrypoint (str or list, optional) – Entrypoint for the job, by default None (the entrypoint defined in the image will be used).
cmd (str, optional) – Command for the job, by default None.
- Returns:
The runtime instance.
- Return type:
ads.jobs.builders.runtimes.python_runtime module
- class ads.jobs.builders.runtimes.python_runtime.CondaRuntime(spec: Optional[Dict] = None, **kwargs)
Bases:
Runtime
Represents a job runtime with conda pack This is the base class for Runtime using conda environment. The
CondaRuntime
is not designed to be used directly when creating a job.To initialize the object, user can either pass in the specification as a dictionary or through keyword arguments.
- Parameters:
spec (dict, optional) – Object specification, by default None
kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.
- CONST_CONDA = 'conda'
- CONST_CONDA_REGION = 'region'
- CONST_CONDA_SLUG = 'slug'
- CONST_CONDA_TYPE = 'type'
- CONST_CONDA_TYPE_CUSTOM = 'published'
- CONST_CONDA_TYPE_SERVICE = 'service'
- CONST_CONDA_URI = 'uri'
- attribute_map = {'conda': 'conda', 'env': 'env', 'freeformTags': 'freeform_tags'}
- property conda: dict
The conda environment specification.
For service conda environment, the specification contains:
type
, the type of the conda environment. This is alwaysservice
for service conda environment.slug
, the slug of the conda environment.
For custom conda environment, the specification contains:
type
, the type of the conda environment. This is alwayspublished
for custom conda environment.uri
, the uri of the conda environment, e.g. oci://bucket@namespace/prefix/to/condaregion
, the region of the bucket in which the conda environment is stored. By default, ADS will determine the region based on the authenticated API key or resource principal. This is only needed if your conda environment is stored in a different region.
- Returns:
A dictionary containing the conda environment specifications.
- Return type:
dict
- with_custom_conda(uri: str, region: Optional[str] = None)
Specifies the custom conda pack for running the job Make sure you have configured the IAM policy for the job run to access the conda environment.
- Parameters:
uri (str) – The OCI object storage URI for the conda pack, e.g. “oci://your_bucket@namespace/object_name.” In the Environment Explorer of an OCI notebook session, this is shown as the “source” of the conda pack.
region (str, optional) –
The region of the bucket storing the custom conda pack, by default None. If region is not specified, ADS will use the region from your authentication credentials:
For API Key, config[“region”] is used.
For Resource Principal, signer.region is used.
This is required if the conda pack is stored in a different region.
- Returns:
The runtime instance.
- Return type:
self
See also
https
//docs.oracle.com/en-us/iaas/data-science/using/conda_publishs_object.htm
- with_service_conda(slug: str)
Specifies the service conda pack for running the job
- Parameters:
slug (str) – The slug name of the service conda pack
- Returns:
The runtime instance.
- Return type:
self
- class ads.jobs.builders.runtimes.python_runtime.DataFlowNotebookRuntime(spec: Optional[Dict] = None, **kwargs)
Bases:
DataFlowRuntime
,NotebookRuntime
To initialize the object, user can either pass in the specification as a dictionary or through keyword arguments.
- Parameters:
spec (dict, optional) – Object specification, by default None
kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.
- convert(overwrite=False)
- class ads.jobs.builders.runtimes.python_runtime.DataFlowRuntime(spec: Optional[Dict] = None, **kwargs)
Bases:
CondaRuntime
To initialize the object, user can either pass in the specification as a dictionary or through keyword arguments.
- Parameters:
spec (dict, optional) – Object specification, by default None
kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.
- CONST_ARCHIVE_BUCKET = 'archiveBucket'
- CONST_ARCHIVE_URI = 'archiveUri'
- CONST_CONDA_AUTH_TYPE = 'condaAuthType'
- CONST_CONFIGURATION = 'configuration'
- CONST_OVERWRITE = 'overwrite'
- CONST_SCRIPT_BUCKET = 'scriptBucket'
- CONST_SCRIPT_PATH = 'scriptPathURI'
- property archive_bucket: str
Bucket to save archive zip
- property archive_uri
The Uri of archive zip
- attribute_map = {'archiveUri': 'archive_uri', 'condaAuthType': 'conda_auth_type', 'configuration': 'configuration', 'env': 'env', 'freeformTags': 'freeform_tags', 'overwrite': 'overwrite', 'scriptBucket': 'script_bucket', 'scriptPathURI': 'script_path_uri'}
- property configuration: dict
Configuration for Spark
- convert(**kwargs)
- property overwrite: str
Whether to overwrite the existing script in object storage (script bucket).
- property script_bucket: str
Bucket to save script
- property script_uri: str
The URI of the source code
- with_archive_bucket(bucket) DataFlowRuntime
Set object storage bucket to save the archive zip, in case archive uri given is local.
- Parameters:
bucket (str) – name of the bucket
- Returns:
runtime instance itself
- Return type:
- with_archive_uri(uri: str) DataFlowRuntime
Set archive uri (which is a zip file containing dependencies).
- Parameters:
uri (str) – uri to the archive zip
- Returns:
runtime instance itself
- Return type:
- with_conda(conda_spec: Optional[dict] = None)
- with_configuration(config: dict) DataFlowRuntime
Set Configuration for Spark.
- Parameters:
config (dict) – dictionary of configuration details https://spark.apache.org/docs/latest/configuration.html#available-properties. Example: { “spark.app.name” : “My App Name”, “spark.shuffle.io.maxRetries” : “4” }
- Returns:
runtime instance itself
- Return type:
- with_custom_conda(uri: str, region: Optional[str] = None, auth_type: Optional[str] = None)
Specifies the custom conda pack for running the job
- Parameters:
uri (str) – The OCI object storage URI for the conda pack, e.g. “oci://your_bucket@namespace/object_name.” In the Environment Explorer of an OCI notebook session, this is shown as the “source” of the conda pack.
region (str, optional) – The region of the bucket storing the custom conda pack, by default None. If region is not specified, ADS will use the region from your authentication credentials, * For API Key, config[“region”] is used. * For Resource Principal, signer.region is used. This is required if the conda pack is stored in a different region.
auth_type (str, (="resource_principal")) – One of “resource_principal”, “api_keys”, “instance_principal”, etc. Auth mechanism used to read the conda back uri provided.
- Returns:
The runtime instance.
- Return type:
self
See also
https
//docs.oracle.com/en-us/iaas/data-science/using/conda_publishs_object.htm
- with_overwrite(overwrite: bool) DataFlowRuntime
Whether to overwrite the existing script in object storage (script bucket). If the Object Storage bucket already contains a script with the same name, then it will be overwritten with the new one if the overwrite flag equal to True.
- Parameters:
overwrite (bool) – Whether to overwrite the existing script in object storage (script bucket).
- Returns:
The DataFlowRuntime instance (self).
- Return type:
- with_script_bucket(bucket) DataFlowRuntime
Set object storage bucket to save the script, in case script uri given is local.
- Parameters:
bucket (str) – name of the bucket
- Returns:
runtime instance itself
- Return type:
- with_script_uri(path: str) DataFlowRuntime
Set script uri.
- Parameters:
path (str) – uri to the script
- Returns:
runtime instance itself
- Return type:
- with_service_conda(slug: str)
Specifies the service conda pack for running the job
- Parameters:
slug (str) – The slug name of the service conda pack
- Returns:
The runtime instance.
- Return type:
self
- class ads.jobs.builders.runtimes.python_runtime.GitPythonRuntime(spec: Optional[Dict] = None, **kwargs)
Bases:
CondaRuntime
,_PythonRuntimeMixin
Represents a job runtime with source code from git repository
Example:
runtime = ( GitPythonRuntime() .with_environment_variable(GREETINGS="Welcome to OCI Data Science") # Specify the service conda environment by slug name. .with_service_conda("pytorch19_p37_gpu_v1") # Specify the git repository # Optionally, you can specify the branch or commit .with_source("https://github.com/pytorch/tutorials.git") # Entrypoint is a relative path from the root of the git repo. .with_entrypoint("beginner_source/examples_nn/polynomial_nn.py") # Copy files in "beginner_source/examples_nn" to object storage after job finishes. .with_output( output_dir="beginner_source/examples_nn", output_uri="oci://bucket_name@namespace/path/to/dir" ) )
To initialize the object, user can either pass in the specification as a dictionary or through keyword arguments.
- Parameters:
spec (dict, optional) – Object specification, by default None
kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.
- CONST_BRANCH = 'branch'
- CONST_COMMIT = 'commit'
- CONST_GIT_SSH_SECRET_ID = 'gitSecretId'
- CONST_GIT_URL = 'url'
- CONST_SKIP_METADATA = 'skipMetadataUpdate'
- attribute_map = {'branch': 'branch', 'commit': 'commit', 'conda': 'conda', 'entryFunction': 'entry_function', 'entrypoint': 'entrypoint', 'env': 'env', 'freeformTags': 'freeform_tags', 'gitSecretId': 'git_secret_id', 'outputDir': 'output_dir', 'outputUri': 'output_uri', 'pythonPath': 'python_path', 'skipMetadataUpdate': 'skip_metadata_update', 'url': 'url', 'workingDir': 'working_dir'}
- property branch: str
Git branch name.
- property commit: str
Git commit ID (SHA1 hash)
- property skip_metadata_update
Indicate if the metadata update should be skipped after the job run
By default, the job run metadata will be updated with the following freeform tags: * repo: The URL of the Git repository * commit: The Git commit ID * module: The entry script/module * method: The entry function/method * outputs. The prefix of the output files in object storage.
This update step also requires resource principals to have the permission to update the job run.
- Returns:
True if the metadata update will be skipped. Otherwise False.
- Return type:
bool
- property ssh_secret_ocid: str
The OCID of the OCI Vault secret storing the Git SSH key.
- property url: str
URL of the Git repository.
- with_source(url: str, branch: Optional[str] = None, commit: Optional[str] = None, secret_ocid: Optional[str] = None)
Specifies the Git repository and branch/commit for the job source code.
- Parameters:
url (str) – URL of the Git repository.
branch (str, optional) – Git branch name, by default None, the default branch will be used.
commit (str, optional) – Git commit ID (SHA1 hash), by default None, the most recent commit will be used.
secret_ocid (str) – The secret OCID storing the SSH key content for checking out the Git repository.
- Returns:
The runtime instance.
- Return type:
self
- class ads.jobs.builders.runtimes.python_runtime.NotebookRuntime(spec: Optional[Dict] = None, **kwargs)
Bases:
CondaRuntime
Represents a job runtime with Jupyter notebook
To run a job with a single Jupyter notebook, you can define the run time as:
runtime = ( NotebookRuntime() .with_notebook( path="https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb", encoding='utf-8' ) .with_service_conda("tensorflow28_p38_cpu_v1") .with_environment_variable(GREETINGS="Welcome to OCI Data Science") .with_exclude_tag(["ignore", "remove"]) .with_output("oci://bucket_name@namespace/path/to/dir") )
Note that the notebook path can be local or remote path supported by fsspec, including OCI object storage path like
oci://bucket@namespace/path/to/notebook
To initialize the object, user can either pass in the specification as a dictionary or through keyword arguments.
- Parameters:
spec (dict, optional) – Object specification, by default None
kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.
- CONST_ENTRYPOINT = 'entrypoint'
- CONST_EXCLUDE_TAG = 'excludeTags'
- CONST_NOTEBOOK_ENCODING = 'notebookEncoding'
- CONST_NOTEBOOK_PATH = 'notebookPathURI'
- CONST_OUTPUT_URI = 'outputUri'
- CONST_OUTPUT_URI_ALT = 'outputURI'
- CONST_SOURCE = 'source'
- attribute_map = {'conda': 'conda', 'entrypoint': 'entrypoint', 'env': 'env', 'excludeTags': 'exclude_tags', 'freeformTags': 'freeform_tags', 'notebookEncoding': 'notebook_encoding', 'notebookPathURI': 'notebook_path_uri', 'outputUri': 'output_uri', 'source': 'source'}
- property exclude_tag: list
A list of cell tags indicating cells to be excluded from the job
- property notebook: str
The path of the notebook relative to the source.
- property notebook_encoding: str
The encoding of the notebook
- property notebook_uri: str
The URI of the notebook
- property output_uri: list
URI for storing the output notebook and files
- property source: str
The source code location.
- with_exclude_tag(*tags) NotebookRuntime
Specifies the cell tags in the notebook to exclude cells from the job script.
- Parameters:
*tags (list) – A list of tags (strings).
- Returns:
The runtime instance.
- Return type:
self
- with_notebook(path: str, encoding='utf-8') NotebookRuntime
Specifies the notebook to be run as a job. Use this method if you would like to run a single notebook. Use
with_source()
method if you would like to run a notebook with additional dependency files.- Parameters:
path (str) – The path of the Jupyter notebook
encoding (str) – The encoding for opening the notebook. Defaults to utf-8.
- Returns:
The runtime instance.
- Return type:
self
- with_output(output_uri: str) NotebookRuntime
Specifies the output URI for storing the output notebook and files. All files in the directory containing the notebook will be saved.
- Parameters:
output_uri (str) – URI for a directory storing the output notebook and files. For example, oci://bucket@namespace/path/to/dir
- Returns:
The runtime instance.
- Return type:
self
- with_source(uri: str, notebook: str, encoding='utf-8')
Specify source code directory containing the notebook and dependencies for the job. Use this method if you would like to run a notebook with additional dependency files. Use the with_notebook() method if you would like to run a single notebook.
In the following example, local folder “path/to/source” contains the notebook and dependencies, The local path of the notebook is “path/to/source/relative/path/to/notebook.ipynb”:
runtime.with_source(uri="path/to/source", notebook="relative/path/to/notebook.ipynb")
- Parameters:
uri (str) – URI of the source code directory. This can be local or on OCI object storage.
notebook (str) – The relative path of the notebook from the source URI.
encoding (str) – The encoding for opening the notebook. Defaults to utf-8.
- Returns:
The runtime instance.
- Return type:
Self
- class ads.jobs.builders.runtimes.python_runtime.PythonRuntime(spec: Optional[Dict] = None, **kwargs)
Bases:
ScriptRuntime
,_PythonRuntimeMixin
Represents a job runtime using ADS driver script to run Python code
Example:
runtime = ( PythonRuntime() # Specify the service conda environment by slug name. .with_service_conda("pytorch110_p38_cpu_v1") # The job artifact can be a single Python script, a directory or a zip file. .with_source("local/path/to/code_dir") # Environment variable .with_environment_variable(NAME="Welcome to OCI Data Science.") # Command line argument, arg1 --key arg2 .with_argument("arg1", key="arg2") # Set the working directory # When using a directory as source, the default working dir is the parent of code_dir. # Working dir should be a relative path beginning from the source directory (code_dir) .with_working_dir("code_dir") # The entrypoint is applicable only to directory or zip file as source # The entrypoint should be a path relative to the working dir. # Here my_script.py is a file in the code_dir/my_package directory .with_entrypoint("my_package/my_script.py") # Add an additional Python path, relative to the working dir (code_dir/other_packages). .with_python_path("other_packages") # Copy files in "code_dir/output" to object storage after job finishes. .with_output("output", "oci://bucket_name@namespace/path/to/dir") )
To initialize the object, user can either pass in the specification as a dictionary or through keyword arguments.
- Parameters:
spec (dict, optional) – Object specification, by default None
kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.
- attribute_map = {'conda': 'conda', 'entryFunction': 'entry_function', 'entrypoint': 'entrypoint', 'env': 'env', 'freeformTags': 'freeform_tags', 'outputDir': 'output_dir', 'outputUri': 'output_uri', 'pythonPath': 'python_path', 'scriptPathURI': 'script_path_uri', 'workingDir': 'working_dir'}
- class ads.jobs.builders.runtimes.python_runtime.ScriptRuntime(spec: Optional[Dict] = None, **kwargs)
Bases:
CondaRuntime
Represents job runtime with scripts and conda pack.
This runtime is designed to define job artifacts and configurations supported by OCI Data Science Jobs natively. It can be used with any script types that is supported by the OCI Data Science Jobs, including shell scripts and python scripts.
To run a script with all dependencies contained in a local folder:
runtime = ( ScriptRuntime() # Specify the service conda environment by slug name. .with_service_conda("pytorch110_p38_cpu_v1") # The job artifact can be a single Python script, a directory or a zip file. .with_source("local/path/to/code_dir") # Environment variable .with_environment_variable(NAME="Welcome to OCI Data Science.") # Command line argument .with_argument("100 linux "hi there"") # The entrypoint is applicable only to directory or zip file as source # The entrypoint should be a path relative to the working dir. # Here my_script.sh is a file in the code_dir/my_package directory .with_entrypoint("my_package/my_script.sh") )
References
https://docs.oracle.com/en-us/iaas/data-science/using/jobs-artifact.htm
To initialize the object, user can either pass in the specification as a dictionary or through keyword arguments.
- Parameters:
spec (dict, optional) – Object specification, by default None
kwargs (dict) – Specification as keyword arguments. If spec contains the same key as the one in kwargs, the value from kwargs will be used.
- CONST_ENTRYPOINT = 'entrypoint'
- CONST_SCRIPT_PATH = 'scriptPathURI'
- attribute_map = {'conda': 'conda', 'entrypoint': 'entrypoint', 'env': 'env', 'freeformTags': 'freeform_tags', 'scriptPathURI': 'script_path_uri'}
- property entrypoint: str
The relative path of the script to be set as entrypoint when source is a zip/tar/directory.
- property script_uri: str
The URI of the source code
- property source_uri: str
The URI of the source code
- with_entrypoint(entrypoint: str)
Specify the entrypoint for the job
- Parameters:
entrypoint (str) – The relative path of the script to be set as entrypoint when source is a zip/tar/directory.
- Returns:
The runtime instance.
- Return type:
self
- with_script(uri: str)
Specifies the source code script for the job
- with_source(uri: str, entrypoint: Optional[str] = None)
Specifies the source code for the job
- Parameters:
uri (str) – URI to the source code, which can be a (.py/.sh) script, a zip/tar file or directory containing the scripts/modules If the source code is a single file, URI can be any URI supported by fsspec, including http://, https:// and OCI object storage. For example: oci://your_bucket@your_namespace/path/to/script.py URI can also be a folder or a zip file containing the source code. In that case, entrypoint is required.
entrypoint (str, optional) – The relative path of the script to be set as entrypoint when source is a zip/tar/directory. By default None. This is not needed when the source is a single script.
- Returns:
The runtime instance.
- Return type:
self