Oracle Accelerated Data Science SDK (ADS)

Overview

The SklearnModel class in ADS is designed to allow you to rapidly get a Scikit-learn model into production. The .prepare() method creates the model artifacts that are needed to deploy a functioning model without you having to configure it or write code. However, you can customize the required score.py file.

The .verify() method simulates a model deployment by calling the load_model() and predict() methods in the score.py file. With the .verify() method, you can debug your score.py file without deploying any models. The .save() method deploys a model artifact to the model catalog. The .deploy() method deploys a model to a REST endpoint.

The following steps take your trained scikit-learn model and deploy it into production with a few lines of code.

Create a Scikit-learn Model

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

seed = 42

X, y = make_classification(n_samples=10000, n_features=15, n_classes=2, flip_y=0.05)
trainx, testx, trainy, testy = train_test_split(X, y, test_size=30, random_state=seed)
model = RandomForestClassifier(
        n_estimators=100, random_state=42
    )
model.fit(
        trainx,
        trainy,
    )

Prepare Model Artifact

from ads.model.framework.sklearn_model import SklearnModel
from ads.common.model_metadata import UseCaseType

sklearn_model = SklearnModel(estimator=model, artifact_dir="~/sklearn_artifact_dir")
sklearn_model.prepare(
    inference_conda_env="generalml_p38_cpu_v1",
    training_conda_env="generalml_p38_cpu_v1",
    X_sample=trainx,
    y_sample=trainy,
    use_case_type=UseCaseType.BINARY_CLASSIFICATION,
)

Instantiate a ads.model.framework.sklearn_model.SklearnModel() object with an Scikit-learn model. Each instance accepts the following parameters:

artifact_dir: str: Artifact directory to store the files needed for deployment.
auth: (Dict, optional): Defaults to None. The default authentication is set using the ads.set_auth API. To override the default, use ads.common.auth.api_keys() or ads.common.auth.resource_principal() and create the appropriate authentication signer and the **kwargs required to instantiate the IdentityClient object.
estimator: (Callable): Trained Scikit-learn model or Scikit-learn pipeline.
properties: (ModelProperties, optional): Defaults to None. The ModelProperties object required to save and deploy a model.

The properties is an instance of the ModelProperties class and has the following predefined fields:

bucket_uri: str
compartment_id: str
deployment_access_log_id: str
deployment_bandwidth_mbps: int
deployment_instance_count: int
deployment_instance_shape: str
deployment_log_group_id: str
deployment_predict_log_id: str
deployment_memory_in_gbs: Union[float, int]
deployment_ocpus: Union[float, int]
inference_conda_env: str
inference_python_version: str
overwrite_existing_artifact: bool
project_id: str
remove_existing_artifact: bool
training_conda_env: str
training_id: str
training_python_version: str
training_resource_id: str
training_script_path: str

By default, properties is populated from the environment variables when not specified. For example, in notebook sessions the environment variables are preset and stored in project id (PROJECT_OCID) and compartment id (NB_SESSION_COMPARTMENT_OCID). So properties populates these environment variables, and uses the values in methods such as .save() and .deploy(). Pass in values to overwrite the defaults. When you use a method that includes an instance of properties, then properties records the values that you pass in. For example, when you pass inference_conda_env into the .prepare() method, then properties records the value. To reuse the properties file in different places, you can export the properties file using the .to_yaml() method then reload it into a different machine using the .from_yaml() method.

Summary Status

You can call the .summary_status() method after a model serialization instance such as GenericModel, SklearnModel, TensorFlowModel, or PyTorchModel is created. The .summary_status() method returns a Pandas dataframe that guides you through the entire workflow. It shows which methods are available to call and which ones aren’t. Plus it outlines what each method does. If extra actions are required, it also shows those actions.

The following image displays an example summary status table created after a user initiates a model instance. The table’s Step column displays a Status of Done for the initiate step. And the Details column explains what the initiate step did such as generating a score.py file. The Step column also displays the prepare(), verify(), save(), deploy(), and predict() methods for the model. The Status column displays which method is available next. After the initiate step, the prepare() method is available. The next step is to call the prepare() method.

Register Model

>>> # Register the model
>>> model_id = sklearn_model.save()

Start loading model.joblib from model directory /tmp/tmphl0uhtbb ...
Model is successfully loaded.
['output_schema.json', 'runtime.yaml', 'model.joblib', 'score.py', 'input_schema.json']

'ocid1.datasciencemodel.oc1.xxx.xxxxx'

Deploy and Generate Endpoint

>>> # Deploy and create an endpoint for the Random Forest model
>>> sklearn_model.deploy(
        display_name="Random Forest Model For Classification",
        deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
        deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
        deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
    )
>>> print(f"Endpoint: {sklearn_model.model_deployment.url}")
https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx

Run Prediction against Endpoint

>>> # Generate prediction by invoking the deployed endpoint
>>> sklearn_model.predict(testx)['prediction']
[1,0,...,1]

Run Prediction with oci raw-request command

Model deployment endpoints can be invoked with the OCI-CLI. The below examples invoke a model deployment with the CLI with different types of payload: json, numpy.ndarray, pandas.core.frame.DataFrame or dict.

json payload example

>>> # Prepare data sample for prediction
>>> data = testx[[10]]
>>> data
array([[ 0.41330051,  0.67658927, -0.39189561,  0.21879805, -0.79208514,
     0.0906022 , -1.60595137,  1.65853693, -1.61337437,  0.82302124,
    -0.87032051,  0.70721209, -1.81956653, -0.26537296, -0.25471684]])

Use output of the data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data": [[ 0.41330051,  0.67658927, ... , -0.25471684]]}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

numpy.ndarray payload example

>>> # Prepare data sample for prediction
>>> from io import BytesIO
>>> import base64
>>> import numpy as np

>>> data = testx[[10]]
>>> np_bytes = BytesIO()
>>> np.save(np_bytes, data, allow_pickle=True)
>>> data = base64.b64encode(np_bytes.getvalue()).decode("utf-8")
>>> print(data)
k05VTVBZAQB2AHsnZGVzY......4UdN0L8=

Use printed output of base64 data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data":"k05VTVBZAQB2AHsnZGVzY......4UdN0L8=", "data_type": "numpy.ndarray"}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

pandas.core.frame.DataFrame payload example

>>> # Prepare data sample for prediction
>>> import pandas as pd

>>> df = pd.DataFrame(testx[[10]])
>>> print(json.dumps(df.to_json())
"{\"0\":{\"0\":0.4133005141},\"1\":{\"0\":0.676589266},...,\"14\":{\"0\":-0.2547168443}}"

Use printed output of DataFrame data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data":"{\"0\":{\"0\":0.4133005141},...,\"14\":{\"0\":-0.2547168443}}","data_type":"pandas.core.frame.DataFrame"}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

dict payload example

>>> # Prepare data sample for prediction
>>> import pandas as pd

>>> df = pd.DataFrame(testx[[10]])
>>> print(json.dumps(df.to_dict()))
{"0": {"0": 0.413300514080485}, "1": {"0": 0.6765892660311731}, ...,"14": {"0": -0.2547168443271222}}

Use printed output of dict data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data": {"0": {"0": 0.413300514080485}, ...,"14": {"0": -0.2547168443271222}}}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

Expected output of raw-request command

{
  "data": {
    "prediction": [
      0
    ]
  },
  "headers": {
    "Connection": "keep-alive",
    "Content-Length": "18",
    "Content-Type": "application/json",
    "Date": "Wed, 07 Dec 2022 18:31:39 GMT",
    "X-Content-Type-Options": "nosniff",
    "opc-request-id": "E1125E17AE084DFAB6BCCFA045C16966/0BBB65235292EE0817B67BD9141A620A/5956FE428CBAB2878EBA605CEECAD39D",
    "server": "uvicorn"
  },
  "status": "200 OK"
}

Examples

from ads.model.framework.sklearn_model import SklearnModel
from ads.common.model_metadata import UseCaseType

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

import tempfile

seed = 42

# Create a classification dataset
X, y = make_classification(n_samples=10000, n_features=15, n_classes=2, flip_y=0.05)
trainx, testx, trainy, testy = train_test_split(X, y, test_size=30, random_state=seed)

# Train LGBM model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(
    trainx,
    trainy,
)


# Deploy the model, test it and clean up.
# Prepare Model Artifact for RandomForest Classifier model
artifact_dir = tempfile.mkdtemp()
sklearn_model = SklearnModel(estimator=model, artifact_dir=artifact_dir)
sklearn_model.prepare(
    inference_conda_env="generalml_p38_cpu_v1",
    training_conda_env="generalml_p38_cpu_v1",
    use_case_type=UseCaseType.BINARY_CLASSIFICATION,
    X_sample=trainx,
    y_sample=trainy,
    force_overwrite=True,
)

# Check if the artifacts are generated correctly.
# The verify method invokes the ``predict`` function defined inside ``score.py`` in the artifact_dir

sklearn_model.verify(testx[:10])["prediction"]
sklearn_model.save(display_name="SKLearn Model")

# Deploy and create an endpoint for the RandomForest model
sklearn_model.deploy(
    display_name="Random Forest Model For Classification",
    deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxx",
    deployment_access_log_id="ocid1.log.oc1.xxx.xxxx",
    deployment_predict_log_id="ocid1.log.oc1.xxx.xxxx",
)


print(f"Endpoint: {sklearn_model.model_deployment.url}")

sklearn_model.predict(testx)["prediction"]

# To delete the deployed endpoint uncomment the following line
# sklearn_model.delete_deployment(wait_for_completion=True)

PyTorchModel

Overview

The ads.model.framework.pytorch_model.PyTorchModel class in ADS is designed to allow you to rapidly get a PyTorch model into production. The .prepare() method creates the model artifacts that are needed to deploy a functioning model without you having to configure it or write code. However, you can customize the required score.py file.

The .verify() method simulates a model deployment by calling the load_model() and predict() methods in the score.py file. With the .verify() method, you can debug your score.py file without deploying any models. The .save() method deploys a model artifact to the model catalog. The .deploy() method deploys a model to a REST endpoint.

The following steps take your trained PyTorch model and deploy it into production with a few lines of code.

Create a PyTorch Model

Load a ResNet18 model and put it into evaluation mode.

import torch
import torchvision

model = torchvision.models.resnet18(pretrained=True)
model.eval()

Prepare Model Artifact

Save as TorchScript

New in version 2.6.9.

Serializing model in TorchScript program by setting use_torch_script to True, you can load the model and run inference without defining the model class.

from ads.common.model_metadata import UseCaseType
from ads.model.framework.pytorch_model import PyTorchModel

import tempfile

# Prepare the model
artifact_dir = "pytorch_model_artifact"
pytorch_model = PyTorchModel(model, artifact_dir=artifact_dir)
pytorch_model.prepare(
    inference_conda_env="pytorch110_p38_cpu_v1",
    training_conda_env="pytorch110_p38_cpu_v1",
    use_case_type=UseCaseType.IMAGE_CLASSIFICATION,
    force_overwrite=True,
    use_torch_script=True,
)
# You don't need to modify the score.py generated. The model can be loaded without defining the model class.
# More info here - https://pytorch.org/tutorials/beginner/saving_loading_models.html#export-load-model-in-torchscript-format

Save state_dict

from ads.common.model_metadata import UseCaseType
from ads.model.framework.pytorch_model import PyTorchModel

import tempfile

# Prepare the model
artifact_dir = "pytorch_model_artifact"
pytorch_model = PyTorchModel(model, artifact_dir=artifact_dir)
pytorch_model.prepare(
    inference_conda_env="pytorch110_p38_cpu_v1",
    training_conda_env="pytorch110_p38_cpu_v1",
    use_case_type=UseCaseType.IMAGE_CLASSIFICATION,
    force_overwrite=True,
)

# The score.py generated requires you to create the class instance of the Model before the weights are loaded.
# More info here - https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-state-dict-recommended

Open pytorch_model_artifact/score.py and edit the code to instantiate the model class. The edits are highlighted -

import os
import sys
from functools import lru_cache
import torch
import json
from typing import Dict, List
import numpy as np
import pandas as pd
from io import BytesIO
import base64
import logging

import torchvision
the_model = torchvision.models.resnet18()

model_name = 'model.pt'



"""
Inference script. This script is used for prediction by scoring server when schema is known.
"""

@lru_cache(maxsize=10)
def load_model(model_file_name=model_name):
    """
    Loads model from the serialized format

    Returns
    -------
    model:  a model instance on which predict API can be invoked
    """
    model_dir = os.path.dirname(os.path.realpath(__file__))
    if model_dir not in sys.path:
        sys.path.insert(0, model_dir)
    contents = os.listdir(model_dir)
    if model_file_name in contents:
        print(f'Start loading {model_file_name} from model directory {model_dir} ...')
        model_state_dict = torch.load(os.path.join(model_dir, model_file_name))
        print(f"loading {model_file_name} is complete.")
    else:
        raise Exception(f'{model_file_name} is not found in model directory {model_dir}')

    # User would need to provide reference to the TheModelClass and
    # construct the the_model instance first before loading the parameters.
    # the_model = TheModelClass(*args, **kwargs)
    try:
        the_model.load_state_dict(model_state_dict)
    except NameError as e:
        raise NotImplementedError("TheModelClass instance must be constructed before loading the parameters. Please modify the load_model() function in score.py." )
    except Exception as e:
        raise e

    the_model.eval()
    print("Model is successfully loaded.")

    return the_model

Instantiate a PyTorchModel() object with a PyTorch model. Each instance accepts the following parameters:

artifact_dir: str. Artifact directory to store the files needed for deployment.
auth: (Dict, optional): Defaults to None. The default authentication is set using the ads.set_auth API. To override the default, use ads.common.auth.api_keys() or ads.common.auth.resource_principal() and create the appropriate authentication signer and the **kwargs required to instantiate the IdentityClient object.
estimator: Callable. Any model object generated by the PyTorch framework.
properties: (ModelProperties, optional). Defaults to None. The ModelProperties object required to save and deploy model.

The properties is an instance of the ModelProperties class and has the following predefined fields:

bucket_uri: str
compartment_id: str
deployment_access_log_id: str
deployment_bandwidth_mbps: int
deployment_instance_count: int
deployment_instance_shape: str
deployment_log_group_id: str
deployment_predict_log_id: str
deployment_memory_in_gbs: Union[float, int]
deployment_ocpus: Union[float, int]
inference_conda_env: str
inference_python_version: str
overwrite_existing_artifact: bool
project_id: str
remove_existing_artifact: bool
training_conda_env: str
training_id: str
training_python_version: str
training_resource_id: str
training_script_path: str

By default, properties is populated from the environment variables when not specified. For example, in notebook sessions the environment variables are preset and stored in project id (PROJECT_OCID) and compartment id (NB_SESSION_COMPARTMENT_OCID). So properties populates these environment variables, and uses the values in methods such as .save() and .deploy(). Pass in values to overwrite the defaults. When you use a method that includes an instance of properties, then properties records the values that you pass in. For example, when you pass inference_conda_env into the .prepare() method, then properties records the value. To reuse the properties file in different places, you can export the properties file using the .to_yaml() method then reload it into a different machine using the .from_yaml() method.

Verify Changes to Score.py

Download and load an image for prediction

# Download an image
import urllib.request
url, filename = ("https://github.com/pytorch/hub/raw/master/images/dog.jpg", "dog.jpg")
try: urllib.URLopener().retrieve(url, filename)
except: urllib.request.urlretrieve(url, filename)

# Preprocess the image and convert to torch.Tensor
from PIL import Image
from torchvision import transforms
input_image = Image.open(filename)
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
input_tensor = preprocess(input_image)
input_batch = input_tensor.unsqueeze(0) # create a mini-batch as expected by the model

Verify score.py changes by running inference locally

>>> prediction = pytorch_model.verify(input_batch)["prediction"]
>>> import numpy as np
>>> np.argmax(prediction)
258

Summary Status

You can call the .summary_status() method after a model serialization instance such as GenericModel, SklearnModel, TensorFlowModel, or PyTorchModel is created. The .summary_status() method returns a Pandas dataframe that guides you through the entire workflow. It shows which methods are available to call and which ones aren’t. Plus it outlines what each method does. If extra actions are required, it also shows those actions.

The following image displays an example summary status table created after a user initiates a model instance. The table’s Step column displays a Status of Done for the initiate step. And the Details column explains what the initiate step did such as generating a score.py file. The Step column also displays the prepare(), verify(), save(), deploy(), and predict() methods for the model. The Status column displays which method is available next. After the initiate step, the prepare() method is available. The next step is to call the prepare() method.

Register Model

>>> # Register the model
>>> model_id = pytorch_model.save()

Start loading model.pt from model directory /tmp/tmpf11gnx9c ...
loading model.pt is complete.
Model is successfully loaded.
['.score.py.swp', 'score.py', 'model.pt', 'runtime.yaml']


'ocid1.datasciencemodel.oc1.xxx.xxxxx'

Deploy and Generate Endpoint

>>> # Deploy and create an endpoint for the PyTorch model
>>> pytorch_model.deploy(
        display_name="PyTorch Model For Classification",
        deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
        deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
        deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
    )


>>> print(f"Endpoint: {pytorch_model.model_deployment.url}")

https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx

Run Prediction against Endpoint

# Download an image
import urllib.request
url, filename = ("https://github.com/pytorch/hub/raw/master/images/dog.jpg", "dog.jpg")
try: urllib.URLopener().retrieve(url, filename)
except: urllib.request.urlretrieve(url, filename)

# Preprocess the image and convert to torch.Tensor
from PIL import Image
from torchvision import transforms
input_image = Image.open(filename)
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
input_tensor = preprocess(input_image)
input_batch = input_tensor.unsqueeze(0) # create a mini-batch as expected by the model


# Generate prediction by invoking the deployed endpoint
prediction = pytorch_model.predict(input_batch)['prediction']
print(np.argmax(prediction))

Predict with Image

New in version 2.6.7.

Predict Image by passing a uri, which can be http(s), local path, or other URLs (e.g. starting with “oci://”, “s3://”, and “gcs://”), of the image or a PIL.Image.Image object using the image argument in predict() to predict a single image. The image will be converted to a tensor and then serialized so it can be passed to the endpoint. You can catch the tensor in score.py to perform further transformation.

Note

The payload size limit is 10 MB. Read more about invoking a model deployment here.

Given the size limitation, the example below is with resized image. To pass an image and invoke prediction, additional code inside score.py is required to preprocess the data. Open pytorch_model_artifact/score.py and update the pre_inference() method. The edits are highlighted:

def pre_inference(data, input_schema_path):
    """
    Preprocess json-serialized data to feed into predict function.

    Parameters
    ----------
    data: Data format as expected by the predict API of the core estimator.
    input_schema_path: path of input schema.

    Returns
    -------
    data: Data format after any processing.
    """
    data = deserialize(data, input_schema_path)
    import torchvision.transforms as transforms
    preprocess = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225]
        ),
    ])
    input_tensor = preprocess(data)
    input_batch = input_tensor.unsqueeze(0)
    return input_batch

Save score.py and verify prediction works:

>>> uri = ("https://github.com/oracle-samples/oci-data-science-ai-samples/tree/master/model_deploy_examples/images/dog_resized.jpg")
>>> prediction = pytorch_model.verify(image=uri)["prediction"]
>>> import numpy as np
>>> np.argmax(prediction)
258

Re-deploy model with updated score.py:

pytorch_model.deploy(
    display_name="PyTorch Model For Classification",
    deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxxx",
    deployment_access_log_id="ocid1.log.oc1.xxx.xxxxxx",
    deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxxx",
)

Run prediction with the image provided:

uri = ("https://github.com/oracle-samples/oci-data-science-ai-samples/tree/master/model_deploy_examples/images/dog_resized.jpg")

# Generate prediction by invoking the deployed endpoint
prediction = pytorch_model.predict(image=uri)['prediction']

Run Prediction with oci raw-request command

Model deployment endpoints can be invoked with the OCI-CLI. This example invokes a model deployment with the CLI with a torch.Tensor payload:

torch.Tensor payload example

>>> # Prepare data sample for prediction and save it to file 'data-payload'
>>> from io import BytesIO
>>> import base64

>>> buffer = BytesIO()
>>> torch.save(input_batch, buffer)
>>> data = base64.b64encode(buffer.getvalue()).decode("utf-8")
>>> with open('data-payload', 'w') as f:
>>>     f.write('{"data": "' + data + '", "data_type": "torch.Tensor"}')

File data-payload will have this information:

{"data": "UEsDBAAACAgAAAAAAAAAAAAAAAAAAAAAAAAQ ........................
.......................................................................
...AAAAEAAABQSwUGAAAAAAMAAwC3AAAA0jEJAAAA", "data_type": "torch.Tensor"}

Use file data-payload with data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body file://data-payload

Expected output of raw-request command

{
  "data": {
    "prediction": [
      [
        0.0159152802079916,
        -1.5496551990509033,
        .......
        2.5116958618164062
      ]
    ]
  },
  "headers": {
    "Connection": "keep-alive",
    "Content-Length": "19398",
    "Content-Type": "application/json",
    "Date": "Thu, 08 Dec 2022 18:28:41 GMT",
    "X-Content-Type-Options": "nosniff",
    "opc-request-id": "BD80D931A6EA4C718636ECE00730B255/86111E71C1B33C24988C59C27F15ECDE/E94BBB27AC3F48CB68F41135073FF46B",
    "server": "uvicorn"
  },
  "status": "200 OK"
}

Copy prediction output into argmax to retrieve result of the image prediction:

>>> print(np.argmax([[0.0159152802079916,
>>>                   -1.5496551990509033,
>>>                   .......
>>>                   2.5116958618164062]]))
258

Example

from ads.common.model_metadata import UseCaseType
from ads.model.framework.pytorch_model import PyTorchModel

import numpy as np
from PIL import Image

import tempfile
import torchvision
from torchvision import transforms

import urllib

# Load a pretrained PyTorch Model
model = torchvision.models.resnet18(pretrained=True)
model.eval()


# Prepare Model Artifact for PyTorch Model
artifact_dir = tempfile.mkdtemp()
pytorch_model = PyTorchModel(model, artifact_dir=artifact_dir)
pytorch_model.prepare(
    inference_conda_env="pytorch110_p38_cpu_v1",
    training_conda_env="pytorch110_p38_cpu_v1",
    use_case_type=UseCaseType.IMAGE_CLASSIFICATION,
    force_overwrite=True,
    use_torch_script=True
)


# Download an image for running inference
url, filename = ("https://github.com/pytorch/hub/raw/master/images/dog.jpg", "dog.jpg")
urllib.request.urlretrieve(url, filename)

# Load image
input_image = Image.open(filename)
preprocess = transforms.Compose(
    [
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ]
)
input_tensor = preprocess(input_image)
input_batch = input_tensor.unsqueeze(0)  # create a mini-batch as expected by the model

# Check if the artifacts are generated correctly.
# The verify method invokes the ``predict`` function defined inside ``score.py`` in the artifact_dir
prediction = pytorch_model.verify(input_batch)["prediction"]
print(np.argmax(prediction))

# Register the model
model_id = pytorch_model.save(display_name="PyTorch Model")

# Deploy and create an endpoint for the PyTorch model
pytorch_model.deploy(
    display_name="PyTorch Model For Classification",
    deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxxx",
    deployment_access_log_id="ocid1.log.oc1.xxx.xxxxxx",
    deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxxx",
)

# Generate prediction by invoking the deployed endpoint
prediction = pytorch_model.predict(input_batch)["prediction"]

print(np.argmax(prediction))

# To delete the deployed endpoint uncomment the line below
# pytorch_model.delete_deployment(wait_for_completion=True)

TensorFlowModel

Overview

The ads.model.framework.tensorflow_model.TensorFlowModel class in ADS is designed to allow you to rapidly get a TensorFlow model into production. The .prepare() method creates the model artifacts that are needed to deploy a functioning model without you having to configure it or write code. However, you can customize the required score.py file.

The .verify() method simulates a model deployment by calling the load_model() and predict() methods in the score.py file. With the .verify() method, you can debug your score.py file without deploying any models. The .save() method deploys a model artifact to the model catalog. The .deploy() method deploys a model to a REST endpoint.

The following steps take your trained TensorFlow model and deploy it into production with a few lines of code.

Create a TensorFlow Model

import tensorflow as tf

mnist = tf.keras.datasets.mnist
(trainx, trainy), (testx, testy) = mnist.load_data()
trainx, testx = trainx / 255.0, testx / 255.0

model = tf.keras.models.Sequential(
    [
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(128, activation="relu"),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10),
])
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer="adam", loss=loss_fn, metrics=["accuracy"])
model.fit(trainx, trainy, epochs=1)

Prepare Model Artifact

from ads.common.model_metadata import UseCaseType
from ads.model.framework.tensorflow_model import TensorFlowModel
from uuid import uuid4

tensorflow_model = TensorFlowModel(estimator=model, artifact_dir=f"./model-artifact-{str(uuid4())}")
tensorflow_model.prepare(
    inference_conda_env="tensorflow28_p38_cpu_v1",
    training_conda_env="tensorflow28_p38_cpu_v1",
    X_sample=trainx,
    y_sample=trainy,
    use_case_type=UseCaseType.MULTINOMIAL_CLASSIFICATION,
)

Instantiate a ads.model.framework.tensorflow_model.TensorFlowModel() object with a TensorFlow model. Each instance accepts the following parameters:

artifact_dir: str: Artifact directory to store the files needed for deployment.
auth: (Dict, optional): Defaults to None. The default authentication is set using the ads.set_auth API. To override the default, use ads.common.auth.api_keys() or ads.common.auth.resource_principal() and create the appropriate authentication signer and the **kwargs required to instantiate the IdentityClient object.
estimator: Callable: Any model object generated by the TensorFlow framework.
properties: (ModelProperties, optional): Defaults to None. The ModelProperties object required to save and deploy a model.

The properties is an instance of the ModelProperties class and has the following predefined fields:

bucket_uri: str
compartment_id: str
deployment_access_log_id: str
deployment_bandwidth_mbps: int
deployment_instance_count: int
deployment_instance_shape: str
deployment_log_group_id: str
deployment_predict_log_id: str
deployment_memory_in_gbs: Union[float, int]
deployment_ocpus: Union[float, int]
inference_conda_env: str
inference_python_version: str
overwrite_existing_artifact: bool
project_id: str
remove_existing_artifact: bool
training_conda_env: str
training_id: str
training_python_version: str
training_resource_id: str
training_script_path: str

By default, properties is populated from the environment variables when not specified. For example, in notebook sessions the environment variables are preset and stored in project id (PROJECT_OCID) and compartment id (NB_SESSION_COMPARTMENT_OCID). So properties populates these environment variables, and uses the values in methods such as .save() and .deploy(). Pass in values to overwrite the defaults. When you use a method that includes an instance of properties, then properties records the values that you pass in. For example, when you pass inference_conda_env into the .prepare() method, then properties records the value. To reuse the properties file in different places, you can export the properties file using the .to_yaml() method then reload it into a different machine using the .from_yaml() method.

Summary Status

You can call the .summary_status() method after a model serialization instance such as GenericModel, SklearnModel, TensorFlowModel, or PyTorchModel is created. The .summary_status() method returns a Pandas dataframe that guides you through the entire workflow. It shows which methods are available to call and which ones aren’t. Plus it outlines what each method does. If extra actions are required, it also shows those actions.

The following image displays an example summary status table created after a user initiates a model instance. The table’s Step column displays a Status of Done for the initiate step. And the Details column explains what the initiate step did such as generating a score.py file. The Step column also displays the prepare(), verify(), save(), deploy(), and predict() methods for the model. The Status column displays which method is available next. After the initiate step, the prepare() method is available. The next step is to call the prepare() method.

In TensorFlowModel, data serialization is supported for JSON serializable objects. Plus, there is support for a dictionary, string, list, np.ndarray, and tf.python.framework.ops.EagerTensor. Not all these objects are JSON serializable, however, support to automatically serializes and deserialized is provided.

Register Model

>>> # Register the model
>>> model_id = tensorflow_model.save()

Start loading model.h5 from model directory /tmp/tmpapjjzeol ...
Model is successfully loaded.
['runtime.yaml', 'model.h5', 'score.py']

'ocid1.datasciencemodel.oc1.xxx.xxxxx'

Deploy and Generate Endpoint

>>> # Deploy and create an endpoint for the TensorFlow model
>>> tensorflow_model.deploy(
        display_name="TensorFlow Model For Classification",
        deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
        deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
        deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
    )


>>> print(f"Endpoint: {tensorflow_model.model_deployment.url}")

https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx

Run Prediction against Endpoint

# Generate prediction by invoking the deployed endpoint
tensorflow_model.predict(testx[:3])['prediction']

[[-2.9461750984191895, -5.293642997741699, 0.4030594229698181, 3.0270071029663086, -6.470805644989014, -2.07453989982605, -9.646402359008789, 9.256569862365723, -2.6433541774749756, -0.8167083263397217],
[-3.4297854900360107, 2.4863781929016113, 8.968724250793457, 3.162344217300415, -11.153030395507812, 0.15335027873516083, -0.5451826453208923, -7.817524433135986, -1.0585914850234985, -10.736929893493652],
[-4.420501232147217, 5.841022491455078, -0.17066864669322968, -1.0071465969085693, -2.261953592300415, -3.0983355045318604, -2.0874621868133545, 1.0745809078216553, -1.2511857748031616, -2.273810625076294]]

Predict with Image

New in version 2.6.7.

Predict Image by passing a uri, which can be http(s), local path, or other URLs (e.g. starting with “oci://”, “s3://”, and “gcs://”), of the image or a PIL.Image.Image object using the image argument in predict() to predict a single image. The image will be converted to a tensor and then serialized so it can be passed to the endpoint. You can catch the tensor in score.py to perform further transformation.

Common example for model deployment prediction with image passed:

# Generate prediction by invoking the deployed endpoint
prediction = tensorflow_model.predict(image=<uri>)['prediction']

See the “Predict with Image” example here.

Run Prediction with oci raw-request command

Model deployment endpoints can be invoked with the OCI-CLI. This example invokes a model deployment with the CLI with a numpy.ndarray payload:

numpy.ndarray payload example

>>> # Prepare data sample for prediction and save it to file 'data-payload'
>>> from io import BytesIO
>>> import base64
>>> import numpy as np

>>> data = testx[:3]
>>> np_bytes = BytesIO()
>>> np.save(np_bytes, data, allow_pickle=True)
>>> data = base64.b64encode(np_bytes.getvalue()).decode("utf-8")
>>> with open('data-payload', 'w') as f:
>>>     f.write('{"data": "' + data + '", "data_type": "numpy.ndarray"}')

File data-payload will have this information:

{"data": "k05VTVBZAQB2AHsnZGVzY3InOiAnPGY4JywgJ2ZvcnRyYW5fb3JkZXInOi...
.......................................................................
...................AAAAAAAAAAAAAAAAAAA=", "data_type": "numpy.ndarray"}

Use file data-payload with data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body file://data-payload

Expected output of raw-request command

{
  "data": {
    "prediction": [
      [
        -1.8818638324737549,
        -6.04175329208374,
        ..................,
        -1.1257498264312744
      ],
      [
        1.2170600891113281,
        1.6379727125167847,
        ..................,
        -9.877771377563477
      ],
      [
        -4.255424499511719,
        5.320354461669922,
        ..................,
        -2.858555555343628
      ]
    ]
  },
  "headers": {
    "Connection": "keep-alive",
    "Content-Length": "594",
    "Content-Type": "application/json",
    "Date": "Thu, 08 Dec 2022 23:25:47 GMT",
    "X-Content-Type-Options": "nosniff",
    "opc-request-id": "E70002DAA3024F46B074F9B53DB6BEBB/421B34FCB12CF33F23C85D5619A62926/CAABF4A269C63B112482B2E57463CA13",
    "server": "uvicorn"
  },
  "status": "200 OK"
}

Example

from ads.common.model_metadata import UseCaseType
from ads.model.framework.tensorflow_model import TensorFlowModel

import tensorflow as tf
from uuid import uuid4

# Load MNIST Data
mnist = tf.keras.datasets.mnist
(trainx, trainy), (testx, testy) = mnist.load_data()
trainx, testx = trainx / 255.0, testx / 255.0

# Train TensorFlow model
model = tf.keras.models.Sequential(
    [
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(128, activation="relu"),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10),
    ]
)
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer="adam", loss=loss_fn, metrics=["accuracy"])
model.fit(trainx, trainy, epochs=1)

# Prepare Model Artifact for TensorFlow model
tensorflow_model = TensorFlowModel(estimator=model, artifact_dir=f"./model-artifact-{str(uuid4())}")
tensorflow_model.prepare(
    inference_conda_env="tensorflow28_p38_cpu_v1",
    training_conda_env="tensorflow28_p38_cpu_v1",
    X_sample=trainx,
    y_sample=trainy,
    use_case_type=UseCaseType.MULTINOMIAL_CLASSIFICATION,
)

# Check if the artifacts are generated correctly.
# The verify method invokes the ``predict`` function defined inside ``score.py`` in the artifact_dir
tensorflow_model.verify(testx[:10])["prediction"]

# Register the model
model_id = tensorflow_model.save(display_name="TensorFlow Model")

# Deploy and create an endpoint for the TensorFlow model
tensorflow_model.deploy(
    display_name="TensorFlow Model For Classification",
    deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
    deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
    deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
)

# Generate prediction by invoking the deployed endpoint
tensorflow_model.predict(testx)["prediction"]

# To delete the deployed endpoint uncomment the line below
# tensorflow_model.delete_deployment(wait_for_completion=True)

SparkPipelineModel

Overview

The SparkPipelineModel class in ADS is designed to allow you to rapidly get a PySpark model into production. The .prepare() method creates the model artifacts that are needed to deploy a functioning model without you having to configure it or write code. However, you can customize the required score.py file.

The .verify() method simulates a model deployment by calling the load_model() and predict() methods in the score.py file. With the .verify() method, you can debug your score.py file without deploying any models. The .save() method deploys a model artifact to the model catalog. The .deploy() method deploys a model to a REST endpoint.

The following steps take your trained PySpark model and deploy it into production with a few lines of code.

Create a Spark Pipeline Model

Generate a synthetic dataset:

from pyspark.sql import SparkSession


spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .getOrCreate()
training = spark.createDataFrame(
    [
        (0, "a b c d e spark", 1.0),
        (1, "b d", 0.0),
        (2, "spark f g h", 1.0),
        (3, "hadoop mapreduce", 0.0),
    ],
    ["id", "text", "label"],
)
test = spark.createDataFrame(
    [
        (4, "spark i j k"),
        (5, "l m n"),
        (6, "spark hadoop spark"),
        (7, "apache hadoop"),
    ],
    ["id", "text"],
)

Create a Spark Pipeline. (Note that a Spark Pipeline can be made with just 1 stage.)

from pyspark.ml import Pipeline
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import HashingTF, Tokenizer

tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features")
lr = LogisticRegression(maxIter=10, regParam=0.001)

pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])
model = pipeline.fit(training)

Prepare Model Artifact

import tempfile
from ads.model.framework.spark_model import SparkPipelineModel
from ads.common.model_metadata import UseCaseType

artifact_dir=tempfile.mkdtemp()
spark_model = SparkPipelineModel(estimator=model, artifact_dir=artifact_dir)

spark_model.prepare(inference_conda_env="pyspark32_p38_cpu_v2",
                    X_sample=training,
                    force_overwrite=True,
                    use_case_type=UseCaseType.BINARY_CLASSIFICATION)

Instantiate a SparkPipelineModel() object with a PySpark model. Each instance accepts the following parameters:

artifact_dir: str. Artifact directory to store the files needed for deployment.
auth: (Dict, optional): Defaults to None. The default authentication is set using the ads.set_auth API. To override the default, use ads.common.auth.api_keys() or ads.common.auth.resource_principal() and create the appropriate authentication signer and the **kwargs required to instantiate the IdentityClient object.
estimator: Callable. Any model object generated by the PySpark framework.
properties: (ModelProperties, optional). Defaults to None. The ModelProperties object required to save and deploy model.

The properties is an instance of the ModelProperties class and has the following predefined fields:

bucket_uri: str
compartment_id: str
deployment_access_log_id: str
deployment_bandwidth_mbps: int
deployment_instance_count: int
deployment_instance_shape: str
deployment_log_group_id: str
deployment_predict_log_id: str
deployment_memory_in_gbs: Union[float, int]
deployment_ocpus: Union[float, int]
inference_conda_env: str
inference_python_version: str
overwrite_existing_artifact: bool
project_id: str
remove_existing_artifact: bool
training_conda_env: str
training_id: str
training_python_version: str
training_resource_id: str
training_script_path: str

By default, properties is populated from the environment variables when not specified. For example, in notebook sessions the environment variables are preset and stored in project id (PROJECT_OCID) and compartment id (NB_SESSION_COMPARTMENT_OCID). So properties populates these environment variables, and uses the values in methods such as .save() and .deploy(). Pass in values to overwrite the defaults. When you use a method that includes an instance of properties, then properties records the values that you pass in. For example, when you pass inference_conda_env into the .prepare() method, then properties records the value. To reuse the properties file in different places, you can export the properties file using the .to_yaml() method then reload it into a different machine using the .from_yaml() method.

Summary Status

You can call the .summary_status() method after a model serialization instance such as GenericModel, SklearnModel, TensorFlowModel, or PyTorchModel is created. The .summary_status() method returns a Pandas dataframe that guides you through the entire workflow. It shows which methods are available to call and which ones aren’t. Plus it outlines what each method does. If extra actions are required, it also shows those actions.

The following image displays an example summary status table created after a user initiates a model instance. The table’s Step column displays a Status of Done for the initiate step. And the Details column explains what the initiate step did such as generating a score.py file. The Step column also displays the prepare(), verify(), save(), deploy(), and predict() methods for the model. The Status column displays which method is available next. After the initiate step, the prepare() method is available. The next step is to call the prepare() method.

Register Model

model_id = spark_model.save()

Start loading model.joblib from model directory /tmp/tmphdo8dfn3 ...
Model is successfully loaded.
['input_schema.json', 'runtime.yaml', 'model_input_data_schema.json', 'model', 'score.py']

'ocid1.datasciencemodel.oc1.xxx.xxxxx'

Deploy and Generate Endpoint

spark_model.deploy(
    display_name="Spark Pipeline Model For Classification",
    deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
    deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
    deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
)

print(f"Endpoint: {spark_model.model_deployment.url}")

# https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx

Run Prediction against Endpoint

spark_model.predict(test)['prediction']
# [0.0, 0.0, 1.0, 0.0]

Run Prediction with oci raw-request command

Model deployment endpoints can be invoked with the OCI-CLI. This example invokes a model deployment with the CLI with a json payload:

json payload example

>>> # Prepare data sample for prediction and save it to file 'payload'
>>> print(json.dumps(test.toJSON().collect()))
["{\"id\":4,\"text\":\"spark i j k\"}", "{\"id\":5,\"text\":\"l m n\"}",
"{\"id\":6,\"text\":\"spark hadoop spark\"}", "{\"id\":7,\"text\":\"apache hadoop\"}"]

Use printed output of the data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data": ["{\"id\":4,\"text\":\"spark i j k\"}", ... "{\"id\":7,\"text\":\"apache hadoop\"}"]}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

Expected output of raw-request command

{
  "data": {
    "prediction": [
      0.0,
      0.0,
      1.0,
      0.0
    ]
  },
  "headers": {
    "Connection": "keep-alive",
    "Content-Length": "32",
    "Content-Type": "application/json",
    "Date": "Thu, 08 Dec 2022 18:45:12 GMT",
    "X-Content-Type-Options": "nosniff",
    "opc-request-id": "C2E73B1679B34BAD8358B49D20619055/0EE2E5F93F48142725525D7A5BA7F5FB/049A66AA38AA0163DBBC70F225285851",
    "server": "uvicorn"
  },
  "status": "200 OK"
}

Example

Adapted from an example provided by Apache in the PySpark API Reference Documentation.

import tempfile
from pyspark.ml import Pipeline
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import HashingTF, Tokenizer
from pyspark.sql import SparkSession
from ads.model.framework.spark_model import SparkPipelineModel
from ads.common.model_metadata import UseCaseType

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .getOrCreate()

artifact_dir=tempfile.mkdtemp()

training = spark.createDataFrame(
    [
        (0, "a b c d e spark", 1.0),
        (1, "b d", 0.0),
        (2, "spark f g h", 1.0),
        (3, "hadoop mapreduce", 0.0),
    ],
    ["id", "text", "label"],
)

test = spark.createDataFrame(
    [
        (4, "spark i j k"),
        (5, "l m n"),
        (6, "spark hadoop spark"),
        (7, "apache hadoop"),
    ],
    ["id", "text"],
)

tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features")
lr = LogisticRegression(maxIter=10, regParam=0.001)
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

model = pipeline.fit(training)

spark_model = SparkPipelineModel(estimator=model, artifact_dir=artifact_dir)

spark_model.prepare(inference_conda_env="pyspark32_p38_cpu_v2",
                    X_sample=training,
                    force_overwrite=True,
                    use_case_type=UseCaseType.BINARY_CLASSIFICATION)

# Check if the artifacts are generated correctly.
# The verify method invokes the ``predict`` function defined inside ``score.py`` in the artifact_dir
prediction = spark_model.verify(test)


# Register the model
spark_model.save(display_name="Spark Pipeline Model")

# Deploy and create an endpoint for the Spark model
spark_model.deploy(
    display_name="Spark Pipeline Model For Classification",
    deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
    deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
    deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
)

# Generate prediction by invoking the deployed endpoint
spark_model.predict(test)["prediction"]

# To delete the deployed endpoint uncomment the line below
# spark_model.delete_deployment(wait_for_completion=True)

LightGBMModel

Overview

The ads.model.framework.lightgbm_model.LightGBMModel class in ADS is designed to allow you to rapidly get a LightGBM model into production. The .prepare() method creates the model artifacts that are needed to deploy a functioning model without you having to configure it or write code. However, you can customize the required score.py file.

The .verify() method simulates a model deployment by calling the load_model() and predict() methods in the score.py file. With the .verify() method, you can debug your score.py file without deploying any models. The .save() method deploys a model artifact to the model catalog. The .deploy() method deploys a model to a REST endpoint.

The following steps take your trained LightGBM model and deploy it into production with a few lines of code.

The LightGBMModel module in ADS supports serialization for models generated from both the Training API using lightgbm.train() and the Scikit-Learn API using lightgbm.LGBMClassifier(). Both of these interfaces are defined by LightGBM.

The Training API in LightGBM contains training and cross-validation routines. The Dataset class is an internal data structure that is used by LightGBM when using the lightgbm.train() method. You can also create LightGBM models using the Scikit-Learn Wrapper interface. The LightGBMModel class handles the differences between the LightGBM Training and SciKit-Learn APIs seamlessly.

Create LightGBM Model

import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

seed = 42

X, y = make_classification(n_samples=10000, n_features=15, n_classes=2, flip_y=0.05)
trainx, testx, trainy, testy = train_test_split(X, y, test_size=30, random_state=seed)
model = lgb.LGBMClassifier(
        n_estimators=100, learning_rate=0.01, random_state=42
    )
model.fit(
        trainx,
        trainy,
    )

Prepare Model Artifact

from ads.common.model_metadata import UseCaseType
from ads.model.framework.lightgbm_model import LightGBMModel

artifact_dir = tempfile.mkdtemp()
lightgbm_model = LightGBMModel(estimator=model, artifact_dir=artifact_dir)
lightgbm_model.prepare(
    inference_conda_env="generalml_p38_cpu_v1",
    training_conda_env="generalml_p38_cpu_v1",
    X_sample=trainx,
    y_sample=trainy,
    use_case_type=UseCaseType.BINARY_CLASSIFICATION,
)

Instantiate a ads.model.framework.lightgbm_model.LightGBMModel() object with a LightGBM model. Each instance accepts the following parameters:

artifact_dir: str: Artifact directory to store the files needed for deployment.
auth: (Dict, optional): Defaults to None. The default authentication is set using the ads.set_auth API. To override the default, use ads.common.auth.api_keys() or ads.common.auth.resource_principal() and create the appropriate authentication signer and the **kwargs required to instantiate the IdentityClient object.
estimator: (Callable): Trained LightGBM model using the Training API or the Scikit-Learn Wrapper interface.
properties: (ModelProperties, optional): Defaults to None. The ModelProperties object required to save and deploy a model.

The properties is an instance of the ModelProperties class and has the following predefined fields:

bucket_uri: str
compartment_id: str
deployment_access_log_id: str
deployment_bandwidth_mbps: int
deployment_instance_count: int
deployment_instance_shape: str
deployment_log_group_id: str
deployment_predict_log_id: str
deployment_memory_in_gbs: Union[float, int]
deployment_ocpus: Union[float, int]
inference_conda_env: str
inference_python_version: str
overwrite_existing_artifact: bool
project_id: str
remove_existing_artifact: bool
training_conda_env: str
training_id: str
training_python_version: str
training_resource_id: str
training_script_path: str

By default, properties is populated from the environment variables when not specified. For example, in notebook sessions the environment variables are preset and stored in project id (PROJECT_OCID) and compartment id (NB_SESSION_COMPARTMENT_OCID). So properties populates these environment variables, and uses the values in methods such as .save() and .deploy(). Pass in values to overwrite the defaults. When you use a method that includes an instance of properties, then properties records the values that you pass in. For example, when you pass inference_conda_env into the .prepare() method, then properties records the value. To reuse the properties file in different places, you can export the properties file using the .to_yaml() method then reload it into a different machine using the .from_yaml() method.

Summary Status

You can call the .summary_status() method after a model serialization instance such as GenericModel, SklearnModel, TensorFlowModel, or PyTorchModel is created. The .summary_status() method returns a Pandas dataframe that guides you through the entire workflow. It shows which methods are available to call and which ones aren’t. Plus it outlines what each method does. If extra actions are required, it also shows those actions.

The following image displays an example summary status table created after a user initiates a model instance. The table’s Step column displays a Status of Done for the initiate step. And the Details column explains what the initiate step did such as generating a score.py file. The Step column also displays the prepare(), verify(), save(), deploy(), and predict() methods for the model. The Status column displays which method is available next. After the initiate step, the prepare() method is available. The next step is to call the prepare() method.

Register Model

>>> # Register the model
>>> model_id = lightgbm_model.save()

Start loading model.joblib from model directory /tmp/tmphl0uhtbb ...
Model is successfully loaded.
['runtime.yaml', 'model.joblib', 'score.py', 'input_schema.json']

'ocid1.datasciencemodel.oc1.xxx.xxxxx'

Deploy and Generate Endpoint

>>> # Deploy and create an endpoint for the LightGBM model
>>> lightgbm_model.deploy(
        display_name="LightGBM Model For Classification",
        deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
        deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
        deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
    )


>>> print(f"Endpoint: {lightgbm_model.model_deployment.url}")

https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx

Run Prediction against Endpoint

# Generate prediction by invoking the deployed endpoint
lightgbm_model.predict(testx)['prediction']

[1,0,...,1]

Run Prediction with oci raw-request command

Model deployment endpoints can be invoked with the OCI-CLI. The below examples invoke a model deployment with the CLI with different types of payload: json, numpy.ndarray, pandas.core.frame.DataFrame or dict.

json payload example

>>> # Prepare data sample for prediction
>>> data = testx[[11]]
>>> data
array([[ 0.59990614,  0.95516275, -1.22366985, -0.89270887, -1.14868768,
    -0.3506047 ,  0.28529227,  2.00085413,  0.31212668,  0.39411511,
     0.87301082, -0.01743923, -1.15089633,  1.03461823,  1.50228029]])

Use printed output of the data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data": [[ 0.59990614,  0.95516275, ... , 1.50228029]]}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

numpy.ndarray payload example

>>> # Prepare data sample for prediction
>>> from io import BytesIO
>>> import base64
>>> import numpy as np

>>> data = testx[[10]]
>>> np_bytes = BytesIO()
>>> np.save(np_bytes, data, allow_pickle=True)
>>> data = base64.b64encode(np_bytes.getvalue()).decode("utf-8")
>>> print(data)
k05VTVBZAQB2AHsnZGVzY......D1cJ+D8=

Use printed output of base64 data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data":"k05VTVBZAQB2AHsnZGVzY......4UdN0L8=", "data_type": "numpy.ndarray"}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

pandas.core.frame.DataFrame payload example

>>> # Prepare data sample for prediction
>>> import pandas as pd

>>> df = pd.DataFrame(testx[[10]])
>>> print(json.dumps(df.to_json())
"{\"0\":{\"0\":0.4133005141},\"1\":{\"0\":0.676589266},...,\"14\":{\"0\":-0.2547168443}}"

Use printed output of DataFrame data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data":"{\"0\":{\"0\":0.4133005141},...,\"14\":{\"0\":-0.2547168443}}","data_type":"pandas.core.frame.DataFrame"}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

dict payload example

>>> # Prepare data sample for prediction
>>> import pandas as pd

>>> df = pd.DataFrame(testx[[11]])
>>> print(json.dumps(df.to_dict()))
{"0": {"0": 0.5999061426438217}, "1": {"0": 0.9551627492226553}, ...,"14": {"0": 1.5022802918908846}}

Use printed output of dict data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data": {"0": {"0": 0.5999061426438217}, ...,"14": {"0": 1.5022802918908846}}}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

Expected output of raw-request command

{
  "data": {
    "prediction": [
      1
    ]
  },
  "headers": {
    "Connection": "keep-alive",
    "Content-Length": "18",
    "Content-Type": "application/json",
    "Date": "Wed, 07 Dec 2022 18:31:05 GMT",
    "X-Content-Type-Options": "nosniff",
    "opc-request-id": "B67EED723ADD43D7BBA1AD5AFCCBD0C6/03F218FF833BC8A2D6A5BDE4AB8B7C12/6ACBC4E5C5127AC80DA590568D628B60",
    "server": "uvicorn"
  },
  "status": "200 OK"
}

Example

from ads.model.framework.lightgbm_model import LightGBMModel
from ads.common.model_metadata import UseCaseType

import lightgbm as lgb

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

import tempfile

seed = 42

# Create a classification dataset
X, y = make_classification(n_samples=10000, n_features=15, n_classes=2, flip_y=0.05)

trainx, testx, trainy, testy = train_test_split(X, y, test_size=30, random_state=seed)

# Train LGBM model
model = lgb.LGBMClassifier(n_estimators=100, learning_rate=0.01, random_state=42)
model.fit(
    trainx,
    trainy,
)

# Prepare Model Artifact for LightGBM model
artifact_dir = tempfile.mkdtemp()
lightgbm_model = LightGBMModel(estimator=model, artifact_dir=artifact_dir)
lightgbm_model.prepare(
    inference_conda_env="generalml_p38_cpu_v1",
    training_conda_env="generalml_p38_cpu_v1",
    X_sample=trainx,
    y_sample=trainy,
    force_overwrite=True,
    use_case_type=UseCaseType.BINARY_CLASSIFICATION,
)

# Check if the artifacts are generated correctly.
# The verify method invokes the ``predict`` function defined inside ``score.py`` in the artifact_dir
lightgbm_model.verify(testx[:10])["prediction"]

# Register the model
model_id = lightgbm_model.save(display_name="LightGBM Model")

# Deploy and create an endpoint for the LightGBM model
lightgbm_model.deploy(
    display_name="LightGBM Model For Classification",
    deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
    deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
    deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
)


print(f"Endpoint: {lightgbm_model.model_deployment.url}")

# Generate prediction by invoking the deployed endpoint
lightgbm_model.predict(testx)["prediction"]

# To delete the deployed endpoint uncomment the line below
# lightgbm_model.delete_deployment(wait_for_completion=True)

XGBoostModel

Overview

The ads.model.framework.xgboost_model.XGBoostModel class in ADS is designed to allow you to rapidly get a XGBoost model into production. The .prepare() method creates the model artifacts that are needed to deploy a functioning model without you having to configure it or write code. However, you can customize the required score.py file.

The .verify() method simulates a model deployment by calling the load_model() and predict() methods in the score.py file. With the .verify() method, you can debug your score.py file without deploying any models. The .save() method deploys a model artifact to the model catalog. The .deploy() method deploys a model to a REST endpoint.

The following steps take your trained XGBoost model and deploy it into production with a few lines of code.

The XGBoostModel module in ADS supports serialization for models generated from both the Learning API using xgboost.train() and the Scikit-Learn API using xgboost.XGBClassifier(). Both of these interfaces are defined by XGBoost.

Create XGBoost Model

from ads.model.framework.xgboost_model import XGBoostModel
from ads.common.model_metadata import UseCaseType

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

import tempfile

import xgboost

seed = 42

X, y = make_classification(n_samples=10000, n_features=15, n_classes=2, flip_y=0.05)
trainx, testx, trainy, testy = train_test_split(X, y, test_size=30, random_state=seed)
model = xgboost.XGBClassifier(
        n_estimators=100, learning_rate=0.01, random_state=42, use_label_encoder=False
    )
model.fit(
        trainx,
        trainy,
    )

Prepare Model Artifact

from ads.model.framework.xgboost_model import XGBoostModel
from ads.common.model_metadata import UseCaseType

artifact_dir = tempfile.mkdtemp()
xgb_model = XGBoostModel(estimator=model, artifact_dir=artifact_dir)
xgb_model.prepare(
    inference_conda_env="generalml_p38_cpu_v1",
    training_conda_env="generalml_p38_cpu_v1",
    X_sample=trainx,
    y_sample=trainy,
    use_case_type=UseCaseType.BINARY_CLASSIFICATION,
)

Instantiate a ads.model.framework.xgboost_model.XGBoostModel object with an XGBoost model. Each instance accepts the following parameters:

artifact_dir: str: Artifact directory to store the files needed for deployment.
auth: (Dict, optional): Defaults to None. The default authentication is set using the ads.set_auth API. To override the default, use ads.common.auth.api_keys() or ads.common.auth.resource_principal() and create the appropriate authentication signer and the **kwargs required to instantiate the IdentityClient object.
estimator: (Callable): Trained XGBoost model either using the Learning API or the Scikit-Learn Wrapper interface.
properties: (ModelProperties, optional): Defaults to None. The ModelProperties object required to save and deploy a model.

The properties is an instance of the ModelProperties class and has the following predefined fields:

bucket_uri: str
compartment_id: str
deployment_access_log_id: str
deployment_bandwidth_mbps: int
deployment_instance_count: int
deployment_instance_shape: str
deployment_log_group_id: str
deployment_predict_log_id: str
deployment_memory_in_gbs: Union[float, int]
deployment_ocpus: Union[float, int]
inference_conda_env: str
inference_python_version: str
overwrite_existing_artifact: bool
project_id: str
remove_existing_artifact: bool
training_conda_env: str
training_id: str
training_python_version: str
training_resource_id: str
training_script_path: str

By default, properties is populated from the environment variables when not specified. For example, in notebook sessions the environment variables are preset and stored in project id (PROJECT_OCID) and compartment id (NB_SESSION_COMPARTMENT_OCID). So properties populates these environment variables, and uses the values in methods such as .save() and .deploy(). Pass in values to overwrite the defaults. When you use a method that includes an instance of properties, then properties records the values that you pass in. For example, when you pass inference_conda_env into the .prepare() method, then properties records the value. To reuse the properties file in different places, you can export the properties file using the .to_yaml() method then reload it into a different machine using the .from_yaml() method.

Summary Status

You can call the .summary_status() method after a model serialization instance such as GenericModel, SklearnModel, TensorFlowModel, or PyTorchModel is created. The .summary_status() method returns a Pandas dataframe that guides you through the entire workflow. It shows which methods are available to call and which ones aren’t. Plus it outlines what each method does. If extra actions are required, it also shows those actions.

The following image displays an example summary status table created after a user initiates a model instance. The table’s Step column displays a Status of Done for the initiate step. And the Details column explains what the initiate step did such as generating a score.py file. The Step column also displays the prepare(), verify(), save(), deploy(), and predict() methods for the model. The Status column displays which method is available next. After the initiate step, the prepare() method is available. The next step is to call the prepare() method.

Register Model

>>> # Register the model
>>> model_id = xgb_model.save()

Start loading model.joblib from model directory /tmp/tmphl0uhtbb ...
Model is successfully loaded.
['runtime.yaml', 'model.joblib', 'score.py', 'input_schema.json']

'ocid1.datasciencemodel.oc1.xxx.xxxxx'

Deploy and Generate Endpoint

>>> # Deploy and create an endpoint for the XGBoost model
>>> xgb_model.deploy(
        display_name="XGBoost Model For Classification",
        deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
        deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
        deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
    )


>>> print(f"Endpoint: {xgb_model.model_deployment.url}")

https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx

Run Prediction against Endpoint

# Generate prediction by invoking the deployed endpoint
>>> xgb_model.predict(testx)['prediction']
[0.22879330813884735, 0.2054443359375, 0.20657016336917877,...,0.8005291223526001]

Run Prediction with oci raw-request command

Model deployment endpoints can be invoked with the OCI-CLI. The below examples invoke a model deployment with the CLI with different types of payload: json, numpy.ndarray, pandas.core.frame.DataFrame or dict.

json payload example

>>> # Prepare data sample for prediction
>>> data = testx[[12]]
>>> data
array([[ 0.66098176, -1.06487896, -0.88581208,  0.05667259,  0.42884393,
    -0.52552184,  0.75322749, -0.58112776, -0.81102029, -1.35854886,
    -1.16440502, -0.67791303, -0.04810906,  0.72970972, -0.24120756]])

Use printed output of the data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data": [[ 0.66098176, -1.06487896, ... , -0.24120756]]}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

numpy.ndarray payload example

>>> # Prepare data sample for prediction
>>> from io import BytesIO
>>> import base64
>>> import numpy as np

>>> data = testx[[12]]
>>> np_bytes = BytesIO()
>>> np.save(np_bytes, data, allow_pickle=True)
>>> data = base64.b64encode(np_bytes.getvalue()).decode("utf-8")
>>> print(data)
k05VTVBZAQB2AHsnZGVzY......pePfzr8=

Use printed output of base64 data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data":"k05VTVBZAQB2AHsnZGVzY......pePfzr8=", "data_type": "numpy.ndarray"}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

pandas.core.frame.DataFrame payload example

>>> # Prepare data sample for prediction
>>> import pandas as pd

>>> df = pd.DataFrame(testx[[12]])
>>> print(json.dumps(df.to_json())
"{\"0\":{\"0\":0.6609817554},\"1\":{\"0\":-1.0648789569},...,\"14\":{\"0\":-0.2412075575}}"

Use printed output of DataFrame data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data":"{\"0\":{\"0\":0.6609817554},...,\"14\":{\"0\":-0.2412075575}}","data_type":"pandas.core.frame.DataFrame"}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

dict payload example

>>> # Prepare data sample for prediction
>>> import pandas as pd

>>> df = pd.DataFrame(testx[[12]])
>>> print(json.dumps(df.to_dict()))
{"0": {"0": -0.6712208871908425}, "1": {"0": 0.5266565978285116}, ...,"14": {"0": 0.9062102978188604}}

Use printed output of dict data and endpoint to invoke prediction with raw-request command in terminal:

export uri=https://modeldeployment.{region}.oci.customer-oci.com/ocid1.datasciencemodeldeployment.oc1.xxx.xxxxx/predict
export data='{"data": {"0": {"0": -0.6712208871908425}, ...,"14": {"0": 0.9062102978188604}}}'
oci raw-request \
    --http-method POST \
    --target-uri $uri \
    --request-body "$data"

Expected output of raw-request command

{
  "data": {
    "prediction": [
      0.5611757040023804
    ]
  },
  "headers": {
    "Connection": "keep-alive",
    "Content-Length": "35",
    "Content-Type": "application/json",
    "Date": "Wed, 07 Dec 2022 17:27:17 GMT",
    "X-Content-Type-Options": "nosniff",
    "opc-request-id": "19E90A7F2AFE401BB437DBC6168D2F1C/4A1CC9969F40B9F0656DD0497A28B51A/FEB42D1B690E8A665244046C7A151AB5",
    "server": "uvicorn"
  },
  "status": "200 OK"
}

Example

from ads.model.framework.xgboost_model import XGBoostModel
from ads.common.model_metadata import UseCaseType

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

import tempfile

import xgboost

seed = 42

# Create a classification dataset
X, y = make_classification(n_samples=10000, n_features=15, n_classes=2, flip_y=0.05)
trainx, testx, trainy, testy = train_test_split(X, y, test_size=30, random_state=seed)

# Train XGBoost model
model = xgboost.XGBClassifier(n_estimators=100, learning_rate=0.01, random_state=42)
model.fit(
    trainx,
    trainy,
)

artifact_dir = tempfile.mkdtemp()
xgb_model = XGBoostModel(estimator=model, artifact_dir=artifact_dir)
xgb_model.prepare(
    inference_conda_env="generalml_p38_cpu_v1",
    training_conda_env="generalml_p38_cpu_v1",
    X_sample=trainx,
    y_sample=trainy,
    use_case_type=UseCaseType.BINARY_CLASSIFICATION,
)

# Check if the artifacts are generated correctly.
# The verify method invokes the ``predict`` function defined inside ``score.py`` in the artifact_dir
xgb_model.verify(testx)

# Register the model
model_id = xgb_model.save(display_name="XGBoost Model")

# Deploy and create an endpoint for the XGBoost model
xgb_model.deploy(
    display_name="XGBoost Model For Classification",
    deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
    deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
    deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
)

print(f"Endpoint: {xgb_model.model_deployment.url}")

# Generate prediction by invoking the deployed endpoint
xgb_model.predict(testx)["prediction"]

# To delete the deployed endpoint uncomment the line below
# xgb_model.delete_deployment(wait_for_completion=True)

HuggingFacePipelineModel

New in version 2.8.2.

Overview

The ads.model.framework.huggingface_model.HuggingFacePipelineModel class in ADS is designed to allow you to rapidly get a HuggingFace pipelines into production. The .prepare() method creates the model artifacts that are needed to deploy a functioning pipeline without you having to configure it or write code. However, you can customize the required score.py file.

The .verify() method simulates a model deployment by calling the load_model() and predict() methods in the score.py file. With the .verify() method, you can debug your score.py file without deploying any models. The .save() method deploys a model artifact to the model catalog. The .deploy() method deploys a model to a REST endpoint.

The following steps take your trained HuggingFacePipelineModel model and deploy it into production with a few lines of code.

Create a HuggingFace Pipeline

Load a ImageSegmentationPipeline pretrained model.

>>> from transformers import pipeline

>>> segmenter = pipeline(task="image-segmentation", model="facebook/detr-resnet-50-panoptic", revision="fc15262")
>>> preds = segmenter(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds
[{'score': 0.987885,
    'label': 'LABEL_184',
    'mask': <PIL.Image.Image image mode=L size=960x686>},
    {'score': 0.997345,
    'label': 'snow',
    'mask': <PIL.Image.Image image mode=L size=960x686>},
    {'score': 0.997247,
    'label': 'cat',
    'mask': <PIL.Image.Image image mode=L size=960x686>}]

Prepare a Custom Conda Pack

To do a image segmentation model, you can start with the PyTorch conda pack with slug pytorch110_p38_cpu_v1.

Run pip install timm since image segmentation model requires timm.
Then use odsc conda init -b your_bucket_name -n bucket_namespace to config where to store the published conda pack if you have not done this yet.
Lastly, run odsc conda publish -s pytorch110_p38_cpu_v1.
Once it’s done, you can find the path of the pusblished conda pack under the Environment Explorer Published tab. Refresh the page if you cannot find it.

Prepare Model Artifact

>>> from ads.common.model_metadata import UseCaseType
>>> from ads.model import HuggingFacePipelineModel

>>> import tempfile

>>> # Prepare the model
>>> artifact_dir = "huggingface_pipeline_model_artifact"
>>> huggingface_pipeline_model = HuggingFacePipelineModel(model, artifact_dir=artifact_dir)
>>> huggingface_pipeline_model.prepare(
...    inference_conda_env="<your-conda-pack-path>",
...    inference_python_version="<your-python-version>",
...    training_conda_env="<your-conda-pack-path>",
...    use_case_type=UseCaseType.OTHER,
...    force_overwrite=True,
...)
# You don't need to modify the score.py generated. The model can be loaded by the transformers.pipeline.
# More info here - https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.pipeline

Instantiate a HuggingFacePipelineModel() object with HuggingFace pipelines. All the pipelines related files are saved under the artifact_dir.

For more detailed information on what parameters that HuggingFacePipelineModel takes, refer to the API Documentation

Summary Status

You can call the .summary_status() method after a model serialization instance such as GenericModel, SklearnModel, TensorFlowModel, or PyTorchModel is created. The .summary_status() method returns a Pandas dataframe that guides you through the entire workflow. It shows which methods are available to call and which ones aren’t. Plus it outlines what each method does. If extra actions are required, it also shows those actions.

The following image displays an example summary status table created after a user initiates a model instance. The table’s Step column displays a Status of Done for the initiate step. And the Details column explains what the initiate step did such as generating a score.py file. The Step column also displays the prepare(), verify(), save(), deploy(), and predict() methods for the model. The Status column displays which method is available next. After the initiate step, the prepare() method is available. The next step is to call the prepare() method.

Register Model

>>> # Register the model
>>> model_id = huggingface_pipeline_model.save()

Model is successfully loaded.
['.model-ignore', 'score.py', 'config.json', 'runtime.yaml', 'preprocessor_config.json', 'pytorch_model.bin']

'ocid1.datasciencemodel.oc1.xxx.xxxxx'

Deploy and Generate Endpoint

>>> # Deploy and create an endpoint for the huggingface_pipeline_model
>>> huggingface_pipeline_model.deploy(
...     display_name="HuggingFace Pipeline Model For Image Segmentation",
...     deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
...     deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
...     deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
... )
>>> print(f"Endpoint: {huggingface_pipeline_model.model_deployment.url}")

Run Prediction against Endpoint

>>> # Download an image
>>> import PIL.Image
>>> import requests
>>> import cloudpickle
>>> image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
>>> image = PIL.Image.open(requests.get(image_url, stream=True).raw)

>>> # Generate prediction by invoking the deployed endpoint
>>> preds = huggingface_pipeline_model.predict(image)["prediction"]
>>> print([{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds])
[{'score': 0.9879, 'label': 'LABEL_184'},
{'score': 0.9973, 'label': 'snow'},
{'score': 0.9972, 'label': 'cat'}]

Predict with Image

Predict Image by passing a PIL.Image.Image object using the data argument in predict() to predict a single image. The image will be converted to bytes using cloudpickle so it can be passed to the endpoint. It will be loaded back to PIL.Image.Image in score.py before pass into the pipeline.

Note

The payload size limit is 10 MB. Read more about invoking a model deployment here.
Model deployment currently does not support internet(coming soon), hence you cannot pass in a url.

Predict with Multiple Arguments

If your model takes more than one argument, you can pass in through dictionary with the keys as the argument name and values as the value of the arguement.

>>> your_huggingface_pipeline_model.verify({"parameter_name_1": "parameter_value_1", ..., "parameter_name_n": "parameter_value_n"})
>>> your_huggingface_pipeline_model.predict({"parameter_name_1": "parameter_value_1", ..., "parameter_name_n": "parameter_value_n"})

Run Prediction with oci sdk

Model deployment endpoints can be invoked with the oci sdk. This example invokes a model deployment with the oci sdk with a bytes payload:

`bytes` payload example

>>> # The OCI SDK must be installed for this example to function properly.
>>> # Installation instructions can be found here: https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/pythonsdk.htm

>>> import requests
>>> import oci
>>> from ads.common.auth import default_signer
>>> import cloudpickle
>>> import PIL.Image
>>> import cloudpickle
>>> headers = {"Content-Type": "application/octet-stream"}
>>> endpoint = huggingface_pipeline_model.model_deployment.url + "/predict"

>>> ## download the image
>>> image_url = "https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png"
>>> image = PIL.Image.open(requests.get(image_link, stream=True).raw)
>>> image_bytes = cloudpickle.dumps(image)

>>> preds = requests.post(endpoint, data=image_bytes, auth=default_signer()['signer'], headers=headers).json()
>>> print([{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds['prediction']])
[{'score': 0.9879, 'label': 'LABEL_184'},
{'score': 0.9973, 'label': 'snow'},
{'score': 0.9972, 'label': 'cat'}]

Example

from transformers import pipeline
from ads.model import HuggingFacePipelineModel

import tempfile
import PIL.Image
from ads.common.auth import default_signer
import requests
import cloudpickle

## download the image
image_url = "https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png"
image = PIL.Image.open(requests.get(image_url, stream=True).raw)

## download the pretrained model
classifier = pipeline(model="openai/clip-vit-large-patch14")
classifier(
        images=image,
        candidate_labels=["animals", "humans", "landscape"],
    )

## Initiate a HuggingFacePipelineModel instance
zero_shot_image_classification_model = HuggingFacePipelineModel(classifier, artifact_dir=tempfile.mkdtemp())

# Autogenerate score.py, serialized model, runtime.yaml
conda_pack_path = "oci://bucket@namespace/path/to/conda/pack" # your published conda pack
python_version = "3.x" # Remember to update 3.x with your actual python version, e.g. 3.8
zero_shot_image_classification_model.prepare(inference_conda_env=conda_pack_path, inference_python_version = python_version, force_overwrite=True)

## Convert payload to bytes
data = {"images": image, "candidate_labels": ["animals", "humans", "landscape"]}
body = cloudpickle.dumps(data) # convert image to bytes

# Verify generated artifacts
zero_shot_image_classification_model.verify(data=data)
zero_shot_image_classification_model.verify(data=body)

# Register HuggingFace Pipeline model
zero_shot_image_classification_model.save()

## Deploy
log_group_id = "<log_group_id>"
log_id = "<log_id>"
zero_shot_image_classification_model.deploy(deployment_bandwidth_mbps=100,
                wait_for_completion=False,
                deployment_log_group_id = log_group_id,
                deployment_access_log_id = log_id,
                deployment_predict_log_id = log_id)
zero_shot_image_classification_model.predict(data)
zero_shot_image_classification_model.predict(body)

### Invoke the model by sending bytes
auth = default_signer()['signer']
endpoint = zero_shot_image_classification_model.model_deployment.url + "/predict"
headers = {"Content-Type": "application/octet-stream"}
requests.post(endpoint, data=body, auth=auth, headers=headers).json()

AutoMLModel

Working with AutoML has moved from within ADS to working directly with the automlx library. To deploy an AutoMlx model, use GenericModel class.

The following example takes your trained AutoML model using GenericModel and deploys it into production.

Example

import pandas as pd
import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

import ads
import automl
from automl import init
from ads.model import GenericModel
from ads.common.model_metadata import UseCaseType

dataset = fetch_openml(name='adult', as_frame=True)
df, y = dataset.data, dataset.target

# Several of the columns are incorrectly labeled as category type in the original dataset
numeric_columns = ['age', 'capitalgain', 'capitalloss', 'hoursperweek']
for col in df.columns:
    if col in numeric_columns:
        df[col] = df[col].astype(int)

X_train, X_test, y_train, y_test = train_test_split(df,
                                                    y.map({'>50K': 1, '<=50K': 0}).astype(int),
                                                    train_size=0.7,
                                                    random_state=0)

init(engine='local')
est = automl.Pipeline(task='classification')
est.fit(X_train, y_train)

ads.set_auth("resource_principal")
automl_model = GenericModel(estimator=est, artifact_dir="automl_model_artifact")
automl_model.prepare(inference_conda_env="automlx_p38_cpu_v1",
                     training_conda_env="automlx_p38_cpu_v1",
                     use_case_type=UseCaseType.BINARY_CLASSIFICATION,
                     X_sample=X_test,
                     force_overwrite=True)

Open automl_model_artifact/score.py and edit the code to instantiate the model class. The edits are highlighted -

# score.py 1.0 generated by ADS 2.8.1 on 20230226_214703
import json
import os
import sys
import importlib
from cloudpickle import cloudpickle
from functools import lru_cache
from io import StringIO
import logging
import sys
import automl
import pandas as pd
import numpy as np

model_name = 'model.pkl'

"""
Inference script. This script is used for prediction by scoring server when schema is known.
"""

def init_automl_logger():
    logger = logging.getLogger("automl")
    handler = logging.StreamHandler(sys.stdout)
    handler.setLevel(logging.ERROR)
    formatter = logging.Formatter(
        "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
    )
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    automl.init(engine="local", engine_opts={"n_jobs": 1}, logger=logger)

@lru_cache(maxsize=10)
def load_model(model_file_name=model_name):
    """
    Loads model from the serialized format

    Returns
    -------
    model:  a model instance on which predict API can be invoked
    """
    init_automl_logger()
    model_dir = os.path.dirname(os.path.realpath(__file__))
    if model_dir not in sys.path:
        sys.path.insert(0, model_dir)
    contents = os.listdir(model_dir)
    if model_file_name in contents:
        print(f'Start loading {model_file_name} from model directory {model_dir} ...')
        with open(os.path.join(os.path.dirname(os.path.realpath(__file__)), model_file_name), "rb") as file:
            loaded_model = cloudpickle.load(file)

        print("Model is successfully loaded.")
        return loaded_model
    else:
        raise Exception(f'{model_file_name} is not found in model directory {model_dir}')

@lru_cache(maxsize=1)
def fetch_data_type_from_schema(input_schema_path=os.path.join(os.path.dirname(os.path.realpath(__file__)), "input_schema.json")):
    """
    Returns data type information fetch from input_schema.json.

    Parameters
    ----------
    input_schema_path: path of input schema.

    Returns
    -------
    data_type: data type fetch from input_schema.json.

    """
    data_type = {}
    if os.path.exists(input_schema_path):
        schema = json.load(open(input_schema_path))
        for col in schema['schema']:
            data_type[col['name']] = col['dtype']
    else:
        print("input_schema has to be passed in in order to recover the same data type. pass `X_sample` in `ads.model.framework.automl_model.AutoMLModel.prepare` function to generate the input_schema. Otherwise, the data type might be changed after serialization/deserialization.")
    return data_type

def deserialize(data, input_schema_path, task=None):
    """
    Deserialize json serialization data to data in original type when sent to predict.

    Parameters
    ----------
    data: serialized input data.
    input_schema_path: path of input schema.
    task: Machine learning task, supported: classification, regression, anomaly_detection, forecasting. Defaults to None.

    Returns
    -------
    data: deserialized input data.

    """

    if isinstance(data, bytes):
        return pd.read_json(StringIO(data.decode("utf-8")))

    data_type = data.get('data_type', '') if isinstance(data, dict) else ''
    json_data = data.get('data', data) if isinstance(data, dict) else data

    if task and task == "forecasting":
        try:
            data_type = data_type.split("'")[1]
            module, spec = ".".join(data_type.split(".")[:-1]), data_type.split(".")[-1]
            lib = importlib.import_module(name=module)
            func = getattr(lib, spec)
            return pd.DataFrame(index=func(json_data))
        except:
            logging.warning("Cannot autodetect the type of the model input data. By default, convert input data to pd.DatetimeIndex and feed the model with an empty pandas DataFrame with index as input data. If assumption is not correct, modify the score.py and check with .verify() before saving model with .save().")
            return pd.DataFrame(index=pd.DatetimeIndex(json_data))
    if "pandas.core.series.Series" in data_type:
        return pd.Series(json_data)
    if "pandas.core.frame.DataFrame" in data_type or isinstance(json_data, str):
        return pd.read_json(json_data, dtype=fetch_data_type_from_schema(input_schema_path))
    if isinstance(json_data, dict):
        return pd.DataFrame.from_dict(json_data)

    return json_data

def pre_inference(data, input_schema_path, task=None):
    """
    Preprocess data

    Parameters
    ----------
    data: Data format as expected by the predict API of the core estimator.
    input_schema_path: path of input schema.
    task: Machine learning task, supported: classification, regression, anomaly_detection, forecasting. Defaults to None.

    Returns
    -------
    data: Data format after any processing.

    """
    data = deserialize(data, input_schema_path, task)
    return data

def post_inference(yhat):
    """
    Post-process the model results

    Parameters
    ----------
    yhat: Data format after calling model.predict.

    Returns
    -------
    yhat: Data format after any processing.

    """
    if isinstance(yhat, pd.core.frame.DataFrame):
        yhat = yhat.values
    return yhat.tolist()

def predict(data, model=load_model(), input_schema_path=os.path.join(os.path.dirname(os.path.realpath(__file__)), "input_schema.json")):
    """
    Returns prediction given the model and data to predict

    Parameters
    ----------
    model: Model instance returned by load_model API
    data: Data format as expected by the predict API of the core estimator. For eg. in case of sckit models it could be numpy array/List of list/Pandas DataFrame
    input_schema_path: path of input schema.

    Returns
    -------
    predictions: Output from scoring server
        Format: {'prediction': output from model.predict method}

    """
    task = model.task if hasattr(model, "task") else None
    features = pre_inference(data, input_schema_path, task)
    yhat = post_inference(
        model.predict(features)
    )
    return {'prediction': yhat}

Verify score.py changes by running inference locally.

automl_model.verify(X_test.iloc[:2], auto_serialize_data=True)

Save the model, and Deploy the model. After it is successfully deployed, invoke the endpoint by calling .predict() function.

model_id = automl_model.save(display_name='Demo AutoMLModel model')
deploy = automl_model.deploy(display_name='Demo AutoMLModel deployment')
automl_model.predict(X_test.iloc[:2], auto_serialize_data=True)

Other Frameworks

Getting Started with Data Flow

Overview

The ads.model.generic_model.GenericModel class in ADS provides an efficient way to serialize almost any model class. This section demonstrates how to use the GenericModel class to prepare model artifacts, verify models, save models to the model catalog, deploy models, and perform predictions on model deployment endpoints.

The GenericModel class works with any unsupported model framework that has a .predict() method. For the most common model classes such as scikit-learn, XGBoost, LightGBM, TensorFlow, and PyTorch, we recommend that you use the ADS provided, framework-specific serializations models. For example, for a scikit-learn model, use SKLearnmodel. For other models, use the GenericModel class.

The .verify() method simulates a model deployment by calling the load_model() and predict() methods in the score.py file. With the .verify() method, you can debug your score.py file without deploying any models. The .save() method deploys a model artifact to the model catalog. The .deploy() method deploys a model to a REST endpoint.

These simple steps take your trained model and will deploy it into production with just a few lines of code.

Prepare Model Artifact

Instantiate a GenericModel() object by giving it any model object. It accepts the following parameters:

artifact_dir: str: Artifact directory to store the files needed for deployment.
auth: (Dict, optional): Defaults to None. The default authentication is set using the ads.set_auth API. To override the default, use ads.common.auth.api_keys() or ads.common.auth.resource_principal() and create the appropriate authentication signer and the **kwargs required to instantiate the IdentityClient object.
estimator: (Callable): Trained model.
properties: (ModelProperties, optional): Defaults to None. ModelProperties object required to save and deploy the model.
serialize: (bool, optional): Defaults to True. If True the model will be serialized into a pickle file. If False, you must set the model_file_name in the .prepare() method, serialize the model manually, and save it in the artifact_dir. You will also need to update the score.py file to work with this model.

The properties is an instance of the ModelProperties class and has the following predefined fields:

bucket_uri: str
compartment_id: str
deployment_access_log_id: str
deployment_bandwidth_mbps: int
deployment_instance_count: int
deployment_instance_shape: str
deployment_log_group_id: str
deployment_predict_log_id: str
deployment_memory_in_gbs: Union[float, int]
deployment_ocpus: Union[float, int]
inference_conda_env: str
inference_python_version: str
overwrite_existing_artifact: bool
project_id: str
remove_existing_artifact: bool
training_conda_env: str
training_id: str
training_python_version: str
training_resource_id: str
training_script_path: str

By default, properties is populated from the environment variables when not specified. For example, in notebook sessions the environment variables are preset and stored in project id (PROJECT_OCID) and compartment id (NB_SESSION_COMPARTMENT_OCID). So properties populates these environment variables, and uses the values in methods such as .save() and .deploy(). Pass in values to overwrite the defaults. When you use a method that includes an instance of properties, then properties records the values that you pass in. For example, when you pass inference_conda_env into the .prepare() method, then properties records the value. To reuse the properties file in different places, you can export the properties file using the .to_yaml() method then reload it into a different machine using the .from_yaml() method.

Summary Status

You can call the .summary_status() method after a model serialization instance such as GenericModel, SklearnModel, TensorFlowModel, or PyTorchModel is created. The .summary_status() method returns a Pandas dataframe that guides you through the entire workflow. It shows which methods are available to call and which ones aren’t. Plus it outlines what each method does. If extra actions are required, it also shows those actions.

The following image displays an example summary status table created after a user initiates a model instance. The table’s Step column displays a Status of Done for the initiate step. And the Details column explains what the initiate step did such as generating a score.py file. The Step column also displays the prepare(), verify(), save(), deploy(), and predict() methods for the model. The Status column displays which method is available next. After the initiate step, the prepare() method is available. The next step is to call the prepare() method.

Example

By default, the GenericModel serializes to a pickle file. The following example, the user creates a model. In the prepare step, the user saves the model as a pickle file with the name toy_model.pkl. Then the user verifies the model, saves it to the model catalog, deploys the model and makes a prediction. Finally, the user deletes the model deployment and then deletes the model.

import tempfile
from ads.model.generic_model import GenericModel

class Toy:
    def predict(self, x):
        return x ** 2
model = Toy()

generic_model = GenericModel(estimator=model, artifact_dir=tempfile.mkdtemp())
generic_model.summary_status()

generic_model.prepare(
        inference_conda_env="dbexp_p38_cpu_v1",
        model_file_name="toy_model.pkl",
        force_overwrite=True
     )

# Check if the artifacts are generated correctly.
# The verify method invokes the ``predict`` function defined inside ``score.py`` in the artifact_dir
generic_model.verify(2)

# Register the model
model_id = generic_model.save(display_name="Custom Model")

# Deploy and create an endpoint for the XGBoost model
generic_model.deploy(
    display_name="My Custom Model",
    deployment_log_group_id="ocid1.loggroup.oc1.xxx.xxxxx",
    deployment_access_log_id="ocid1.log.oc1.xxx.xxxxx",
    deployment_predict_log_id="ocid1.log.oc1.xxx.xxxxx",
)

print(f"Endpoint: {generic_model.model_deployment.url}")

# Generate prediction by invoking the deployed endpoint
generic_model.predict(2)

# To delete the deployed endpoint uncomment the line below
# generic_model.delete_deployment(wait_for_completion=True)

You can also use the shortcut .prepare_save_deploy() instead of calling .prepare(), .save() and .deploy() seperately.

import tempfile
from ads.model.generic_model import GenericModel

class Toy:
    def predict(self, x):
        return x ** 2
estimator = Toy()

model = GenericModel(estimator=estimator)
model.summary_status()

# If you are running the code inside a notebook session and using a service pack, `inference_conda_env` can be omitted.
model.prepare_save_deploy(inference_conda_env="dbexp_p38_cpu_v1")
model.verify(2)

# Generate prediction by invoking the deployed endpoint
model.predict(2)

# To delete the deployed endpoint uncomment the line below
# model.delete_deployment(wait_for_completion=True)

Example – CatBoost

Here is a more realistic example using CatBoost model.

import tempfile
import ads
from ads.model.generic_model import GenericModel
from catboost import CatBoostRegressor

ads.set_auth(auth="resource_principal")

# Initialize data

X_train = [[1, 4, 5, 6],
            [4, 5, 6, 7],
            [30, 40, 50, 60]]

X_test = [[2, 4, 6, 8],
            [1, 4, 50, 60]]

y_train = [10, 20, 30]

# Initialize CatBoostRegressor
catboost_estimator = CatBoostRegressor(iterations=2,
                        learning_rate=1,
                        depth=2)
# Train a CatBoostRegressor model
catboost_estimator.fit(X_train, y_train)

# Get predictions
preds = catboost_estimator.predict(X_test)

# Instantiate ads.model.generic_model.GenericModel using the trained Custom Model using the trained CatBoost Classifier  model
catboost_model = GenericModel(estimator=catboost_estimator,
                            artifact_dir=tempfile.mkdtemp(),
                            model_save_serializer="cloudpickle",
                            model_input_serializer="json")

# Autogenerate score.py, pickled model, runtime.yaml, input_schema.json and output_schema.json
catboost_model.prepare(
    inference_conda_env="oci://bucket@namespace/path/to/your/conda/pack",
    inference_python_version="your_python_version",
    X_sample=X_train,
    y_sample=y_train,
)

# Verify generated artifacts. Payload looks like this: [[2, 4, 6, 8], [1, 4, 50, 60]]
catboost_model.verify(X_test, auto_serialize_data=True)

# Register CatBoostRegressor model
model_id = catboost_model.save(display_name="CatBoost Model")
catboost_model.deploy()
catboost_model.predict(X_test)
catboost_model.delete_deployment(wait_for_completion=True)
catboost_model.delete() # delete the model

Example – Save Your Own Model

By default, the serialize in GenericModel class is True, and it will serialize the model using cloudpickle. However, you can set serialize=False to disable it. And serialize the model on your own. You just need to copy the serialized model into the .artifact_dir. This example shows step by step how you can do that. The example is illustrated using a Sklearn model.

import tempfile
from ads import set_auth
from ads.model import GenericModel
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

set_auth(auth="resource_principal")

# Load dataset and Prepare train and test split
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)

# Train a LogisticRegression model
sklearn_estimator = LogisticRegression()
sklearn_estimator.fit(X_train, y_train)

# Serialize your model. You can choose your own way to serialize your model.
import cloudpickle
with open("./model.pkl", "wb") as f:
    cloudpickle.dump(sklearn_estimator, f)

model = GenericModel(sklearn_estimator, artifact_dir = "model_artifact_folder", serialize=False)
model.prepare(inference_conda_env="generalml_p38_cpu_v1",force_overwrite=True, model_file_name="model.pkl", X_sample=X_test)

Now copy the model.pkl file and paste into the model_artifact_folder folder. And open the score.py in the model_artifact_folder folder to add implementation of the load_model function. You can also add your preprocessing steps in pre_inference function and postprocessing steps in post_inference function. Below is an example implementation of the score.py. Replace your score.py with the code below.

# score.py 1.0 generated by ADS 2.8.2 on 20230301_065458
import os
import sys
import json
from functools import lru_cache

model_name = 'model.pkl'


"""
Inference script. This script is used for prediction by scoring server when schema is known.
"""

@lru_cache(maxsize=10)
def load_model(model_file_name=model_name):
    """
    Loads model from the serialized format

    Returns
    -------
    model:  a model instance on which predict API can be invoked
    """
    model_dir = os.path.dirname(os.path.realpath(__file__))
    if model_dir not in sys.path:
        sys.path.insert(0, model_dir)
    contents = os.listdir(model_dir)
    if model_file_name in contents:
        import cloudpickle
        with open(os.path.join(model_dir, model_name), "rb") as f:
            model = cloudpickle.load(f)
        return model
    else:
        raise Exception(f'{model_file_name} is not found in model directory {model_dir}')

@lru_cache(maxsize=1)
def fetch_data_type_from_schema(input_schema_path=os.path.join(os.path.dirname(os.path.realpath(__file__)), "input_schema.json")):
    """
    Returns data type information fetch from input_schema.json.

    Parameters
    ----------
    input_schema_path: path of input schema.

    Returns
    -------
    data_type: data type fetch from input_schema.json.

    """
    data_type = {}
    if os.path.exists(input_schema_path):
        schema = json.load(open(input_schema_path))
        for col in schema['schema']:
            data_type[col['name']] = col['dtype']
    else:
        print("input_schema has to be passed in in order to recover the same data type. pass `X_sample` in `ads.model.framework.sklearn_model.SklearnModel.prepare` function to generate the input_schema. Otherwise, the data type might be changed after serialization/deserialization.")
    return data_type

def deserialize(data, input_schema_path):
    """
    Deserialize json serialization data to data in original type when sent to predict.

    Parameters
    ----------
    data: serialized input data.
    input_schema_path: path of input schema.

    Returns
    -------
    data: deserialized input data.

    """

    import pandas as pd
    import numpy as np
    import base64
    from io import BytesIO
    if isinstance(data, bytes):
        return data

    data_type = data.get('data_type', '') if isinstance(data, dict) else ''
    json_data = data.get('data', data) if isinstance(data, dict) else data

    if "numpy.ndarray" in data_type:
        load_bytes = BytesIO(base64.b64decode(json_data.encode('utf-8')))
        return np.load(load_bytes, allow_pickle=True)
    if "pandas.core.series.Series" in data_type:
        return pd.Series(json_data)
    if "pandas.core.frame.DataFrame" in data_type or isinstance(json_data, str):
        return pd.read_json(json_data, dtype=fetch_data_type_from_schema(input_schema_path))
    if isinstance(json_data, dict):
        return pd.DataFrame.from_dict(json_data)
    return json_data

def pre_inference(data, input_schema_path):
    """
    Preprocess data

    Parameters
    ----------
    data: Data format as expected by the predict API of the core estimator.
    input_schema_path: path of input schema.

    Returns
    -------
    data: Data format after any processing.

    """
    return deserialize(data, input_schema_path)

def post_inference(yhat):
    """
    Post-process the model results

    Parameters
    ----------
    yhat: Data format after calling model.predict.

    Returns
    -------
    yhat: Data format after any processing.

    """
    return yhat.tolist()

def predict(data, model=load_model(), input_schema_path=os.path.join(os.path.dirname(os.path.realpath(__file__)), "input_schema.json")):
    """
    Returns prediction given the model and data to predict

    Parameters
    ----------
    model: Model instance returned by load_model API.
    data: Data format as expected by the predict API of the core estimator. For eg. in case of sckit models it could be numpy array/List of list/Pandas DataFrame.
    input_schema_path: path of input schema.

    Returns
    -------
    predictions: Output from scoring server
        Format: {'prediction': output from model.predict method}

    """
    features = pre_inference(data, input_schema_path)
    yhat = post_inference(
        model.predict(features)
    )
    return {'prediction': yhat}

Save the score.py and now call .verify() to check if it works locally.

model.verify(X_test[:2], auto_serialize_data=True)

After verify run successfully, you can save the model to model catalog, deploy and call predict to invoke the endpoint.

model_id = model.save(display_name='Demo Sklearn model')
deploy = model.deploy(display_name='Demo Sklearn deployment')
model.predict(X_test[:2].tolist())

You can also use the shortcut .prepare_save_deploy() instead of calling .prepare(), .save() and .deploy() seperately.

import tempfile
from ads.catalog.model import ModelCatalog
from ads.model.generic_model import GenericModel

class Toy:
    def predict(self, x):
        return x ** 2
estimator = Toy()

model = GenericModel(estimator=estimator)
model.summary_status()
# If you are running the code inside a notebook session and using a service pack, `inference_conda_env` can be omitted.
model.prepare_save_deploy(inference_conda_env="dataexpl_p37_cpu_v3")
model.verify(2)
model.predict(2)
model.delete_deployment(wait_for_completion=True)
model.delete()

Apache Spark

DataFlow

Oracle Cloud Infrastructure (OCI) Data Flow is a fully managed, serverless, and on-demand Apache Spark Service that performs data processing or model training tasks on extremely large datasets without infrastructure to deploy or manage.

ads integrates with OCI Data Flow allowing users to submit and monitor Spark jobs from Notebook Sessions or from their local machine.

Quick Start

Data Flow is a hosted Apache Spark server. It is quick to start, and can scale to handle large datasets in parallel. ADS provides a convenient API for creating and maintaining workloads on Data Flow.

Submit a Toy Python Script to DataFlow

From a Python Environment

Submit a python script to DataFlow entirely from your python environment. The following snippet uses a toy python script that prints “Hello World” followed by the spark version, 3.2.1.

from ads.jobs import DataFlow, Job, DataFlowRuntime
from uuid import uuid4
import os
import tempfile

SCRIPT_CONTENT = """
import pyspark

def main():
    print("Hello World")
    print("Spark version is", pyspark.__version__)

if __name__ == "__main__":
    main()
"""

with tempfile.TemporaryDirectory() as td:
    with open(os.path.join(td, "script.py"), "w") as f:
        f.write(SCRIPT_CONTENT)
    name = f"dataflow-app-{str(uuid4())}"
    dataflow_configs = (
        DataFlow()
        .with_compartment_id("oci.xx.<compartment_id>")
        .with_logs_bucket_uri("oci://<mybucket>@<mynamespace>/<dataflow-logs-prefix>")
        .with_driver_shape("VM.Standard.E4.Flex")
        .with_driver_shape_config(ocpus=2, memory_in_gbs=32)
        .with_executor_shape("VM.Standard.E4.Flex")
        .with_executor_shape_config(ocpus=4, memory_in_gbs=64)
        .with_spark_version("3.2.1")
    )
    runtime_config = (
        DataFlowRuntime()
        .with_script_uri(os.path.join(td, "script.py"))
        .with_script_bucket("oci://<mybucket>@<mynamespace>/<subdir_to_put_and_get_script>")
    )
    df = Job(name=name, infrastructure=dataflow_configs, runtime=runtime_config)
    df.create()
    df_run = df.run()

From the Command Line

The same result can be achieved from the command line using ads CLI and a yaml file.

Assuming you have the following two files written in your current directory as script.py and dataflow.yaml respectively:

# script.py
import pyspark
def main():
    print("Hello World")
    print("Spark version is", pyspark.__version__)
if __name__ == "__main__":
    main()

# dataflow.yaml
kind: job
spec:
    name: dataflow-app-<uuid>
    infrastructure:
        kind: infrastructure
        spec:
            compartmentId: oci.xx.<compartment_id>
            logsBucketUri: oci://<mybucket>@<mynamespace>/<dataflow-logs-prefix>
            driverShape: VM.Standard.E4.Flex
            driverShapeConfig:
              ocpus: 2
              memory_in_gbs: 32
            executorShape: VM.Standard.E4.Flex
            executorShapeConfig:
              ocpus: 4
              memory_in_gbs: 64
            sparkVersion: 3.2.1
            numExecutors: 1
        type: dataFlow
    runtime:
        kind: runtime
        spec:
            scriptUri: script.py
            scriptBucket: oci://<mybucket>@<mynamespace>/<subdir_to_put_and_get_script>

ads jobs run -f dataflow.yaml

Real Data Flow Example with Conda Environment

From PySpark v3.0.0 and onwards, Data Flow allows a published conda environment as the Spark runtime environment when built with ADS. Data Flow supports published conda environments only. Conda packs are tar’d conda environments. When you publish your own conda packs to object storage, ensure that the DataFlow Resource has access to read the object or bucket. Below is a more built-out example using conda packs:

From a Python Environment

from ads.jobs import DataFlow, Job, DataFlowRuntime
from uuid import uuid4
import os
import tempfile

with tempfile.TemporaryDirectory() as td:
    with open(os.path.join(td, "script.py"), "w") as f:
        f.write(
'''
from pyspark.sql import SparkSession
import click

@click.command()
@click.argument("app_name")
@click.option(
    "--limit", "-l", help="max number of row to print", default=10, required=False
)
@click.option("--verbose", "-v", help="print out result in verbose mode", is_flag=True)
def main(app_name, limit, verbose):
    Create a Spark session
    spark = SparkSession.builder.appName(app_name).getOrCreate()

    Load a csv file from dataflow public storage
    df = (
        spark.read.format("csv")
        .option("header", "true")
        .option("multiLine", "true")
        .load(
            "oci://oow_2019_dataflow_lab@bigdatadatasciencelarge/usercontent/kaggle_berlin_airbnb_listings_summary.csv"
        )
    )

    Create a temp view and do some SQL operations
    df.createOrReplaceTempView("berlin")
    query_result_df = spark.sql(
        """
        SELECT
            city,
            zipcode,
            CONCAT(latitude,',', longitude) AS lat_long
        FROM berlin
        """
    ).limit(limit)

    # Convert the filtered Spark DataFrame into JSON format
    # Note: we are writing to the spark stdout log so that we can retrieve the log later at the end of the notebook.
    if verbose:
        rows = query_result_df.toJSON().collect()
        for i, row in enumerate(rows):
            print(f"record {i}")
            print(row)

if __name__ == "__main__":
    main()
'''
    )
    name = f"dataflow-app-{str(uuid4())}"
    dataflow_configs = (
        DataFlow()
        .with_compartment_id("oci.xx.<compartment_id>")
        .with_logs_bucket_uri("oci://<mybucket>@<mynamespace>/<dataflow-logs-prefix>")
        .with_driver_shape("VM.Standard.E4.Flex")
        .with_driver_shape_config(ocpus=2, memory_in_gbs=32)
        .with_executor_shape("VM.Standard.E4.Flex")
        .with_executor_shape_config(ocpus=4, memory_in_gbs=64)
        .with_spark_version("3.2.1")
    )
    runtime_config = (
        DataFlowRuntime()
        .with_script_uri(os.path.join(td, "script.py"))
        .with_script_bucket("oci://<mybucket>@<mynamespace>/<subdir_to_put_and_get_script>")
        .with_custom_conda(uri="oci://<mybucket>@<mynamespace>/<path_to_conda_pack>")
        .with_arguments(["run-test", "-v", "-l", "5"])
    )
    df = Job(name=name, infrastructure=dataflow_configs, runtime=runtime_config)
    df.create()
    df_run = df.run()

From the Command Line

Again, assume you have the following two files written in your current directory as script.py and dataflow.yaml respectively:

# script.py
from pyspark.sql import SparkSession
import click

@click.command()
@click.argument("app_name")
@click.option(
    "--limit", "-l", help="max number of row to print", default=10, required=False
)
@click.option("--verbose", "-v", help="print out result in verbose mode", is_flag=True)
def main(app_name, limit, verbose):
    Create a Spark session
    spark = SparkSession.builder.appName(app_name).getOrCreate()

    Load a csv file from dataflow public storage
    df = (
        spark.read.format("csv")
        .option("header", "true")
        .option("multiLine", "true")
        .load(
            "oci://oow_2019_dataflow_lab@bigdatadatasciencelarge/usercontent/kaggle_berlin_airbnb_listings_summary.csv"
        )
    )

    Create a temp view and do some SQL operations
    df.createOrReplaceTempView("berlin")
    query_result_df = spark.sql(
        """
        SELECT
            city,
            zipcode,
            CONCAT(latitude,',', longitude) AS lat_long
        FROM berlin
        """
    ).limit(limit)

    # Convert the filtered Spark DataFrame into JSON format
    # Note: we are writing to the spark stdout log so that we can retrieve the log later at the end of the notebook.
    if verbose:
        rows = query_result_df.toJSON().collect()
        for i, row in enumerate(rows):
            print(f"record {i}")
            print(row)


if __name__ == "__main__":
    main()

# dataflow.yaml
kind: job
spec:
    name: dataflow-app-<uuid>
    infrastructure:
        kind: infrastructure
        spec:
            compartmentId: oci.xx.<compartment_id>
            logsBucketUri: oci://<mybucket>@<mynamespace>/<dataflow-logs-prefix>
            driverShape: VM.Standard.E4.Flex
            driverShapeConfig:
                ocpus: 2
                memory_in_gbs: 32
            executorShape: VM.Standard.E4.Flex
            executorShapeConfig:
                ocpus: 4
                memory_in_gbs: 64
            sparkVersion: 3.2.1
            numExecutors: 1
        type: dataFlow
    runtime:
        kind: runtime
        spec:
            scriptUri: script.py
            scriptBucket: oci://<mybucket>@<mynamespace>/<subdir_to_put_and_get_script>
            conda:
                uri: oci://<mybucket>@<mynamespace>/<path_to_conda_pack>
                type: published
            args:
                - "run-test"
                - "-v"
                - "-l"
                - "5"

ads jobs run -f dataflow.yaml

Setup and Installation

Notebook Session Development

Follow these set up instructions to submit Spark Jobs to Data Flow from an OCI Notebook Session.

Pyspark Environment

To setup PySpark environment, install one of the PySpark conda environments from the Environment Explorer

Example -

odsc conda install -s pyspark30_p37_cpu_v5

Find the information about the latest pyspark conda environment here

Activate the conda environment to upgrade to the latest oracle-ads

conda activate /home/datascience/conda/pyspark30_p37_cpu_v5
pip install oracle-ads[data_science, data, opctl] --upgrade

Configuring core-site.xml

When the conda environment is installed, a templated version of core-site.xml is also installed. You can update the core-site.xml file using an automated configuration or manually.

Authentication with Resource Principals

Authentication to Object Storage can be done with a resource principal.

For automated configuration, run the following command in a terminal

odsc core-site config -a resource_principal

This command will populate the file ~/spark_conf_dir/core-site.xml with the values needed to connect to Object Storage.

The following command line options are available:

-a, –authentication Authentication mode. Supports resource_principal and api_key (default).
-r, –region Name of the region.
-o, –overwrite Overwrite core-site.xml.
-O, –output Output path for core-site.xml.
-q, –quiet Suppress non-error output.
-h, –help Show help message and exit.

To manually configure the core-site.xml file, you edit the file, and then specify these values:

fs.oci.client.hostname: The Object Storage endpoint for your region. See https://docs.oracle.com/iaas/api/#/en/objectstorage/20160918/ for available endpoints.

fs.oci.client.custom.authenticator: Set the value to com.oracle.bmc.hdfs.auth.ResourcePrincipalsCustomAuthenticator.

When using resource principals, these properties don’t need to be configured:

fs.oci.client.auth.tenantId
fs.oci.client.auth.userId
fs.oci.client.auth.fingerprint
fs.oci.client.auth.pemfilepath

The following example core-site.xml file illustrates using resource principals for authentication to access Object Storage:

<?xml version="1.0"?>
<configuration>
  <property>
    <name>fs.oci.client.hostname</name>
    <value>https://objectstorage.us-ashburn-1.oraclecloud.com</value>
  </property>
  <property>
    <name>fs.oci.client.custom.authenticator</name>
    <value>com.oracle.bmc.hdfs.auth.ResourcePrincipalsCustomAuthenticator</value>
  </property>
</configuration>

For details, see HDFS connector for Object Storage #using resource principals for authentication.

Authentication with API Keys

When using authentication with API keys, the core-site.xml file is be updated in two ways, automated or manual configuration.

For automated configuration, you use the odsc command line tool. With an OCI configuration file, you can run

odsc core-site config -o

By default, this command uses the OCI configuration file stored in ~/.oci/config, automatically populates the core-site.xml file, and then saves it to ~/spark_conf_dir/core-site.xml.

The following command line options are available:

-a, –authentication Authentication mode. Supports resource_principal and api_key (default).
-c, –configuration Path to the OCI configuration file.
-p, –profile Name of the profile.
-r, –region Name of the region.
-o, –overwrite Overwrite core-site.xml.
-O, –output Output path for core-site.xml.
-q, –quiet Suppress non-error output.
-h, --help Show help message and exit.

To manually configure the core-site.xml file, you must specify these parameters:

fs.oci.client.hostname: Address of Object Storage. For example, https://objectstorage.us-ashburn-1.oraclecloud.com. You must replace us-ashburn-1 with the region you are in. fs.oci.client.auth.tenantId: OCID of your tenancy. fs.oci.client.auth.userId: Your user OCID. fs.oci.client.auth.fingerprint: Fingerprint for the key pair. fs.oci.client.auth.pemfilepath: The fully qualified file name of the private key used for authentication.

The values of these parameters are found in the OCI configuration file.

Local Development

Follow these set up instructions to submit Spark Jobs to Data Flow from your local machine.

PySpark Environment

Prerequisite

You have completed Local Development Environment Setup

Use ADS CLI to setup a PySpark conda environment. Currently, the ADS CLI only supports fetching conda packs published by you. If you haven’t already published a conda pack, you can create one using ADS CLI

To install from your published environment source -

ads conda install oci://mybucket@mynamespace/path/to/pyspark/env

To create a conda pack for your local use -

cat <<EOF> pyspark.yaml

    dependencies:
        - pyspark
        - pip
        - pip:
            - oracle-ads
    name: pysparkenv
EOF

ads create -f pyspark.yaml

ads publish -s pysparkenv

Developing in Visual Studio Code

Prerequisites

Setup Visual Studio Code development environment by following steps from Local Development Environment Setup
ads conda install <oci uri of pyspark conda environment>. Currently, we cannot access service pack directly. You can instead publish a pyspark service pack to your object storage and use the URI for the pack in OCI Object Storage.

Once the development environment is setup, you could write your code and run it from the terminal of the Visual Studio Code.

core-site.xml is setup automatically when you install a pyspark conda pack.

Logging From DataFlow

If using the ADS Python SDK,

To create and run a Data Flow application, you must specify a compartment and a bucket for storing logs under the same compartment:

compartment_id = "<compartment_id>"
logs_bucket_uri = "<logs_bucket_uri>"

Ensure that you set up the correct policies. For instance, for Data Flow to access logs bucket, use a policy like:

ALLOW SERVICE dataflow TO READ objects IN tenancy WHERE target.bucket.name='dataflow-logs'

For more information, see the Data Flow documentation.

Running your Spark Application on OCI Data Flow

Submit your code to DataFlow for workloads that require larger resources.

Notebook Extension

For most Notebook users, local or OCI Notebook Sessions, the notebook extension is the most straightforward integration with dataflow. It’s a “Set It and Forget It” API with options to update ad-hoc. You can configure your dataflow runs by running ads opctl configure in the terminal.

After setting up your dataflow config, you can return to the Notebook. Import ads and DataFlowConfig:

import ads
from ads.jobs.utils import DataFlowConfig

Load the dataflow extension inside the notebook cell -

%load_ext ads.jobs.extension

Define config. If you have not yet configured your dataflow setting, or would like to amend the defaults, you can modify as shown below:

dataflow_config = DataFlowConfig()
dataflow_config.compartment_id = "ocid1.compartment.<your compartment ocid>"
dataflow_config.driver_shape = "VM.Standard.E4.Flex"
dataflow_config.driver_shape_config = oci.data_flow.models.ShapeConfig(ocpus=2, memory_in_gbs=32)
dataflow_config.executor_shape = "VM.Standard.E4.Flex"
dataflow_config.executor_shape_config = oci.data_flow.models.ShapeConfig(ocpus=4, memory_in_gbs=64)
dataflow_config.logs_bucket_uri = "oci://<my-bucket>@<my-tenancy>/"
dataflow_config.spark_version = "3.2.1"
dataflow_config.configuration = {"spark.driver.memory": "512m"}
dataflow_config.private_endpoint_id = "ocid1.dataflowprivateendpoint.oc1.iad.<your private endpoint ocid>"

Use the config defined above to submit the cell.

Tip

Get more information about the dataflow extension by running %dataflow -h

Call the dataflow magic command in the first line of your cell to run it on dataflow.

%%dataflow run -f a_script.py -c {dataflow_config} -w -o -- abc -l 5 -v

This header will: - save the cell as a file called script.py and store it in your dataflow_config.script_bucket - After the -- notation, all parameters are sent to your script. For example abc is a positional argument, and l and v are named arguments.

Below is a full example:

%%dataflow run -f a_script.py -c {dataflow_config} -w -o -- abc -l 5 -v
from pyspark.sql import SparkSession
import click


@click.command()
@click.argument("app_name")
@click.option(
    "--limit", "-l", help="max number of rows to print", default=10, required=False
)
@click.option("--verbose", "-v", help="print out result in verbose mode", is_flag=True)
def main(app_name, limit, verbose):
    # Create a Spark session
    spark = SparkSession.builder.appName(app_name).getOrCreate()

    # Load a csv file from dataflow public storage
    df = (
        spark.read.format("csv")
        .option("header", "true")
        .option("multiLine", "true")
        .load(
            "oci://oow_2019_dataflow_lab@bigdatadatasciencelarge/usercontent/kaggle_berlin_airbnb_listings_summary.csv"
        )
    )

    # Create a temp view and do some SQL operations
    df.createOrReplaceTempView("berlin")
    query_result_df = spark.sql(
        """
        SELECT
            city,
            zipcode,
            CONCAT(latitude,',', longitude) AS lat_long
        FROM berlin
    """
    ).limit(limit)

    # Convert the filtered Spark DataFrame into JSON format
    # Note: we are writing to the spark stdout log so that we can retrieve the log later at the end of the notebook.
    if verbose:
        rows = query_result_df.toJSON().collect()
        for i, row in enumerate(rows):
            print(f"record {i}")
            print(row)


if __name__ == "__main__":
    main()

ADS CLI

Prerequisites

Tip

If, for some reason, you are unable to use CLI, instead skip to the Create, Run Data Flow Application Using ADS Python SDK section below.

Sometimes your code is too complex to run in a single cell, and it’s better run as a notebook or file. In that case, use the ADS Opctl CLI.

To submit your notebook to DataFlow using the ads CLI, run:

ads opctl run -s <folder where notebook is located> -e <notebook name> -b dataflow

Tip

You can avoid running cells that are not DataFlow environment compatible by tagging the cells and then providing the tag names to ignore. In the following example cells that are tagged ignore and remove will be ignored - --exclude-tag ignore --exclude-tag remove

Tip

You can run the notebook in your local pyspark environment before submitting to DataFlow using the same CLI with -b local

# Activate the Pyspark conda environment in local
ads opctl run -s <notebook directory> -e <notebook file> -b local

You could submit a notebook using ADS SDK APIs. Here is an example to submit a notebook -

from ads.jobs import Job, DataFlow, DataFlowNotebookRuntime

df = (
    DataFlow()
    .with_compartment_id(
        "ocid1.compartment.oc1.<your compartment id>"
    )
    .with_driver_shape("VM.Standard.E4.Flex")
            .with_driver_shape_config(ocpus=2, memory_in_gbs=32)
            .with_executor_shape("VM.Standard.E4.Flex")
            .with_executor_shape_config(ocpus=4, memory_in_gbs=64)
    .with_logs_bucket_uri("oci://mybucket@mytenancy/")
    .with_private_endpoint_id("ocid1.dataflowprivateendpoint.oc1.iad.<your private endpoint ocid>")
    .with_configuration({
        "spark.driverEnv.myEnvVariable": "value1",
        "spark.executorEnv.myEnvVariable": "value2",
    })
)
rt = (
    DataFlowNotebookRuntime()
    .with_notebook(
        "<path to notebook>"
    )  # This could be local path or http path to notebook ipynb file
    .with_script_bucket("<my-bucket>")
    .with_exclude_tag(["ignore", "remove"])  # Cells to Ignore
)
job = Job(infrastructure=df, runtime=rt).create(overwrite=True)
df_run = job.run(wait=True)

ADS Python SDK

To create a Data Flow application using the ADS Python API you need two components:

DataFlow, a subclass of Infrastructure.
DataFlowRuntime, a subclass of Runtime.

DataFlow stores properties specific to Data Flow service, such as compartment_id, logs_bucket_uri, and so on. You can set them using the with_{property} functions:

with_compartment_id
with_configuration
with_driver_shape
with_driver_shape_config
with_executor_shape
with_executor_shape_config
with_language
with_logs_bucket_uri
with_metastore_id (doc)
with_num_executors
with_spark_version
with_warehouse_bucket_uri
with_private_endpoint_id (doc)

For more details, see DataFlow class documentation.

DataFlowRuntime stores properties related to the script to be run, such as the path to the script and CLI arguments. Likewise all properties can be set using with_{property}. The DataFlowRuntime properties are:

with_script_uri
with_script_bucket
with_archive_uri (doc)
with_archive_bucket
with_custom_conda
with_configuration

For more details, see the runtime class documentation.

Since service configurations remain mostly unchanged across multiple experiments, a DataFlow object can be reused and combined with various DataFlowRuntime parameters to create applications.

In the following “hello-world” example, DataFlow is populated with compartment_id, driver_shape, driver_shape_config, executor_shape, executor_shape_config and spark_version. DataFlowRuntime is populated with script_uri and script_bucket. The script_uri specifies the path to the script. It can be local or remote (an Object Storage path). If the path is local, then script_bucket must be specified additionally because Data Flow requires a script to be available in Object Storage. ADS performs the upload step for you, as long as you give the bucket name or the Object Storage path prefix to upload the script. Either can be given to script_bucket. For example, either with_script_bucket("<bucket_name>") or with_script_bucket("oci://<bucket_name>@<namespace>/<prefix>") is accepted. In the next example, the prefix is given for script_bucket.

from ads.jobs import DataFlow, Job, DataFlowRuntime
from uuid import uuid4
import os
import tempfile

with tempfile.TemporaryDirectory() as td:
    with open(os.path.join(td, "script.py"), "w") as f:
        f.write(
            """
import pyspark

def main():
    print("Hello World")
    print("Spark version is", pyspark.__version__)

if __name__ == "__main__":
    main()
        """
        )
    name = f"dataflow-app-{str(uuid4())}"
    dataflow_configs = (
        DataFlow()
        .with_compartment_id("oci.xx.<compartment_id>")
        .with_logs_bucket_uri("oci://mybucket@mynamespace/dflogs")
        .with_driver_shape("VM.Standard.E4.Flex")
                .with_driver_shape_config(ocpus=2, memory_in_gbs=32)
                .with_executor_shape("VM.Standard.E4.Flex")
                .with_executor_shape_config(ocpus=4, memory_in_gbs=64)
        .with_spark_version("3.0.2")
    )
    runtime_config = (
        DataFlowRuntime()
        .with_script_uri(os.path.join(td, "script.py"))
        .with_script_bucket("oci://mybucket@namespace/prefix")
        .with_custom_conda("oci://<mybucket>@<mynamespace>/<path/to/conda_pack>")
        .with_configuration({
            "spark.driverEnv.myEnvVariable": "value1",
            "spark.executorEnv.myEnvVariable": "value2",
        })
    )
    df = Job(name=name, infrastructure=dataflow_configs, runtime=runtime_config)
    df.create()

To run this application, you could use:

df_run = df.run()

After the run completes, check the stdout log from the application by running:

print(df_run.logs.application.stdout)

You should this in the log:

Hello World
Spark version is 3.0.2

Note on Policy

ALLOW SERVICE dataflow TO READ objects IN tenancy WHERE target.bucket.name='dataflow-logs'

Data Flow supports adding third-party libraries using a ZIP file, usually called archive.zip, see the Data Flow documentation about how to create ZIP files. Similar to scripts, you can specify an archive ZIP for a Data Flow application using with_archive_uri. In the next example, archive_uri is given as an Object Storage location. archive_uri can also be local so you must specify with_archive_bucket and follow the same rule as with_script_bucket.

from ads.jobs import DataFlow, DataFlowRun, DataFlowRuntime, Job
from uuid import uuid4
import tempfile
import os

with tempfile.TemporaryDirectory() as td:
    with open(os.path.join(td, "script.py"), "w") as f:
        f.write(
            '''
from pyspark.sql import SparkSession
import click


@click.command()
@click.argument("app_name")
@click.option(
    "--limit", "-l", help="max number of row to print", default=10, required=False
)
@click.option("--verbose", "-v", help="print out result in verbose mode", is_flag=True)
def main(app_name, limit, verbose):
    # Create a Spark session
    spark = SparkSession.builder.appName(app_name).getOrCreate()

    # Load a csv file from dataflow public storage
    df = (
        spark.read.format("csv")
        .option("header", "true")
        .option("multiLine", "true")
        .load(
            "oci://oow_2019_dataflow_lab@bigdatadatasciencelarge/usercontent/kaggle_berlin_airbnb_listings_summary.csv"
        )
    )

    # Create a temp view and do some SQL operations
    df.createOrReplaceTempView("berlin")
    query_result_df = spark.sql(
        """
        SELECT
            city,
            zipcode,
            CONCAT(latitude,',', longitude) AS lat_long
        FROM berlin
    """
    ).limit(limit)

    # Convert the filtered Spark DataFrame into JSON format
    # Note: we are writing to the spark stdout log so that we can retrieve the log later at the end of the notebook.
    if verbose:
        rows = query_result_df.toJSON().collect()
        for i, row in enumerate(rows):
            print(f"record {i}")
            print(row)


if __name__ == "__main__":
    main()
        '''
        )

    name = f"dataflow-app-{str(uuid4())}"
    dataflow_configs = (
        DataFlow()
        .with_compartment_id("oci1.xxx.<compartment_ocid>")
        .with_logs_bucket_uri("oci://mybucket@mynamespace/prefix")
        .with_driver_shape("VM.Standard.E4.Flex")
                .with_driver_shape_config(ocpus=2, memory_in_gbs=32)
                .with_executor_shape("VM.Standard.E4.Flex")
                .with_executor_shape_config(ocpus=4, memory_in_gbs=64)
        .with_spark_version("3.0.2")
        .with_configuration({
            "spark.driverEnv.myEnvVariable": "value1",
            "spark.executorEnv.myEnvVariable": "value2",
        })
    )
    runtime_config = (
        DataFlowRuntime()
        .with_script_uri(os.path.join(td, "script.py"))
        .with_script_bucket("oci://<bucket>@<namespace>/prefix/path")
        .with_archive_uri("oci://<bucket>@<namespace>/prefix/archive.zip")
        .with_custom_conda(uri="oci://<mybucket>@<mynamespace>/<my-conda-uri>")
    )
    df = Job(name=name, infrastructure=dataflow_configs, runtime=runtime_config)
    df.create()

You can pass arguments to a Data Flow run as a list of strings:

df_run = df.run(args=["run-test", "-v", "-l", "5"])

You can save the application specification into a YAML file for future reuse. You could also use the json format.

print(df.to_yaml("sample-df.yaml"))

You can also load a Data Flow application directly from the YAML file saved in the previous example:

df2 = Job.from_yaml(uri="sample-df.yaml")

Creating a new job and a run:

df_run2 = df2.create().run()

Deleting a job cancels associated runs:

df2.delete()
df_run2.status

You can also load a Data Flow application from an OCID:

df3 = Job.from_dataflow_job(df.id)

Creating a run under the same application:

df_run3 = df3.run()

Now there are 2 runs under the df application:

assert len(df.run_list()) == 2

When you run a Data Flow application, a DataFlowRun object is created. You can check the status, wait for a run to finish, check its logs afterwards, or cancel a run in progress. For example:

df_run.status
df_run.wait()

watch is an alias of wait, so you can also call df_run.watch().

There are three types of logs for a run:

application log
driver log
executor log

Each log consists of stdout and stderr. For example, to access stdout from application log, you could use:

df_run.logs.application.stdout

Then you could check it with:

df_run.logs.application.stderr
df_run.logs.executor.stdout
df_run.logs.executor.stderr

You can also examine head or tail of the log, or download it to a local path. For example,

log = df_run.logs.application.stdout
log.head(n=1)
log.tail(n=1)
log.download(<local-path>)

For the sample script, the log prints first five rows of a sample dataframe in JSON and it looks like:

record 0
{"city":"Berlin","zipcode":"10119","lat_long":"52.53453732241747,13.402556926822387"}
record 1
{"city":"Berlin","zipcode":"10437","lat_long":"52.54851279221664,13.404552826587466"}
record 2
{"city":"Berlin","zipcode":"10405","lat_long":"52.534996191586714,13.417578665333295"}
record 3
{"city":"Berlin","zipcode":"10777","lat_long":"52.498854933130026,13.34906453348717"}
record 4
{"city":"Berlin","zipcode":"10437","lat_long":"52.5431572633131,13.415091104515707"}

Calling log.head(n=1) returns this:

'record 0'

Calling log.tail(n=1) returns this:

{"city":"Berlin","zipcode":"10437","lat_long":"52.5431572633131,13.415091104515707"}

A link to run the page in the OCI Console is given using the run_details_link property:

df_run.run_details_link

To list Data Flow applications, a compartment id must be given with any optional filtering criteria. For example, you can filter by name of the application:

Job.dataflow_job(compartment_id=compartment_id, display_name=name)

YAML

You can create a Data Flow job directly from a YAML string. You can pass a YAML string into the Job.from_yaml() function to build a Data Flow job:

kind: job
spec:
  id: <dataflow_app_ocid>
  infrastructure:
    kind: infrastructure
    spec:
      compartmentId: <compartment_id>
      driverShape: VM.Standard.E4.Flex
      driverShapeConfig:
        ocpus: 2
        memory_in_gbs: 32
      executorShape: VM.Standard.E4.Flex
      executorShapeConfig:
        ocpus: 4
        memory_in_gbs: 64
      id: <dataflow_app_ocid>
      language: PYTHON
      logsBucketUri: <logs_bucket_uri>
      numExecutors: 1
      sparkVersion: 3.2.1
      privateEndpointId: <private_endpoint_ocid>
    type: dataFlow
  name: dataflow_app_name
  runtime:
    kind: runtime
    spec:
      configuration:
          spark.driverEnv.myEnvVariable: value1
          spark.executorEnv.myEnvVariable: value2
      scriptBucket: bucket_name
      scriptPathURI: oci://<bucket_name>@<namespace>/<prefix>
    type: dataFlow

Data Flow Infrastructure YAML Schema

kind:
    allowed:
        - infrastructure
    required: true
    type: string
spec:
    required: true
    type: dict
    schema:
        compartmentId:
            required: false
            type: string
        displayName:
            required: false
            type: string
        driverShape:
            required: false
            type: string
        driverShapeConfig:
            required: false
            type: dict
            schema:
                ocpus:
                    required: true
                    type: float
                memory_in_gbs:
                    required: true
                    type: float
        executorShape:
            required: false
            type: string
        executorShapeConfig:
            required: false
            type: dict
            schema:
                ocpus:
                    required: true
                    type: float
                memory_in_gbs:
                    required: true
                    type: float
        id:
            required: false
            type: string
        language:
            required: false
            type: string
        logsBucketUri:
            required: false
            type: string
        metastoreId:
            required: false
            type: string
        numExecutors:
            required: false
            type: integer
        sparkVersion:
            required: false
            type: string
        privateEndpointId:
            required: false
            type: string
        configuration:
            required: false
            type: dict
type:
    allowed:
        - dataFlow
    required: true
    type: string

Data Flow Runtime YAML Schema

kind:
    allowed:
        - runtime
    required: true
    type: string
spec:
    required: true
    type: dict
    schema:
        archiveBucket:
            required: false
            type: string
        archiveUri:
            required: false
            type: string
        args:
            nullable: true
            required: false
            schema:
                type: string
            type: list
        conda:
            nullable: false
            required: false
            type: dict
            schema:
                slug:
                    required: true
                    type: string
                type:
                    allowed:
                        - service
                    required: true
                    type: string
        configuration:
            required: false
            type: dict
        freeform_tag:
            required: false
            type: dict
        scriptBucket:
            required: false
            type: string
        scriptPathURI:
            required: false
            type: string
type:
    allowed:
        - dataFlow
    required: true
    type: string

`spark-defaults.conf`

The spark-defaults.conf file is used to define the properties that are used by Spark. This file can be configured manually or with the aid of the odsc command-line tool. The best practice is to use the odsc data-catalog config command-line tool when you want to connect to Data Catalog. It gathers information about your environment and uses that to build the file.

The odsc data-catalog config command-line tool uses the --metastore option to define the Data Catalog Metastore OCID. There are no required command-line options. Default values are used or values are taken from your notebook session environment and OCI configuration file. Below is a discussion of common parameters that you may need to override.

The --authentication option sets the authentication mode. It supports resource principal and API keys. The preferred method for authentication is resource principal and this is sent with --authentication resource_principal. If you want to use API keys then used the option --authentication api_key. If the --authentication is not specified, API keys will be used. When API keys are used, information from the OCI configuration file is used to create the spark-defaults.conf file.

The Object Storage and Data Catalog are regional services. By default, the region is set to the region that your notebook session is in. This information is taken from the environment variable NB_REGION. Use the --region option to override this behavior.

The default location of the spark-defaults.conf file is in the ~/spark_conf_dir directory, as defined in the SPARK_CONF_DIR environment variable. Use the --output option to define the directory where the file is to be written.

`odsc` Command-line

The odsc data-catalog config command-line tool is ideal for setting up the spark-defaults.conf file as it gathers information about your environment and uses that to build the file.

You will need to determine what settings are appropriate for your configuration. However, the following will work for most configurations.

odsc data-catalog config --authentication resource_principal

If the option --authentication api_key is used, it will extract information from the OCI configuration file that is stored in ~/.oci/config. Use the --config option to change the path and the --profile option to specify what OCI configuration profile will be used. The default profile is DEFAULT.

A default Data Catalog Metastore OCID can be set using the --metastore option. This value can be overridden at run-time.

odsc data-catalog config --authentication resource_principal --metastore <metastore_id>

The <metastore_id> must be replaced with the OCID for the Data Catalog Metastore that is to be used.

For details on the command-line option use the command:

odsc data-catalog config --help

Manual

The odsc command-line tool is the preferred method for configuring the spark-defaults.conf file. However, if you are not in a notebook session or if you have special requirements, you may need to manually configure the file. This section will guide you through the steps.

When a Data Science Conda environment is installed, it includes a template of the spark-defaults.conf file. The following sections provide guidance to make the required changes.

These parameters define the Object Storage address that backs the Data Catalog entry. This is the location of the data warehouse. You also need to define the address of the Data Catalog Metastore.

spark.hadoop.fs.oci.client.hostname: Address of Object Storage for the data warehouse. For example, https://objectstorage.us-ashburn-1.oraclecloud.com. Replace us-ashburn-1 with the region you are in.
spark.hadoop.oci.metastore.uris: The address of Data Catalog Metastore. For example, https://datacatalog.us-ashburn-1.oci.oraclecloud.com/ Replace us-ashburn-1 with the region you are in.

You can set a default metastore with the following parameter. This can be overridden at run time. Setting it is optional.

spark.hadoop.oracle.dcat.metastore.id: The OCID of Data Catalog Metastore. For example, ocid1.datacatalogmetastore..<unique_id>

Depending on the authentication method that is to be used there are additional parameters that need to be set. See the following sections for guidance.

Resource Principal

Update the spark-defaults.conf file parameters to use resource principal to authenticate:

spark.hadoop.fs.oci.client.custom.authenticator: Set the value to com.oracle.bmc.hdfs.auth.ResourcePrincipalsCustomAuthenticator.
spark.hadoop.oracle.dcat.metastore.client.custom.authentication_provider: Set the value to com.oracle.bmc.hdfs.auth.ResourcePrincipalsCustomAuthenticator.

API Keys

Update the spark-defaults.conf file parameters to use API keys to authenticate:

spark.hadoop.OCI_FINGERPRINT_METADATA: Fingerprint for the key pair being used.
spark.hadoop.OCI_PASSPHRASE_METADATA: Passphrase used for the key if it is encrypted.
spark.hadoop.OCI_PVT_KEY_FILE_PATH: The full path and file name of the private key used for authentication.
spark.hadoop.OCI_REGION_METADATA: An Oracle Cloud Infrastructure region. Example: us-ashburn-1
spark.hadoop.OCI_USER_METADATA: Your user OCID.
spark.hadoop.fs.oci.client.auth.fingerprint: Fingerprint for the key pair being used.
spark.hadoop.fs.oci.client.auth.passphrase: Passphrase used for the key if it is encrypted.
spark.hadoop.fs.oci.client.auth.pemfilepath: The full path and file name of the private key used for authentication.
spark.hadoop.fs.oci.client.auth.tenantId: OCID of your tenancy.
spark.hadoop.fs.oci.client.auth.userId: Your user OCID.
spark.hadoop.fs.oci.client.custom.authenticator: Set the value to com.oracle.pic.dcat.metastore.commons.auth.provider.UserPrincipalsCustomAuthenticationDetailsProvider
spark.hadoop.spark.hadoop.OCI_TENANT_METADATA: OCID of your tenancy.

The values of these parameters are found in the OCI configuration file.

Data Catalog Metastore

This section demonstrates how to configure the spark-defaults.conf file so that you can connect with the Oracle Cloud Infrastructure (OCI) Data Catalog Metastore. This connection is used to run a PySpark application using OCI Data Flow and Data Science Jobs. The data will be stored in OCI Object Storage. Thus, you will work with data that is stored in Object Storage, information about the location and structure of that data will be managed by Data Catalog Metastore, compute will be provided by Data Flow and all of this will be run in a Job.

OCI Data Catalog is a metadata management service that helps data professionals discover data and support data governance. The Data Catalog Metastore provides schema definitions for objects in structured and unstructured data assets that reside in Object Storage. Use the metastore as a central metadata repository to manage data tables that are backed by files in Object Storage.

OCI Data Flow is a fully managed Apache Spark service. This section demonstrates how to use PySpark to create Spark applications.

Data Science Jobs allows you to run customized tasks outside of a notebook session. A Job is a template that describes a task that you want to perform. In this section, that task is to run a PySpark application using Data Flow. Since the Job is run outside of a notebook, command-line arguments can be passed to the Job such that it performs customized activities. OCI Logging is used to capture events. You can also read and write data to Object Storage directly or with the aid of Data Catalog.

Data Flow can access the Data Catalog Metastore to securely store and retrieve schema definitions for unstructured and structured data assets in Object Storage. For integration with Data Flow, the metastore provides an invocation endpoint. This endpoint is a Hive Metastore interface.

Apache Hive is a data warehousing framework that facilitates read, write, or manage operations on large datasets residing in distributed file systems. The Data Catalog Metastore is backed by the Apache Hive Metastore. A Hive Metastore is the central repository of metadata for a Hive cluster. It stores metadata for data structures such as databases, tables, and partitions in a relational database, backed by files maintained in Object Storage. Apache Spark SQL makes use of a Hive Metastore for this purpose.

Prerequisite

To access the data in the Data Catalog or work with Data Flow, there are a number of steps that need to be completed.

To configure Data Flow you will need to:

DataFlow requires a bucket to store the logs, and a data warehouse bucket. Refer to the Data Flow documentation for setting up storage.
DataFlow requires policies to be set in IAM to access resources to manage and run applications/sessions. Refer to the Data Flow documentation on how to setup policies.
DataFlow natively supports conda packs published to OCI Object Storage. Ensure the Data Flow Resource has read access to the bucket or path of your published conda pack, and that the spark version >= 3 when running your Data Flow Application/Session.

The core-site.xml file needs to be configured.

To configure Data Catalog you will need to:

Data Catalog requires policies to be set in IAM. Refer to the Data Catalog documentation on how to setup policies.

The spark-defaults.conf file needs to be configured.

Quick Start

Data Flow

from ads.jobs import DataFlow, DataFlowRun, DataFlowRuntime

# Update these values
job_name = "<job_name>"
logs_bucket = "oci://<bucket_name>@<namespace>/<prefix>"
metastore_id = "<metastore_id>"
script_bucket = "oci://<bucket_name>@<namespace>/<prefix>"

compartment_id = os.environ.get("NB_SESSION_COMPARTMENT_OCID")
driver_shape = "VM.Standard.E4.Flex"
driver_shape_config = {"ocpus":2, "memory_in_gbs":32}
executor_shape = "VM.Standard.E4.Flex"
executor_shape_config = {"ocpus":4, "memory_in_gbs":64}
spark_version = "3.2.1"

# A python script to be run in Data Flow
script = '''
from pyspark.sql import SparkSession

def main():

    database_name = "employee_attrition"
    table_name = "orcl_attrition"

    # Create a Spark session
    spark = SparkSession \\
        .builder \\
        .appName("Python Spark SQL basic example") \\
        .enableHiveSupport() \\
        .getOrCreate()

    # Load a CSV file from a public Object Storage bucket
    df = spark \\
        .read \\
        .format("csv") \\
        .option("header", "true") \\
        .option("multiLine", "true") \\
        .load("oci://hosted-ds-datasets@bigdatadatasciencelarge/synthetic/orcl_attrition.csv")

    print(f"Creating {database_name}")
    spark.sql(f"DROP DATABASE IF EXISTS {database_name} CASCADE")
    spark.sql(f"CREATE DATABASE IF NOT EXISTS {database_name}")

    # Write the data to the database
    df.write.mode("overwrite").saveAsTable(f"{database_name}.{table_name}")

    # Use Spark SQL to read from the database.
    query_result_df = spark.sql(f"""
                                SELECT EducationField, SalaryLevel, JobRole FROM {database_name}.{table_name} limit 10
                                """)

    # Convert the filtered Apache Spark DataFrame into JSON format and write it out to stdout
    # so that it can be captured in the log.
    print('\\n'.join(query_result_df.toJSON().collect()))

if __name__ == '__main__':
    main()
'''

# Saves the python script to local path.
dataflow_base_folder = tempfile.mkdtemp()
script_uri = os.path.join(dataflow_base_folder, "example.py")

with open(script_uri, 'w') as f:
    print(script.strip(), file=f)

dataflow_configs = DataFlow(
    {
        "compartment_id": compartment_id,
        "driver_shape": driver_shape,
        "driver_shape_config": driver_shape_config,
        "executor_shape": executor_shape,
        "executor_shape_config": executor_shape_config,
        "logs_bucket_uri": log_bucket_uri,
        "metastore_id": metastore_id,
        "spark_version": spark_version
    }
)

runtime_config = DataFlowRuntime(
    {
        "script_uri": pyspark_file_path,
        "script_bucket": script_uri
    }
)

# creates a Data Flow application with DataFlow and DataFlowRuntime.
df_job = Job(name=job_name,
             infrastructure=dataflow_configs,
             runtime=runtime_config)
df_app = df_job.create()
df_run = df_app.run()

# check a job log
df_run.watch()

Interactive Spark

from pyspark.sql import SparkSession

# Update these values
warehouse_uri = "<warehouse_uri>"
metastore_id = "<metastore_id>"

database_name = "ODSC_DEMO"
table_name = "ODSC_PYSPARK_METASTORE_DEMO"

# create a spark session
spark = SparkSession \
    .builder \
    .appName("Python Spark SQL Hive integration example") \
    .config("spark.sql.warehouse.dir", warehouse_uri) \
    .config("spark.hadoop.oracle.dcat.metastore.id", metastore_id) \
    .enableHiveSupport() \
    .getOrCreate()
spark.sparkContext.setLogLevel("ERROR")

# show the databases in the warehouse:
spark.sql("SHOW DATABASES").show()
spark.sql(f"DROP DATABASE IF EXISTS {database_name} CASCADE")
spark.sql(f"CREATE DATABASE {database_name}")

# Load the Employee Attrition data file from OCI Object Storage into a Spark DataFrame:
file_path = "oci://hosted-ds-datasets@bigdatadatasciencelarge/synthetic/orcl_attrition.csv"
input_dataframe = spark.read.option("header", "true").csv(file_path)
input_dataframe.write.mode("overwrite").saveAsTable(f"{database_name}.{table_name}")

# explore data
spark_df = spark.sql(f"""
                     SELECT EducationField, SalaryLevel, JobRole FROM {database_name}.{table_name} limit 10
                     """)
spark_df.show()

Data Flow

This example demonstrates how to create a Data Flow application that is connected to the Data Catalog Metastore. It creates a PySpark script, then a Data Flow application. This application can be run by directly by Data Flow or as part of a Job.

This section runs Hive queries using Data Flow. When the Data Catalog is being used the only changes that need to be made are to provide the metastore OCID.

PySpark Script

A PySpark script is needed for the Data Flow application. The following code creates that script. The script will use Spark to load a CSV file from a public Object Storage bucket. It will then create a database and write the file to Object Storage. Finally, it will use Spark SQL to query the database and print the records in JSON format.

There is nothing in the PySpark script that is specific to using Data Catalog Metastore. The script treats the database as a standard Hive database.

script = '''
from pyspark.sql import SparkSession

def main():

    database_name = "employee_attrition"
    table_name = "orcl_attrition"

    # Create a Spark session
    spark = SparkSession \\
        .builder \\
        .appName("Python Spark SQL basic example") \\
        .enableHiveSupport() \\
        .getOrCreate()

    # Load a CSV file from a public Object Storage bucket
    df = spark \\
        .read \\
        .format("csv") \\
        .option("header", "true") \\
        .option("multiLine", "true") \\
        .load("oci://hosted-ds-datasets@bigdatadatasciencelarge/synthetic/orcl_attrition.csv")

    print(f"Creating {database_name}")
    spark.sql(f"DROP DATABASE IF EXISTS {database_name} CASCADE")
    spark.sql(f"CREATE DATABASE IF NOT EXISTS {database_name}")

    # Write the data to the database
    df.write.mode("overwrite").saveAsTable(f"{database_name}.{table_name}")

    # Use Spark SQL to read from the database.
    query_result_df = spark.sql(f"""
                                SELECT EducationField, SalaryLevel, JobRole FROM {database_name}.{table_name} limit 10
                                """)

    # Convert the filtered Apache Spark DataFrame into JSON format and write it out to stdout
    # so that it can be captured in the log.
    print('\\n'.join(query_result_df.toJSON().collect()))

if __name__ == '__main__':
    main()
'''

# Save the PySpark script to a file
dataflow_base_folder = tempfile.mkdtemp()
script_uri = os.path.join(dataflow_base_folder, "example.py")

with open(script_uri, 'w') as f:
    print(script.strip(), file=f)

Create Application

To create a Data Flow application you will need DataFlow and DataFlowRuntime objects. A DataFlow object stores the properties that are specific to the Data Flow service. These would be things such as the compartment OCID, the URI to the Object Storage bucket for the logs, the type of hardware to be used, the version of Spark, and much more. If you are using a Data Catalog Metastore to manage a database, the metastore OCID is stored in this object. The DataFlowRuntime object stores properties related to the script to be run. This would be the bucket to be used for the script, the location of the PySpark script, and any command-line arguments.

Update the script_bucket, log_bucket, and metastore_id variables to match your tenancy’s configuration.

# Update values
log_bucket_uri = "oci://<bucket_name>@<namespace>/<prefix>"
metastore_id = "<metastore_id>"
script_bucket = "oci://<bucket_name>@<namespace>/<prefix>"

compartment_id = os.environ.get("NB_SESSION_COMPARTMENT_OCID")
driver_shape = "VM.Standard.E4.Flex"
driver_shape_config = {"ocpus":2, "memory_in_gbs":32}
executor_shape = "VM.Standard.E4.Flex"
executor_shape_config = {"ocpus":4, "memory_in_gbs":64}
spark_version = "3.2.1"

In the following example, a DataFlow is created and populated with the information that it needs to define the Data Flow service. Since, we are connecting to the Data Catalog Metastore to work with a Hive database, the metastore OCID must be given.

from ads.jobs import DataFlow, DataFlowRun, DataFlowRuntime

dataflow_configs = DataFlow(
    {"compartment_id": compartment_id,
     "driver_shape": driver_shape,
     "driver_shape_config": driver_shape_config,
     "executor_shape": executor_shape,
     "executor_shape_config": executor_shape_config,
     "logs_bucket_uri": log_bucket_uri,
     "metastore_id": metastore_id,
     "spark_version": spark_version}
)

In the following example, a DataFlowRuntime is created and populated with the URI to the PySpark script and the URI for the script bucket. The script URI specifies the path to the script. It can be local or remote (an Object Storage path). If the path is local, then a URI to the script bucket must also be specified. This is because Data Flow requires a script to be in Object Storage. If the specified path to the PySpark script is on a local drive, ADS will upload it for you.

runtime_config = DataFlowRuntime(
    {
        "script_bucket": script_uri
        "script_uri": pyspark_file_path,
    }
)

Run

The recommended approach for running Data Flow applications is to use a Job. This will prevent your notebook from being blocked.

A Job requires a name, infrastructure, and runtime settings. Update the following code to give the job a unique name. The infrastructure takes a DataFlow object and the runtime parameter takes a DataFlowRuntime object.

# Update values
job_name = "<job_name>"

df_job = Job(name=job_name,
             infrastructure=dataflow_configs,
             runtime=runtime_config)
df_app = df_job.create()
df_run = df_app.run()

Interactive Spark

This section demonstrates how to make connections to the Data Catalog Metastore and Object Storage. It uses Spark to load data from a public Object Storage file and creates a database. The metadata for the database is managed by the Data Catalog Metastore and the data is copied to your data warehouse bucket. Finally, Spark is used to make a Spark SQL query on the database.

Specify the bucket URI that will act as the data warehouse. Use the warehouse_uri variable and it should have the following format oci://<bucket_name>@<namespace_name>/<prefix>. Update the variable metastore_id with the OCID of the Data Catalog Metastore.

Create a Spark session that connects to the Data Catalog Metastore and the Object Storage that will act as the data warehouse.

from pyspark.sql import SparkSession

warehouse_uri = "<warehouse_uri>"
metastore_id = "<metastore_id>"

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL Hive integration example") \
    .config("spark.sql.warehouse.dir", warehouse_uri) \
    .config("spark.hadoop.oracle.dcat.metastore.id", metastore_id) \
    .enableHiveSupport() \
    .getOrCreate()
spark.sparkContext.setLogLevel("ERROR")

Load a data file from Object Storage into a Spark DataFrame. Create a database in the Data Catalog Metastore and then save the dataframe as a table. This will write the files to the location specified by the warehouse_uri variable.

database_name = "ODSC_DEMO"
table_name = "ODSC_PYSPARK_METASTORE_DEMO"
file_path = "oci://hosted-ds-datasets@bigdatadatasciencelarge/synthetic/orcl_attrition.csv"

input_dataframe = spark.read.option("header", "true").csv(file_path)
spark.sql(f"DROP DATABASE IF EXISTS {database_name} CASCADE")
spark.sql(f"CREATE DATABASE {database_name}")
input_dataframe.write.mode("overwrite").saveAsTable(f"{database_name}.{table_name}")

Use Spark SQL to read from the database.

spark_df = spark.sql(f"""
                     SELECT EducationField, SalaryLevel, JobRole FROM {database_name}.{table_name} limit 10
                     """)
spark_df.show()

OCI Data Flow Studio

This section demonstrates how to run interactive Spark workloads on a long lasting Oracle Cloud Infrastructure Data Flow cluster through Apache Livy integration.

Data Flow Studio allows you to:

Run Spark code against a Data Flow remote Spark cluster
Create a Data Flow Spark session with SparkContext and HiveContext against a Data Flow remote Spark cluster
Capture the output of Spark queries as a local Pandas data frame to interact easily with other Python libraries (e.g. matplotlib)

Key Features & Benefits:

Data Flow sessions support auto-scaling Data Flow cluster capabilities
Data Flow sessions support the use of conda environments as customizable Spark runtime environments

Limitations:

Data Flow sessions can last up to 7 days or 10,080 mins (maxDurationInMinutes).
Data Flow Sessions can only be accessed through OCI Data Science Notebook Sessions.
Not all SparkMagic commands are currently supported. To see the full list, run the %help command in a notebook cell.

Notebook Examples:

Prerequisite

Data Flow Sessions are accessible through the following conda environment:

PySpark 3.2 and Data Flow 2.0 (pyspark32_p38_cpu_v2)

You can customize pypspark32_p38_cpu_v1, publish it, and use it as a runtime environment for a Data Flow Session.

Policies

Data Flow requires policies to be set in IAM to access resources to manage and run sessions. Refer to the Data Flow Studio Policies documentation on how to setup policies.

Quick Start

import ads
ads.set_auth("resource_principal")

%load_ext dataflow.magics

%create_session -l python -c '{\
  "compartmentId":"<compartment_id>",\
  "displayName":"TestDataFlowSession",\
  "sparkVersion":"3.2.1",\
  "driverShape":"VM.Standard.E4.Flex",\
  "executorShape":"VM.Standard.E4.Flex",\
  "numExecutors":1,\
  "driverShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "executorShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "logsBucketUri" : "oci://<bucket_name>@<namespace>/"}'

%%spark
print(sc.version)

Data Flow Spark Magic

Data Flow Spark Magic is used for interactively working with remote Spark clusters through Livy, a Spark REST server, in Jupyter notebooks. It is a JupyterLab extension that you need to activate in your notebook.

%load_ext dataflow.magics

Use the %help method to get a list of all the available commands, along with a list of their arguments and example calls.

%help

Tip

To access the docstrings of any magic command and figure out what arguments to provide, simply add ? at the end of the command. For instance: %create_session?

Create Session

Example command for Flex shapes

To create a new Data Flow cluster session use the %create_session magic command.

%create_session -l python -c '{\
  "compartmentId":"<compartment_id>",\
  "displayName":"TestDataFlowSession",\
  "sparkVersion":"3.2.1",\
  "driverShape":"VM.Standard.E4.Flex",\
  "executorShape":"VM.Standard.E4.Flex",\
  "numExecutors":1,\
  "driverShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "executorShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "logsBucketUri" : "oci://<bucket_name>@<namespace>/"}'

Example command for Spark dynamic allocation (aka auto-scaling)

To help you save resources and reduce time on management, Spark dynamic allocation is now enabled in Data Flow. You can define a Data Flow cluster based on a range of executors, instead of just a fixed number of executors. Spark provides a mechanism to dynamically adjust the resources the application occupies based on the workload. The application might relinquish resources if they are no longer used and request them again later when there is demand.

%create_session -l python -c '{\
  "compartmentId":"<compartment_id>",\
  "displayName":"TestDataFlowSession",\
  "sparkVersion":"3.2.1",\
  "driverShape":"VM.Standard.E4.Flex",\
  "executorShape":"VM.Standard.E4.Flex",\
  "numExecutors":1,\
  "driverShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "executorShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "logsBucketUri" : "oci://<bucket_name>@<namespace>/"\
  "configuration":{\
    "spark.dynamicAllocation.enabled":"true",\
      "spark.dynamicAllocation.shuffleTracking.enabled":"true",\
      "spark.dynamicAllocation.minExecutors":"1",\
      "spark.dynamicAllocation.maxExecutors":"4",\
      "spark.dynamicAllocation.executorIdleTimeout":"60",\
      "spark.dynamicAllocation.schedulerBacklogTimeout":"60",\
      "spark.dataflow.dynamicAllocation.quotaPolicy":"min"}}'

Example command with third-party libraries

The Data Flow Sessions support custom dependencies in the form of Python wheels or virtual environments. You might want to make native code or other assets available within your Spark runtime. The dependencies can be attached by using the archiveUri attribute.

%create_session -l python -c '{\
  "compartmentId":"<compartment_id>",\
  "displayName":"TestDataFlowSession",\
  "sparkVersion":"3.2.1",\
  "driverShape":"VM.Standard.E4.Flex",\
  "executorShape":"VM.Standard.E4.Flex",\
  "numExecutors":1,\
  "driverShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "executorShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "archiveUri":"oci://<bucket_name>@<namespace>/<zip_archive>",\
  "logsBucketUri" : "oci://<bucket_name>@<namespace>/"}'

Example command with the Data Catalog Hive Metastore

The Data Catalog Hive Metastore provides schema definitions for objects in structured and unstructured data assets. Use the metastoreId to access the Data Catalog Metastore.

%create_session -l python -c '{\
  "compartmentId":"<compartment_id>",\
  "displayName":"TestDataFlowSession",\
  "sparkVersion":"3.2.1",\
  "driverShape":"VM.Standard.E4.Flex",\
  "executorShape":"VM.Standard.E4.Flex",\
  "numExecutors":1,\
  "driverShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "executorShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "metastoreId": "<ocid1.datacatalogmetastore...>",\
  "logsBucketUri" : "oci://<bucket_name>@<namespace>/"}'

Example command with the published conda environment

You can use a published conda environment as a Data Flow runtime environment.

The path to the published conda environment can be copied from the Environment Explorer.

Example path : oci://<your-bucket>@<your-tenancy-namespace>/conda_environments/cpu/PySpark 3.2 and Data Flow/2.0/pyspark32_p38_cpu_v2#conda

%create_session -l python -c '{\
  "compartmentId":"<compartment_id>",\
  "displayName":"TestDataFlowSession",\
  "sparkVersion":"3.2.1",\
  "driverShape":"VM.Standard.E4.Flex",\
  "executorShape":"VM.Standard.E4.Flex",\
  "numExecutors":1,\
  "driverShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "executorShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "logsBucketUri" : "oci://<bucket_name>@<namespace>/"\
  "configuration":{\
    "spark.archives": "oci://<your-bucket>@<your-tenancy-namespace>/conda_environments/cpu/PySpark 3.2 and Data Flow/2.0/pyspark32_p38_cpu_v2#conda>"}}'

Update Session

You can modify the configuration of your running session using the %update_session command. For example, Data Flow sessions can last up to 7 days or 10080 mins (168 hours) (maxDurationInMinutes) and have default idle timeout value of 480 mins (8 hours)(idleTimeoutInMinutes). Only those two can be updated on a running cluster without re-creating the cluster.

%update_session -i '{"maxDurationInMinutes": 1440, "idleTimeoutInMinutes": 420}'

Configure Session

The existing session can be reconfigured with the %configure_session command. The new configuration will be applied the next time the session is started. Use the force flag -f to immediately drop and recreate the running cluster session.

%configure_session -f -i '{\
  "driverShape":"VM.Standard.E4.Flex",\
  "executorShape":"VM.Standard.E4.Flex",\
  "numExecutors":2,\
  "driverShapeConfig":{"ocpus":1,"memoryInGBs":16},\
  "executorShapeConfig":{"ocpus":1,"memoryInGBs":16}}'

Stop Session

To stop the current session, use the %stop_session magic command. You don’t need to provide any arguments for this command. The current active cluster will be stopped. All data in memory will be lost.

%stop_session

Activate Session

To re-activate the existing session, use the %activate_session magic command. The application_id can be taken from the console UI.

%activate_session -l python -c '{\
  "compartmentId":"<compartment_id>",\
  "displayName":"TestDataFlowSession",\
  "applicationId":"<application_id>"}'

Use Existing Session

To connect to the existing session use the %use_session magic command.

%use_session -s <application_id>

Basic Spark Usage Examples

A SparkContext (sc) and HiveContext (sqlContext) are automatically created in the session cluster. The magic commands include the %%spark command to run Spark commands in the cluster. You can access information about the Spark application, define a dataframe where results are to be stored, modify the configuration, and so on.

The %%spark magic command comes with a number of parameters that allow you to interact with the Data Flow Spark cluster. Any cell content that starts with the %%spark command will be executed in the remote Spark cluster.

Check the Spark context version:

%%spark
print(sc.version)

A toy example of how to use sc in a Data Flow Spark Magic cell:

%%spark
numbers = sc.parallelize([4, 3, 2, 1])
print(f"First element of numbers is {numbers.first()}")
print(f"The RDD, numbers, has the following description\n{numbers.toDebugString()}")

Spark SQL

Using the -c sql option allows you to run Spark SQL commands in a cell. In this section, the NYC Taxi and Limousine Commission (TLC) Data dataset is used. The size of the dataset is around 35GB.

The next cell reads the dataset into a Spark dataframe, and then saves it as a view used to demonstrate Spark SQL.

Use the -c sql option to run Spark SQL commands in a cell.

The next example demonstrates how a dataset can be created on the fly:

%%spark
df_nyc_tlc = spark.read.parquet("oci://hosted-ds-datasets@bigdatadatasciencelarge/nyc_tlc/201[1,2,3,4,5,6,7,8]/**/data.parquet", header=False, inferSchema=True)
df_nyc_tlc.show()
df_nyc_tlc.createOrReplaceTempView("nyc_tlc")

The following cell uses the -c sql option to tell Data Flow Spark Magic that the contents of the cell is SparkSQL. The -o <variable> option takes the results of the Spark SQL operation and stores it in the defined variable. In this case, the df_people will be a Pandas dataframe that is available to be used in the notebook.

%%spark -c sql -o df_nyc_tlc
SELECT vendor_id, passenger_count, trip_distance, payment_type FROM nyc_tlc LIMIT 1000;

Check the result:

print(type(df_nyc_tlc))
df_nyc_tlc.head()

[Legacy]

Deprecated since version v2.6.3: July 2022

ADS can be used to to create and run PySpark applications directly from a notebook session.

Prerequisite

To access , there are a number of steps that are needed to be completed.

DataFlow requires a bucket to store the logs, and a data warehouse bucket. Refer to the Data Flow documentation for setting up storage.
DataFlow requires policies to be set in IAM to access resources to manage and run applications/sessions. Refer to the Data Flow documentation on how to setup policies.
DataFlow natively supports conda packs published to OCI Object Storage. Ensure the Data Flow Resource has read access to the bucket or path of your published conda pack, and that the spark version >= 3 when running your Data Flow Application/Session.

The core-site.xml file needs to be configured.

Create a Instance

First, you create a DataFlow object instance.

By default, all artifacts are stored using the dataflow_base_folder optional argument. By default, all artifacts are stored in /home/datascience/dataflow. The dataflow_base_folder directory contains multiple subdirectories, each one corresponds to a different application. The name of the subdirectory corresponds to the application name that a random string is added as a suffix. In each application directory, artifacts generated by separate runs are stored in different folders. Each folder is identified by the run display name and the run creation time. All the run specific artifacts including the script, the run configuration, and the run logs are saved in the corresponding run folder.

Also, you can choose to use a specific compartment using the optional compartment_id argument when creating the dataflow instance. Otherwise, it uses the same compartment as your notebook session to create the instance.

from ads.dataflow.dataflow import DataFlow
data_flow = DataFlow(
  compartment_id="<compartmentA_OCID>",
  dataflow_base_folder="<my_dataflow_dir>"
)

Generate a Script Using a Template

We provide simple PySpark or sparksql templates for you to get started with . You can use data_flow.template() to generate a pre-written template.

We support these templates:

The standard_pyspark template is used for standard PySpark jobs.

The sparksql template is used for sparksql jobs.

from ads.dataflow.dataflow import DataFlow
data_flow = DataFlow()
data_flow.template(job_type='standard_pyspark')

data_flow.template() returns the local path to the script you have generated.

Create a Application

The application creation process has two stages, preparation and creation.

In the preparation stage, you prepare the configuration object necessary to create a application. You must provide values for these three parameters:

display_name: The name you give your application.
pyspark_file_path: The local path to your PySpark script.
script_bucket: The bucket used to read/write the PySpark script in Object Storage.

ADS checks that the bucket exists, and that you can write to it from your notebook sesssion. Optionally, you can change values for these parameters:

compartment_id: The OCID of the compartment to create a application. If it’s not provided, the same compartment as your dataflow object is used.
driver_shape: The driver shape used to create the application. The default value is "VM.Standard2.4".
executor_shape: The executor shape to create the application. The default value is "VM.Standard2.4".
logs_bucket: The bucket used to store run logs in Object Storage. The default value is "dataflow-logs".
num_executors: The number of executor VMs requested. The default value is 1.

Note

If you want to use a private bucket as the logs_bucket, ensure that you add a corresponding service policy using ` Identity: Policy Set Up <https://docs.cloud.oracle.com/en-us/iaas/data-flow/using/dfs_getting_started.htm#policy_set_up>`_.

Then you can use prepare_app() to create the configuration object necessary to create the application.

from ads.dataflow.dataflow import DataFlow

data_flow = DataFlow()
app_config = data_flow.prepare_app(
  display_name="<app-display-name>",
  script_bucket="<your-script-bucket>" ,
  pyspark_file_path="<your-scirpt-path>"
)

After you have the application configured, you can create a application using create_app:

app = data_flow.create_app(app_config)

Your local script is uploaded to the script bucket in this application creation step. Object Storage supports file versioning that creates an object version when the content changes, or the object is deleted. You can enable Object Versioning in your bucket in the OCI Console to prevent overwriting of existing files in Object Storage.

You can create an application with a script file that exists in Object Storage by setting overwrite_script=True in create_app. Similarly, you can set overwrite_archive=True to create an application with an archive file that exists in Object Storage. By default, the overwrite_script and overwrite_archive options are set to false.

app = data_flow.create_app(app_config, overwrite_script=True, overwrite_archive=True)

You can explore a few attributes of the DataFlowApp object.

First , you can look at the configuration of the application.

app.config

Next, you could get a URL link to the OCI Console Application Details page.

app.oci_link

Load an Existing Application

As an alternative to creating applications in ADS, you can load existing applications created elsewhere. These applications must be Python applications. To load an existing applications, you need the application’s OCID.

existing_app = data_flow.load_app(app_id, target_folder)

You can find the app_id in the the OCI Console or by listing existing applications.

Optionally, you could assign a value to the parameter target_folder. This parameter is the directory you want to store the local artifacts of this application in. If target_folder is not provided, then the local artifacts of this application are stored in the dataflow_base_folder folder defined by the dataflow object instance.

Listing Applications

From ADS you can list applications, that are returned a as a list of dictionaries, with a function to provide the data in a Pandas dataframe. The default sort order is the most recent run first.

For example, to list the most recent five applications use this code:

from ads.dataflow.dataflow import DataFlow
data_flow = DataFlow()
data_flow.list_apps().to_dataframe().head(5)

Create a Run

After an application is created or loaded in your notebook session, the next logical step is to execute a run of that application. The process of running (or creating) a run is similar to creating an application.

First, you configure the run using the prepare_run() method of the DataFlowApp object. You only need to provide a value for the name of your run using run_display_name:

run_config = app.prepare_run(run_display_name="<run-display-name>")

You could use a compartment different from your application to create a run by specifying the compartment_id in prepare_run. By default, it uses the same compartment as your application to create the run.

Optionally, you can specify the logs_bucket to store the logs of your run. By default, the run inherits the logs_bucket from the parent application, but you can overwrite that option.

Every time the application launches a run, a local folder representing this run is created. This folder stores all the information including the script, the run configuration, and any logs that are stored in the logs bucket.

Then, you can create a run using the run_config generated in the preparation stage. During this process, you can monitor the run while the job is running. You can also pull logs into your local directories by setting, save_log_to_local=True.

run = app.run(run_config, save_log_to_local=True)

The DataFlowRun object has some useful attributes similar to the DataFlowApp object.

You can check the status of the run with:

run.status

You can get the configuration file that created this run. The run configuration and the PySpark script used in this run are also saved in the corresponding run directory in your notebook environment.

run.config

You can get the run directory where the artifacts are stored in your notebook environment with:

run.local_dir

Similarly, you can get a clickable link to the OCI Console Run Details page with:

run.oci_link

Fetching Logs

After a run has completed, you can examine the logs using ADS. There are two types of logs, stdout and stderr.

run.log_stdout.head()   # show first rows of stdout
run.log_stdout.tail()   # show last lines of stdout

# where the logs are stored on OCI Storage
run.log_stdout.oci_path

# the path to the saved logs in the notebook environment if ``save_log_to_local`` was ``True`` when you create this run
run.log_stdout.local_path

If save_log_to_local is set to False during app.run(...), you can fetch logs by calling the fetch_log(...).save() method on the DataFlowRun object with the correct logs type.

run.fetch_log("stdout").save()
run.fetch_log("stderr").save()

Note

Due to a limitation of PySpark (specifically Python applications in Spark), both stdout and stderr are merged into the stdout stream.

Edit and Synchronize PySpark Script

The integration with ADS supports the edit-run-edit cycle, so the local PySpark script can be edited, and is automatically synchronized to Object Storage each time the application is run.

obtains the PySpark script from Object Storage

so the local files in the notebook session are not visible to . The app.run(...) method compares the content hash of the local file with the remote copy on Object Storage. If any change is detected, the new local version is copied over to the remote. For the first run the synchronization creates the remote file and generates a fully qualified URL with namespace that’s required for .

Synchronizing is the default setting in app.run(...). If you don’t want the application to sync with the local modified files, you need to include sync=False as an argument parameter in app.run(...).

Arguments and Parameters

Passing arguments to PySpark scripts is done with the arguments value in prepare_app. Additional to the arguments supports, is a parameter dictionary that you can use to interpolate arguments. To just pass arguments, the script_parameter section may be ignored. However, any key-value pair defined in script_parameter can be referenced in arguments using the ${key} syntax, and the value of that key is passed as the argument value.

from ads.dataflow.dataflow import DataFlow

data_flow = DataFlow()
app_config = data_flow.prepare_app(
  display_name,
  script_bucket,
  pyspark_file_path,
  arguments = ['${foo}', 'bar', '-d', '--file', '${filename}'],
  script_parameters={
    'foo': 'val1 val2',
    'filename': 'file1',
  }
)
app = data_flow.create_app(app_config)

run_config = app.prepare_run(run_display_name="test-run")
run = app.run(run_config)

Note

The arguments in the format of ${arg} are replaced by the value provided in script parameters when passed in, while arguments not in this format are passed into the script verbatim.

You can override the values of some or all script parameters in each run by passing different values to prepare_run().

run_config = app.prepare_run(run_display_name="test-run", foo='val3')
run = app.run(run_config)

Add Third-Party Libraries

Your PySpark applications might have custom dependencies in the form of Python wheels or virtual environments, see Adding Third-Party Libraries to Applications.

Pass the archive file to your applications with archive_path and archive_bucket values in prepare_app.

archive_path: The local path to archive file.
archive_bucket: The bucket used to read and write the archive file in Object Storage; if not provided, archive_bucket will use the bucket for PySpark bucket by default.

Use prepare_app() to create the configuration object necessary to create the application.

from ads.dataflow.dataflow import DataFlow

data_flow = DataFlow()
app_config = data_flow.prepare_app(
  display_name="<app-display-name>",
  script_bucket="<your-script-bucket>",
  pyspark_file_path="<your-scirpt-path>",
  archive_path="<your-archive-path>",
  archive_bucket="<your-archive-bucket>"
)

The behavior of the archive file is very similar to the PySpark script when creating:

An application, the local archive file is uploaded to the specified bucket Object Storage.
A run, the latest local archive file is synchronized to the remote file in Object Storage. The sync parameter controls this behavior.
Loading an existing application created with archive_uri, the archive file is obtained from Object Storage, and saved in the local directory.

Fetching PySpark Output

After the application has run and any stdout captured in the log file, the PySpark script likely produces some form of output. Usually a PySpark script batch processes something. For example, sampling data, aggregating data, preprocessing data. You can load the resulting output as an ADSDataset.open() using the ocis:// protocol handler.

The only way to get output from PySpark back into the notebook session is to create files in Object Storage that is read into the notebook, or use the stdout stream.

Following is a simple example of a PySpark script producing output printed in a portable JSON-L format, though CSV works too. This method, while convenient as an example, is not a recommended for large data.

from pyspark.sql import SparkSession

def main():

    # create a spark session
    spark = SparkSession \
        .builder \
        .appName("Python Spark SQL basic example") \
        .getOrCreate()

    # load an example csv file from dataflow public storage into DataFrame
    original_df = spark\
          .read\
          .format("csv")\
          .option("header", "true")\
          .option("multiLine", "true")\
          .load("oci://oow_2019_dataflow_lab@bigdatadatasciencelarge/usercontent/kaggle_berlin_airbnb_listings_summary.csv")

    # the dataframe as a sql view so we can perform SQL on it
    original_df.createOrReplaceTempView("berlin")

    query_result_df = spark.sql("""
                      SELECT
                        city,
                        zipcode,
                        number_of_reviews,
                        CONCAT(latitude, ',', longitude) AS lat_long
                      FROM
                        berlin"""
                    )

    # Convert the filtered Spark DataFrame into JSON format
    # Note: we are writing to the spark stdout log so that we can retrieve the log later at the end of the notebook.

    print('\n'\
            .join(query_result_df\
            .toJSON()\
            .collect()))

if __name__ == '__main__':
    main()

After you run the stdout stream (which contains CSV formatted data), it can be interpreted as a string using Pandas.

import io
import pandas as pd

# the PySpark script wrote to the log as jsonL, and we read the log back as a pandas dataframe
df = pd.read_json((str(run.log_stdout)), lines=True)

df.head()

Example Notebook: Develop Pyspark jobs locally - from local to remote workflows

This notebook provides spark operations for customers by bridging the existing local spark workflows with cloud based capabilities. Data scientists can use their familiar local environments with JupyterLab, and work with remote data and remote clusters simply by selecting a kernel. The operations demonstrated are, how to:

Use the interactive spark environment and produce a spark script,
Prepare and create an application,
Prepare and create a run,
List existing dataflow applications,
Retrieve and display the logs,

The purpose of the dataflow module is to provide an efficient and convenient way for you to launch a Spark application, and run Spark jobs. The interactive Spark kernel provides a simple and efficient way to edit and build your Spark script, and easy access to read from an OCI filesystem.

import io
import matplotlib.pyplot as plt
import os
from os import path
import pandas as pd
import tempfile
import uuid

from ads.dataflow.dataflow import DataFlow

from pyspark.sql import SparkSession

Build your PySPark Script Using an Interactive Spark kernel

Set up spark session in your PySPark conda environment:

# create a spark session
spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.driver.cores", "4") \
    .config("spark.executor.cores", "4") \
    .getOrCreate()

Load the Employee Attrition data file from OCI Object Storage into a Spark DataFrame:

emp_attrition = spark\
      .read\
      .format("csv")\
      .option("header", "true")\
      .option("inferSchema", "true")\
      .option("multiLine", "true")\
      .load("oci://hosted-ds-datasets@bigdatadatasciencelarge/synthetic/orcl_attrition.csv") \
      .cache() # cache the dataset to increase computing speed
emp_attrition.createOrReplaceTempView("emp_attrition")

Next, explore the dataframe:

spark.sql('select * from emp_attrition limit 5').toPandas()

	Age	Attrition	TravelForWork	...	name
0	42	Yes	infrequent	...	Tracy Moore
1	50	No	often	...	Andrew Hoover
2	38	Yes	infrequent	...	Julie Bell
3	34	No	often	...	Thomas Adams
4	28	No	infrequent	...	Johnathan Burnett

5 rows × 36 columns

Visualize how monthly income and age relate to one another in the context of years in industry:

fig, ax = plt.subplots()
plot = spark.sql("""
          SELECT
              Age,
              MonthlyIncome,
              YearsInIndustry
          FROM
            emp_attrition
          """).toPandas().plot.scatter(x="Age", y="MonthlyIncome", title='Age vs Monthly Income',
                                       c="YearsInIndustry", cmap="viridis", figsize=(12,12), ax=ax)
plot.set_xlabel("Age")
plot.set_ylabel("Monthly Income")
plot

<AxesSubplot:title={'center':'Age vs Monthly Income'}, xlabel='Age', ylabel='Monthly Income'>

View all of the columns in the table:

spark.sql("show columns from emp_attrition").show()

+--------------------+
|            col_name|
+--------------------+
|                 Age|
|           Attrition|
|       TravelForWork|
|         SalaryLevel|
|         JobFunction|
|       CommuteLength|
|    EducationalLevel|
|      EducationField|
|             Directs|
|      EmployeeNumber|
| EnvironmentSatisf..|
|              Gender|
|          HourlyRate|
|      JobInvolvement|
|            JobLevel|
|             JobRole|
|     JobSatisfaction|
|       MaritalStatus|
|       MonthlyIncome|
|         MonthlyRate|
+--------------------+
only showing top 20 rows

Select a few columns using Spark, and convert it into a Pandas dataframe:

df = spark.sql("""
         SELECT
            Age,
            MonthlyIncome,
            YearsInIndustry
          FROM
            emp_attrition """).limit(10).toPandas()
df

	Age	MonthlyIncome	YearsInIndustry
0	42	5993	8
1	50	5130	10
2	38	2090	7
3	34	2909	8
4	28	3468	6
5	33	3068	8
6	60	2670	12
7	31	2693	1
8	39	9526	10
9	37	5237	17

You can work with different compression formats within . For example, snappy Parquet:

# Writing to a snappy parquet file
df.to_parquet('emp_attrition.parquet.snappy', compression='snappy')
pd.read_parquet('emp_attrition.parquet.snappy')

	Age	MonthlyIncome	YearsInIndustry
0	42	5993	8
1	50	5130	10
2	38	2090	7
3	34	2909	8
4	28	3468	6
5	33	3068	8
6	60	2670	12
7	31	2693	1
8	39	9526	10
9	37	5237	17

# We are able to read in this snappy parquet file to a spark dataframe
read_snappy_df = SparkSession \
    .builder \
    .appName("Snappy Compression Loading Example") \
    .config("spark.io.compression.codec", "org.apache.spark.io.SnappyCompressionCodec") \
    .getOrCreate() \
    .read \
    .format("parquet") \
    .load(f"{os.getcwd()}/emp_attrition.parquet.snappy")

read_snappy_df.first()

Row(Age=42, MonthlyIncome=5993, YearsInIndustry=8)

Other compression formats that supports include snappy Parquet, and Gzip on both CSV and Parquet.

You might have query that you want to run in from previous explorations, review the dataflow.ipynb notebook example that shows you how to submit a job to .

dataflow_base_folder = tempfile.mkdtemp()
data_flow = DataFlow(dataflow_base_folder=dataflow_base_folder)
print("Data flow directory: {}".format(dataflow_base_folder))

Data flow directory: /tmp/tmpe18x_qbr

pyspark_file_path = path.join(dataflow_base_folder, "example-{}.py".format(str(uuid.uuid4())[-6:]))
script = '''
from pyspark.sql import SparkSession

def main():

    # Create a Spark session
    spark = SparkSession \\
        .builder \\
        .appName("Python Spark SQL basic example") \\
        .getOrCreate()

    # Load a csv file from dataflow public storage
    df = spark \\
        .read \\
        .format("csv") \\
        .option("header", "true") \\
        .option("multiLine", "true") \\
        .load("oci://hosted-ds-datasets@bigdatadatasciencelarge/synthetic/orcl_attrition.csv")

    # Create a temp view and do some SQL operations
    df.createOrReplaceTempView("emp_attrition")
    query_result_df = spark.sql("""
        SELECT
            Age,
            MonthlyIncome,
            YearsInIndustry
        FROM emp_attrition
    """)

    # Convert the filtered Spark DataFrame into JSON format
    # Note: we are writing to the spark stdout log so that we can retrieve the log later at the end of the notebook.
    print('\\n'.join(query_result_df.toJSON().collect()))

if __name__ == '__main__':
    main()
'''

with open(pyspark_file_path, 'w') as f:
    print(script.strip(), file=f)

print("Script path: {}".format(pyspark_file_path))

Script path: /tmp/example.py

script_bucket = "test"                     # Update the value
logs_bucket = "dataflow-log"               # Update the value
display_name = "sample_Data_Flow_app"

app_config = data_flow.prepare_app(display_name=display_name,
                                   script_bucket=script_bucket,
                                   pyspark_file_path=pyspark_file_path,
                                   logs_bucket=logs_bucket)

app = data_flow.create_app(app_config)

run_display_name = "sample_Data_Flow_run"
run_config = app.prepare_run(run_display_name=run_display_name)

run = app.run(run_config, save_log_to_local=True)

run.status

'SUCCEEDED'

run.config

{'compartment_id': 'ocid1.compartment..<unique_ID>',
 'script_bucket': 'test',
 'pyspark_file_path': '/tmp/tmpe18x_qbr/example-0054ed.py',
 'archive_path': None,
 'archive_bucket': None,
 'run_display_name': 'sample_Data_Flow_run',
 'logs_bucket': 'dataflow-log',
 'logs_bucket_uri': 'oci://dataflow-log@ociodscdev',
 'driver_shape': 'VM.Standard2.4',
 'executor_shape': 'VM.Standard2.4',
 'num_executors': 1}

run.oci_link

Saving processed data to jdbc:oracle:thin:@database_high?TNS_ADMIN=/tmp/

Read from the Database Using PySpark

PySpark can be used to load data from an Oracle Autonomous Database (ADB) into a Spark application. The next cell makes a JDBC connection to the database defined using the adb_url variable, and accesses the table defined with table_name. The credentials stored in the vault and previously read into memory are used. After this command is run, you can perform Spark operations on it.

The table is relatively small so the notebook uses PySpark in the notebook session. However, for larger jobs, we recommended that you use the Oracle service.

if "adb_url" in globals():
    output_dataframe = sc.read \
        .format("jdbc") \
        .option("url", adb_url) \
        .option("dbtable", table_name) \
        .option("user", user) \
        .option("password", password) \
        .load()
else:
    print("Skipping as it appears that you do not have adb_url configured.")

The database table is loaded into Spark so that you can perform operations to transform, model, and more. In the next cell, the notebook prints the table demonstrating that it was successfully loaded into Spark from the ADB.

if "adb_url" in globals():
    output_dataframe.show()
else:
    print("Skipping as it appears that you do not have output_dataframe configured.")

+----+----------+--------------+------------+-------------------+-------------+-----------------+---------------+--------+---------------+------------------------+-------+-----------+---------------+---------+---------------------+---------------+--------------+--------------+------------+-------------------+-------+---------+------------------+------------------+-------------------------+------------------+-----------------+----------------+----------------------+----------------+-----------+--------------------+------------------------+---------------------+------------------+
| Age| Attrition| TravelForWork| SalaryLevel|       JobFunction| CommuteLength| EducationalLevel| EducationField| Directs| EmployeeNumber| EnvironmentSatisfaction| Gender| HourlyRate| JobInvolvement| JobLevel|             JobRole| JobSatisfaction| MaritalStatus| MonthlyIncome| MonthlyRate| NumCompaniesWorked| Over18| OverTime| PercentSalaryHike| PerformanceRating| RelationshipSatisfaction| WeeklyWorkedHours| StockOptionLevel| YearsinIndustry| TrainingTimesLastYear| WorkLifeBalance| YearsOnJob| YearsAtCurrentLevel| YearsSinceLastPromotion| YearsWithCurrManager|              name|
+----+----------+--------------+------------+-------------------+-------------+-----------------+---------------+--------+---------------+------------------------+-------+-----------+---------------+---------+---------------------+---------------+--------------+--------------+------------+-------------------+-------+---------+------------------+------------------+-------------------------+------------------+-----------------+----------------+----------------------+----------------+-----------+--------------------+------------------------+---------------------+------------------+
|  42|       Yes|    infrequent|        5054| Product Management|            2|               L2|  Life Sciences|       1|              1|                       2| Female|         94|              3|        2|      Sales Executive|              4|        Single|          5993|       19479|                  8|      Y|      Yes|                11|                 3|                        1|                80|                0|               8|                     0|               1|          6|                   4|                       0|                    5|       Tracy Moore|
|  50|        No|         often|        1278| Software Developer|            9|               L1|  Life Sciences|       1|              2|                       3|   Male|         61|              2|        2|   Research Scientist|              2|       Married|          5130|       24907|                  1|      Y|       No|                23|                 4|                        4|                80|                1|              10|                     3|               3|         10|                   7|                       1|                    7|     Andrew Hoover|
|  38|       Yes|    infrequent|        6296| Software Developer|            3|               L2|          Other|       1|              4|                       4|   Male|         92|              2|        1| Laboratory Techni...|              3|        Single|          2090|        2396|                  6|      Y|      Yes|                15|                 3|                        2|                80|                0|               7|                     3|               3|          0|                   0|                       0|                    0|        Julie Bell|
|  34|        No|         often|        6384| Software Developer|            4|               L4|  Life Sciences|       1|              5|                       4| Female|         56|              3|        1|   Research Scientist|              3|       Married|          2909|       23159|                  1|      Y|      Yes|                11|                 3|                        3|                80|                0|               8|                     3|               3|          8|                   7|                       3|                    0|      Thomas Adams|
|  28|        No|    infrequent|        2710| Software Developer|            3|               L1|        Medical|       1|              7|                       1|   Male|         40|              3|        1| Laboratory Techni...|              2|       Married|          3468|       16632|                  9|      Y|       No|                12|                 3|                        4|                80|                1|               6|                     3|               3|          2|                   2|                       2|                    2| Johnathan Burnett|
|  33|        No|         often|        4608| Software Developer|            3|               L2|  Life Sciences|       1|              8|                       4|   Male|         79|              3|        1| Laboratory Techni...|              4|        Single|          3068|       11864|                  0|      Y|       No|                13|                 3|                        3|                80|                0|               8|                     2|               2|          7|                   7|                       3|                    6|      Rhonda Grant|
|  60|        No|    infrequent|        6072| Software Developer|            4|               L3|        Medical|       1|             10|                       3| Female|         81|              4|        1| Laboratory Techni...|              1|       Married|          2670|        9964|                  4|      Y|      Yes|                20|                 4|                        1|                80|                3|              12|                     3|               2|          1|                   0|                       0|                    0|      Brandon Gill|
|  31|        No|    infrequent|        6228| Software Developer|           25|               L1|  Life Sciences|       1|             11|                       4|   Male|         67|              3|        1| Laboratory Techni...|              3|      Divorced|          2693|       13335|                  1|      Y|       No|                22|                 4|                        2|                80|                1|               1|                     2|               3|          1|                   0|                       0|                    0|       Debbie Chan|
|  39|        No|         often|         990| Software Developer|           24|               L3|  Life Sciences|       1|             12|                       4|   Male|         44|              2|        3| Manufacturing Dir...|              3|        Single|          9526|        8787|                  0|      Y|       No|                21|                 4|                        2|                80|                0|              10|                     2|               3|          9|                   7|                       1|                    8|        Kayla Ward|
|  37|        No|    infrequent|        5958| Software Developer|           28|               L3|        Medical|       1|             13|                       3|   Male|         94|              3|        2| Healthcare Repres...|              3|       Married|          5237|       16577|                  6|      Y|       No|                13|                 3|                        2|                80|                2|              17|                     3|               2|          7|                   7|                       7|                    7|      Angel Vaughn|
|  36|        No|    infrequent|        3710| Software Developer|           17|               L3|        Medical|       1|             14|                       1|   Male|         84|              4|        1| Laboratory Techni...|              2|       Married|          2426|       16479|                  0|      Y|       No|                13|                 3|                        3|                80|                1|               6|                     5|               3|          5|                   4|                       0|                    3|   Samantha Parker|
|  30|        No|    infrequent|         700| Software Developer|           16|               L2|  Life Sciences|       1|             15|                       4| Female|         49|              2|        2| Laboratory Techni...|              3|        Single|          4193|       12682|                  0|      Y|      Yes|                12|                 3|                        4|                80|                0|              10|                     3|               3|          9|                   5|                       0|                    8|   Melanie Mcbride|
|  32|        No|    infrequent|        3072| Software Developer|           27|               L1|  Life Sciences|       1|             16|                       1|   Male|         31|              3|        1|   Research Scientist|              3|      Divorced|          2911|       15170|                  1|      Y|       No|                17|                 3|                        4|                80|                1|               5|                     1|               2|          5|                   2|                       4|                    3|      Bradley Hall|
|  35|        No|    infrequent|        6172| Software Developer|           20|               L2|        Medical|       1|             18|                       2|   Male|         93|              3|        1| Laboratory Techni...|              4|      Divorced|          2661|        8758|                  0|      Y|       No|                11|                 3|                        3|                80|                1|               3|                     2|               3|          2|                   2|                       1|                    2|       Patrick Lee|
|  29|       Yes|    infrequent|         472| Software Developer|           25|               L3|  Life Sciences|       1|             19|                       3|   Male|         50|              2|        1| Laboratory Techni...|              3|        Single|          2028|       12947|                  5|      Y|      Yes|                14|                 3|                        2|                80|                0|               6|                     4|               3|          4|                   2|                       0|                    3|    Jessica Willis|
|  30|        No|    infrequent|        6370| Software Developer|           22|               L4|  Life Sciences|       1|             20|                       2| Female|         51|              4|        3| Manufacturing Dir...|              1|      Divorced|          9980|       10195|                  1|      Y|       No|                11|                 3|                        3|                80|                1|              10|                     1|               3|         10|                   9|                       8|                    8|        Chad Scott|
|  33|        No|    infrequent|        1530| Software Developer|            6|               L2|  Life Sciences|       1|             21|                       1|   Male|         80|              4|        1|   Research Scientist|              2|      Divorced|          3298|       15053|                  0|      Y|      Yes|                12|                 3|                        4|                80|                2|               7|                     5|               2|          6|                   2|                       0|                    5|   Gregory Bennett|
|  23|        No|          none|        5150| Software Developer|           17|               L2|        Medical|       1|             22|                       4|   Male|         96|              4|        1| Laboratory Techni...|              4|      Divorced|          2935|        7324|                  1|      Y|      Yes|                13|                 3|                        2|                80|                2|               1|                     2|               2|          1|                   0|                       0|                    0|      Jesse Palmer|
|  54|        No|    infrequent|        5590| Product Management|            3|               L4|  Life Sciences|       1|             23|                       1| Female|         78|              2|        4|              Manager|              4|       Married|         15427|       22021|                  2|      Y|       No|                16|                 3|                        3|                80|                0|              31|                     3|               3|         25|                   8|                       3|                    7| Dr. Erin Good DDS|
|  39|        No|    infrequent|        1700| Software Developer|            3|               L3|  Life Sciences|       1|             24|                       4|   Male|         45|              3|        1|   Research Scientist|              4|        Single|          3944|        4306|                  5|      Y|      Yes|                11|                 3|                        3|                80|                0|               6|                     3|               3|          3|                   2|                       1|                    2|     Kathy Patrick|
+----+----------+--------------+------------+-------------------+-------------+-----------------+---------------+--------+---------------+------------------------+-------+-----------+---------------+---------+---------------------+---------------+--------------+--------------+------------+-------------------+-------+---------+------------------+------------------+-------------------------+------------------+-----------------+----------------+----------------------+----------------+-----------+--------------------+------------------------+---------------------+------------------+
only showing top 20 rows

Cleaning Up Artifacts

This example created a number of artifacts, such as unzipping the wallet file, creating a database table, and starting a Spark cluster. Next, you remove these resources.

if wallet_path != "<wallet_path>":
    connection.update_repository(key="pyspark_adb", value=adb_creds)
    connection.import_wallet(wallet_path=wallet_path, key="pyspark_adb")
    conn = cx_Oracle.connect(user, password, tnsname)
    cursor = conn.cursor()
    cursor.execute(f"DROP TABLE {table_name}")
    cursor.close()
    conn.close()
else:
    print("Skipping as it appears that you do not have wallet_path specified.")

if "tns_path" in globals():
    shutil.rmtree(tns_path)

sc.stop()

Example Notebook: Using the ADB with PySpark

This notebook demonstrates how to use PySpark to process data in Object Storage, and save the results to an ADB. It also demonstrates how to query data from an ADB using a local PySpark session.

import base64
import cx_Oracle
import oci
import os
import shutil
import tempfile
import zipfile

from ads.database import connection
from ads.vault.vault import Vault
from pyspark import SparkConf
from pyspark.sql import SparkSession
from urllib.parse import urlparse

Introduction

It has become a common practice to store structured and semi-structured data using services such as Object Storage. This provides a scalable solution to store vast quantities of data that can be post-processed. However, using a relational database management system (RDMS) such as the Oracle ADB provides advantages like ACID compliance, rapid relational joins, support for complex business logic, and more. It is important to be able to access information stored in Object Storage, process that information, and load it into an RBMS. This notebook demonstrates how to use PySpark, a Python interface to Apache Spark, to perform these operations.

This notebook uses a publicly accessible Object Storage location to read from. However, an ADB needs to be configured with permissions to create a table, write to that table, and read from it. It also assumes that the credentials to access the database are stored in the Vault. This is the best practice as it prevents the credentials from being stored locally or in the notebook where they may be accessible to others. If you do not have credentials stored in the Vault. Once credentials to the database, are stored in the Vault, you need the OCIDs for the Vault, encryption key, and the secret.

ADBs have an additional level of security that is needed to access them and are wallet file. You can obtain the wallet file from your account administrator or download it using the steps that are outlined in the [downloading a wallet(https://docs.oracle.com/en-us/iaas/Content/Database/Tasks/adbconnecting.htm#access). The wallet file is a ZIP file. This notebook unzips the wallet and updates the configuration settings so you don’t have to.

The database connection also needs the TNS name of the database. Your database administrator can give you the TNS name of the database that you have access to.

Setup the Required Variables

The required variables to set up are:

vault_id, key_id, secret_ocid: The OCID of the secret by storing the username and password required to connect to your ADB in a secret within the OCI Vault service. Note that the secret is the credential needed to access a database. This notebook is designed so that any secret can be stored as long as it is in the form of a dictionary. To store your secret, just modify the dictionary, see the vault.ipynb example notebook for detailed steps to generate this OCID.
tnsname: A TNS name valid for the database.
wallet_path: The local path to your wallet ZIP file, see the autonomous_database.ipynb example notebook for instructions on accessing the wallet file.

secret_ocid = "secret_ocid"
tnsname = "tnsname"
wallet_path = "wallet_path"
vault_id = "vault_id"
key_id = "key_id"

Obtain Credentials from the Vault

If the vault_id, key_id, and secret_id have been updated, then the notebook obtains a handle to the vault with a variable called vault. This uses the get_secret() method to return a dictionary with the user credentials. The approach assumes that the Accelerated Data Science (ADS) library was used to store the secret.

if vault_id != "<vault_id>" and key_id != "<key_id>" and secret_ocid != "<secret_ocid>":
    print("Getting wallet username and password")
    vault = Vault(vault_id=vault_id, key_id=key_id)
    adb_creds = vault.get_secret(secret_ocid)
    user = adb_creds["username"]
    password = adb_creds["password"]
else:
    print("Skipping as it appears that you do not have vault, key, and secret ocid specified.")

Getting wallet username and password

Setup the Wallet

An ADB requires a wallet file to access the database. The wallet_path variable defines the location of this file. The next cell prepares the wallet file to make a connection to the database. It also creates the ADB connection string, adb_url.

def setup_wallet(wallet_path):
    """
    Prepare ADB wallet file for use in PySpark.
    """

    temporary_directory = tempfile.mkdtemp()
    zip_file_path = os.path.join(temporary_directory, "wallet.zip")

    # Extract everything locally.
    with zipfile.ZipFile(wallet_path, "r") as zip_ref:
        zip_ref.extractall(temporary_directory)

    return temporary_directory

if wallet_path != "<wallet_path>":
    print("Setting up wallet")
    tns_path = setup_wallet(wallet_path)
else:
    print("Skipping as it appears that you do not have wallet_path specified.")

Setting up wallet

if "tns_path" in globals() and tnsname != "<tnsname>":
    adb_url = f"jdbc:oracle:thin:@{tnsname}?TNS_ADMIN={tns_path}"
else:
    print("Skipping, as the tns_path or tnsname are not defined.")

Reading Data from Object Storage

This notebook uses PySpark to access the Object Storage file. The next cell creates a Spark application called “Python Spark SQL Example” and returns a SparkContext. The SparkContext, normally called sc, is a handle to the Spark application.

The data file that is used is relatively small so the notebook uses PySpark by running a version of Spark in local mode. That means, it is running in the notebook session. For larger jobs, we recommended that you use the Oracle service, which is an Oracle managed Spark service.

# create a spark session
sc = SparkSession \
    .builder \
    .appName("Python Spark SQL Example") \
    .getOrCreate()

This notebook reads in a data file that is stored in an Oracle Object Storage file. This is defined with the file_path variable. The SparkContext with the read.option().csv() methods is used to read in the CSV file from Object Storage into a data frame.

file_path = "oci://hosted-ds-datasets@bigdatadatasciencelarge/synthetic/orcl_attrition.csv"
input_dataframe = sc.read.option("header", "true").csv(file_path)

Save the Data to the Database

This notebook creates a table in your database with the name specified with table_name. The name that is defined should be unique so that it does not interfere with any existing table in your database. If it does, change the value to something that is unique.

table_name = "ODSC_PYSPARK_ADB_DEMO"

if tnsname != "<tnsname>" and "adb_url" in globals():
    print("Saving processed data to " + adb_url)
    properties = {
        "oracle.net.tns_admin": tnsname,
        "password": password,
        "user": user,
    }
    input_dataframe.write.jdbc(
        url=adb_url, table=table_name, properties=properties
    )
else:
    print("Skipping as it appears that you do not have tnsname specified.")

Big Data Service

New in version 2.5.10.

The Oracle Big Data Service (BDS) is an Oracle Cloud Infrastructure (OCI) service designed for a diverse set of big data use cases and workloads. From short-lived clusters used to tackle specific tasks to long-lived clusters that manage data lakes. BDS scales to meet an organization’s requirements at a low cost and with the highest levels of security. To be able to connect to the BDS from the notebook session, the cluster created must have Kerberos enabled.

Quick Start

Set Up A Conda Environment

The following are the recommended steps to create a conda environment to connect to BDS:

Open a terminal window then run the following commands:
odsc conda install -s pyspark30_p37_cpu_v5: Install the PySpark conda environment.

Connect from a Notebook

Using the Vault

import ads
import os

from ads.bds.auth import krbcontext
from ads.secrets.big_data_service import BDSSecretKeeper
from pyhive import hive

ads.set_auth('resource_principal')
with BDSSecretKeeper.load_secret("<secret_id>") as cred:
    with krbcontext(principal=cred["principal"], keytab_path=cred['keytab_path']):
        cursor = hive.connect(host=cred["hive_host"],
                              port=cred["hive_port"],
                              auth='KERBEROS',
                              kerberos_service_name="hive").cursor()

Without Using the Vault

import ads
import fsspec
import os

from ads.bds.auth import refresh_ticket

ads.set_auth('resource_principal')
refresh_ticket(principal="<your_principal>", keytab_path="<your_local_keytab_file_path>",
               kerb5_path="<your_local_kerb5_config_file_path>")
cursor = hive.connect(host="<hive_host>", port="<hive_port>",
                      auth='KERBEROS', kerberos_service_name="hive").cursor()

Conda Environment

To work with BDS in a notebook session or job, you must have a conda environment that supports the BDS module in ADS along with support for PySpark. This section demonstrates how to modify a PySpark Data Science conda environment to work with BDS. It also demonstrates how to publish this conda environment so that you can be share it with team members and use it in jobs.

Create

The following are the recommended steps to create a conda environment to connect to BDS:

Open a terminal window then run the following commands:
odsc conda install -s pyspark30_p37_cpu_v5: Install the PySpark conda environment.

Publish

Create an Object Storage bucket to store published conda environments.
Open a terminal window then run the following commands and actions:
odsc conda init -b <bucket_name> -b <namespace> -a <resource_principal or api_key>: Initialize the environment so that you can work with Published Conda Environments.
odsc conda publish -s pyspark30_p37_cpu_v3: Publish the conda environment.
In the OCI Console, open Data Science.
Select a project.
Select a click the notebook session’s name, or the Actions menu, and click Open to open the notebook session’s JupyterLab interface in another tab..
Click Published Conda Environments in the Environment Explorer tab to list all the published conda environments that are available in your designated Object Storage bucket.
Select the Environment Version that you specified.
Click the copy button adjacent to the Source conda environment to copy the file source path to use when installing the conda environment in other notebook sessions or to use with jobs.

Connect

Notebook Session

Notebook sessions require a conda environment that has the BDS module of ADS installed.

Using the Vault

The preferred method to connect to a BDS cluster is to use the BDSSecretKeeper class. This allows you to store the BDS credentials in the vault and not the notebook. It also provides a greater level of access control to the secrets and allows for credential rotation without breaking connections from various sources.

import ads
import os

from ads.bds.auth import krbcontext
from ads.secrets.big_data_service import BDSSecretKeeper
from pyhive import hive

ads.set_auth('resource_principal')
with BDSSecretKeeper.load_secret("<secret_id>") as cred:
    with krbcontext(principal=cred["principal"], keytab_path=cred['keytab_path']):
        cursor = hive.connect(host=cred["hive_host"],
                              port=cred["hive_port"],
                              auth='KERBEROS',
                              kerberos_service_name="hive").cursor()

Without Using the Vault

BDS requires a Kerberos ticket to authenticate to the service. The preferred method is to use the vault and BDSSecretKeeper because it is more secure, and prevents private information from being stored in a notebook. However, if this is not possible, you can use the refresh_ticket() method to manually create the Kerberos ticket. This method requires the following parameters:

kerb5_path: The path to the krb5.conf file. You can copy this file from the master node of the BDS cluster located in /etc/krb5.conf.
keytab_path: The path to the principal’s keytab file. You can download this file from the master node on the BDS cluster.
principal: The unique identity to that Kerberos can assign tickets to.

import ads
import fsspec
import os

from ads.bds.auth import refresh_ticket

ads.set_auth('resource_principal')
refresh_ticket(principal="<your_principal>", keytab_path="<your_local_keytab_file_path>",
               kerb5_path="<your_local_kerb5_config_file_path>")
cursor = hive.connect(host="<hive_host>", port="<hive_port>",
                      auth='KERBEROS', kerberos_service_name="hive").cursor()

Jobs

A job requires a conda environment that has the BDS module of ADS installed. It also requires secrets and configuration information that can be used to obtain a Kerberos ticket for authentication. You must copy the keytab and krb5.conf files to the jobs instance and can be copied as part of the job. We recommend that you save them into the vault then use BDSSecretKeeper to access them. This is secure because the vault provides access control and allows for key rotation without breaking exiting jobs. You can use the notebook to load configuration parameters like hdfs_host, hdfs_port, hive_host, hive_port, and so on. The keytab and krb5.conf files are securely loaded from the vault then saved in the jobs instance. The krbcontext() method is then used to create the Kerberos ticket. Once the ticket is created, you can query BDS.

File Management

This section demonstrates various methods to work with files on BDS’ HDFS, see the individual framework’s documentation for details.

A Kerberos ticket is needed to connect to the BDS cluster. This authentication ticket can be obtained with the refresh_ticket() method or with the use of the Vault and a BDSSercretKeeper object. This section will demonstrate the use of the BDSSecretKeeper object as this is more secure and is the preferred method.

FSSpec

The fsspec or Filesystem Spec is an interface that allows access to local, remote, and embedded file systems. You use it to access data stored in the BDS’ HDFS. This connection is made with the WebHDFS protocol.

The fsspec library must be able to access BDS so a Kerberos ticket must be generated. The secure and recommended method to do this is to use BDSSecretKeeper that stores the BDS credentials in the vault not the notebook session.

This section outlines some common file operations, see the fsspec API Reference for complete details on the features that are demonstrated and additional functionality.

Pandas and PyArrow can also use fsspec to perform file operations.

Connect

Credentials and configuration information is stored in the vault. This information is used to obtain a Kerberos ticket and define the hdfs_config dictionary. This configuration dictionary is passed to the fsspec.filesystem() method to make a connection to the BDS’ underlying HDFS storage.

import ads
import fsspec

from ads.secrets.big_data_service import BDSSecretKeeper
from ads.bds.auth import has_kerberos_ticket, krbcontext

ads.set_auth("resource_principal")
with BDSSecretKeeper.load_secret("<secret_id>") as cred:
    with krbcontext(principal = cred["principal"], keytab_path = cred['keytab_path']):
        hdfs_config = {
            "protocol": "webhdfs",
            "host": cred["hdfs_host"],
            "port": cred["hdfs_port"],
            "kerberos": "True"
        }

fs = fsspec.filesystem(**hdfs_config)

Delete

Delete files from HDFS using the .rm() method. It accepts a path of the files to delete.

fs.rm("/data/biketrips/2020??-tripdata.csv", recursive=True)

Download

Download files from HDFS to a local storage device using the .get() method. It takes the HDFS path of the files to download, and the local path to store the files.

fs.get("/data/biketrips/20190[123456]-tripdata.csv", local_path="./first_half/", overwrite=True)

List

The .ls() method lists files. It returns the matching file names as a list.

fs.ls("/data/biketrips/2019??-tripdata.csv")

['201901-tripdata.csv',
 '201902-tripdata.csv',
 '201903-tripdata.csv',
 '201904-tripdata.csv',
 '201905-tripdata.csv',
 '201906-tripdata.csv',
 '201907-tripdata.csv',
 '201908-tripdata.csv',
 '201909-tripdata.csv',
 '201910-tripdata.csv',
 '201911-tripdata.csv',
 '201912-tripdata.csv']

Upload

The .put() method is used to upload files from local storage to HDFS. The first parameter is the local path of the files to upload. The second parameter is the HDFS path where the files are to be stored. .upload() is an alias of .put(). .. code-block:: python3

fs.put(
lpath=”./first_half/20200[456]-tripdata.csv”, rpath=”/data/biketrips/second_quarter/”

)

Ibis

Ibis is an open-source library by Cloudera that provides a Python framework to access data and perform analytical computations from different sources. Ibis allows access to the data ising HDFS. You use the ibis.impala.hdfs_connect() method to make a connection to HDFS, and it returns a handler. This handler has methods such as .ls() to list, .get() to download, .put() to upload, and .rm() to delete files. These operations support globbing. Ibis’ HDFS connector supports a variety of additional operations.

Connect

After obtaining a Kerberos ticket, the hdfs_connect() method allows access to the HDFS. It is a thin wrapper around a fsspec file system. Depending on your system configuration, you may need to define the ibis.options.impala.temp_db and ibis.options.impala.temp_hdfs_path options.

import ibis

with BDSSecretKeeper.load_secret("<secret_id>") as cred:
    with krbcontext(principal=cred["principal"], keytab_path=cred['keytab_path']):
        hdfs = ibis.impala.hdfs_connect(host=cred['hdfs_host'], port=cred['hdfs_port'],
                                             use_https=False, verify=False,
                                             auth_mechanism='GSSAPI', protocol='webhdfs')

Delete

Delete files from HDFS using the .rm() method. It accepts a path of the files to delete.

hdfs.rm("/data/biketrips/2020??-tripdata.csv", recursive=True)

Download

Download files from HDFS to a local storage device using the .get() method. It takes the HDFS path of the files to download, and the local path to store the files.

hdfs.get("/data/biketrips/20190[123456]-tripdata.csv", local_path="./first_half/", overwrite=True)

List

The .ls() method lists files. It returns the matching file names as a list.

hdfs.ls("/data/biketrips/2019??-tripdata.csv")

['201901-tripdata.csv',
 '201902-tripdata.csv',
 '201903-tripdata.csv',
 '201904-tripdata.csv',
 '201905-tripdata.csv',
 '201906-tripdata.csv',
 '201907-tripdata.csv',
 '201908-tripdata.csv',
 '201909-tripdata.csv',
 '201910-tripdata.csv',
 '201911-tripdata.csv',
 '201912-tripdata.csv']

Upload

Use the .put() method to upload files from local storage to HDFS. The first parameter is the HDFS path where the files are to be stored. The second parameter is the local path of the files to upload.

hdfs.put(rpath="/data/biketrips/second_quarter/",
         lpath="./first_half/20200[456]-tripdata.csv",
         overwrite=True, recursive=True)

Pandas

Pandas allows access to BDS’ HDFS system through :ref: FSSpec. This section demonstrates some common operations.

Connect

import ads
import fsspec

from ads.secrets.big_data_service import BDSSecretKeeper
from ads.bds.auth import has_kerberos_ticket, krbcontext

ads.set_auth("resource_principal")
with BDSSecretKeeper.load_secret("<secret_id>") as cred:
    with krbcontext(principal = cred["principal"], keytab_path = cred['keytab_path']):
        hdfs_config = {
            "protocol": "webhdfs",
            "host": cred["hdfs_host"],
            "port": cred["hdfs_port"],
            "kerberos": "True"
        }

fs = fsspec.filesystem(**hdfs_config)

File Handle

You can use the fsspec .open() method to open a data file. It returns a file handle. That file handle, f, can be passed to any Pandas’ methods that support file handles. In this example, a file on a BDS’ HDFS cluster is read into a Pandas dataframe.

with fs.open("/data/biketrips/201901-tripdata.csv", "r") as f:
    df = pd.read_csv(f)

URL

Pandas supports fsspec so you can preform file operations by specifying a protocol string. The WebHDFS protocol is used to access files on BDS’ HDFS system. The protocol string has this format:

webhdfs://host:port/path/to/data

The host and port parameters can be passed in the protocol string as follows:

df = pd.read_csv(f"webhdfs://{hdfs_config['host']}:{hdfs_config['port']}/data/biketrips/201901-tripdata.csv",
                 storage_options={'kerberos': 'True'})

You can also pass the host and port parameters in the dictionary used by the storage_options parameter. The sample code for hdfs_config defines the host and port with the keyes host and port respectively.

hdfs_config = {
    "protocol": "webhdfs",
    "host": cred["hdfs_host"],
    "port": cred["hdfs_port"],
    "kerberos": "True"
}

In this case, Pandas uses the following syntax to read a file on BDS’ HDFS cluster:

df = pd.read_csv(f"webhdfs:///data/biketrips/201901-tripdata.csv",
                 storage_options=hdfs_config)

PyArrow

PyArrow is a Python interface to Apache Arrow. Apache Arrow is an in-memory columnar analytical tool that is designed to process data at scale. PyArrow supports the fspec.filesystem() through the use of the filesystem parameter in many of its data operation methods.

Connect

Make a connection to BDS’ HDFS using fsspec:

import ads
import fsspec

from ads.secrets.big_data_service import BDSSecretKeeper
from ads.bds.auth import has_kerberos_ticket, krbcontext

ads.set_auth("resource_principal")
with BDSSecretKeeper.load_secret("<secret_id>") as cred:
    with krbcontext(principal = cred["principal"], keytab_path = cred['keytab_path']):
        hdfs_config = {
            "protocol": "webhdfs",
            "host": cred["hdfs_host"],
            "port": cred["hdfs_port"],
            "kerberos": "True"
        }

fs = fsspec.filesystem(**hdfs_config)

Filesystem

The following sample code shows several different PyArrow methods for working with BDS’ HDFS using the filesystem parameter:

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

ds = ds.dataset("/path/on/BDS/HDFS/data.csv", format="csv", filesystem=fs)
pq.write_table(ds.to_table(), '/path/on/BDS/HDFS/data.parquet', filesystem=fs)

import pandas as pd
import numpy as np

idx = pd.date_range('2022-01-01 12:00:00.000', '2022-03-01 12:00:00.000', freq='T')

df = pd.DataFrame({
        'numeric_col': np.random.rand(len(idx)),
        'string_col': pd._testing.rands_array(8,len(idx))},
        index = idx
    )
df["dt"] = df.index
df["dt"] = df["dt"].dt.date

table = pa.Table.from_pandas(df)
pq.write_to_dataset(table, root_path="/path/on/BDS/HDFS", partition_cols=["dt"],
                    flavor="spark", filesystem=fs)

SQL Data Management

This section demonstrates how to perform standard SQL-based data management operations in BDS using various frameworks, see the individual framework’s documentation for details.

A Kerberos ticket is needed to connect to the BDS cluster. You can obtain this authentication ticket with the refresh_ticket() method, or with the use of the vault and a BDSSercretKeeper object. This section demonstrates the use of the BDSSecretKeeper object because this is more secure and is the recommended method.

Ibis

Ibis is an open-source library by Cloudera that provides a Python framework to access data and perform analytical computations from different sources. The Ibis project is designed to provide an abstraction over different dialects of SQL. It enables the data scientist to interact with many different data systems. Some of these systems are Dask, MySQL, Pandas, PostgreSQL, PySpark, and most importantly for use with BDS, Hadoop clusters.

Connect

Obtaining a Kerberos ticket, depending on your system configuration, you may need to define the ibis.options.impala.temp_db and ibis.options.impala.temp_hdfs_path options. The ibis.impala.connect() method makes a connection to the Impala execution backend. The .sql() allows you to run SQL commands on the data.

import ibis

with BDSSecretKeeper.load_secret("<secret_id>") as cred:
    with krbcontext(principal=cred["principal"], keytab_path=cred['keytab_path']):
        ibis.options.impala.temp_db = '<temp_db>'
        ibis.options.impala.temp_hdfs_path = '<temp_hdfs_path>'
        hdfs = ibis.impala.hdfs_connect(host=cred['hdfs_host'], port=cred['hdfs_port'],
                                         use_https=False, verify=False,
                                         auth_mechanism='GSSAPI', protocol='webhdfs')
        client = ibis.impala.connect(host=cred['hive_host'], port=cred['hive_port'],
                                     hdfs_client=hdfs, auth_mechanism="GSSAPI",
                                     use_ssl=False, kerberos_service_name="hive")

Query

To query the data using ibis use an SQL DML command like SELECT. Pass the string to the .sql() method, and then call .execute() on the returned object. The output is a Pandas dataframe.

df = client.sql("SELECT * FROM bikes.trips LIMIT 100").execute(limit=None)

Close a Connection

It is important to close sessions when you don’t need them anymore. This frees up resources in the system. Use the .close() method close sessions.

client.close()

Impala

Impala is a Python client for HiveServer2 implementations (i.e. Impala, Hive). Both Impala and PyHive clients are HiveServer2 compliant so the connection syntax is very similar. The difference is that the Impala client uses the Impala query engine and PyHive uses Hive. In practical terms, Hive is best suited for long-running batch queries and Impala is better suited for real-time interactive querying, see more about the differences between Hive and Impala.

The Impala dbapi module is a Python DB-API interface.

Connect

After obtaining a Kerberos ticket, use the connect() method to make the connection. It returns a connection, and the .cursor() method returns a cursor object. The cursor has the method .execute() that allows you to run Impala SQL commands on the data.

from impala.dbapi import connect

with BDSSecretKeeper.load_secret("<secret_id>") as cred:
    with krbcontext(principal=cred["principal"], keytab_path=cred['keytab_path']):
        cursor = connect(host=cred["hive_host"], port=cred["hive_port"],
                         auth_mechanism="GSSAPI", kerberos_service_name="hive").cursor()

Create a Table

To create an Impala table and insert data, use the .execute() method on the cursor object, and pass in Impala SQL commands to perform these operations.

cursor.execute("CREATE TABLE default.location (city STRING, province STRING)")
cursor.execute("INSERT INTO default.location VALUES ('Halifax', 'Nova Scotia')")

Query

To query an Impala table, use an Impala SQL DML command like SELECT. Pass this string to the .execute() method on the cursor object to create a record set in the cursor. You can obtain a Pandas dataframe with the as_pandas() function.

from impala.util import as_pandas

cursor.execute("SELECT * FROM default.location")
df = as_pandas(cursor)

Drop a Table

To drop an Impala table, use an Impala SQL DDL command like DROP TABLE. Pass this string to the .execute() method on the cursor object.

cursor.execute("DROP TABLE IF EXISTS default.location")

Close a Connection

It is important to close sessions when you don’t need them anymore. This frees up resources in the system. Use the .close() method on the cursor object to close a connection.

cursor.close()

PyHive

PyHive is a set of interfaces to Presto and Hive. It is based on the SQLAlchemy and Python DB-API interfaces for Presto and Hive.

Connect

After obtaining a Kerberos ticket, call the hive.connect() method to make the connection. It returns a connection, and the .cursor() method returns a cursor object. The cursor has the .execute() method that allows you to run Hive SQL commands on the data.

import ads
import os

from ads.bds.auth import krbcontext
from ads.secrets.big_data_service import BDSSecretKeeper
from pyhive import hive

ads.set_auth('resource_principal')
with BDSSecretKeeper.load_secret("<secret_id>") as cred:
    with krbcontext(principal=cred["principal"], keytab_path=cred['keytab_path']):
        cursor = hive.connect(host=cred["hive_host"],
                              port=cred["hive_port"],
                              auth='KERBEROS',
                              kerberos_service_name="hive").cursor()

Create a Table

To create a Hive table and insert data, use the .execute() method on the cursor object and pass in Hive SQL commands to perform these operations.

cursor.execute("CREATE TABLE default.location (city STRING, province STRING)")
cursor.execute("INSERT INTO default.location VALUES ('Halifax', 'Nova Scotia')")

Query

To query a Hive table, use a Hive SQL DML command like SELECT. Pass this string to the .execute() method on the cursor object. This creates a record set in the cursor. You can access the actual records with methods like .fetchall(), .fetchmany(), and .fetchone().

In the following example, the .fetchall() method is used in a pd.DataFrame() call to return all the records in Pandas dataframe: .

import pandas as pd

cursor.execute("SELECT * FROM default.location")
df = pd.DataFrame(cursor.fetchall(), columns=[col[0] for col in cursor.description])

Drop a Table

To drop a Hive table, use a Hive SQL DDL command like DROP TABLE. Pass this string to the .execute() method on the cursor object.

cursor.execute("DROP TABLE IF EXISTS default.location")

Close a Connection

It is important to close sessions when you don’t need them anymore. This frees up resources in the system. Use the .close() method on the cursor object to close a connection.

cursor.close()

Data Science Jobs

Oracle Cloud Infrastructure (OCI) Data Science Jobs (Jobs) enables you to define and run repeatable machine learning tasks on a fully managed infrastructure. You can create a compute resource on demand and run applications that perform tasks such as data preparation, model training, hyperparameter tuning, and batch inference.

Running workloads with Data Science Jobs involves two types resources: Job and Job Run.

A Job is a template that describes the training task. It contains configurations about the infrastructure, such as Compute Shape, Block Storage, Logging, and information about the runtime, such as the source code of your workload, environment variables, and CLI arguments.

A Job Run is an instantiation of a job. In each job run, you can override some of the job configurations, such as environment variables and CLI arguments. You can use the same job as a template and launch multiple simultaneous job runs to parallelize a large task. You can also sequence jobs and keep the state by writing state information to Object Storage

For example, you may want to experiment with how different model classes perform on the same training data by using the ADSTuner to perform hyperparameter tuning on each model class. You could do this in parallel by having a different job run for each class of models. For a given job run, you could pass an environment variable that identifies the model class that you want to use. Each model can write its results to the Logging service or Object Storage. Then you can run a final sequential job that uses the best model class, and trains the final model on the entire dataset.

The following sections provides details on running workloads with OCI Data Science Jobs using ADS Jobs APIs. You can use similar APIs to Run a OCI DataFlow Application.

Quick Start

Prerequisite

Before creating a job, ensure that you have policies configured for Data Science resources.

See IAM Policies and About Data Science Policies.

Define a Job

In ADS, a job is defined by Infrastructure and Runtime. The Data Science Job infrastructure is configured through a DataScienceJob instance. The runtime can be an instance of:

PythonRuntime for Python code stored locally, OCI object storage, or other remote location supported by fsspec. See Run a Python Workload.
GitPythonRuntime for Python code from a Git repository. See Run Code from Git Repo.
NotebookRuntime for a single Jupyter notebook stored locally, OCI object storage, or other remote location supported by fsspec. See Run a Notebook.
ScriptRuntime for bash or shell scripts stored locally, OCI object storage, or other remote location supported by fsspec. See Run a Script.

ContainerRuntime for container images.

Here is an example to define and run a Python Job.

Note that a job can be defined either using Python APIs or YAML. See the next section for how to load and save the job with YAML.

Python
YAML

from ads.jobs import Job, DataScienceJob, PythonRuntime

job = (
    Job(name="My Job")
    .with_infrastructure(
        DataScienceJob()
        # Configure logging for getting the job run outputs.
        .with_log_group_id("<log_group_ocid>")
        # Log resource will be auto-generated if log ID is not specified.
        .with_log_id("<log_ocid>")
        # If you are in an OCI data science notebook session,
        # the following configurations are not required.
        # Configurations from the notebook session will be used as defaults.
        .with_compartment_id("<compartment_ocid>")
        .with_project_id("<project_ocid>")
        .with_subnet_id("<subnet_ocid>")
        .with_shape_name("VM.Standard.E3.Flex")
        # Shape config details are applicable only for the flexible shapes.
        .with_shape_config_details(memory_in_gbs=16, ocpus=1)
        # Minimum/Default block storage size is 50 (GB).
        .with_block_storage_size(50)
    )
    .with_runtime(
        PythonRuntime()
        # Specify the service conda environment by slug name.
        .with_service_conda("pytorch110_p38_cpu_v1")
        # Source code of the job, can be local or remote.
        .with_source("path/to/script.py")
        # Environment variable
        .with_environment_variable(NAME="Welcome to OCI Data Science.")
        # Command line argument
        .with_argument(greeting="Good morning")
    )
)

kind: job
spec:
  name: "My Job"
  infrastructure:
    kind: infrastructure
    type: dataScienceJob
    spec:
      blockStorageSize: 50
      compartmentId: <compartment_ocid>
      jobInfrastructureType: STANDALONE
      logGroupId: <log_group_ocid>
      logId: <log_ocid>
      projectId: <project_ocid>
      shapeConfigDetails:
        memoryInGBs: 16
        ocpus: 1
      shapeName: VM.Standard.E3.Flex
      subnetId: <subnet_ocid>
  runtime:
    kind: runtime
    type: python
    spec:
      args:
      - --greeting
      - Good morning
      conda:
        slug: pytorch110_p38_cpu_v1
        type: service
      env:
      - name: NAME
        value: Welcome to OCI Data Science.
      scriptPathURI: path/to/script.py

The PythonRuntime is designed for Running a Python Workload. The source code is specified by with_source() (path/to/script.py). It can be a script, a Jupyter notebook, a folder or a zip file. The source code location can be a local or remote, including HTTP URL and OCI Object Storage. An example Python script is available on Data Science AI Sample GitHub Repository.

For more details, see Infrastructure and Runtime configurations. You can also Run a Notebook, Run a Script and Run Code from Git Repo.

YAML

A job can be defined using YAML, as shown in the “YAML” tab in the example above. Here are some examples to load/save the YAML job configurations:

# Load a job from a YAML file
job = Job.from_yaml(uri="oci://bucket_name@namespace/path/to/job.yaml")

# Save a job to a YAML file
job.to_yaml(uri="oci://bucket_name@namespace/path/to/job.yaml")

# Save a job to YAML in a string
yaml_string = job.to_yaml()

# Load a job from a YAML string
job = Job.from_yaml("""
kind: job
spec:
  infrastructure:
  kind: infrastructure
    ...
""")

The uri can be a local file path or a remote location supported by fsspec, including OCI object storage.

With the YAML file, you can create and run the job with ADS CLI:

ads opctl run -f your_job.yaml

For more details on ads opctl, see Working with the CLI.

The job infrastructure, runtime and job run also support YAML serialization/deserialization.

Run a Job and Monitor outputs

Once the job is defined or loaded from YAML, you can call the create() method to create the job on OCI. To start a job run, you can call the run() method, which returns a DataScienceJobRun instance. Once the job or job run is created, the job OCID can be accessed through job.id or run.id.

# Create the job on OCI Data Science
job.create()
# Start a job run
run = job.run()
# Stream the job run outputs
run.watch()

The watch() method is useful to monitor the progress of the job run. It will stream the logs to terminal and return once the job is finished. Logging configurations are required for this method to show logs. Here is an example of the logs:

Job OCID: <job_ocid>
Job Run OCID: <job_run_ocid>
2023-02-27 15:58:01 - Job Run ACCEPTED
2023-02-27 15:58:11 - Job Run ACCEPTED, Infrastructure provisioning.
2023-02-27 15:59:06 - Job Run ACCEPTED, Infrastructure provisioned.
2023-02-27 15:59:29 - Job Run ACCEPTED, Job run bootstrap starting.
2023-02-27 16:01:08 - Job Run ACCEPTED, Job run bootstrap complete. Artifact execution starting.
2023-02-27 16:01:18 - Job Run IN_PROGRESS, Job run artifact execution in progress.
2023-02-27 16:01:11 - Good morning, your environment variable has value of (Welcome to OCI Data Science.)
2023-02-27 16:01:11 - Job Run 02-27-2023-16:01:11
2023-02-27 16:01:11 - Job Done.
2023-02-27 16:01:22 - Job Run SUCCEEDED, Job run artifact execution succeeded. Infrastructure de-provisioning.

Load Existing Job or Job Run

You can load an existing job or job run using the OCID from OCI:

from ads.jobs import Job, DataScienceJobRun

# Load a job
job = Job.from_datascience_job("<job_ocid>")

# Load a job run
job_run = DataScienceJobRun.from_ocid("<job_run_ocid>"")

List Existing Jobs or Job Runs

To get a list of existing jobs in a specific compartment:

from ads.jobs import Job

# Get a list of jobs in a specific compartment.
jobs = Job.datascience_job("<compartment_ocid>")

With a Job object, you can get a list of job runs:

# Gets a list of job runs for a specific job.
runs = job.run_list()

Delete a Job or Job Run

You can delete a job or job run by calling the delete() method.

# Delete a job and the corresponding job runs.
job.delete()
# Delete a job run
run.delete()

You can also cancel a job run:

run.cancel()

Variable Substitution

When defining a job or starting a job run, you can use environment variable substitution for the names and output_uri argument of the with_output() method.

For example, the following job specifies the name based on the environment variable DATASET_NAME, and output_uri based on the environment variables JOB_RUN_OCID:

Python
YAML

from ads.jobs import Job, DataScienceJob, PythonRuntime

job = (
    Job(name="Training on ${DATASET_NAME}")
    .with_infrastructure(
        DataScienceJob()
        .with_log_group_id("<log_group_ocid>")
        .with_log_id("<log_ocid>")
        .with_compartment_id("<compartment_ocid>")
        .with_project_id("<project_ocid>")
        .with_shape_name("VM.Standard.E3.Flex")
        .with_shape_config_details(memory_in_gbs=16, ocpus=1)
    )
    .with_runtime(
        PythonRuntime()
        .with_service_conda("pytorch110_p38_gpu_v1")
        .with_environment_variable(DATASET_NAME="MyData")
        .with_source("local/path/to/training_script.py")
        .with_output("output", "oci://bucket_name@namespace/prefix/${JOB_RUN_OCID}")
    )
)

kind: job
spec:
  name: Training on ${DATASET_NAME}
  infrastructure:
    kind: infrastructure
    type: dataScienceJob
    spec:
      compartmentId: <compartment_ocid>
      logGroupId: <log_group_ocid>
      logId: <log_ocid>
      projectId: <project_ocid>
      shapeConfigDetails:
        memoryInGBs: 16
        ocpus: 1
      shapeName: VM.Standard.E3.Flex
  runtime:
    kind: runtime
    type: python
    spec:
      conda:
        slug: pytorch110_p38_cpu_v1
        type: service
      env:
      - name: DATASET_NAME
        value: MyData
      outputDir: output
      outputUri: oci://bucket_name@namespace/prefix/${JOB_RUN_OCID}
      scriptPathURI: local/path/to/training_script.py

Note that JOB_RUN_OCID is an environment variable provided by the service after the job run is created. It is available for the output_uri but cannot be used in the job name.

See also:

IAM Policies

Oracle Cloud Infrastructure Identity and Access Management (IAM) lets you specify policies to control the access to your cloud resources. This section describe the policies you might need for running Data Science Jobs.

Warning

The policies presented in this page are intended to show the resource_type used by the job and job runs. You should further restrict the access to the resources base on your needs.

Policy subject

In the following example, group <your_data_science_users> is the subject of the policy when using OCI API keys for authentication. For resource principal authentication, the subject should be a dynamic-group, for example, dynamic-group <your_resources>

Here is an example defining a dynamic group for all job runs in a compartment:

all { resource.type='datasciencejobrun', resource.compartment.id='<job_run_compartment_ocid>' }

The following policies are for creating and managing Data Science Jobs and Job Runs:

Allow group <your_data_science_users> to manage data-science-jobs in compartment <your_compartment_name>
Allow group <your_data_science_users> to manage data-science-job-runs in compartment <your_compartment_name>
Allow group <your_data_science_users> to use virtual-network-family in compartment <your_compartment_name>
Allow group <your_data_science_users> to manage log-groups in compartment <your_compartment_name>
Allow group <your_data_science_users> to use logging-family in compartment <your_compartment_name>
Allow group <your_data_science_users> to use read metrics in compartment <your_compartment_name>

The following policies are for job runs to access other OCI resources:

Allow dynamic-group <your_resources> to read repos in compartment <your_compartment_name>
Allow dynamic-group <your_resources> to use data-science-family in compartment <your_compartment_name>
Allow dynamic-group <your_resources> to use virtual-network-family in compartment <your_compartment_name>
Allow dynamic-group <your_resources> to use log-groups in compartment <your_compartment_name>
Allow dynamic-group <your_resources> to use logging-family in compartment <your_compartment_name>
Allow dynamic-group <your_resources> to manage objects in compartment <your_compartment_name> where all {target.bucket.name=<your_bucket_name>}
Allow dynamic-group <your_resources> to use buckets in compartment <your_compartment_name> where all {target.bucket.name=<your_bucket_name>}

The following policy is needed for running a container job:

Allow dynamic-group <your_resources> to read repos in compartment <your_compartment_name>

See also:

Infrastructure and Runtime

This page describes the configurations of Infrastructure and Runtime defining a Data Science Job.

Example

The following example configures the infrastructure and runtime to run a Python script.

Python
YAML

from ads.jobs import Job, DataScienceJob, PythonRuntime

job = (
    Job(name="My Job")
    .with_infrastructure(
        DataScienceJob()
        # Configure logging for getting the job run outputs.
        .with_log_group_id("<log_group_ocid>")
        # Log resource will be auto-generated if log ID is not specified.
        .with_log_id("<log_ocid>")
        # If you are in an OCI data science notebook session,
        # the following configurations are not required.
        # Configurations from the notebook session will be used as defaults.
        .with_compartment_id("<compartment_ocid>")
        .with_project_id("<project_ocid>")
        .with_subnet_id("<subnet_ocid>")
        .with_shape_name("VM.Standard.E3.Flex")
        # Shape config details are applicable only for the flexible shapes.
        .with_shape_config_details(memory_in_gbs=16, ocpus=1)
        # Minimum/Default block storage size is 50 (GB).
        .with_block_storage_size(50)
    )
    .with_runtime(
        PythonRuntime()
        # Specify the service conda environment by slug name.
        .with_service_conda("pytorch110_p38_cpu_v1")
        # Source code of the job, can be local or remote.
        .with_source("path/to/script.py")
        # Environment variable
        .with_environment_variable(NAME="Welcome to OCI Data Science.")
        # Command line argument
        .with_argument(greeting="Good morning")
    )
)

kind: job
spec:
  name: "My Job"
  infrastructure:
    kind: infrastructure
    type: dataScienceJob
    spec:
      blockStorageSize: 50
      compartmentId: <compartment_ocid>
      jobInfrastructureType: STANDALONE
      logGroupId: <log_group_ocid>
      logId: <log_ocid>
      projectId: <project_ocid>
      shapeConfigDetails:
        memoryInGBs: 16
        ocpus: 1
      shapeName: VM.Standard.E3.Flex
      subnetId: <subnet_ocid>
  runtime:
    kind: runtime
    type: python
    spec:
      args:
      - --greeting
      - Good morning
      conda:
        slug: pytorch110_p38_cpu_v1
        type: service
      env:
      - name: NAME
        value: Welcome to OCI Data Science.
      scriptPathURI: path/to/script.py

Infrastructure

The Data Science Job infrastructure is defined by a DataScienceJob instance. For example:

Python
YAML

from ads.jobs import Job, DataScienceJob, GitPythonRuntime

infrastructure = (
    DataScienceJob()
    # Configure logging for getting the job run outputs.
    .with_log_group_id("<log_group_ocid>")
    # Log resource will be auto-generated if log ID is not specified.
    .with_log_id("<log_ocid>")
    # If you are in an OCI data science notebook session,
    # the following configurations are not required.
    # Configurations from the notebook session will be used as defaults.
    .with_compartment_id("<compartment_ocid>")
    .with_project_id("<project_ocid>")
    .with_subnet_id("<subnet_ocid>")
    .with_shape_name("VM.Standard.E3.Flex")
    # Shape config details are applicable only for the flexible shapes.
    .with_shape_config_details(memory_in_gbs=16, ocpus=1)
    # Minimum/Default block storage size is 50 (GB).
    .with_block_storage_size(50)
)

kind: infrastructure
type: dataScienceJob
spec:
  blockStorageSize: 50
  compartmentId: <compartment_ocid>
  logGroupId: <log_group_ocid>
  logId: <log_ocid>
  projectId: <project_ocid>
  shapeConfigDetails:
    memoryInGBs: 16
    ocpus: 1
  shapeName: VM.Standard.E3.Flex
  subnetId: <subnet_ocid>

When creating a DataScienceJob instance, the following configurations are required:

Compartment ID
Project ID
Compute Shape

The following configurations are optional:

Block Storage Size, defaults to 50 (GB)
Log Group ID
Log ID

For more details about the mandatory and optional parameters, see DataScienceJob.

Using Configurations from Notebook

If you are creating a job from an OCI Data Science Notebook Session, the same infrastructure configurations from the notebook session will be used as defaults, including:

Compartment ID
Project ID
Subnet ID
Compute Shape
Block Storage Size

You can initialize the DataScienceJob with the logging configurations and override the other options as needed. For example:

Python
YAML

from ads.jobs import DataScienceJob

infrastructure = (
    DataScienceJob()
    .with_log_group_id("<log_group_ocid>")
    .with_log_id("<log_ocid>")
    # Use a GPU shape for the job,
    # regardless of the shape used by the notebook session
    .with_shape_name("VM.GPU3.1")
    # compartment ID, project ID, subnet ID and block storage will be
    # the same as the ones set in the notebook session
)

kind: infrastructure
type: dataScienceJob
spec:
  logGroupId: <log_group_ocid>
  logId: <log_ocid>
  shapeName: VM.GPU3.1

Compute Shapes

The DataScienceJob class provides two static methods to obtain the support compute shapes:

You can get a list of currently supported compute shapes by calling instance_shapes().
can get a list of shapes that are available for fast launch by calling fast_launch_shapes(). Specifying a fast launch shape will allow your job to start as fast as possible.

Networking

Data Science Job offers two types of networking: default networking (managed egress) and custom networking. Default networking allows job runs to access public internet through a NAT gateway and OCI service through a service gateway, both are configured automatically. Custom networking requires you to specify a subnet ID. You can control the network access through the subnet and security lists.

If you specified a subnet ID, your job will be configured to have custom networking. Otherwise, default networking will be used. Note that when you are in a Data Science Notebook Session, the same networking configuration is be used by default. You can specify the networking manually by calling with_job_infrastructure_type().

Logging

Logging is not required to create the job. However, it is highly recommended to enable logging for debugging and monitoring.

In the preceding example, both the log OCID and corresponding log group OCID are specified with the DataScienceJob instance. If your administrator configured the permission for you to search for logging resources, you can skip specifying the log group OCID because ADS can automatically retrieve it.

If you specify only the log group OCID and no log OCID, a new Log resource is automatically created within the log group to store the logs, see also ADS Logging.

With logging configured, you can call watch() method to stream the logs.

Runtime

The runtime of a job defines the source code of your workload, environment variables, CLI arguments and other configurations for the environment to run the workload.

Depending on the source code, ADS provides different types of runtime for defining a data science job, including:

PythonRuntime for Python code stored locally, OCI object storage, or other remote location supported by fsspec. See Run a Python Workload.
GitPythonRuntime for Python code from a Git repository. See Run Code from Git Repo.
NotebookRuntime for a single Jupyter notebook stored locally, OCI object storage, or other remote location supported by fsspec. See Run a Notebook.
ScriptRuntime for bash or shell scripts stored locally, OCI object storage, or other remote location supported by fsspec. See Run a Script.

ContainerRuntime for container images.

Environment Variables

You can set environment variables for a runtime by calling with_environment_variable(). Environment variables enclosed by ${...} will be substituted. For example:

Python
YAML

from ads.jobs import PythonRuntime

runtime = (
    PythonRuntime()
    .with_environment_variable(
        HOST="10.0.0.1",
        PORT="443",
        URL="http://${HOST}:${PORT}/path/",
        ESCAPED_URL="http://$${HOST}:$${PORT}/path/",
        MISSING_VAR="This is ${UNDEFINED}",
        VAR_WITH_DOLLAR="$10",
        DOUBLE_DOLLAR="$$10"
    )
)

kind: runtime
type: python
spec:
  env:
  - name: HOST
    value: 10.0.0.1
  - name: PORT
    value: '443'
  - name: URL
    value: http://${HOST}:${PORT}/path/
  - name: ESCAPED_URL
    value: http://$${HOST}:$${PORT}/path/
  - name: MISSING_VAR
    value: This is ${UNDEFINED}
  - name: VAR_WITH_DOLLAR
    value: $10
  - name: DOUBLE_DOLLAR
    value: $$10

for k, v in runtime.environment_variables.items():
    print(f"{k}: {v}")

will show the following environment variables for the runtime:

HOST: 10.0.0.1
PORT: 443
URL: http://10.0.0.1:443/path/
ESCAPED_URL: http://${HOST}:${PORT}/path/
MISSING_VAR: This is ${UNDEFINED}
VAR_WITH_DOLLAR: $10
DOUBLE_DOLLAR: $10

Note that:

You can use $$ to escape the substitution.
Undefined variable enclosed by ${...} will be ignored.
Double dollar signs $$ will be substituted by a single one $.

Command Line Arguments

The command line arguments for running your script or function can be configured by calling with_argument(). For example:

Python
YAML

from ads.jobs import PythonRuntime

runtime = (
    PythonRuntime()
    .with_source("oci://bucket_name@namespace/path/to/script.py")
    .with_argument(
        "arg1", "arg2",
        key1="val1",
        key2="val2"
    )
)

kind: runtime
type: python
spec:
  scriptPathURI: oci://bucket_name@namespace/path/to/script.py
  args:
  - arg1
  - arg2
  - --key1
  - val1
  - --key2
  - val2

will configured the job to call your script by:

python script.py arg1 arg2 --key1 val1 --key2 val2

You can call with_argument() multiple times to set the arguments to your desired order. You can check runtime.args to see the added arguments.

Here are a few more examples:

Python
YAML

runtime = PythonRuntime()
runtime.with_argument(key1="val1", key2="val2")
runtime.with_argument("pos1")

kind: runtime
type: python
spec:
  args:
  - --key1
  - val1
  - --key2
  - val2
  - pos1

print(runtime.args)
# ['--key1', 'val1', '--key2', 'val2', 'pos1']

Python
YAML

runtime = PythonRuntime()
runtime.with_argument("pos1")
runtime.with_argument(key1="val1", key2="val2.1 val2.2")
runtime.with_argument("pos2")

kind: runtime
type: python
spec:
  args:
  - pos1
  - --key1
  - val1
  - --key2
  - val2.1 val2.2
  - pos2

print(runtime.args)
# ['pos1', '--key1', 'val1', '--key2', 'val2.1 val2.2', 'pos2']

Python
YAML

runtime = PythonRuntime()
runtime.with_argument("pos1")
runtime.with_argument(key1=None, key2="val2")
runtime.with_argument("pos2")

kind: runtime
type: python
spec:
  args:
  - pos1
  - --key1
  - --key2
  - val2
  - pos2

print(runtime.args)
# ["pos1", "--key1", "--key2", "val2", "pos2"]

Conda Environment

You can configure a Conda Environment for running your workload. You can use the slug name to specify a conda environment provided by the data science service. For example, to use the TensorFlow conda environment:

Python
YAML

from ads.jobs import PythonRuntime

runtime = (
    PythonRuntime()
    .with_source("oci://bucket_name@namespace/path/to/script.py")
    # Use slug name for conda environment provided by data science service
    .with_service_conda("tensorflow28_p38_cpu_v1")
)

kind: runtime
type: python
spec:
  conda:
    type: service
    slug: tensorflow28_p38_cpu_v1
  scriptPathURI: oci://bucket_name@namespace/path/to/script.py

You can also use a custom conda environment published to OCI Object Storage by passing the uri to with_custom_conda(), for example:

Python
YAML

from ads.jobs import PythonRuntime

runtime = (
    PythonRuntime()
    .with_source("oci://bucket_name@namespace/path/to/script.py")
    .with_custom_conda("oci://bucket@namespace/conda_pack/pack_name")
)

kind: runtime
type: python
spec:
  conda:
    type: published
    uri: oci://bucket@namespace/conda_pack/pack_name
  scriptPathURI: oci://bucket_name@namespace/path/to/script.py

By default, ADS will try to determine the region based on the authenticated API key or resource principal. If your custom conda environment is stored in a different region, you can specify the region when calling with_custom_conda().

For more details on custom conda environment, see Publishing a Conda Environment to an Object Storage Bucket in Your Tenancy.

Override Configurations

When you call ads.jobs.Job.run(), a new job run will be started with the configuration defined in the job. You may want to override the configuration with custom variables. For example, you can customize job run display name, override command line argument, specify additional environment variables, and add free form tags:

job_run = job.run(
    name="<my_job_run_name>",
    args="new_arg --new_key new_val",
    env_var={"new_env": "new_val"},
    freeform_tags={"new_tag": "new_tag_val"}
)

Run a Python Workload

The PythonRuntime is designed for running a Python workload. You can configure the environment variables, command line arguments, and conda environment as described in Infrastructure and Runtime. This section shows the additional enhancements provided by PythonRuntime.

Example

Here is an example to define and run a job using PythonRuntime:

Python
YAML

from ads.jobs import Job, DataScienceJob, PythonRuntime

job = (
    Job(name="My Job")
    .with_infrastructure(
        DataScienceJob()
        # Configure logging for getting the job run outputs.
        .with_log_group_id("<log_group_ocid>")
        # Log resource will be auto-generated if log ID is not specified.
        .with_log_id("<log_ocid>")
        # If you are in an OCI data science notebook session,
        # the following configurations are not required.
        # Configurations from the notebook session will be used as defaults.
        .with_compartment_id("<compartment_ocid>")
        .with_project_id("<project_ocid>")
        .with_subnet_id("<subnet_ocid>")
        .with_shape_name("VM.Standard.E3.Flex")
        # Shape config details are applicable only for the flexible shapes.
        .with_shape_config_details(memory_in_gbs=16, ocpus=1)
        # Minimum/Default block storage size is 50 (GB).
        .with_block_storage_size(50)
    )
    .with_runtime(
        PythonRuntime()
        # Specify the service conda environment by slug name.
        .with_service_conda("pytorch110_p38_cpu_v1")
        # The job artifact can be a single Python script, a directory or a zip file.
        .with_source("local/path/to/code_dir")
        # Environment variable
        .with_environment_variable(NAME="Welcome to OCI Data Science.")
        # Command line argument, arg1 --key arg2
        .with_argument("arg1", key="arg2")
        # Set the working directory
        # When using a directory as source, the default working dir is the parent of code_dir.
        # Working dir should be a relative path beginning from the source directory (code_dir)
        .with_working_dir("code_dir")
        # The entrypoint is applicable only to directory or zip file as source
        # The entrypoint should be a path relative to the working dir.
        # Here my_script.py is a file in the code_dir/my_package directory
        .with_entrypoint("my_package/my_script.py")
        # Add an additional Python path, relative to the working dir (code_dir/other_packages).
        .with_python_path("other_packages")
        # Copy files in "code_dir/output" to object storage after job finishes.
        .with_output("output", "oci://bucket_name@namespace/path/to/dir")
    )
)

kind: job
spec:
  name: "My Job"
  infrastructure:
    kind: infrastructure
    type: dataScienceJob
    spec:
      blockStorageSize: 50
      compartmentId: <compartment_ocid>
      jobInfrastructureType: STANDALONE
      logGroupId: <log_group_ocid>
      logId: <log_ocid>
      projectId: <project_ocid>
      shapeConfigDetails:
        memoryInGBs: 16
        ocpus: 1
      shapeName: VM.Standard.E3.Flex
      subnetId: <subnet_ocid>
  runtime:
    kind: runtime
    type: python
    spec:
      args:
      - arg1
      - --key
      - arg2
      conda:
        slug: pytorch110_p38_cpu_v1
        type: service
      entrypoint: my_package/my_script.py
      env:
      - name: NAME
        value: Welcome to OCI Data Science.
      outputDir: output
      outputUri: oci://bucket_name@namespace/path/to/dir
      pythonPath:
      - other_packages
      scriptPathURI: local/path/to/code_dir
      workingDir: code_dir
      workingDir: code_dir

# Create the job on OCI Data Science
job.create()
# Start a job run
run = job.run()
# Stream the job run outputs
run.watch()

# Create the job on OCI Data Science
job.create()
# Start a job run
run = job.run()
# Stream the job run outputs
run.watch()

The PythonRuntime uses an driver script from ADS for the job run. It performs additional operations before and after invoking your code. You can examine the driver script by downloading the job artifact from the OCI Console.

Source Code

In the with_source() method, you can specify the location of your source code. The location can be a local path or a remote URI supported by fsspec. For example, you can specify files on OCI object storage using URI like oci://bucket@namespace/path/to/prefix. ADS will use the authentication method configured by ads.set_auth() to fetch the files and upload them as job artifact.

The source code can be a single file, a compressed file/archive (zip/tar), or a folder. When the source code is a compressed file/archive (zip/tar) or a folder, you need to also specify the entrypoint using with_entrypoint(). The path of the entrypoint should be a path relative to the working directory.

The entrypoint can be a Python script or a Jupyter Notebook.

Working Directory

The working directory of your workload can be configured by with_working_dir(). By default, PythonRuntime will create a code directory as the working directory in the job run to store your source code (job artifact), for example /home/datascience/decompressed_artifact/code.

When the entrypoint is a Jupyter notebook, the working directory for the code running in the notebook will be the directory containing the notebook.

When the entrypoint is not a notebook, the working directory depends on the source code.

File Source Code

If your source code is a single file, for example, my_script.py, the file structure in the job run will look like:

code  <---This is the working directory
└── my_script.py

You can refer your as ./my_script.py

Folder Source Code

If your source code is a folder, for example my_source_code, ADS will compress the folder as job artifact. In the job run, it will be decompressed under the working directory. The file structure in the job run will look like:

code  <---This is the working directory
└── my_source_code
    ├── my_module.py
    └── my_entrypoint.py

In this case, the working directory is the parent of your source code folder. You will need to specify the entrypoint as my_source_code/my_entrypoint.py.

runtime = (
  PythonRuntime()
  .with_source("path/to/my_source_code")
  .with_entrypoint("my_source_code/my_entrypoint.py")
)

Alternatively, you can specify the working directory as my_source_code and the entrypoint as my_entrypoint.py:

runtime = (
  PythonRuntime()
  .with_source("path/to/my_source_code")
  .with_working_dir("my_source_code")
  .with_entrypoint("my_entrypoint.py")
)

Archive Source Code

If your source code is a zip/tar file, the files in the archive will be decompressed under the working directory. The file structure in the job run depends on whether your archive has a top level directory. For example, you can inspect the structure of your zip file by running the unzip -l command:

unzip -l my_source_code.zip

This will give you outputs similar to the following:

Archive:  path/to/my_source_code.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  02-22-2023 16:38   my_source_code/
     1803  02-22-2023 16:38   my_source_code/my_module.py
       91  02-22-2023 16:38   my_source_code/my_entrypoint.py
---------                     -------
     1894                     3 files

In this case, a top level directory my_source_code/ is presented in the archive. The file structure in the job run will look like:

code  <---This is the working directory
└── my_source_code
    ├── my_module.py
    └── my_entrypoint.py

which is the same as the case when you specified a local folder as source code. You can configure the entrypoint and working directory similar to the examples above.

If a top level directory is not presented, outputs for the archive will look like the following:

Archive:  path/to/my_source_code.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
     1803  02-22-2023 16:38   my_module.py
       91  02-22-2023 16:38   my_entrypoint.py
---------                     -------
     1894                     2 files

In this case, the file structure in the job run will look like:

code  <---This is the working directory
├── my_module.py
└── my_entrypoint.py

And, you can specify the entrypoint with the filename directly:

runtime = (
  PythonRuntime()
  .with_source("path/to/my_source_code.zip")
  .with_entrypoint("my_entrypoint.py")
)

Python Paths

The working directory is added to the Python paths automatically. You can call with_python_path() to add additional python paths as needed. The paths should be relative paths from the working directory.

Outputs

The with_output() method allows you to specify the output path output_path in the job run and a remote URI (output_uri). Files in the output_path are copied to the remote output URI after the job run finishes successfully. Note that the output_path should be a path relative to the working directory.

OCI object storage location can be specified in the format of oci://bucket_name@namespace/path/to/dir. Please make sure you configure the I AM policy to allow the job run dynamic group to use object storage.

Run a Notebook

The NotebookRuntime allows you to run a single Jupyter notebook as a job.

TensorFlow Example

The following example shows you how to run an the TensorFlow 2 quick start for beginner notebook from the internet and save the results to OCI Object Storage. The notebook path points to the raw file link from GitHub. To run the example, ensure that you have internet access to retrieve the notebook:

Python
YAML

from ads.jobs import Job, DataScienceJob, NotebookRuntime

job = (
    Job(name="My Job")
    .with_infrastructure(
        DataScienceJob()
        # Configure logging for getting the job run outputs.
        .with_log_group_id("<log_group_ocid>")
        # Log resource will be auto-generated if log ID is not specified.
        .with_log_id("<log_ocid>")
        # If you are in an OCI data science notebook session,
        # the following configurations are not required.
        # Configurations from the notebook session will be used as defaults.
        .with_compartment_id("<compartment_ocid>")
        .with_project_id("<project_ocid>")
        .with_subnet_id("<subnet_ocid>")
        .with_shape_name("VM.Standard.E3.Flex")
        # Shape config details are applicable only for the flexible shapes.
        .with_shape_config_details(memory_in_gbs=16, ocpus=1)
        # Minimum/Default block storage size is 50 (GB).
        .with_block_storage_size(50)
    )
    .with_runtime(
        NotebookRuntime()
        .with_notebook(
            path="https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb",
            encoding='utf-8'
        )
        .with_service_conda("tensorflow28_p38_cpu_v1")
        .with_environment_variable(GREETINGS="Welcome to OCI Data Science")
        .with_exclude_tag(["ignore", "remove"])
        .with_output("oci://bucket_name@namespace/path/to/dir")
    )
)

kind: job
spec:
  name: "My Job"
  infrastructure:
    kind: infrastructure
    type: dataScienceJob
    spec:
      blockStorageSize: 50
      compartmentId: <compartment_ocid>
      jobInfrastructureType: STANDALONE
      logGroupId: <log_group_ocid>
      logId: <log_ocid>
      projectId: <project_ocid>
      shapeConfigDetails:
        memoryInGBs: 16
        ocpus: 1
      shapeName: VM.Standard.E3.Flex
      subnetId: <subnet_ocid>
  runtime:
    kind: runtime
    type: notebook
    spec:
      conda:
        slug: tensorflow28_p38_cpu_v1
        type: service
      env:
      - name: GREETINGS
        value: Welcome to OCI Data Science
      excludeTags:
      - ignore
      - remove
      notebookEncoding: utf-8
      notebookPathURI: https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb
      outputUri: oci://bucket_name@namespace/path/to/dir

# Create the job on OCI Data Science
job.create()
# Start a job run
run = job.run()
# Stream the job run outputs
run.watch()
# Download the notebook back to local
run.download("/path/to/local/dir")

Working Directory

An empty directory in the job run will be created as the working directory for running the notebook. All relative paths used in the notebook will be base on the working directory.

Download the Outputs

If you specify the output location using with_output(). All files in the working directory, including the notebook with outputs, will be saved to output location (oci://bucket_name@namespace/path/to/dir) after the job finishes running. You can download the output by calling the download() method.

Exclude Cells

The NotebookRuntime also allows you to specify tags to exclude cells from being processed in a job run using with_exclude_tag() method. For example, you could do exploratory data analysis and visualization in a notebook, and you may want to exclude the visualization when running the notebook in a job.

To tag cells in a notebook, see Adding tags using notebook interfaces.

The with_exclude_tag() take a list of tags as argument Cells with any matching tags are excluded from the job run. In the above example, cells with ignore or remove are excluded.

Notebook with Dependencies

If your notebook needs extra dependencies like custom module or data files, you can use PythonRuntime or GitPythonRuntime and set your notebook as the entrypoint.

See also:

Here is an example of running the minGPT demo notebook.

Python
YAML

from ads.jobs import Job, DataScienceJob, GitPythonRuntime

job = (
    Job(name="My Job")
    .with_infrastructure(
        DataScienceJob()
        # Configure logging for getting the job run outputs.
        .with_log_group_id("<log_group_ocid>")
        # Log resource will be auto-generated if log ID is not specified.
        .with_log_id("<log_ocid>")
        # If you are in an OCI data science notebook session,
        # the following configurations are not required.
        # Configurations from the notebook session will be used as defaults.
        .with_compartment_id("<compartment_ocid>")
        .with_project_id("<project_ocid>")
        .with_subnet_id("<subnet_ocid>")
        .with_shape_name("VM.Standard.E3.Flex")
        # Shape config details are applicable only for the flexible shapes.
        .with_shape_config_details(memory_in_gbs=16, ocpus=1)
        # Minimum/Default block storage size is 50 (GB).
        .with_block_storage_size(50)
    )
    .with_runtime(
        GitPythonRuntime()
        # Use service conda pack
        .with_service_conda("pytorch110_p38_gpu_v1")
        # Specify training source code from GitHub
        .with_source(url="https://github.com/karpathy/minGPT.git")
        # Entrypoint is a relative path from the root of the Git repository
        .with_entrypoint("demo.ipynb")
    )
)

kind: job
spec:
  name: "My Job"
  infrastructure:
    kind: infrastructure
    type: dataScienceJob
    spec:
      blockStorageSize: 50
      compartmentId: <compartment_ocid>
      jobInfrastructureType: STANDALONE
      logGroupId: <log_group_ocid>
      logId: <log_ocid>
      projectId: <project_ocid>
      shapeConfigDetails:
        memoryInGBs: 16
        ocpus: 1
      shapeName: VM.Standard.E3.Flex
      subnetId: <subnet_ocid>
  runtime:
    kind: runtime
    type: gitPython
    spec:
      conda:
        slug: pytorch19_p37_gpu_v1
        type: service
      entrypoint: demo.ipynb
      url: https://github.com/karpathy/minGPT.git

# Create the job on OCI Data Science
job.create()
# Start a job run
run = job.run()
# Stream the job run outputs
run.watch()

Run a Script

This section shows how to create a job to run a script.

The ScriptRuntime is designed for you to define job artifacts and configurations supported by OCI Data Science Jobs natively. It can be used with any script types that is supported by the OCI Data Science Jobs, including shell scripts and python scripts.

The source code can be a single script, files in a folder or a zip/tar file.

Working Directory

The working directory is the parent directory where the job artifacts are decompressed, for example /home/datascience/decompressed_artifact/. When the source code is a compressed file/archive (zip/tar) or a folder, you need to also specify the entrypoint using with_entrypoint(). The path of the entrypoint should be a path relative to the working directory. Note that this directory cannot be changed when using ScriptRuntime. See Python Runtime Working Directory for more details.

Run a Container

The ContainerRuntime class allows you to run a container image using OCI data science jobs.

OCI Container Registry

To use the ContainerRuntime, you need to first push the image to OCI container registry.

Note that you cannot build a docker image inside an OCI Data Science Notebook Session.

For more details, see:

Here is an example to create and run a container job:

Python
YAML

from ads.jobs import Job, DataScienceJob, ContainerRuntime

job = (
    Job(name="My Job")
    .with_infrastructure(
        DataScienceJob()
        # Configure logging for getting the job run outputs.
        .with_log_group_id("<log_group_ocid>")
        # Log resource will be auto-generated if log ID is not specified.
        .with_log_id("<log_ocid>")
        # If you are in an OCI data science notebook session,
        # the following configurations are not required.
        # Configurations from the notebook session will be used as defaults.
        .with_compartment_id("<compartment_ocid>")
        .with_project_id("<project_ocid>")
        .with_subnet_id("<subnet_ocid>")
        .with_shape_name("VM.Standard.E3.Flex")
        # Shape config details are applicable only for the flexible shapes.
        .with_shape_config_details(memory_in_gbs=16, ocpus=1)
        # Minimum/Default block storage size is 50 (GB).
        .with_block_storage_size(50)
    )
    .with_runtime(
        ContainerRuntime()
        .with_image("<region>.ocir.io/<your_tenancy>/<your_image>")
        .with_environment_variable(GREETINGS="Welcome to OCI Data Science")
        .with_entrypoint(["/bin/sh", "-c"])
        .with_cmd("sleep 5 && echo $GREETINGS")
    )
)

kind: job
spec:
  name: "My Job"
  infrastructure:
    kind: infrastructure
    type: dataScienceJob
    spec:
      blockStorageSize: 50
      compartmentId: <compartment_ocid>
      jobInfrastructureType: STANDALONE
      logGroupId: <log_group_ocid>
      logId: <log_ocid>
      projectId: <project_ocid>
      shapeConfigDetails:
        memoryInGBs: 16
        ocpus: 1
      shapeName: VM.Standard.E3.Flex
      subnetId: <subnet_ocid>
  runtime:
    kind: runtime
    type: container
    spec:
      cmd: sleep 5 && echo $GREETINGS
      entrypoint:
      - /bin/sh
      - -c
      env:
      - name: GREETINGS
        value: Welcome to OCI Data Science
      image: <region>.ocir.io/<your_tenancy>/<your_image>

# Create the job on OCI Data Science
job.create()
# Start a job run
run = job.run()
# Stream the job run outputs
run.watch()

To configure ContainerRuntime, you must specify the container image. Similar to other runtime, you can add environment variables. You can optionally specify the entrypoint and cmd for running the container.

See also:

Run Code from Git Repo

The GitPythonRuntime allows you to run source code from a Git repository as a job.

PyTorch Example

The following example shows how to run a PyTorch Neural Network Example to train third order polynomial predicting y=sin(x).

Python
YAML

from ads.jobs import Job, DataScienceJob, GitPythonRuntime

job = (
    Job(name="My Job")
    .with_infrastructure(
        DataScienceJob()
        # Configure logging for getting the job run outputs.
        .with_log_group_id("<log_group_ocid>")
        # Log resource will be auto-generated if log ID is not specified.
        .with_log_id("<log_ocid>")
        # If you are in an OCI data science notebook session,
        # the following configurations are not required.
        # Configurations from the notebook session will be used as defaults.
        .with_compartment_id("<compartment_ocid>")
        .with_project_id("<project_ocid>")
        .with_subnet_id("<subnet_ocid>")
        .with_shape_name("VM.Standard.E3.Flex")
        # Shape config details are applicable only for the flexible shapes.
        .with_shape_config_details(memory_in_gbs=16, ocpus=1)
        # Minimum/Default block storage size is 50 (GB).
        .with_block_storage_size(50)
    )
    .with_runtime(
        GitPythonRuntime()
        .with_environment_variable(GREETINGS="Welcome to OCI Data Science")
        # Specify the service conda environment by slug name.
        .with_service_conda("pytorch19_p37_gpu_v1")
        # Specify the git repository
        # Optionally, you can specify the branch or commit
        .with_source("https://github.com/pytorch/tutorials.git")
        # Entrypoint is a relative path from the root of the git repo.
        .with_entrypoint("beginner_source/examples_nn/polynomial_nn.py")
        # Copy files in "beginner_source/examples_nn" to object storage after job finishes.
        .with_output(
          output_dir="beginner_source/examples_nn",
          output_uri="oci://bucket_name@namespace/path/to/dir"
        )
    )
)

kind: job
spec:
  name: "My Job"
  infrastructure:
    kind: infrastructure
    type: dataScienceJob
    spec:
      blockStorageSize: 50
      compartmentId: <compartment_ocid>
      jobInfrastructureType: STANDALONE
      logGroupId: <log_group_ocid>
      logId: <log_ocid>
      projectId: <project_ocid>
      shapeConfigDetails:
        memoryInGBs: 16
        ocpus: 1
      shapeName: VM.Standard.E3.Flex
      subnetId: <subnet_ocid>
  runtime:
    kind: runtime
    type: gitPython
    spec:
      conda:
        slug: pytorch19_p37_gpu_v1
        type: service
      entrypoint: beginner_source/examples_nn/polynomial_nn.py
      env:
      - name: GREETINGS
        value: Welcome to OCI Data Science
      outputDir: beginner_source/examples_nn
      outputUri: oci://bucket_name@namespace/path/to/dir
      url: https://github.com/pytorch/tutorials.git

# Create the job on OCI Data Science
job.create()
# Start a job run
run = job.run()
# Stream the job run outputs
run.watch()

Git Repository

To configure the GitPythonRuntime, you must specify the source code url and the entrypoint. The default branch from the Git repository is used unless you specify a different branch or commit in the with_source() method.

For a public repository, we recommend the “http://” or “https://” URL. Authentication may be required for the SSH URL even if the repository is public.

To use a private repository, you must first save an SSH key to OCI Vault as a secret, and provide the secret_ocid when calling with_source(). For more information about creating and using secrets, see Managing Secret with Vault. For repository on GitHub, you could setup the GitHub Deploy Key as secret.

Git Version for Private Repository

Git version of 2.3+ is required to use a private repository.

Entrypoint

The entrypoint specifies how the source code is invoked. The with_entrypoint() supports the following arguments:

path: Required. The relative path of the script/module from the root of the Git repository.
func: Optional. The function in the script specified by path to call. If you don’t specify it, then the script specified by path is run as a Python script in a subprocess.

The arguments for the entrypoint can be specified through with_argument(). For running a script, the arguments are passed in as command line arguments. See Runtime Command Line Arguments for more details. For running a function, the arguments are passed into the function call.

The following example shows how you can define a runtime using Python function from a git repository as an entrypoint. Here my_function is a function in the my_source/my_module.py module.

Python
YAML

runtime = (
  GitPythonRuntime()
  .with_environment_variable(GREETINGS="Welcome to OCI Data Science")
  # Specify the service conda environment by slug name.
  .with_service_conda("pytorch19_p37_gpu_v1")
  # Specify the git repository
  .with_source("https://example.com/your_repository.git")
  # Entrypoint is a relative path from the root of the git repo.
  .with_entrypoint("my_source/my_module.py", func="my_function")
  .with_argument("arg1", "arg2", key1="val1", key2="val2")
)

kind: runtime
type: gitPython
spec:
  args:
  - arg1
  - arg2
  - --key1
  - val1
  - --key2
  - val2
  conda:
    slug: pytorch19_p37_gpu_v1
    type: service
  entryFunction: my_function
  entrypoint: my_source/my_module.py
  env:
  - name: GREETINGS
    value: Welcome to OCI Data Science
  url: https://example.com/your_repository.git

The function will be called as my_function("arg1", "arg2", key1="val1", key2="val2").

The arguments can be strings, list of strings or dict containing only strings.

GitPythonRuntime also support Jupyter notebook as entrypoint. Arguments are not used when the entrypoint is a notebook.

Working Directory

By default, the working directory is the root of the git repository. This can be configured by can be configured by with_working_dir() using a relative path from the root of the Git repository.

Note that the entrypoint should always specified as a relative path from the root of the Git repository, regardless of the working directory. The python paths and output directory should be specified relative to the working directory.

Python Paths

The working directory is the root of the git repository. The working directory is added to the Python paths automatically. You can call with_python_path() to add additional python paths as needed. The paths should be relative paths from the working directory.

Outputs

The with_output() method allows you to specify the output path output_dir in the job run and a remote URI (output_uri). Files in the output_dir are copied to the remote output URI after the job run finishes successfully. Note that the output_dir should be a path relative to the working directory.

OCI object storage location can be specified in the format of oci://bucket_name@namespace/path/to/dir. Please make sure you configure the I AM policy to allow the job run dynamic group to use object storage.

Metadata

The GitPythonRuntime updates metadata as free-form tags of the job run after the job run finishes. The following tags are added automatically:

commit: The Git commit ID.
method: The entry function or method.
module: The entry script or module.
outputs: The prefix of the output files in Object Storage.
repo: The URL of the Git repository.

The new values overwrite any existing tags. If you want to skip the metadata update, set skip_metadata_update to True when initializing the runtime:

runtime = GitPythonRuntime(skip_metadata_update=True)

Working with the CLI

Prerequisite

Install ADS CLI

Running a Pre Defined Job

ads opctl run -j <job ocid>

Delete Job or Job Run

ads opctl delete <job-id or run-id>

Cancel Job Run

ads opctl cancel <run-id>

Cancel Distributed Training Job

Stop a running cluster using cancel subcommand.

Option 1: Using Job OCID and Work Dir

ads opctl cancel -j <job ocid> --work-dir <Object storage working directory specified when the cluster was created>

Option 2: Using cluster info file

Cluster info file is a yaml file with output generated from ads opctl run -f

ads opctl cancel -j <job ocid> --work-dir <Object storage working directory specified when the cluster was created>

This command requires an api key or resource principal setup. The logs are streamed from the logging service. If your job is not attached to logging service, this option will show only the lifecycle state.

Monitoring With CLI

watch

You can tail the logs generated by OCI Data Science Job Runs or OCI DataFlow Application Runs using the watch subcommand.

ads opctl watch <job run ocid or dataflow application run ocid>

This command requires an api key or resource principal setup. The logs are streamed from the logging service. If your job is not attached to logging service, this option will show only the lifecycle state.

Data Science Pipelines

New in version 2.8.0.

Overview

Oracle Cloud Infrastructure (OCI) Data Science Machine Learning (ML) Pipelines lets you define and run an end-to-end machine learning orchestration covering all the steps of machine learning lifecycle that can be executed in a repeatable, continuous ML pipeline.

The machine learning lifecycle is composed of several steps: data acquisition and extraction, data preparation, featurization, model training including algorithm selection and hyper-parameter tuning, model evaluation, deployment, and finally monitoring the deployed model and possible retraining.

Pipeline Step

A pipeline step is a task in a pipeline. A pipeline step can be either a Data Science Job step or a Custom Script step.

Pipeline

A pipeline is a workflow of tasks, called steps. Steps can run in sequence or in parallel resulting in a Directed Acyclic Graph (DAG) of the steps.

In a machine learning context, ML Pipelines provide a workflow of data import, data transformation, model training, and model evaluation.

Pipeline Run

Pipeline Run is the execution instance of a pipeline. Each pipeline run includes its step runs.

Quick Start

ADS Python SDK

The following sections provide sample code to define, create, and run a pipeline, including a visualization to track the pipeline run status.

The following example shows creating and runnning a pipeline with multiple steps. The steps step_1 and step_2 run in parallel and step_3 runs after step_1 and step_2 are complete.

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: An example pipeline
  projectId: {project_id}
  dag:
  - (step_1, step_2) >> step_3
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: step_1
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
  - kind: customScript
    spec:
      description: A step running a notebook
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: step_2
      runtime:
        kind: runtime
        spec:
          conda:
            slug: tensorflow26_p37_cpu_v2
            type: service
          notebookEncoding: utf-8
          notebookPathURI: https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb
        type: notebook
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: step_3
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()      # create the pipeline
pipeline.show()       # visualize the pipeline

pipeline_run = pipeline.run()   # run the pipeline

pipeline_run.show()     # watch the pipeline run status

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, ScriptRuntime, NotebookRuntime
import os

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

script_runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

notebook_runtime = (
    NotebookRuntime()
    .with_notebook(
        path="https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb",
        encoding='utf-8'
    )
    .with_service_conda("tensorflow26_p37_cpu_v2")
)

pipeline_step_1 = (
    PipelineStep("step_1")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(script_runtime)
)

pipeline_step_2 = (
    PipelineStep("step_2")
    .with_description("A step running a notebook")
    .with_infrastructure(infrastructure)
    .with_runtime(notebook_runtime)
)

pipeline_step_3 = (
    PipelineStep("step_3")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(script_runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
      Pipeline("An example pipeline")
      .with_compartment_id(compartment_id)
      .with_project_id(project_id)
      .with_step_details([pipeline_step_1, pipeline_step_2, pipeline_step_3])
      .with_dag(["(step_1, step_2) >> step_3"])
  )

pipeline.create()      # create the pipeline
pipeline.show()       # visualize the pipeline

pipeline_run = pipeline.run()   # run the pipeline

pipeline_run.show()     # watch the pipeline run status

ADS CLI

Prerequisites

Create / Run

To build a brand new Data Science Pipeline and run it, provide the path to the pipeline YAML file with the --file option

ads opctl run --file <path_to_pipeline_yaml>

Alternatively, to run an existing pipeline, provide the pipeline OCID with the --ocid option

ads opctl run --ocid <pipeline_ocid>

Monitor

To monitor a pipeline run, provide the pipeline run OCID and provide the log type with the -l option

Below is an example to stream the custom log

ads opctl watch <pipeline_run_ocid> -l custom_log

Tip

The allowed values for -l option are custom_log, service_log, or None.

Cancel

To cancel a pipeline run, provide the pipeline run OCID

ads opctl cancel <pipeline_run_ocid>

Data Science Pipeline Runs can only be canceled when they are in the ACCEPTED or IN_PROGRESS state.

Delete

To delete a pipeline run, provide the pipeline run OCID

ads opctl delete <pipeline_run_ocid>

Data Science Pipeline Runs can only be deleted when they are in the SUCCEEDED, FAILED, or CANCELED state.

To delete a pipeline, provide the pipeline OCID

ads opctl delete <pipeline_ocid>

Data Science Pipelines can only be deleted when their associated pipeline runs are all deleted.

ADS Magic Commands

Tip

Get more information about the pipeline extension by running %pipeline -h

Installation

Install the pipeline extension by running the following command

%load_ext ads.pipeline.extension

Create / Run

To build a brand new Data Science Pipeline and run it, provide the path to the pipeline YAML file with the --file option

%pipeline run --file <path_to_pipeline_yaml>

Alternatively, to run an existing pipeline, provide the pipeline OCID with the --ocid option

%pipeline run --ocid <pipeline_ocid>

Visualize

To visualize a pipeline in a graph, use the pipeline OCID

%pipeline show <pipeline_ocid>

Watch status

To watch the status of pipeline run, use the pipeline run OCID

Tip

Get more information about watching pipeline status by running %pipeline status -h

Below is an example of watching the status of pipeline run in graph mode until it finishes

%pipeline status <pipeline_run_ocid> -w

Below is an example of watching the status of pipeline run in text mode

%pipeline status <pipeline_run_ocid> -x

Monitor logs

To monitor a pipeline run, use the pipeline run OCID.

Tip

Get more information about monitoring pipeline logs by running %pipeline log -h

Below is an example of streaming the custom_log

%pipeline log <pipeline_run_ocid> -l custom_log

Below is an example of viewing the last 10 consolidated logs with tail

%pipeline log <pipeline_run_ocid> -t -n 10

Cancel

To cancel a pipeline run, use the pipeline run OCID

%pipeline cancel <pipeline_run_ocid>

Data Science Pipeline Runs can only be canceled when they are in the ACCEPTED or IN_PROGRESS state.

Delete

Tip

Get more information about deleting pipelines and pipeline runs by running %pipeline delete -h

To delete a pipeline run, use the pipeline run OCID

%pipeline delete <pipeline_run_ocid>

Data Science Pipeline Runs can only be deleted when they are in the SUCCEEDED, FAILED, or CANCELED state.

To delete a pipeline, use the pipeline OCID

%pipeline delete <pipeline_ocid>

Data Science Pipelines can only be deleted when their associated pipeline runs are all deleted.

Pipeline

A pipeline is a workflow of tasks, called steps. Steps can run in sequence or in parallel, creating a Directed Acyclic Graph (DAG) of the steps.

Define

In an ADS pipeline module, you can either use the Python API or YAML to define a pipeline. In addition to the configuration of the pipeline, you provide the details of Pipeline Step and the DAG of the pipeline steps (which defines the dependencies between the steps).

DAG

DAG is used to define the dependencies between the steps.

>> denotes the tasks running in sequence, A >> B means that A is followed by B.
() denotes the tasks running in parallel.
If DAG not provided, all the steps will run in parallel.

In the example below, step_name_1 and step_name_2 will run in parallel, and step_name_3 will start after both step_name_1 and step_name_2 are complete.

(step_name_1, step_name_2) >> step_name_3

Logs

Both the log OCID and corresponding log group OCID can be specified in the Pipeline instance. If you specify only the log group OCID and no log OCID, a new Log resource is automatically created within the log group to store the logs, see ADS Logging.

There are two types of logs for pipeline runs, service log and custom log. When defining a pipeline:

to enable custom log, specify log_id and log_group_id.
to enable service log, specify log_group_id and set enable_service_log to True.
to enable both types of logs, specify log_id and log_group_id, and set enable_service_log to True.

With the specified DAG and pre-created pipeline steps, you can define a pipeline and give it a name. A Pipeline instance will be created.

YAML
Python

from ads.pipeline import Pipeline

yaml_string = """
kind: pipeline
spec:
  compartmentId: ocid1.compartment..<unique_id>
  dag:
  - pipeline_step_name_1 >> pipeline_step_name_2
  description: <pipeline_description>
  displayName: <pipeline_display_name>
  logGroupId: ocid1.loggroup.oc1..<unique_id>
  logId: ocid1.log..<unique_id>
  enableServiceLog: True
  maximumRuntimeInMinutes: 20
  projectId: ocid1.datascienceproject..<unique_id>
  stepDetails:
  - kind: dataScienceJob
    spec:
      description: <pipeline_step_description>
      jobId: ocid1.datasciencejob..<unique_id>
      name: pipeline_step_name_1
  - kind: customScript
    spec:
      description: <pipeline_step_description>
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: pipeline_step_name_2
      runtime:
        kind: runtime
        spec:
          conda:
            slug: <slug>
            type: service
          scriptPathURI: oci://<bucket_name>@<namespace>/<prefix>/<script.py>
        type: script
type: pipeline
"""

pipeline = Pipeline.from_yaml(yaml_string)

from ads.pipeline import PipelineStep, Pipeline

pipeline_step_one = (
  PipelineStep("<pipeline_step_name_1>")
  .with_description("<pipeline_step_description>")
  .with_job_id("<job_id>")
)

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

runtime = (
    ScriptRuntime()
    .with_source("oci://<bucket_name>@<namespace>/<prefix>/<script.py>")
    .with_service_conda("<slug>")
)

pipeline_step_two = (
    PipelineStep("<pipeline_step_name_2>")
    .with_description("<step_description>")
    .with_infrastructure(infrastructure)
    .with_runtime(runtime)
)

pipeline = (
    Pipeline("<pipeline_name>")
    .with_compartment_id("<compartment_id>")
    .with_project_id("<project_id>")
    .with_log_group_id("<log_group_id>")
    .with_log_id("<log_id>")
    .with_enable_service_log(True)         # to stream service log in pipeline runs
    .with_step_details([pipeline_step_one, pipeline_step_two])
    .with_dag(["pipeline_step_name_1 >> pipeline_step_name_2"])
)

Create

You can call the create() method of the Pipeline instance to create a pipeline.

# Create a pipeline
pipeline.create()

Run

You can call the run() method of the Pipeline instance to launch a new Pipeline Run. It returns a PipelineRun instance.

The run() method gives you the option to override the configurations in a pipeline run. It takes the following optional parameters:

display_name: str, optional. Defaults to None. The display name of the run.

project_id: str, optional. Defaults to None. The project id to override the one defined previously.

compartment_id: str, optional. Defaults to None. The compartment id to override the one defined previously.

configuration_override_details: dict, optional. Defaults to None. The configuration details dictionary to override the one defined previously. The configuration_override_details contains the following keys:

type: str, only DEFAULT is allowed; environment_variables: dict, the environment variables; command_line_arguments: str, the command line arguments; maximum_runtime_in_minutes: int, the maximum runtime allowed in minutes. - log_configuration_override_details: dict, optional. Defaults to None. The log configuration details dictionary to override the one defined previously. - step_override_details: list[PipelineStepOverrideDetails], optional. Defaults to None. The step details list to override the one defined previously. - free_form_tags: dict(str, str), optional. Defaults to None. The free from tags dictionary to override the one defined previously. - defined_tags: dict(str, dict(str, object)), optional. Defaults to None. The defined tags dictionary to override the one defined previously. - system_tags: dict(str, dict(str, object)), optional. Defaults to None. The system tags dictionary to override the one defined previously.

# Run a pipeline, a pipeline run will be created and started
pipeline_run = pipeline.run()

Load

Use the from_ocid() method from the Pipeline class to load an existing pipeline with its OCID provided. It returns a Pipeline instance.

from ads.pipeline import Pipeline

pipeline = Pipeline.from_ocid("ocid1.datasciencepipeline..<unique_id>")

Visualize

Use the show() method on the Pipeline instance to visualize the pipeline in a graph.

The show() method takes the following optional parameter:

rankdir: (str, optional). Defaults to TB. The allowed values are TB or LR. This parameter is applicable only for graph mode and it renders the direction of the graph as either top to bottom (TB) or left to right (LR).

pipeline.show()

Below is an example of the output.

Delete

Use the delete() method on the Pipeline instance to delete a pipeline. It takes the following optional parameters:

delete_related_pipeline_runs: (bool, optional). Specify whether to delete related PipelineRuns or not. Defaults to True.

delete_related_job_runs: (bool, optional). Specify whether to delete related JobRuns or not. Defaults to True.

max_wait_seconds: (int, optional). The maximum time to wait, in seconds. Defaults to 1800.

A pipeline can only be deleted when its associated pipeline runs are all deleted, or alternatively, set the parameter delete_related_pipeline_runs to delete all associated runs in the same operation. Delete fails if a PipelineRun is in progress.

pipeline.delete()

Pipeline Step

Pipeline step is a task in a pipeline. A pipeline step can be one of two types:

Data Science Job: the OCID of an existing Data Science Job must be provided.
Custom Script: the artifact of the Python script and the execution configuration must be specified.

This section shows how you can use the ADS Pipeline APIs to create pipeline steps.

Data Science Job Step

Create a Data Science Job step with the OCID of an existing Job.

YAML
Python

kind: pipeline
spec:
  ...
  stepDetails:
  ...
  - kind: dataScienceJob
    spec:
      description: <pipeline_step_description>
      jobId: ocid1.datasciencejob..<unique_id>
      name: <pipeline_step_name>
  ...
type: pipeline

from ads.pipeline import PipelineStep

pipeline_step = (
    PipelineStep("<pipeline_step_name>")
    .with_description("<pipeline_step_description>")
    .with_job_id("<job_id>")
)

Custom Script Step

To create a Custom Script step, infrastructure and runtime must be specified.

The Custom Script step infrastructure is defined by a CustomScriptStep instance. When constructing a Custom Scrip step infrastructure, you specify the Compute shape, Block Storage size in the CustomScriptStep instance. For example:

YAML
Python

kind: pipeline
spec:
  ...
  stepDetails:
  - kind: customScript
    spec:
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Python_Script_Step
   ...
type: pipeline

from ads.pipeline import CustomScriptStep

infrastructure = (
  CustomScriptStep()
  .with_block_storage_size(200)
  .with_shape_name("VM.Standard3.Flex")
  .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

A Custom Script step can have different types of runtime depending on the source code you run:

PythonRuntime for Python code stored locally, OCI object storage, or other remote location supported by fsspec. See Run a Python Workload.
GitPythonRuntime for Python code from a Git repository. See Run Code from Git Repo.
NotebookRuntime for a single Jupyter notebook stored locally, OCI object storage, or other remote location supported by fsspec. See Run a Notebook.
ScriptRuntime for bash or shell scripts stored locally, OCI object storage, or other remote location supported by fsspec. See Run a Script.

All of these runtime options allow you to configure a Data Science Conda Environment for running your code.

To define a Custom Script step with GitPythonRuntime you can use:

YAML
Python

kind: runtime
spec:
  conda:
        slug: pytorch19_p37_gpu_v1
        type: service
  entrypoint: beginner_source/examples_nn/polynomial_nn.py
  env:
  - name: GREETINGS
        value: Welcome to OCI Data Science
  outputDir: ~/Code/tutorials/beginner_source/examples_nn
  outputUri: oci://<bucket_name>@<namespace>/<prefix>
  url: https://github.com/pytorch/tutorials.git
type: gitPython

from ads.pipeline import GitPythonRuntime

runtime = (
    GitPythonRuntime()
    .with_environment_variable(GREETINGS="Welcome to OCI Data Science")
    .with_service_conda("pytorch19_p37_gpu_v1")
    .with_source("https://github.com/pytorch/tutorials.git")
    .with_entrypoint("beginner_source/examples_nn/polynomial_nn.py")
    .with_output(
    output_dir="~/Code/tutorials/beginner_source/examples_nn",
    output_uri="oci://<bucket_name>@<namespace>/<prefix>"
  )
)

To define a Custom Script step with NotebookRuntime you can use:

YAML
Python

kind: runtime
spec:
  conda:
        slug: tensorflow26_p37_cpu_v2
        type: service
  env:
  - name: GREETINGS
        value: Welcome to OCI Data Science
  notebookEncoding: utf-8
  notebookPathURI: https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb
  outputURI: oci://bucket_name@namespace/path/to/dir
type: notebook

from ads.pipeline import NotebookRuntime

runtime = (
    NotebookRuntime()
      .with_notebook(
          path="https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb",
          encoding='utf-8'
      )
      .with_service_conda("tensorflow26_p37_cpu_v2")
      .with_environment_variable(GREETINGS="Welcome to OCI Data Science")
      .with_output("oci://bucket_name@namespace/path/to/dir")
)

To define a Custom Script step with PythonRuntime you can use:

YAML
Python

kind: runtime
spec:
  conda:
        slug: pytorch110_p38_cpu_v1
        type: service
  entrypoint: zip_or_dir/my_package/entry.py
  outputDir: output
  outputUri: oci://bucket_name@namespace/path/to/dir
  pythonPath:
  - my_python_packages
  scriptPathURI: local/path/to/zip_or_dir
  workingDir: zip_or_dir
type: python

from ads.pipeline import PythonRuntime

runtime = (
    PythonRuntime()
    .with_service_conda("pytorch110_p38_cpu_v1")
    # The job artifact directory is named "zip_or_dir"
    .with_source("local/path/to/zip_or_dir", entrypoint="zip_or_dir/my_package/entry.py")
    # Change the working directory to be inside the job artifact directory
    # Working directory a relative path from the parent of the job artifact directory
    # Working directory is also added to Python paths
    .with_working_dir("zip_or_dir")
    # Add an additional Python path
    # The "my_python_packages" folder is under "zip_or_dir" (working directory)
    .with_python_path("my_python_packages")
    # Files in "output" directory will be copied to OCI object storage once the job finishes
    # Here we assume "output" is a folder under "zip_or_dir" (working directory)
    .with_output("output", "oci://bucket_name@namespace/path/to/dir")
)

To define a Custom Script step with ScriptRuntime you can use:

YAML
Python

kind: runtime
spec:
  conda:
        slug: tensorflow26_p37_cpu_v2
        type: service
  scriptPathURI: oci://<bucket_name>@<namespace>/<prefix>/<script.py>
type: script

from ads.pipeline import ScriptRuntime

runtime = (
    ScriptRuntime()
    .with_source("oci://<bucket_name>@<namespace>/<prefix>/<script.py>")
    .with_service_conda("tensorflow26_p37_cpu_v2")
)

With Infrastructure and runtime provided, create a pipeline step of the Custom Script type.

YAML
Python

kind: pipeline
spec:
  ...
  stepDetails:
  ...
  - kind: customScript
    spec:
      description: <pipeline_step_description>
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: <pipeline_step_name>
      runtime:
        kind: runtime
        spec:
          conda:
            slug: <slug>
            type: service
          scriptPathURI: oci://<bucket_name>@<namespace>/<prefix>/<script.py>
        type: script
  ...
type: pipeline

from ads.pipeline import PipelineStep

pipeline_step = (
    PipelineStep("<pipeline_step_name>")
    .with_description("<pipeline_step_description>")
    .with_infrastructure(infrastructure)
    .with_runtime(runtime)
)

Pipeline Run

Pipeline Run is the execution instance of a pipeline. Each pipeline run includes its step runs. A pipeline run can be configured to override some of the pipeline’s defaults before starting the execution.

With a PipelineRun instance, you can watch the status of the run and stream logs for the pipeline run and the step runs.

Watch status

Use the show() method of the PipelineRun instance to watch the status of pipeline run.

The show() method takes the following optional parameter:

mode: (str, optional). Defaults to graph. The allowed values are text or graph. This parameter renders the current status of pipeline run as either text or graph.

wait: (bool, optional). Defaults to False and it only renders the current status of each step run in graph. If set to True, it renders the current status of each step run until the entire pipeline is complete.

rankdir: (str, optional). Defaults to TB. The allowed values are TB or LR. This parameter is applicable only for graph mode and it renders the direction of the graph as either top to bottom (TB) or left to right (LR).

To watch the live update of each step run status in text until the entire pipeline is complete

pipeline_run.show(mode="text", wait=True)

Step                Status
------------------  ---------
PipelineStepOne:    Succeeded
PipelineStepTwo:    Succeeded
PipelineStepThree:  Succeeded
PipelineStepFour:   In Progress
PipelineStepFive:   Accepted

To watch the live update of each step run status in graph mode

pipeline_run.show(wait=True)

Below is an example of the output.

Monitor Logs

Use the watch() method on the PipelineRun instance to stream the service, custom, or consolidated log of the pipeline run. The watch() method takes the following optional parameters:

steps: (list, optional). Defaults to None and streams the log of the pipeline run. If a list of the step names is provided, the method streams the log of the specified pipeline step runs.
log_type: (str, optional). Defaults to None. The allowed values are custom_log, service_log, or None. If None is provided, the method streams both service and custom logs.
interval: (float, optional). Defaults value is 3. Time interval in seconds between each request to update the logs.

Stream the consolidated log of the pipeline run.

pipeline_run.watch()

[S] - service log, [C] - custom log
[S] - 2022-10-31 08:54:47 - Step PipelineStepOne is starting.
[S] - 2022-10-31 08:55:36 - Step PipelineStepOne is ACCEPTED, lifecycle details: Infrastructure provisioning.
[S] - 2022-10-31 08:56:39 - Step PipelineStepOne is ACCEPTED, lifecycle details: Step run bootstrap starting.
[C] - 2022-10-31 08:57:18 - This is a custom log for PipelineStepOne.
[S] - 2022-10-31 08:57:39 - Step PipelineStepOne is IN_PROGRESS, lifecycle details: Step run artifact execution in progress.
[S] - 2022-10-31 08:58:39 - Step PipelineStepOne is SUCCEEDED.
[S] - 2022-10-31 08:59:54 - Step PipelineStepThree is starting.
[S] - 2022-10-31 09:00:15 - Step PipelineStepTwo is starting.
[S] - 2022-10-31 09:00:44 - Step PipelineStepThree is ACCEPTED, lifecycle details: Infrastructure provisioning.
[S] - 2022-10-31 09:00:53 - Step PipelineStepTwo is ACCEPTED, lifecycle details: Infrastructure provisioning.
[S] - 2022-10-31 09:02:46 - Step PipelineStepThree is ACCEPTED, lifecycle details: Step run bootstrap starting.
[S] - 2022-10-31 09:02:54 - Step PipelineStepTwo is ACCEPTED, lifecycle details: Step run bootstrap starting.
[C] - 2022-10-31 09:03:13 - This is a custom log for PipelineStepThree.
[C] - 2022-10-31 09:03:13 - This is a custom log for PipelineStepTwo.
...

Stream the service log of the pipeline run.

pipeline_run.watch(log_type="service_log")

[S] - service log
[S] - 2022-10-31 08:54:47 - Step PipelineStepOne is starting.
[S] - 2022-10-31 08:55:36 - Step PipelineStepOne is ACCEPTED, lifecycle details: Infrastructure provisioning.
[S] - 2022-10-31 08:56:39 - Step PipelineStepOne is ACCEPTED, lifecycle details: Step run bootstrap starting.
[S] - 2022-10-31 08:57:39 - Step PipelineStepOne is IN_PROGRESS, lifecycle details: Step run artifact execution in progress.
[S] - 2022-10-31 08:58:39 - Step PipelineStepOne is SUCCEEDED.
[S] - 2022-10-31 08:59:54 - Step PipelineStepThree is starting.
[S] - 2022-10-31 09:00:15 - Step PipelineStepTwo is starting.
...

Stream the custom log of the specified steps.

pipeline_run.watch(steps=['<step_name1>', '<step_name2>'], log_type="custom_log")

Load

Use the from_ocid() method from the PipelineRun class to load an existing pipeline run with its OCID provided. The method returns a PipelineRun instance.

from ads.pipeline import PipelineRun

pipeline_run = PipelineRun.from_ocid("ocid1.datasciencepipelinerun..<unique_id>")

Cancel

Use the cancel() method on the PipelineRun instance to cancel a pipeline run.

Pipeline Runs can only be canceled when they are in the ACCEPTED or IN_PROGRESS state.

pipeline_run.cancel()

Delete

Use the delete() method on the PipelineRun instance to delete a pipeline run. It takes the following optional parameter:

delete_related_job_runs: (bool, optional). Specify whether to delete related JobRuns or not. Defaults to True.

max_wait_seconds: (int, optional). The maximum time to wait in seconds. Defaults to 1800.

Pipeline runs can only be deleted when they are already in a SUCCEEDED, FAILED, or CANCELED state.

pipeline_run.delete()

Examples

Create a pipeline

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: A single step pipeline
  projectId: {project_id}
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Python_Script_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, ScriptRuntime
import os

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

pipeline_step = (
    PipelineStep("Python_Script_Step")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
      Pipeline("A single step pipeline")
      .with_compartment_id(compartment_id)
      .with_project_id(project_id)
      .with_step_details([pipeline_step])
  )

pipeline.create()

Run a job as a step

YAML
Python

rom ads.jobs import Job, DataScienceJob, ScriptRuntime
from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    DataScienceJob()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

job = (
    Job()
    .with_infrastructure(infrastructure)
    .with_runtime(runtime)
)
job.create() # create a job

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: A single step pipeline
  projectId: {project_id}
  stepDetails:
  - kind: dataScienceJob
    spec:
      description: A step running a job
      jobId: {job_id}
      name: Job_Step
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id, job_id=job.id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()

pipeline_run = pipeline.run()

from ads.jobs import Job, DataScienceJob, ScriptRuntime
from ads.pipeline import Pipeline, PipelineStep
import os

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    DataScienceJob()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

job = (
    Job()
    .with_infrastructure(infrastructure)
    .with_runtime(runtime)
)
job.create() # create a job

pipeline_step = (
    PipelineStep("Job_Step")
    .with_description("A step running a job")
    .with_job_id(job.id)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
      Pipeline("A single step pipeline")
      .with_compartment_id(compartment_id)
      .with_project_id(project_id)
      .with_step_details([pipeline_step])
  )

pipeline.create()

pipeline_run = pipeline.run()

Run a python script as a step

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: A single step pipeline
  projectId: {project_id}
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Python_Script_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()

pipeline_run = pipeline.run()

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, ScriptRuntime
import os

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

pipeline_step = (
    PipelineStep("Python_Script_Step")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
      Pipeline("A single step pipeline")
      .with_compartment_id(compartment_id)
      .with_project_id(project_id)
      .with_step_details([pipeline_step])
  )

pipeline.create()

pipeline_run = pipeline.run()

Run a notebook as a step

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: A single step pipeline
  projectId: {project_id}
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a notebook
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Notebook_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: tensorflow26_p37_cpu_v2
            type: service
          env:
          - name: GREETINGS
            value: Welcome to OCI Data Science
          notebookEncoding: utf-8
          notebookPathURI: https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb
          outputURI: oci://<bucket_name>@<namespace>/<prefix>
        type: notebook
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()

pipeline_run = pipeline.run()

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, NotebookRuntime
import os

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

runtime = (
    NotebookRuntime()
    .with_notebook(
        path="https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb",
        encoding='utf-8'
    )
    .with_service_conda("tensorflow26_p37_cpu_v2")
    .with_environment_variable(GREETINGS="Welcome to OCI Data Science")
    .with_output("oci://<bucket_name>@<namespace>/<prefix>")
)

pipeline_step = (
    PipelineStep("Notebook_Step")
    .with_description("A step running a notebook")
    .with_infrastructure(infrastructure)
    .with_runtime(runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
    Pipeline("A single step pipeline")
    .with_compartment_id(compartment_id)
    .with_project_id(project_id)
    .with_step_details([pipeline_step])
)

pipeline.create()

pipeline_run = pipeline.run()

Run two steps with the same infrastructure

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: A single step pipeline
  projectId: {project_id}
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Python_Script_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
  - kind: customScript
    spec:
      description: A step running a notebook
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Notebook_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: tensorflow26_p37_cpu_v2
            type: service
          env:
          - name: GREETINGS
            value: Welcome to OCI Data Science
          notebookEncoding: utf-8
          notebookPathURI: https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb
          outputURI: oci://<bucket_name>@<namespace>/<prefix>
        type: notebook
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()

pipeline_run = pipeline.run()

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, ScriptRuntime, NotebookRuntime
import os

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

step_one_runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

pipeline_step_one = (
    PipelineStep("Python_Script_Step")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(step_one_runtime)
)

step_two_runtime = (
    NotebookRuntime()
    .with_notebook(
        path="https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb",
        encoding='utf-8'
    )
    .with_service_conda("tensorflow26_p37_cpu_v2")
    .with_environment_variable(GREETINGS="Welcome to OCI Data Science")
    .with_output("oci://<bucket_name>@<namespace>/<prefix>")
)

pipeline_step_two = (
    PipelineStep("Notebook_Step")
    .with_description("A step running a notebook")
    .with_infrastructure(infrastructure)
    .with_runtime(step_two_runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
      Pipeline("A single step pipeline")
      .with_compartment_id(compartment_id)
      .with_project_id(project_id)
      .with_step_details([pipeline_step_one, pipeline_step_two])
  )

pipeline.create()

pipeline_run = pipeline.run()

Run two steps in parallel

In the example below, when DAG is not specified, the steps in the pipeline run in parallel.

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: A single step pipeline
  projectId: {project_id}
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Python_Script_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
  - kind: customScript
    spec:
      description: A step running a notebook
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Notebook_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: tensorflow26_p37_cpu_v2
            type: service
          env:
          - name: GREETINGS
            value: Welcome to OCI Data Science
          notebookEncoding: utf-8
          notebookPathURI: https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb
          outputURI: oci://<bucket_name>@<namespace>/<prefix>
        type: notebook
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()

pipeline_run = pipeline.run()

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, ScriptRuntime, NotebookRuntime
import os

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

step_one_runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

pipeline_step_one = (
    PipelineStep("Python_Script_Step")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(step_one_runtime)
)

step_two_runtime = (
    NotebookRuntime()
    .with_notebook(
        path="https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb",
        encoding='utf-8'
    )
    .with_service_conda("tensorflow26_p37_cpu_v2")
    .with_environment_variable(GREETINGS="Welcome to OCI Data Science")
    .with_output("oci://<bucket_name>@<namespace>/<prefix>")
)

pipeline_step_two = (
    PipelineStep("Notebook_Step")
    .with_description("A step running a notebook")
    .with_infrastructure(infrastructure)
    .with_runtime(step_two_runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
      Pipeline("A single step pipeline")
      .with_compartment_id(compartment_id)
      .with_project_id(project_id)
      .with_step_details([pipeline_step_one, pipeline_step_two])
  )

pipeline.create()

pipeline_run = pipeline.run()

Run two steps sequentially

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: A single step pipeline
  projectId: {project_id}
  dag:
  - Python_Script_Step >> Notebook_Step
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Python_Script_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
  - kind: customScript
    spec:
      description: A step running a notebook
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Notebook_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: tensorflow26_p37_cpu_v2
            type: service
          env:
          - name: GREETINGS
            value: Welcome to OCI Data Science
          notebookEncoding: utf-8
          notebookPathURI: https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb
          outputURI: oci://<bucket_name>@<namespace>/<prefix>
        type: notebook
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()

pipeline_run = pipeline.run()

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, ScriptRuntime, NotebookRuntime
import os

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

step_one_runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

pipeline_step_one = (
    PipelineStep("Python_Script_Step")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(step_one_runtime)
)

step_two_runtime = (
    NotebookRuntime()
    .with_notebook(
        path="https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb",
        encoding='utf-8'
    )
    .with_service_conda("tensorflow26_p37_cpu_v2")
    .with_environment_variable(GREETINGS="Welcome to OCI Data Science")
    .with_output("oci://<bucket_name>@<namespace>/<prefix>")
)

pipeline_step_two = (
    PipelineStep("Notebook_Step")
    .with_description("A step running a notebook")
    .with_infrastructure(infrastructure)
    .with_runtime(step_two_runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
      Pipeline("A single step pipeline")
      .with_compartment_id(compartment_id)
      .with_project_id(project_id)
      .with_step_details([pipeline_step_one, pipeline_step_two])
      .with_dag(["Python_Script_Step >> Notebook_Step"])
  )

pipeline.create()

pipeline_run = pipeline.run()

Run multiple steps with dependencies specified in DAG

In this example, step_1 and step_2 run in parallel and step_3 runs after step_1 and step_2 are complete.

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: An example pipeline
  projectId: {project_id}
  dag:
  - (step_1, step_2) >> step_3
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: step_1
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
  - kind: customScript
    spec:
      description: A step running a notebook
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: step_2
      runtime:
        kind: runtime
        spec:
          conda:
            slug: tensorflow26_p37_cpu_v2
            type: service
          notebookEncoding: utf-8
          notebookPathURI: https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb
        type: notebook
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: step_3
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()      # create the pipeline
pipeline.show()        # visualize the pipeline

pipeline_run = pipeline.run()   # run the pipeline

pipeline_run.show(wait=True)    # watch the pipeline run status

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, ScriptRuntime, NotebookRuntime
import os

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

script_runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

notebook_runtime = (
    NotebookRuntime()
    .with_notebook(
        path="https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb",
        encoding='utf-8'
    )
    .with_service_conda("tensorflow26_p37_cpu_v2")
)

pipeline_step_1 = (
    PipelineStep("step_1")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(script_runtime)
)

pipeline_step_2 = (
    PipelineStep("step_2")
    .with_description("A step running a notebook")
    .with_infrastructure(infrastructure)
    .with_runtime(notebook_runtime)
)

pipeline_step_3 = (
    PipelineStep("step_3")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(script_runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
      Pipeline("An example pipeline")
      .with_compartment_id(compartment_id)
      .with_project_id(project_id)
      .with_step_details([pipeline_step_1, pipeline_step_2, pipeline_step_3])
      .with_dag(["(step_1, step_2) >> step_3"])
  )

pipeline.create()      # create the pipeline
pipeline.show()        # visualize the pipeline

pipeline_run = pipeline.run()   # run the pipeline

pipeline_run.show(wait=True)    # watch the pipeline run status

Set environment variables in a step

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: A single step pipeline
  projectId: {project_id}
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a notebook
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Notebook_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: tensorflow26_p37_cpu_v2
            type: service
          env:
          - name: GREETINGS
            value: Welcome to OCI Data Science
          notebookEncoding: utf-8
          notebookPathURI: https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb
          outputURI: oci://<bucket_name>@<namespace>/<prefix>
        type: notebook
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()

pipeline_run = pipeline.run()

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, NotebookRuntime
import os

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

runtime = (
    NotebookRuntime()
    .with_notebook(
        path="https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb",
        encoding='utf-8'
    )
    .with_service_conda("tensorflow26_p37_cpu_v2")
    .with_environment_variable(GREETINGS="Welcome to OCI Data Science")
    .with_output("oci://<bucket_name>@<namespace>/<prefix>")
)

pipeline_step = (
    PipelineStep("Notebook_Step")
    .with_description("A step running a notebook")
    .with_infrastructure(infrastructure)
    .with_runtime(runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
    Pipeline("A single step pipeline")
    .with_compartment_id(compartment_id)
    .with_project_id(project_id)
    .with_step_details([pipeline_step])
)

pipeline.create()

pipeline_run = pipeline.run()

Watch status update on a pipeline run

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: A single step pipeline
  projectId: {project_id}
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Python_Script_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()
pipeline_run = pipeline.run()

# pipeline_run.show(mode="text")   # watch pipeline run status in text
pipeline_run.show(wait=True)       # watch pipeline run status in graph

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, ScriptRuntime
import os

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

pipeline_step = (
    PipelineStep("Python_Script_Step")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
      Pipeline("A single step pipeline")
      .with_compartment_id(compartment_id)
      .with_project_id(project_id)
      .with_step_details([pipeline_step])
  )

pipeline.create()
pipeline_run = pipeline.run()

# pipeline_run.show(mode="text")   # watch pipeline run status in text
pipeline_run.show(wait=True)       # watch pipeline run status in graph

Monitor logs of a pipeline run

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

yaml_string = """
kind: pipeline
spec:
  compartmentId: {compartment_id}
  displayName: A single step pipeline
  projectId: {project_id}
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Python_Script_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()
pipeline_run = pipeline.run()

# pipeline_run.watch()  # stream the consolidated log of the pipeline run
pipeline_run.watch(log_type="service_log")      # stream service log of the pipeline run
pipeline_run.watch("Python_Script_Step", log_type="custom_log") # stream custom log of the step run

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, ScriptRuntime
import os

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

pipeline_step = (
    PipelineStep("Python_Script_Step")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
      Pipeline("A single step pipeline")
      .with_compartment_id(compartment_id)
      .with_project_id(project_id)
      .with_step_details([pipeline_step])
  )

pipeline.create()
pipeline_run = pipeline.run()

# pipeline_run.watch()  # stream the consolidated log of the pipeline run
pipeline_run.watch(log_type="service_log")      # stream service log of the pipeline run
pipeline_run.watch("Python_Script_Step", log_type="custom_log") # stream custom log of the step run

Override configurations when creating a pipeline run

YAML
Python

from ads.pipeline import Pipeline
import os

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

yaml_string = """
kind: pipeline
spec:
  commandLineArguments: argument --key value
  environmentVariables:
    env: value
  compartmentId: {compartment_id}
  displayName: A single step pipeline
  projectId: {project_id}
  stepDetails:
  - kind: customScript
    spec:
      description: A step running a python script
      infrastructure:
        kind: infrastructure
        spec:
          blockStorageSize: 200
          shapeConfigDetails:
            memoryInGBs: 32
            ocpus: 4
          shapeName: VM.Standard3.Flex
      name: Python_Script_Step
      runtime:
        kind: runtime
        spec:
          conda:
            slug: generalml_p37_cpu_v1
            type: service
          scriptPathURI: script.py
        type: script
type: pipeline
""".format(compartment_id=compartment_id, project_id=project_id)

pipeline = Pipeline.from_yaml(yaml_string)

pipeline.create()

# Override configurations when creating a pipeline run
display_override_name = "RunOverrideName"
configuration_override_details = {
    "maximum_runtime_in_minutes": 30,
    "type": "DEFAULT",
    "environment_variables": {"a": "b"},
    "command_line_arguments": "ARGUMENT --KEY VALUE",
}

step_override_details = [
{
    "step_name": "Python_Script_Step",
    "step_configuration_details": {
        "maximum_runtime_in_minutes": 200,
        "environment_variables": {"1": "2"},
        "command_line_arguments": "argument --key value",
    },
}
]
pipeline_run = pipeline.run(
    display_name=display_override_name,
    configuration_override_details=configuration_override_details,
    step_override_details=step_override_details,
)

from ads.pipeline import Pipeline, PipelineStep, CustomScriptStep, ScriptRuntime
import os

with open("script.py", "w") as f:
    f.write("print('Hello World!')")

infrastructure = (
    CustomScriptStep()
    .with_block_storage_size(200)
    .with_shape_name("VM.Standard3.Flex")
    .with_shape_config_details(ocpus=4, memory_in_gbs=32)
)

runtime = (
    ScriptRuntime()
    .with_source("script.py")
    .with_service_conda("generalml_p37_cpu_v1")
)

pipeline_step = (
    PipelineStep("Python_Script_Step")
    .with_description("A step running a python script")
    .with_infrastructure(infrastructure)
    .with_runtime(runtime)
)

compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']
project_id = os.environ["PROJECT_OCID"]

pipeline = (
      Pipeline("A single step pipeline")
      .with_compartment_id(compartment_id)
      .with_project_id(project_id)
      .with_step_details([pipeline_step])
      .with_argument("argument", key="value")
      .with_environment_variable(env="value")
  )

pipeline.create()

# Override configurations when creating a pipeline run
display_override_name = "RunOverrideName"
configuration_override_details = {
    "maximum_runtime_in_minutes": 30,
    "type": "DEFAULT",
    "environment_variables": {"a": "b"},
    "command_line_arguments": "ARGUMENT --KEY VALUE",
}

step_override_details = [
{
    "step_name": "Python_Script_Step",
    "step_configuration_details": {
        "maximum_runtime_in_minutes": 200,
        "environment_variables": {"1": "2"},
        "command_line_arguments": "argument --key value",
    },
}
]
pipeline_run = pipeline.run(
    display_name=display_override_name,
    configuration_override_details=configuration_override_details,
    step_override_details=step_override_details,
)

Store Credentials

Services such as OCI Database and Streaming require users to provide credentials. These credentials must be safely accessed at runtime. OCI Vault provides a mechanism for safe storage and access of secrets. SecretKeeper uses Vault as a backend to store and retrieve the credentials. The data structure of the credentials varies from service to service. There is a SecretKeeper specific to each data structure.

These classes are provided:

ADBSecretKeeper: Stores credentials for the Oracle Autonomous Database, with or without the wallet file.
AuthTokenSecretKeeper: Stores an Auth Token or Access Token string. This could be an Auth Token to use to connect to Streaming, Github, or other systems that used Auth Tokens or Access Token strings.
BDSSecretKeeper: Stores credentials for Oracle Big Data Service with or without Keytab and kerb5 configuration files.
MySQLDBSecretKeeper: Stores credentials for the MySQL database. This class will work with many databases that authenticate with a username and password only.
OracleDBSecretKeeper: Stores credentials for the Oracle Database.

Quick Start

Auth Tokens

Save Credentials

import ads
from ads.secrets.auth_token import AuthTokenSecretKeeper

ads.set_auth('resource_principal') # If using resource principal authentication

ocid_vault = "ocid1.vault..<unique_ID>"
ocid_master_key = "ocid1.key..<unique_ID>"
ocid_mycompartment = "ocid1.compartment..<unique_ID>"

authtoken2 = AuthTokenSecretKeeper(
                vault_id=ocid_vault,
                key_id=ocid_master_key,
                compartment_id=ocid_mycompartment,
                auth_token="<your_auth_token>"
               ).save(
                    "my_xyz_auth_token2",
                    "This is my key for git repo xyz",
                    freeform_tags={"gitrepo":"xyz"}
                )
print(authtoken2.secret_id)

'ocid1.vaultsecret..<unique_ID>'

Load Credentials

import ads
from ads.secrets.auth_token import AuthTokenSecretKeeper

ads.set_auth('resource_principal') # If using resource principal authentication

with AuthTokenSecretKeeper.load_secret(source="ocid1.vaultsecret..<unique_ID>",
                               ) as authtoken:
    import os
    print(f"Credentials inside `authtoken` object:  {authtoken}")

Credentials inside `authtoken` object: {'auth_token': '<your_auth_token>'}

Autonomous Database

Save Credentials

import ads
ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.adb import ADBSecretKeeper

connection_parameters={
    "user_name":"admin",
    "password":"<your_password>",
    "service_name":"service_high",
    "wallet_location":"/home/datascience/Wallet_--------.zip"
}

ocid_vault = "ocid1.vault..<unique_ID>"
ocid_master_key = "ocid1.key..<unique_ID>"
ocid_mycompartment = "ocid1.compartment..<unique_ID>"

adw_keeper = ADBSecretKeeper(vault_id=ocid_vault,
                            key_id=ocid_master_key,
                            compartment_id=ocid_mycompartment,
                            **connection_parameters)

# Store the credentials without storing the wallet file
adw_keeper.save("adw_employee_att2",
                    "My DB credentials",
                    freeform_tags={"schema":"emp"},
                    save_wallet=True
                )
print(adw_keeper.secret_id)

'ocid1.vaultsecret..<unique_ID>'

Load Credentials

import ads
ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.adb import ADBSecretKeeper

with ADBSecretKeeper.load_secret("ocid1.vaultsecret..<unique_ID>") as adw_creds2:
    import pandas as pd
    df2 = pd.DataFrame.ads.read_sql("select JOBFUNCTION, ATTRITION from ATTRITION_DATA", connection_parameters=adw_creds2)
    print(df2.head(2))

	JOBFUNCTION	ATTRITION
0	Product Management	No
1	Software Developer	No

Big Data Service

Save Credentials

import ads
import fsspec
import os

from ads.secrets.big_data_service import BDSSecretKeeper
from ads.bds.auth import has_kerberos_ticket, refresh_ticket, krbcontext

ads.set_auth('resource_principal')

principal = "<your_principal>"
hdfs_host = "<your_hdfs_host>"
hive_host = "<your_hive_host>"
hdfs_port = <your_hdfs_port>
hive_port = <your_hive_port>
vault_id = "ocid1.vault..<unique_ID>"
key_id = "ocid1.key..<unique_ID>"

secret = BDSSecretKeeper(
            vault_id=vault_id,
            key_id=key_id,
            principal=principal,
            hdfs_host=hdfs_host,
            hive_host=hive_host,
            hdfs_port=hdfs_port,
            hive_port=hive_port,
            keytab_path=keytab_path,
            kerb5_path=kerb5_path
           )

saved_secret = secret.save(name="your_bds_config_secret_name",
                        description="your bds credentials",
                        freeform_tags={"schema":"emp"},
                        defined_tags={},
                        save_files=True)

Load Credentials

from ads.secrets.big_data_service import BDSSecretKeeper
from pyhive import hive

with BDSSecretKeeper.load_secret(saved_secret.secret_id, keytab_dir="~/path/to/save/keytab_file/") as cred:
    with krbcontext(principal=cred["principal"], keytab_path=cred['keytab_path']):
        hive_cursor = hive.connect(host=cred["hive_host"],
                                   port=cred["hive_port"],
                                   auth='KERBEROS',
                                   kerberos_service_name="hive").cursor()

MySQL

Save Credentials

import ads
from ads.secrets.mysqldb import MySQLDBSecretKeeper

vault_id = "ocid1.vault..<unique_ID>"
key_id = "ocid1.key..<unique_ID>"

ads.set_auth("resource_principal") # If using resource principal for authentication
connection_parameters={
    "user_name":"<your user name>",
    "password":"<your password>",
    "host":"<db host>",
    "port":"<db port>",
   "database":"<database>",
}

mysqldb_keeper = MySQLDBSecretKeeper(vault_id=vault_id,
                                key_id=key_id,
                                **connection_parameters)

mysqldb_keeper.save("mysqldb_employee", "My DB credentials", freeform_tags={"schema":"emp"})
print(mysqldb_keeper.secret_id) # Prints the secret_id of the stored credentials

'ocid1.vaultsecret..<unique_ID>'

Load Credentials

import ads
from ads.secrets.mysqldb import MySQLDBSecretKeeper
ads.set_auth('resource_principal') # If using resource principal authentication

with MySQLDBSecretKeeper.load_secret(source=secret_id) as mysqldb_creds:
    import pandas as pd
    df2 = pd.DataFrame.ads.read_sql("select JOBFUNCTION, ATTRITION from ATTRITION_DATA", connection_parameters=mysqldb_creds)
    print(df2.head(2))

	JOBFUNCTION	ATTRITION
0	Product Management	No
1	Software Developer	No

Oracle Database

Save Credentials

import ads
from ads.secrets.oracledb import OracleDBSecretKeeper

vault_id = "ocid1.vault..<unique_ID>"
key_id = "ocid1.key..<unique_ID>"

ads.set_auth("resource_principal") # If using resource principal for authentication
connection_parameters={
     "user_name":"<your user name>",
     "password":"<your password>",
     "service_name":"service_name",
     "host":"<db host>",
     "port":"<db port>",
}

oracledb_keeper = OracleDBSecretKeeper(vault_id=vault_id,
                                key_id=key_id,
                                **connection_parameters)

oracledb_keeper.save("oracledb_employee", "My DB credentials", freeform_tags={"schema":"emp"})
print(oracledb_keeper.secret_id) # Prints the secret_id of the stored credentials

'ocid1.vaultsecret..<unique_ID>'

Load Credentials

import ads
ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.oracledb import OracleDBSecretKeeper

with OracleDBSecretKeeper.load_secret(source=secret_id) as oracledb_creds:
    import pandas as pd
    df2 = pd.DataFrame.ads.read_sql("select JOBFUNCTION, ATTRITION from ATTRITION_DATA", connection_parameters=oracledb_creds)
    print(df2.head(2))

	JOBFUNCTION	ATTRITION
0	Product Management	No
1	Software Developer	No

Auth Token

The AuthTokenSecretKeeper helps you to save the Auth Token or Access Token string to the OCI Vault service.

See API Documentation for more details

Save Credentials

`AuthTokenSecretKeeper`

The AuthTokenSecretKeeper constructor takes the following parameters:

auth_token (str): Provide the Auth Token or Access Token string to be stored
vault_id (str): ocid of the vault
key_id (str): ocid of the master key used for encrypting the secret
compartment_id (str, optional): Default is None. ocid of the compartment where the vault is located. This will be defaulted to the compartment of the Notebook session, if used within a OCI Data Science notebook session.

Save

The AuthTokenSecretKeeper.save API serializes and stores the credentials to Vault. It takes following parameters -

name (str): Name of the secret when saved in the vault.
description (str): Description of the secret when saved in the vault.
freeform_tags (dict, optional): Freeform tags to use when saving the secret in the OCI Console.
defined_tags (dict, optional.): Save the tags under predefined tags in the OCI Console.

The secret has following information:

auth_token

Examples

Save Auth Token

import ads
from ads.secrets.auth_token import AuthTokenSecretKeeper

ads.set_auth('resource_principal') # If using resource principal authentication

ocid_vault = "ocid1.vault...<unique_ID>"
ocid_master_key = "ocid1.key..<unique_ID>"
ocid_mycompartment = "ocid1.compartment..<unique_ID>"

authtoken2 = AuthTokenSecretKeeper(
                vault_id=ocid_vault,
                key_id=ocid_master_key,
                compartment_id=ocid_mycompartment,
                auth_token="<your_auth_token>"
               ).save(
                    "my_xyz_auth_token2",
                    "This is my key for git repo xyz",
                    freeform_tags={"gitrepo":"xyz"}
                )
print(authtoken2.secret_id)

You can save the vault details in a file for later reference or using it within your code using export_vault_details API. The API currently let us export the information as a yaml file or a json file.

authtoken2.export_vault_details("my_db_vault_info.json", format="json")

Save as a `yaml` File

authtoken2.export_vault_details("my_db_vault_info.yaml", format="yaml")

Load Credentials

Load

The AuthTokenSecretKeeper.load_secret API deserializes and loads the credentials from Vault. You could use this API in one of the following ways:

Using a `with` Statement

with AuthTokenSecretKeeper.load_secret('ocid1.vaultsecret..<unique_ID>') as authtoken:
    print(authtoken['user_name']

This approach is preferred as the secrets are only available within the code block and it reduces the risk that the variable will be leaked.

Without using a `with` Statement

authtoken = AuthTokenSecretKeeper.load_secret('ocid1.vaultsecret..<unique_ID>')
authtokendict = authtoken.to_dict()
print(authtokendict['user_name'])

The .load_secret() takes the following parameters:

auth: Provide overriding authorization information if the authorization information is different from the ads.set_auth setting.
export_env: Default is False. If set to True, the credentials are exported as environment variable when used with
export_prefix: The default name for environment variable is user_name, password, service_name, and wallet_location. You can add a prefix to avoid name collision
format: Optional. If source is a file, then this value must be json or yaml depending on the file format.
source: Either the file that was exported from export_vault_details or the OCID of the secret
the with operator.

Examples

Using a `with` Statement

import ads
from ads.secrets.auth_token import AuthTokenSecretKeeper

ads.set_auth('resource_principal') # If using resource principal authentication

with AuthTokenSecretKeeper.load_secret(source="ocid1.vaultsecret..<unique_ID",
                               ) as authtoken:
    import os
    print(f"Credentials inside `authtoken` object:  {authtoken}")

Credentials inside `authtoken` object: {'auth_token': '<your_auth_token>'}

Export to Environment Variables Using a `with` Statement

To expose credentials through environment variable, set export_env=True. The following keys are exported -

Secret attribute	Environment Variable Name
auth_token	auth_token

import ads
from ads.secrets.auth_token import AuthTokenSecretKeeper
import os

ads.set_auth('resource_principal') # If using resource principal authentication

with AuthTokenSecretKeeper.load_secret(
            source="ocid1.vaultsecret..<unique_ID>",
            export_env=True
        ):
    print(os.environ.get("auth_token")) # Prints the auth token

print(os.environ.get("auth_token")) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

You can avoid name collisions by setting the prefix string using export_prefix along with export_env=True. For example, if you set the prefix to kafka, the exported keys are:

Secret attribute	Environment Variable Name
auth_token	kafka.auth_token

import ads
from ads.secrets.auth_token import AuthTokenSecretKeeper
import os

ads.set_auth('resource_principal') # If using resource principal authentication

with AuthTokenSecretKeeper.load_secret(
            source="ocid1.vaultsecret..<unique_ID>",
            export_env=True,
            export_prefix="kafka"
        ):
    print(os.environ.get("kafka.auth_token")) # Prints the auth token

print(os.environ.get("kafka.auth_token")) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

Autonomous Database

To connect to Autonomous Database you need the following:

user name
password
service name
wallet file

The ADBSecretKeeper class saves the ADB credentials to the OCI Vault service.

See API Documentation for more details

Save Credentials

`ADBSecretKeeper`

The ADBSecretKeeper constructor has the following parameters:

compartment_id (str): OCID of the compartment where the vault is located. This defaults to the compartment of the notebook session when used in a Data Science notebook session.
key_id (str): OCID of the master key used for encrypting the secret.
password (str): The password of the database.
service_name (str): Set the service name of the database.
user_name (str): The user name to be stored.
vault_id (str): OCID of the vault.
wallet_location (str): Path to the wallet ZIP file.

Save

The ADBSecretKeeper.save API serializes and stores the credentials to Vault using the following parameters:

defined_tags (dict, optional): Default None. Save the tags under predefined tags in the OCI Console.
description (str): Description of the secret when saved in Vault.
freeform_tags (dict, optional): Default None. Free form tags to use for saving the secret in the OCI Console.
name (str): Name of the secret when saved in Vault.
save_wallet (bool, optional): Default False. If set to True, then the wallet file is serialized.

When stored without the wallet information, the secret content has following information:

password
service_name
user_name

To store wallet file content, set save_wallet to True. The wallet content is stored by extracting all the files from the wallet ZIP file, and then each file is stored in the vault as a secret. The list of OCIDs corresponding to each file along with username, password, and service name is stored in a separate secret. The secret corresponding to each file content has following information:

filename
content of the file

A meta secret is created to save the username, password, service name, and the secret ids of the files within the wallet file. It has following attributes:

user_name
password
wallet_file_name
wallet_secret_ids

The wallet file is reconstructed when ADBSecretKeeper.load_secret is called using the OCID of the meta secret.

Examples

Without the Wallet File

import ads
ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.adb import ADBSecretKeeper

connection_parameters={
    "user_name":"admin",
    "password":"<your_password>",
    "service_name":"service_high",
    "wallet_location":"/home/datascience/Wallet_--------.zip"
}

ocid_vault = "ocid1.vault..<unique_ID>"
ocid_master_key = "ocid1.key..<unique_ID>"
ocid_mycompartment = "ocid1.compartment..<unique_ID>"

adw_keeper = ADBSecretKeeper(vault_id=ocid_vault,
                            key_id=ocid_master_key,
                            compartment_id=ocid_mycompartment,
                            **connection_parameters)

# Store the credentials without storing the wallet file
adw_keeper.save("adw_employee_att2", "My DB credentials", freeform_tags={"schema":"emp"})
print(adw_keeper.secret_id)

'ocid1.vaultsecret..<unique_ID>'

With the Wallet File

import ads
ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.adb import ADBSecretKeeper

connection_parameters={
    "user_name":"admin",
    "password":"<your_password>",
    "service_name":"service_high",
    "wallet_location":"/home/datascience/Wallet_--------.zip"
}

ocid_vault = "ocid1.vault..<unique_ID>"
ocid_master_key = "ocid1.key..<unique_ID>"
ocid_mycompartment = "ocid1.compartment..<unique_ID>"

adw_keeper = ADBSecretKeeper(vault_id=ocid_vault,
                            key_id=ocid_master_key,
                            compartment_id=ocid_mycompartment,
                            **connection_parameters)

# Set `save_wallet`=True to save wallet file

adw_keeper.save("adw_employee_att2",
    "My DB credentials",
    freeform_tags={"schema":"emp"},
    save_wallet=True
)

print(adw_keeper.secret_id)

'ocid1.vaultsecret..<unique_ID>'

You can save the vault details in a file for later reference or using it within your code using export_vault_details API calls. The API currently enables you to export the information as a YAML file or a JSON file.

adw_keeper.export_vault_details("my_db_vault_info.json", format="json")

To save as a YAML file:

adw_keeper.export_vault_details("my_db_vault_info.yaml", format="yaml")

Load Credentials

Load

The ADBSecretKeeper.load_secret API deserializes and loads the credentials from Vault. You could use this API in one of the following ways:

Using a `with` Statement

with ADBSecretKeeper.load_secret('ocid1.vaultsecret..<unique_ID>') as adwsecret:
    print(adwsecret['user_name'])

This approach is preferred as the secrets are only available within the code block and it reduces the risk that the variable will be leaked.

Without using a `with` Statement

adwsecretobj = ADBSecretKeeper.load_secret('ocid1.vaultsecret..<unique_ID>')
adwsecret = adwsecretobj.to_dict()
print(adwsecret['user_name'])

The .load_secret() method has the following parameters:

auth: Provide overriding authorization information if the authorization information is different from the ads.set_auth setting.
export_env: Default is False. If set to True, the credentials are exported as environment variable when used with the with operator.
export_prefix: The default name for environment variable is user_name, password, service_name, and wallet_location. You can add a prefix to avoid name collision
format: Optional. If source is a file, then this value must be json or yaml depending on the file format.
source: Either the file that was exported from export_vault_details or the OCID of the secret
wallet_dir: Optional. Directory path where the wallet zip file will be saved after the contents are retrieved from Vault. If wallet content is not available in the provided secret OCID, this attribute is ignored.
wallet_location: Optional. Path to the local wallet zip file. If vault secret does not have wallet file content, set this variable so that it will be available in the exported credential. If provided, this path takes precedence over the wallet file information in the secret.

If the wallet file was saved in the vault, then the ZIP file of the same name is created by the .load_secret() method. By default the ZIP file is created in the working directory. To update the location, you can set the directory path with wallet_dir.

Examples

Using a `with` Statement

import ads
ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.adb import ADBSecretKeeper

with ADBSecretKeeper.load_secret(
            "ocid1.vaultsecret..<unique_ID>"
        ) as adw_creds2:
    print (adw_creds2["user_name"]) # Prints the user name

print (adw_creds2["user_name"]) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

Export to Environment Variables Using a `with` Statement

To expose credentials as an environment variable, set export_env=True. The following keys are exported:

Secret attribute	Environment Variable Name
user_name	user_name
password	password
service_name	service_name
wallet_location	wallet_location

import os
import ads

ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.adb import ADBSecretKeeper

with ADBSecretKeeper.load_secret(
            "ocid1.vaultsecret..<unique_ID>",
            export_env=True
        ):
    print(os.environ.get("user_name")) # Prints the user name

print(os.environ.get("user_name")) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

You can avoid name collisions by setting a prefix string using export_prefix along with export_env=True. For example, if you set the prefix to myprocess, then the exported keys are:

Secret attribute	Environment Variable Name
user_name	myprocess.user_name
password	myprocess.password
service_name	myprocess.service_name
wallet_location	myprocess.wallet_location

import os
import ads

ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.adb import ADBSecretKeeper

with ADBSecretKeeper.load_secret(
            "ocid1.vaultsecret..<unique_ID>",
            export_env=True,
            export_prefix="myprocess"
        ):
    print(os.environ.get("myprocess.user_name")) # Prints the user name

print(os.environ.get("myprocess.user_name")) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

Wallet File Location

You can set wallet file location when wallet file is not part of the stored vault secret. To specify a local wallet ZIP file, set the path to the ZIP file with wallet_location:

import ads
ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.adb import ADBSecretKeeper

with ADBSecretKeeper.load_secret(
            "ocid1.vaultsecret..<unique_ID>",
            wallet_location="path/to/my/local/wallet.zip"
        ) as adw_creds2:
    print (adw_creds2["wallet_location"]) # Prints `path/to/my/local/wallet.zip`

print (adw_creds2["wallet_location"]) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

Big Data Service

New in version 2.5.10..

To connect to Oracle Big Data Service (BDS) you need the following:

hdfs host: HDFS hostname which will be used to connect to the HDFS file system.
hdfs port: HDFS port which will be used to connect to the HDFS file system.
hive host: Hive hostname which will be used to connect to the Hive Server.
hive port: Hive port which will be used to connect to the Hive Server.
kerb5 config file: krb5.conf file which can be copied from /etc/krb5.conf from the master node of the BDS cluster. It will be used to generate the kerberos ticket.
keytab file: The principal’s keytab file which can be downloaded from the master node of the BDS cluster. It will be used to generate the kerberos ticket.
principal: The unique identity to which Kerberos can assign tickets. It will be used to generate the kerberos ticket.

The BDSSecretKeeper class saves the BDS credentials to the OCI Vault service.

See API Documentation for more details

Save Credentials

`BDSSecretKeeper`

You can also save the connection parameters as well as the files needed to configure the kerberos authentication into vault. This will allow you to use repetitively in different notebook sessions, machines, and Jobs.

The BDSSecretKeeper constructor requires the following parameters:

compartment_id (str): OCID of the compartment where the vault is located. This defaults to the compartment of the notebook session when used in a Data Science notebook session.
hdfs_host (str): The HDFS hostname from the bds cluster.
hdfs_port (str): The HDFS port from the bds cluster.
hive_host (str): The Hive hostname from the bds cluster.
hive_port (str): The Hive port from the bds cluster.
kerb5_path (str): The krb5.conf file path.
key_id: str (OCID of the master key used for encrypting the secret.
keytab_path (str): The path to the keytab file.
principal (str): The unique identity to which Kerberos can assign tickets.
vault_id: (str): The OCID of the vault.

Save

The BDSSecretKeeper.save API serializes and stores the credentials to Vault using the following parameters:

defined_tags (dict, optional): Default None. Save the tags under predefined tags in the OCI Console.
description (str) – Description of the secret when saved in Vault.
freeform_tags (dict, optional): Default None. Free form tags to use for saving the secret in the OCI Console.
name (str): Name of the secret when saved in Vault.
save_files (bool, optional): Default True. If set to True, then the keytab and kerb5 config files are serialized and saved.

Examples

With the Keytab and kerb5 Config Files

import ads
import fsspec
import os

from ads.secrets.big_data_service import BDSSecretKeeper
from ads.bds.auth import has_kerberos_ticket, refresh_ticket, krbcontext

ads.set_auth('resource_principal')

principal = "<your_principal>"
hdfs_host = "<your_hdfs_host>"
hive_host = "<your_hive_host>"
hdfs_port = <your_hdfs_port>
hive_port = <your_hive_port>
vault_id = "ocid1.vault..<unique_ID>"
key_id = "ocid1.key..<unique_ID>"

secret = BDSSecretKeeper(
            vault_id=vault_id,
            key_id=key_id,
            principal=principal,
            hdfs_host=hdfs_host,
            hive_host=hive_host,
            hdfs_port=hdfs_port,
            hive_port=hive_port,
            keytab_path=keytab_path,
            kerb5_path=kerb5_path
           )

saved_secret = secret.save(name="your_bds_config_secret_name",
                        description="your bds credentials",
                        freeform_tags={"schema":"emp"},
                        defined_tags={},
                        save_files=True)

Without the Keytab and kerb5 Config Files

import ads
import fsspec
import os

from ads.secrets.big_data_service import BDSSecretKeeper
from ads.bds.auth import has_kerberos_ticket, refresh_ticket, krbcontext

ads.set_auth('resource_principal')

principal = "<your_principal>"
hdfs_host = "<your_hdfs_host>"
hive_host = "<your_hive_host>"
hdfs_port = <your_hdfs_port>
hive_port = <your_hive_port>
vault_id = "ocid1.vault..<unique_ID>"
key_id = "ocid1.key..<unique_ID>"

bds_keeper = BDSSecretKeeper(
            vault_id=vault_id,
            key_id=key_id,
            principal=principal,
            hdfs_host=hdfs_host,
            hive_host=hive_host,
            hdfs_port=hdfs_port,
            hive_port=hive_port,
            keytab_path=keytab_path,
            kerb5_path=kerb5_path
           )

saved_secret = bds_keeper.save(name="your_bds_config_secret_name",
                        description="your bds credentials",
                        freeform_tags={"schema":"emp"},
                        defined_tags={},
                        save_files=False)

print(saved_secret.secret_id)

'ocid1.vaultsecret..<unique_ID>'

Load Credentials

Load

The BDSSecretKeeper.load_secret API deserializes and loads the credentials from Vault. You could use this API in one of the following ways:

Using a `with` Statement

with BDSSecretKeeper.load_secret('ocid1.vaultsecret..<unique_ID>') as bdssecret:
    print(bdssecret['hdfs_host'])

This approach is preferred as the secrets are only available within the code block and it reduces the risk that the variable will be leaked.

Without Using a `with` Statement

bdssecretobj = BDSSecretKeeper.load_secret('ocid1.vaultsecret..<unique_ID>')
bdssecret = bdssecretobj.to_dict()
print(bdssecret['hdfs_host'])

The .load_secret() method takes following parameters:

auth: Provide overriding authorization information if the authorization information is different from the ads.set_auth setting.
export_env: Default is False. If set to True, the credentials are exported as environment variable when used with the with operator.
export_prefix: The default name for environment variable is user_name, password, service_name, and wallet_location. You can add a prefix to avoid name collision
format: Optional. If source is a file, then this value must be json or yaml depending on the file format.
keytab_dir: Optional. Directory path where the keytab ZIP file is saved after the contents are retrieved from the vault. If the keytab content is not available in the specified secret OCID, then this attribute is ignored.
source: Either the file that was exported from export_vault_details or the OCID of the secret

If the keytab and kerb5 configuration files were saved in the vault, then a keytab and kerb5 configuration file of the same name is created by .load_secret(). By default, the keytab file is created in the keytab_path specified in the secret. To update the location, set the directory path with key_dir. However, the kerb5 configuration file is always saved in the ~/.bds_config/krb5.conf path.

Note that keytab and kerb5 configuration files are saved only when the content is saved into the vault.

After you load and save the configuration parameters files, you can call the krbcontext context manager to create a Kerberos ticket.

Examples

Using a With Statement

To specify a local keytab file, set the path to the ZIP file with wallet_location:

from pyhive import hive

with BDSSecretKeeper.load_secret(saved_secret.secret_id, keytab_dir="~/path/to/save/keytab_file/") as cred:
    with krbcontext(principal=cred["principal"], keytab_path=cred['keytab_path']):
        hive_cursor = hive.connect(host=cred["hive_host"],
                                   port=cred["hive_port"],
                                   auth='KERBEROS',
                                   kerberos_service_name="hive").cursor()

Now you can query the data from Hive:

hive_cursor.execute("""
    select *
    from your_db.your_table
    limit 10
""")

import pandas as pd
pd.DataFrame(hive_cursor.fetchall(), columns=[col[0] for col in hive_cursor.description])

Without Using a With Statement

Load From Secret OCID

bdssecretobj = BDSSecretKeeper.load_secret(saved_secret.secret_id)
bdssecret = bdssecretobj.to_dict()
print(bdssecret)

Load From a JSON File

bdssecretobj = BDSSecretKeeper.load_secret(source="./my_bds_vault_info.json", format="json")
bdssecretobj.to_dict()

Load From a YAML File

bdssecretobj = BDSSecretKeeper.load_secret(source="./my_bds_vault_info.yaml", format="yaml")
bdssecretobj.to_dict()

MySQL

To connect to a MySQL Database, you need the following:

hostname
password
port, the default is 3306
user name

The MySQLDBSecretKeeper class saves the MySQL database credentials to the OCI Vault service.

See API Documentation for more details

Save Credentials

`MySQLDBSecretKeeper`

The MySQLDBSecretKeeper constructor has the following parameters:

compartment_id (str): OCID of the compartment where the vault is located. Defaults to the compartment of the notebook session when used in a Data Science notebook session.
database (str, optional)): The database name if available.
host (str): The hostname of the database.
key_id (str): OCID of the master key used for encrypting the secret.
password (str): The password of the database.
port (str, optional). Default 3306): Port number of the database service.
user_name (str): The user name to be stored.
vault_id (str): OCID of the vault.

Save

The MySQLDBSecretKeeper.save API serializes and stores the credentials to the vault using the following parameters:

defined_tags (dict, optional): Save the tags under predefined tags in the OCI Console.
description (str): Description of the secret when saved in the vault.
freeform_tags (dict, optional): Freeform tags to be used for saving the secret in the OCI Console.
name (str): Name of the secret when saved in the vault.

The secret has the following information:

database
host
password
port
user_name

Examples

Save Credentials

import ads
from ads.secrets.mysqldb import MySQLDBSecretKeeper

vault_id = "ocid1.vault..<unique_ID>"
key_id = "ocid1.key..<unique_ID>"

ads.set_auth("resource_principal") # If using resource principal for authentication
connection_parameters={
     "user_name":"<your user name>",
     "password":"<your password>",
     "database":"database",
     "host":"<db host>",
     "port":"<db port>",
}

mysqldb_keeper = MySQLDBSecretKeeper(vault_id=vault_id,
                                key_id=key_id,
                                **connection_parameters)

mysqldb_keeper.save("mysqldb_employee", "My DB credentials", freeform_tags={"schema":"emp"})
print(mysqldb_keeper.secret_id) # Prints the secret_id of the stored credentials

'ocid1.vaultsecret..<unique_ID>'

You can save the vault details in a file for later reference, or use it in your code using export_vault_details API calls. The API currently enables you to export the information as a YAML file or a JSON file.

mysqldb_keeper.export_vault_details("my_db_vault_info.json", format="json")

Save as a YAML File

mysqldb_keeper.export_vault_details("my_db_vault_info.yaml", format="yaml")

Load Credentials

Load

The MySQLDBSecretKeeper.load_secret() API deserializes and loads the credentials from the vault. You could use this API in one of the following ways:

Using a `with` Statement

with MySQLDBSecretKeeper.load_secret('ocid1.vaultsecret..<unique_ID>') as mysqldb_secret:
    print(mysqldb_secret['user_name']

Without Using a `with` Statement

mysqldb_secretobj = MySQLDBSecretKeeper.load_secret('ocid1.vaultsecret..<unique_ID>')
mysqldb_secret = mysqldb_secretobj.to_dict()
print(mysqldb_secret['user_name'])

The .load_secret() method has the following parameters:

auth: Provide overriding auth information if the auth information is different from the ads.set_auth setting.
export_env: The default is False. If set to True, the credentials are exported as environment variabled when used with the with operator.
export_prefix: The default name for environment variable is user_name, password, database. and wallet_location. You can add a prefix to avoid name collision.
format: (Optional) If source is a file, then this value must be json or yaml depending on the file format.
source: Either the file that was exported from export_vault_details, or the OCID of the secret.

Examples

Using a `with` Statement

import ads
ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.mysqldb import MySQLDBSecretKeeper

with MySQLDBSecretKeeper.load_secret(
            "ocid1.vaultsecret..<unique_ID>"
        ) as mysqldb_creds2:
    print (mysqldb_creds2["user_name"]) # Prints the user name

print (mysqldb_creds2["user_name"]) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

Export the Environment Variables Using a `with` Statement

To expose credentials as an environment variable, set export_env=True. The following keys are exported:

Secret attribute	Environment Variable Name
user_name	user_name
password	password
host	host
port	port
database	database

import os
import ads

ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.mysqldb import MySQLDBSecretKeeper

with MySQLDBSecretKeeper.load_secret(
            "ocid1.vaultsecret..<unique_ID>",
            export_env=True
        ):
    print(os.environ.get("user_name")) # Prints the user name

print(os.environ.get("user_name")) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

You can avoid name collisions by setting a prefix string using export_prefix along with export_env=True. For example, if you set prefix as myprocess, then the exported keys are:

Secret attribute	Environment Variable Name
user_name	myprocess.user_name
password	myprocess.password
host	myprocess.host
port	myprocess.port
database	myprocess.database

import os
import ads

ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.mysqldb import MySQLDBSecretKeeper

with MySQLDBSecretKeeper.load_secret(
            "ocid1.vaultsecret..<unique_ID>",
            export_env=True,
            export_prefix="myprocess"
        ):
    print(os.environ.get("myprocess.user_name")) # Prints the user name

print(os.environ.get("myprocess.user_name")) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

Oracle Database

To connect to an Oracle Database you need the following:

hostname
password
port. Default is 1521
service name or sid
user name

The OracleDBSecretKeeper class saves the Oracle Database credentials to the OCI Vault service.

See API Documentation for more details

Save Credentials

`OracleDBSecretKeeper`

The OracleDBSecretKeeper constructor has the following parameters:

compartment_id (str): OCID of the compartment where the vault is located. This defaults to the compartment of the notebook session when used in a Data Science notebook session.
dsn (str, optional): The DSN string if available.
host (str): The hostname of the database.
key_id (str): OCID of the master key used for encrypting the secret.
password (str): The password of the database.
port (str, optional). Default 1521. Port number of the database service.
service_name (str, optional): The service name of the database.
sid (str, optional): The SID of the database if the service name is not available.
user_name (str): The user name to be stored.
vault_id (str): OCID of the vault.

Save

The OracleDBSecretKeeper.save() API serializes and stores the credentials to Vault using the following parameters:

defined_tags (dict, optional): Save the tags under predefined tags in the OCI Console.
description (str): Description of the secret when saved in the vault.
freeform_tags (dict, optional): Freeform tags to use when saving the secret in the OCI Console.
name (str): Name of the secret when saved in the vault.

The secret has the following information:

dsn
host
password
port
service_name
sid
user_name

Examples

Save Credentials

import ads
from ads.secrets.oracledb import OracleDBSecretKeeper

vault_id = "ocid1.vault..<unique_ID>"
key_id = "ocid1.key..<unique_ID>"

ads.set_auth("resource_principal") # If using resource principal for authentication
connection_parameters={
     "user_name":"<your user name>",
     "password":"<your password>",
     "service_name":"service_name",
     "host":"<db host>",
     "port":"<db port>",
}

oracledb_keeper = OracleDBSecretKeeper(vault_id=vault_id,
                                key_id=key_id,
                                **connection_parameters)

oracledb_keeper.save("oracledb_employee", "My DB credentials", freeform_tags={"schema":"emp"})
print(oracledb_keeper.secret_id) # Prints the secret_id of the stored credentials

'ocid1.vaultsecret..<unique_ID>'

You can save the vault details in a file for later reference or using it within your code using export_vault_details API calls. The API currently enables you to export the information as a YAML file or a JSON file.

oracledb_keeper.export_vault_details("my_db_vault_info.json", format="json")

Save as a YAML File

oracledb_keeper.export_vault_details("my_db_vault_info.yaml", format="yaml")

Load Credentials

Load

The OracleDBSecretKeeper.load_secret() API deserializes and loads the credentials from the vault. You could use this API in one of the following ways:

Using a `with` Statement

with OracleDBSecretKeeper.load_secret('ocid1.vaultsecret..<unique_ID>') as oracledb_secret:
    print(oracledb_secret['user_name']

Without using a `with` Statement

oracledb_secretobj = OracleDBSecretKeeper.load_secret('ocid1.vaultsecret..<unique_ID>')
oracledb_secret = oracledb_secretobj.to_dict()
print(oracledb_secret['user_name'])

The .load_secret() method has the following parameters:

auth: Provide overriding authorization information if the authorization information is different from the ads.set_auth setting.
export_env: Default is False. If set to True, the credentials are exported as environment variable when used with the with operator.
export_prefix: The default name for environment variable is user_name, password, service_name, and wallet_location. You can add a prefix to avoid name collision.
format: Optional. If source is a file, then this value must be json or yaml depending on the file format.
source: Either the file that was exported from export_vault_details or the OCID of the secret

Examples

Using a `with` Statement

import ads
ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.oracledb import OracleDBSecretKeeper

with OracleDBSecretKeeper.load_secret(
            "ocid1.vaultsecret..<unique_ID>"
        ) as oracledb_creds2:
    print (oracledb_creds2["user_name"]) # Prints the user name

print (oracledb_creds2["user_name"]) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

Export the Environment Variable Using a `with` Statement

To expose credentials as an environment variable, set export_env=True. The following keys are exported:

Secret attribute	Environment Variable Name
user_name	user_name
password	password
host	host
port	port
service user_name	service_name
sid	sid
dsn	dsn

import os
import ads

ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.oracledb import OracleDBSecretKeeper

with OracleDBSecretKeeper.load_secret(
            "ocid1.vaultsecret..<unique_ID>",
            export_env=True
        ):
    print(os.environ.get("user_name")) # Prints the user name

print(os.environ.get("user_name")) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

You can avoid name collisions by setting a prefix string using export_prefix along with export_env=True. For example, if you set prefix as myprocess, then the exported keys are:

Secret attribute	Environment Variable Name
user_name	myprocess.user_name
password	myprocess.password
host	myprocess.host
port	myprocess.port
service user_name	myprocess.service_name
sid	myprocess.sid
dsn	myprocess.dsn

import os
import ads

ads.set_auth('resource_principal') # If using resource principal authentication
from ads.secrets.oracledb import OracleDBSecretKeeper

with OracleDBSecretKeeper.load_secret(
            "ocid1.vaultsecret..<unique_ID>",
            export_env=True,
            export_prefix="myprocess"
        ):
    print(os.environ.get("myprocess.user_name")) # Prints the user name

print(os.environ.get("myprocess.user_name")) # Prints nothing. The credentials are cleared from the dictionary outside the ``with`` block

Class Documentation

ads package

Subpackages

ads.bds package

Submodules

ads.bds.auth module

exception ads.bds.auth.KRB5KinitError

Bases: Exception

KRB5KinitError class when kinit -kt command failed to generate cached ticket with the keytab file and the krb5 config file.

ads.bds.auth.has_kerberos_ticket(): Whether kerberos cache ticket exists.

ads.bds.auth.init_ccache_with_keytab(principal: str, keytab_file: str) → None

Initialize credential cache using keytab file.

Parameters:

principal (str) – The unique identity to which Kerberos can assign tickets.
keytab_path (str) – Path to your keytab file.

Returns:

Nothing.

Return type:

None

ads.bds.auth.krbcontext(principal: str, keytab_path: str, kerb5_path: str = '~/.bds_config/krb5.conf') → None

A context manager for Kerberos-related actions. It provides a Kerberos context that you can put code inside. It will initialize credential cache automatically with keytab if no cached ticket exists. Otherwise, does nothing.

Parameters:

principal (str) – The unique identity to which Kerberos can assign tickets.
keytab_path (str) – Path to your keytab file.
kerb5_path ((str, optional).) – Path to your krb5 config file.

Returns:

Nothing.

Return type:

None

Examples

>>> from ads.bds.auth import krbcontext
>>> from pyhive import hive
>>> with krbcontext(principal = "your_principal", keytab_path = "your_keytab_path"):
>>>    hive_cursor = hive.connect(host="your_hive_host",
...                    port="your_hive_port",
...                    auth='KERBEROS',
...                    kerberos_service_name="hive").cursor()

ads.bds.auth.refresh_ticket(principal: str, keytab_path: str, kerb5_path: str = '~/.bds_config/krb5.conf') → None

generate new cached ticket based on the principal and keytab file path.

Parameters:

principal (str) – The unique identity to which Kerberos can assign tickets.
keytab_path (str) – Path to your keytab file.
kerb5_path ((str, optional).) – Path to your krb5 config file.

Returns:

Nothing.

Return type:

None

Examples

>>> from ads.bds.auth import refresh_ticket
>>> from pyhive import hive
>>> refresh_ticket(principal = "your_principal", keytab_path = "your_keytab_path")
>>> hive_cursor = hive.connect(host="your_hive_host",
...                    port="your_hive_port",
...                    auth='KERBEROS',
...                    kerberos_service_name="hive").cursor()

ads.bds.big_data_service module

class ads.bds.big_data_service.ADSHiveConnection(host: str, port: str = '10000', auth_mechanism: str = 'GSSAPI', driver: str = 'impyla', **kwargs)

Bases: object

Initiate the connection.

Parameters:

host (str) – Hive host name.
port (str) – Hive port. Default to 10000.
auth_mechanism (str) – Default to “GSSAPI”. Using “PLAIN” for unsecure cluster.
driver (str) – Default to “impyla”. Client used to communicate with Hive. Only support impyla by far.
kwargs – Other connection parameters accepted by the client.

insert(table_name: str, df: DataFrame, if_exists: str, batch_size: int = 1000, **kwargs)

insert a table from a pandas dataframe.

Parameters:

(str) (if_exists) – Table Name. Table name contains database name as well. By default it will use ‘default’ database. You can specify the database name by table_name=<db_name>.<tb_name>.
(pd.DataFrame) (df) – Data to be injected to the database.
(str) – Whether to replace, append or fail if the table already exists.
batch_size (int, default 1000) – Inserting in batches improves insertion performance. Choose this value based on available memory and network bandwidth.
(dict) (kwargs) – Other parameters used by pandas.DataFrame.to_sql.

query(sql: str, bind_variables: Optional[Dict] = None, chunksize: Optional[int] = None) → Union[DataFrame, Iterator[DataFrame]]

Query data which support select statement.

Parameters:

(str) (sql) – sql query.
(Optional[Dict]) (bind_variables) – Parameters to be bound to variables in the SQL query, if any. Impyla supports all DB API paramstyle`s, including `qmark, numeric, named, format, pyformat.
(Optional[int]) (chunksize) – chunksize of each of the dataframe in the iterator.

Returns:

A pandas DataFrame or a pandas DataFrame iterator.

Return type:

Union[pd.DataFrame, Iterator[pd.DataFrame]]

class ads.bds.big_data_service.HiveConnection(**params)

Bases: ABC

Base class Interface.

set up hive connection.

abstract get_cursor()

Returns the cursor from the connection.

Returns:: cursor using a specific client.
Return type:: HiveServer2Cursor

abstract get_engine()

Returns engine from the connection.

Return type:: Engine object for the connection.

class ads.bds.big_data_service.HiveConnectionFactory

Bases: object

clientprovider = {'impyla': <class 'ads.bds.big_data_service.ImpylaHiveConnection'>}

classmethod get(driver='impyla')

class ads.bds.big_data_service.ImpylaHiveConnection(**params)

Bases: HiveConnection

ImpalaHiveConnection class which uses impyla client.

set up the impala connection.

get_cursor() → impala.hiveserver2.HiveServer2Cursor

Returns the cursor from the connection.

Returns:: cursor using impyla client.
Return type:: impala.hiveserver2.HiveServer2Cursor

get_engine(schema='default')

return the sqlalchemy engine from the connection.

Parameters:: schema (str) – Default to “default”. The default schema used for query.
Returns:: engine using a specific client.
Return type:: sqlalchemy.engine

Module contents

ads.catalog package

Submodules

ads.catalog.model module

class ads.catalog.model.Model(model: Model, model_etag: str, provenance_metadata: ModelProvenance, provenance_etag: str, ds_client: DataScienceClient, identity_client: IdentityClient)

Bases: object

Class that represents the ADS implementation of model catalog item. Converts the metadata and schema from OCI implememtation to ADS implementation.

to_dataframe(): Converts model to dataframe format.

show_in_notebook(): Shows model in the notebook in dataframe or YAML representation.

activate(): Activates model.

deactivate(): Deactivates model.

commit(): Commits the changes made to the model.

rollback(): Rollbacks the changes made to the model.

load_model(): Loads the model from the model catalog based on model ID.

Initializes the Model.

Parameters:

model (OCIModel) – The OCI model object.
model_etag (str) – The model ETag.
provenance_metadata (ModelProvenance) – The model provenance metadata.
provenance_etag (str) – The model provenance metadata ETag.
ds_client (DataScienceClient) – The Oracle DataScience client.
identity_client (IdentityClient) – The Orcale Identity Service Client.

activate() → None

Activates model.

Returns:: Nothing.
Return type:: None

commit(force: bool = True) → None

Commits model changes.

Parameters:: force (bool) – If True, any remote changes on this model would be lost.
Returns:: Nothing.
Return type:: None

deactivate() → None

Deactivates model.

Returns:: Nothing.
Return type:: None

classmethod load_model(ds_client: DataScienceClient, identity_client: IdentityClient, model_id: str) → Model

Loads the model from the model catalog based on model ID.

Parameters:

ds_client (DataScienceClient) – The Oracle DataScience client.
identity_client (IdentityClient) – The Orcale Identity Service Client.
model_id (str) – The model ID.

Returns:

The ADS model catalog item.

Return type:

Model

Raises:

ServiceError – If error occures while getting model from server.:
KeyError – If model not found.:
ValueError – If error occures while getting model provenance mettadata from server.:

rollback() → None

Rollbacks the changes made to the model.

Returns:: Nothing.
Return type:: None

show_in_notebook(display_format: str = 'dataframe') → None

Shows model in dataframe or yaml format. Supported formats: dataframe and yaml. Defaults to dataframe format.

Returns:: Nothing.
Return type:: None

to_dataframe() → DataFrame

Converts the model to dataframe format.

Returns:: Pandas dataframe.
Return type:: panadas.DataFrame

exception ads.catalog.model.ModelArtifactSizeError(max_artifact_size: str): Bases: Exception

class ads.catalog.model.ModelCatalog(compartment_id: Optional[str] = None, ds_client_auth: Optional[dict] = None, identity_client_auth: Optional[dict] = None, timeout: Optional[int] = None, ds_client: Optional[DataScienceClient] = None, identity_client: Optional[IdentityClient] = None)

Bases: object

Allows to list, load, update, download, upload and delete models from model catalog.

get_model(self, model_id): Loads the model from the model catalog based on model_id.

list_models(self, project_id=None, include_deleted=False, datetime_format=utils.date_format, \*\*kwargs): Lists all models in a given compartment, or in the current project if project_id is specified.

list_model_deployment(self, model_id, config=None, tenant_id=None, limit=500, page=None, \*\*kwargs): Gets the list of model deployments by model Id across the compartments.

update_model(self, model_id, update_model_details=None, \*\*kwargs): Updates a model with given model_id, using the provided update data.

delete_model(self, model, \*\*kwargs): Deletes the model based on model_id.

download_model(self, model_id, target_dir, force_overwrite=False, install_libs=False, conflict_strategy=ConflictStrategy.IGNORE): Downloads the model from model_dir to target_dir based on model_id.

upload_model(self, model_artifact, provenance_metadata=None, project_id=None, display_name=None, description=None): Uploads the model artifact to cloud storage.

Initializes model catalog instance.

Parameters:

compartment_id ((str, optional). Defaults to None.) – Model compartment OCID. If None, the config.NB_SESSION_COMPARTMENT_OCID would be used.
ds_client_auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate DataScienceClient object.
identity_client_auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
timeout ((int, optional). Defaults to 10 seconds.) – The connection timeout in seconds for the client.
ds_client (DataScienceClient) – The Oracle DataScience client.
identity_client (IdentityClient) – The Orcale Identity Service Client.

Raises:

ValueError – If compartment ID not specified.
TypeError – If timeout not an integer.

delete_model(model: Union[str, ads.catalog.Model], **kwargs) → bool

Deletes the model from Model Catalog.

Parameters:

model (Union[str, "ads.catalog.Model"]) – The OCID of the model to delete as a string, or a ads.catalog.Model instance.
kwargs –

delete_associated_model_deployment: (bool, optional). Defaults to False.
Whether associated model deployments need to be deletet or not.

Returns:

True if the model was successfully deleted.

Return type:

bool

Raises:

ModelWithActiveDeploymentError – If model has active model deployments ant inout attribute delete_associated_model_deployment set to False.

download_model(model_id: str, target_dir: str, force_overwrite: bool = False, install_libs: bool = False, conflict_strategy='IGNORE', bucket_uri: Optional[str] = None, remove_existing_artifact: Optional[bool] = True)

Downloads the model from model_dir to target_dir based on model_id.

Parameters:

model_id (str) – The OCID of the model to download.
target_dir (str) – The target location of model after download.
force_overwrite (bool) – Overwrite target_dir if exists.
install_libs (bool, default: False) – Install the libraries specified in ds-requirements.txt which are missing in the current environment.
conflict_strategy (ConflictStrategy, default: IGNORE) – Determines how to handle version conflicts between the current environment and requirements of model artifact. Valid values: “IGNORE”, “UPDATE” or ConflictStrategy. IGNORE: Use the installed version in case of conflict UPDATE: Force update dependency to the version required by model artifact in case of conflict
bucket_uri ((str, optional). Defaults to None.) – The OCI Object Storage URI where model artifacts will be copied to. The bucket_uri is only necessary for downloading large artifacts with size is greater than 2GB. Example: oci://<bucket_name>@<namespace>/prefix/.
remove_existing_artifact ((bool, optional). Defaults to True.) – Whether artifacts uploaded to object storage bucket need to be removed or not.

Returns:

A ModelArtifact instance.

Return type:

ModelArtifact

get_model(model_id)

Loads the model from the model catalog based on model_id.

Parameters:: model_id (str, required) – The model ID.
Returns:: The ads.catalog.Model with the matching ID.
Return type:: ads.catalog.Model

list_model_deployment(model_id: str, config: Optional[dict] = None, tenant_id: Optional[str] = None, limit: int = 500, page: Optional[str] = None, **kwargs)

Gets the list of model deployments by model Id across the compartments.

Parameters:

model_id (str) – The model ID.
config (dict (optional)) – Configuration keys and values as per SDK and Tool Configuration. The from_file() method can be used to load configuration from a file. Alternatively, a dict can be passed. You can validate_config the dict using validate_config(). Defaults to None.
tenant_id (str (optional)) – The tenancy ID, which can be used to specify a different tenancy (for cross-tenancy authorization) when searching for resources in a different tenancy. Defaults to None.
limit (int (optional)) – The maximum number of items to return. The value must be between 1 and 1000. Defaults to 500.
page (str (optional)) – The page at which to start retrieving results.

Return type:

The list of model deployments.

list_models(project_id: Optional[str] = None, include_deleted: bool = False, datetime_format: str = '%Y-%m-%d %H:%M:%S', **kwargs)

Lists all models in a given compartment, or in the current project if project_id is specified.

Parameters:

project_id (str) – The project_id of model.
include_deleted (bool, optional, default=False) – Whether to include deleted models in the returned list.
datetime_format (str, optional, default: '%Y-%m-%d %H:%M:%S') – Change format for date time fields.

Returns:

A list of models.

Return type:

ModelSummaryList

update_model(model_id, update_model_details=None, **kwargs) → Model

Updates a model with given model_id, using the provided update data.

Parameters:

model_id (str) – The model ID.
update_model_details (UpdateModelDetails) – Contains the update model details data to apply. Mandatory unless kwargs are supplied.
kwargs (dict, optional) – Update model details can be supplied instead as kwargs.

Returns:

The ads.catalog.Model with the matching ID.

Return type:

Model

upload_model(model_artifact: ModelArtifact, provenance_metadata: Optional[ModelProvenance] = None, project_id: Optional[str] = None, display_name: Optional[str] = None, description: Optional[str] = None, freeform_tags: Optional[Dict[str, Dict[str, object]]] = None, defined_tags: Optional[Dict[str, Dict[str, object]]] = None, bucket_uri: Optional[str] = None, remove_existing_artifact: Optional[bool] = True, overwrite_existing_artifact: Optional[bool] = True, model_version_set: Optional[Union[str, ModelVersionSet]] = None, version_label: Optional[str] = None)

Uploads the model artifact to cloud storage.

Parameters:

model_artifact (Union[ModelArtifact, GenericModel]) – The model artifacts or generic model instance.
provenance_metadata ((ModelProvenance, optional). Defaults to None.) – Model provenance gives data scientists information about the origin of their model. This information allows data scientists to reproduce the development environment in which the model was trained.
project_id ((str, optional). Defaults to None.) – The project_id of model.
display_name ((str, optional). Defaults to None.) – The name of model. If a display_name is not provided, a randomly generated easy to remember name with timestamp will be generated, like ‘strange-spider-2022-08-17-23:55.02’.
description ((str, optional). Defaults to None.) – The description of model.
freeform_tags ((Dict[str, str], optional). Defaults to None.) – Freeform tags for the model, by default None
defined_tags ((Dict[str, dict[str, object]], optional). Defaults to None.) – Defined tags for the model, by default None.
bucket_uri ((str, optional). Defaults to None.) – The OCI Object Storage URI where model artifacts will be copied to. The bucket_uri is only necessary for uploading large artifacts which size greater than 2GB. Example: oci://<bucket_name>@<namespace>/prefix/.
remove_existing_artifact ((bool, optional). Defaults to True.) – Whether artifacts uploaded to object storage bucket need to be removed or not.
overwrite_existing_artifact ((bool, optional). Defaults to True.) – Overwrite target bucket artifact if exists.
model_version_set ((Union[str, ModelVersionSet], optional). Defaults to None.) – The Model version set OCID, or name, or ModelVersionSet instance.
version_label ((str, optional). Defaults to None.) – The model version label.

Returns:

The ads.catalog.Model with the matching ID.

Return type:

ads.catalog.Model

class ads.catalog.model.ModelSummaryList(model_catalog, model_list, response=None, datetime_format='%Y-%m-%d %H:%M:%S')

Bases: SummaryList

Model Summary List which represents a list of Model Object.

sort_by(self, columns, reverse=False): Performs a multi-key sort on a particular set of columns and returns the sorted ModelSummaryList. Results are listed in a descending order by default.

filter(self, selection, instance=None): Filters the model list according to a lambda filter function, or list comprehension.

filter(selection, instance=None)

Filters the model list according to a lambda filter function, or list comprehension.

Parameters:

selection (lambda function filtering model instances, or a list-comprehension) – function of list filtering projects
instance (list, optional) – list to filter, optional, defaults to self

Returns:

ModelSummaryList

Return type:

A filtered ModelSummaryList

sort_by(columns, reverse=False)

Performs a multi-key sort on a particular set of columns and returns the sorted ModelSummaryList. Results are listed in a descending order by default.

Parameters:

columns (List of string) – A list of columns which are provided to sort on
reverse (Boolean (defaults to false)) – If you’d like to reverse the results (for example, to get ascending instead of descending results)

Returns:

ModelSummaryList

Return type:

A sorted ModelSummaryList

exception ads.catalog.model.ModelWithActiveDeploymentError: Bases: Exception

ads.catalog.notebook module

class ads.catalog.notebook.NotebookCatalog(compartment_id=None)

Bases: object

create_notebook_session(display_name=None, project_id=None, shape=None, block_storage_size_in_gbs=None, subnet_id=None, **kwargs)

Create a new notebook session with the supplied details.

Parameters:

display_name (str, required) – The value to assign to the display_name property of this CreateNotebookSessionDetails.
project_id (str, required) – The value to assign to the project_id property of this CreateNotebookSessionDetails.
shape (str, required) – The value to assign to the shape property of this NotebookSessionConfigurationDetails. Allowed values for this property are: “VM.Standard.E2.2”, “VM.Standard.E2.4”, “VM.Standard.E2.8”, “VM.Standard2.1”, “VM.Standard2.2”, “VM.Standard2.4”, “VM.Standard2.8”, “VM.Standard2.16”,”VM.Standard2.24”.
block_storage_size_in_gbs (int, required) – Size of the block storage drive. Limited to values between 50 (GB) and 1024 (1024GB = 1TB)
subnet_id (str, required) – The OCID of the subnet resource where the notebook is to be created.
kwargs (dict, optional) – Additional kwargs passed to DataScienceClient.create_notebook_session()

Returns:

oci.data_science.models.NotebookSession

Return type:

A new notebook record.

Raises:

KeyError – If the resource was not found or do not have authorization to access that resource.:

delete_notebook_session(notebook, **kwargs)

Deletes the notebook based on notebook_id.

Parameters:: notebook (str ID or oci.data_science.models.NotebookSession,required) – The OCID of the notebook to delete as a string, or a Notebook Session instance
Returns:: Bool
Return type:: True if delete was successful, false otherwise

get_notebook_session(notebook_id)

Get the notebook based on notebook_id

Parameters:: notebook_id (str, required) – The OCID of the notebook to get.
Returns:: oci.data_science.models.NotebookSession
Return type:: The oci.data_science.models.NotebookSession with the matching ID.
Raises:: KeyError – If the resource was not found or do not have authorization to access that resource.:

list_notebook_session(include_deleted=False, datetime_format='%Y-%m-%d %H:%M:%S', **kwargs)

List all notebooks in a given compartment

Parameters:

include_deleted (bool, optional, default=False) – Whether to include deleted notebooks in the returned list
datetime_format (str, optional, default: '%Y-%m-%d %H:%M:%S') – Change format for date time fields

Returns:

NotebookSummaryList

Return type:

A List of notebooks.

Raises:

KeyError – If the resource was not found or do not have authorization to access that resource.:

update_notebook_session(notebook_id, update_notebook_details=None, **kwargs)

Updates a notebook with given notebook_id, using the provided update data

Parameters:

notebook_id (str) – notebook_id OCID to update
update_notebook_details (oci.data_science.models.UpdateNotebookSessionDetails) – contains the new notebook details data to apply
kwargs (dict, optional) – Update notebook session details can be supplied instead as kwargs

Returns:

oci.data_science.models.NotebookSession

Return type:

The updated Notebook record

Raises:

KeyError – If the resource was not found or do not have authorization to access that resource.:

class ads.catalog.notebook.NotebookSummaryList(notebook_list, response=None, datetime_format='%Y-%m-%d %H:%M:%S')

Bases: SummaryList

filter(selection, instance=None)

Filter the notebook list according to a lambda filter function, or list comprehension.

Parameters:

selection (lambda function filtering notebook instances, or a list-comprehension) – function of list filtering notebooks
instance (list, optional) – list to filter, optional, defaults to self

Raises:

ValueError – If selection passed is not correct. For example: selection=oci.data_science.models.NotebookSession.:

sort_by(columns, reverse=False)

Performs a multi-key sort on a particular set of columns and returns the sorted NotebookSummaryList Results are listed in a descending order by default.

Parameters:

columns (List of string) – A list of columns which are provided to sort on
reverse (Boolean (defaults to false)) – If you’d like to reverse the results (for example, to get ascending instead of descending results)

Returns:

NotebookSummaryList

Return type:

A sorted NotebookSummaryList

ads.catalog.project module

class ads.catalog.project.ProjectCatalog(compartment_id=None, ds_client_auth=None, identity_client_auth=None)

Bases: Mapping

create_project(create_project_details=None, **kwargs)

Create a new project with the supplied details. create_project_details contains parameters needed to create a new project, according to oci.data_science.models.CreateProjectDetails.

Parameters:

display_name (str) – The value to assign to the display_name property of this CreateProjectDetails.
description (str) – The value to assign to the description property of this CreateProjectDetails.
compartment_id (str) – The value to assign to the compartment_id property of this CreateProjectDetails.
freeform_tags (dict(str, str)) – The value to assign to the freeform_tags property of this CreateProjectDetails.
defined_tags (dict(str, dict(str, object))) – The value to assign to the defined_tags property of this CreateProjectDetails.
kwargs – New project details can be supplied instead as kwargs

Returns:

oci.data_science.models.Project

Return type:

A new Project record.

delete_project(project, **kwargs)

Deletes the project based on project_id.

Parameters:: project (str ID or oci.data_science.models.Project,required) – The OCID of the project to delete as a string, or a Project instance
Returns:: Bool
Return type:: True if delete was succesful

get_project(project_id)

Get the Project based on project_id

Parameters:: project_id (str, required) – The OCID of the project to get.
Return type:: The oci.data_science.models.Project with the matching ID.
Raises:: KeyError – If the resource was not found or do not have authorization to access that resource.:

list_projects(include_deleted=False, datetime_format='%Y-%m-%d %H:%M:%S', **kwargs)

List all projects in a given compartment, or in the current notebook session’s compartment

Parameters:

include_deleted (bool, optional, default=False) – Whether to include deleted projects in the returned list
datetime_format (str, optional, default: '%Y-%m-%d %H:%M:%S') – Change format for date time fields

Returns:

ProjectSummaryList

Return type:

List of Projects.

Raises:

KeyError – If the resource was not found or do not have authorization to access that resource.:

update_project(project_id, update_project_details=None, **kwargs)

Updates a project with given project_id, using the provided update data update_project_details contains the update project details data to apply, according to oci.data_science.models.UpdateProjectDetails

Parameters:

project_id (str) – project_id OCID to update
display_name (str) – The value to assign to the display_name property of this UpdateProjectDetails.
description (str) – The value to assign to the description property of this UpdateProjectDetails.
freeform_tags (dict(str, str)) – The value to assign to the freeform_tags property of this UpdateProjectDetails.
defined_tags (dict(str, dict(str, object))) – The value to assign to the defined_tags property of this UpdateProjectDetails.
kwargs (dict, optional) – Update project details can be supplied instead as kwargs

Returns:

oci.data_science.models.Project

Return type:

The updated Project record

class ads.catalog.project.ProjectSummaryList(project_list, response=None, datetime_format='%Y-%m-%d %H:%M:%S')

Bases: SummaryList

A class used to represent Project Summary List.

…

df

Summary information for a project.

Type:: data frame

datetime_format

Format used to describe time.

Type:: str

response

A response object with data of type list of ProjectSummaryList.

Type:: oci.response.Response

short_id_index

Mapping of short id and its value.

Type:: (dict of str: str)

sort_by(self, columns, reverse=False):: Sort ProjectSummaryList by columns.

filter(self, selection, instance=None):: Filter the project list according to a lambda filter function, or list comprehension.

filter(selection, instance=None)

Filter the project list according to a lambda filter function, or list comprehension.

Parameters:

selection (lambda function filtering Project instances, or a list-comprehension) – function of list filtering projects
instance (list, optional) – list to filter, optional, defaults to self

Returns:

ProjectSummaryList

Return type:

A filtered ProjectSummaryList

Raises:

ValueError – If selection passed is not correct.:

sort_by(columns, reverse=False)

Sort ProjectSummaryList by columns.

Performs a multi-key sort on a particular set of columns and returns the sorted ProjectSummaryList Results are listed in a descending order by default.

Parameters:

columns (List of string) – A list of columns which are provided to sort on
reverse (Boolean (defaults to false)) – If you’d like to reverse the results (for example, to get ascending instead of descending results)

Returns:

ProjectSummaryList

Return type:

A sorted ProjectSummaryList

ads.catalog.summary module

class ads.catalog.summary.SummaryList(entity_list, datetime_format='%Y-%m-%d %H:%M:%S')

Bases: list

abstract filter(selection, instance=None): Abstract method for filtering, implemented by the derived class

show_in_notebook(datetime_format=None)

Displays the model catalog summary in a Jupyter Notebook cell

Parameters:: date_format (like utils.date_format. Defaults to none.) –
Return type:: None

abstract sort_by(columns, reverse=False): Abstract method for sorting, implemented by the derived class

to_dataframe(datetime_format=None)

Returns the model catalog summary as a pandas dataframe

Parameters:: datatime_format (date_format) – A datetime format, like utils.date_format. Defaults to none.
Returns:: Dataframe
Return type:: The pandas DataFrame repersentation of the model catalog summary

Module contents

ads.common package

Subpackages

ads.common.artifact package

Module contents

ads.common.decorator package

Submodules

ads.common.decorator.argument_to_case module

The module that provides the decorator helping to convert function arguments to specific case (lower/upper).

Examples

>>> @argument_to_case("lower", ["name"])
... def test_function(name: str, last_name: str)
...     print(name)
...
>>> test_function("myname", "mylastname")
... MYNAME

class ads.common.decorator.argument_to_case.ArgumentCase(value)

Bases: Enum

An enumeration.

LOWER = 'lower'

UPPER = 'upper'

ads.common.decorator.argument_to_case.argument_to_case(case: ArgumentCase, arguments: List[str])

The decorator helping to convert function arguments to specific case.

Parameters:

case (ArgumentCase) – The case to convert specific arguments.
arguments (List[str]) – The list of arguments to convert to specific case.

Examples

>>> @argument_to_case("lower", ["name"])
... def test_function(name: str, last_name: str)
...     print(name)
...
>>> test_function("myname", "mylastname")
... MYNAME

ads.common.decorator.deprecate module

exception ads.common.decorator.deprecate.NotSupportedError: Bases: Exception

class ads.common.decorator.deprecate.TARGET_TYPE(value)

Bases: Enum

An enumeration.

ATTRIBUTE = 'Attribute'

CLASS = 'Class'

METHOD = 'Method'

ads.common.decorator.deprecate.deprecated(deprecated_in: Optional[str] = None, removed_in: Optional[str] = None, details: Optional[str] = None, target_type: Optional[str] = None, raise_error: bool = False)

This is a decorator which can be used to mark functions as deprecated. It will result in a warning being emitted when the function is used.

Parameters:

deprecated_in (str) – Version of ADS where this function deprecated.
removed_in (str) – Future version where this function will be removed.
details (str) – More information to be shown.

ads.common.decorator.runtime_dependency module

The module that provides the decorator helping to add runtime dependencies in functions.

Examples

>>> @runtime_dependency(module="pandas", short_name="pd")
... def test_function()
...     print(pd)

>>> @runtime_dependency(module="pandas", object="DataFrame", short_name="df")
... def test_function()
...     print(df)

>>> @runtime_dependency(module="pandas", short_name="pd")
... @runtime_dependency(module="pandas", object="DataFrame", short_name="df")
... def test_function()
...     print(df)
...     print(pd)

>>> @runtime_dependency(module="pandas", object="DataFrame", short_name="df", install_from="ads[optional]")
... def test_function()
...     pass

>>> @runtime_dependency(module="pandas", object="DataFrame", short_name="df", err_msg="Custom error message.")
... def test_function()
...     pass

class ads.common.decorator.runtime_dependency.OptionalDependency

Bases: object

BDS = 'oracle-ads[bds]'

BOOSTED = 'oracle-ads[boosted]'

DATA = 'oracle-ads[data]'

GEO = 'oracle-ads[geo]'

HUGGINGFACE = 'oracle-ads[huggingface]'

LABS = 'oracle-ads[labs]'

MYSQL = 'oracle-ads[mysql]'

NOTEBOOK = 'oracle-ads[notebook]'

ONNX = 'oracle-ads[onnx]'

OPCTL = 'oracle-ads[opctl]'

OPTUNA = 'oracle-ads[optuna]'

PYTORCH = 'oracle-ads[torch]'

SPARK = 'oracle-ads[spark]'

TENSORFLOW = 'oracle-ads[tensorflow]'

TEXT = 'oracle-ads[text]'

VIZ = 'oracle-ads[viz]'

ads.common.decorator.runtime_dependency.runtime_dependency(module: str, short_name: str = '', object: Optional[str] = None, install_from: Optional[str] = None, err_msg: str = '', is_for_notebook_only=False)

The decorator which is helping to add runtime dependencies to functions.

Parameters:

module (str) – The module name to be imported.
short_name ((str, optional). Defaults to empty string.) – The short name for the imported module.
object ((str, optional). Defaults to None.) – The name of the object to be imported. Can be a function or a class, or any variable provided by module.
install_from ((str, optional). Defaults to None.) – The parameter helping to answer from where the required dependency can be installed.
err_msg ((str, optional). Defaults to empty string.) – The custom error message.
is_for_notebook_only ((bool, optional). Defaults to False.) – If the value of this flag is set to True, the dependency will be added only in case when the current environment is a jupyter notebook.

Raises:

ModuleNotFoundError – In case if requested module not found.
ImportError – In case if object cannot be imported from the module.

Examples

>>> @runtime_dependency(module="pandas", short_name="pd")
... def test_function()
...     print(pd)

>>> @runtime_dependency(module="pandas", object="DataFrame", short_name="df")
... def test_function()
...     print(df)

>>> @runtime_dependency(module="pandas", short_name="pd")
... @runtime_dependency(module="pandas", object="DataFrame", short_name="df")
... def test_function()
...     print(df)
...     print(pd)

>>> @runtime_dependency(module="pandas", object="DataFrame", short_name="df", install_from="ads[optional]")
... def test_function()
...     pass

>>> @runtime_dependency(module="pandas", object="DataFrame", short_name="df", err_msg="Custom error message.")
... def test_function()
...     pass

ads.common.decorator.utils module

class ads.common.decorator.utils.class_or_instance_method

Bases: classmethod

Converts a function to be a class method or an instance depending on how it is called at runtime.

To declare a class method, use this idiom:

class C:
@classmethod def f(obj, *args, **kwargs):

…

It can be called either on the class (e.g. C.f()) or on an instance (e.g. C().f()). If it is called on the class C.f(), the first argument (obj) will be the class (aka. cls). If it is called on the instance C().f(), the first argument (obj) will be the instance (aka. self).

Module contents

ads.common.function package

Submodules

ads.common.function.fn_util module

ads.common.function.fn_util.generate_fn_artifacts(path: str, fn_name: Optional[str] = None, fn_attributes=None, artifact_type_generic=False, **kwargs)

Generates artifacts for fn (https://fnproject.io) at the provided path -

func.py
func.yaml
requirements.txt if not there. If exists appends fdk to the file.
score.py

Parameters:

path (str) – Target folder where the artifacts are placed.
fn_attributes (dict) – dictionary specifying all the function attributes as described in https://github.com/fnproject/docs/blob/master/fn/develop/func-file.md
artifact_type_generic (bool) – default is False. This attribute decides which template to pick for score.py. If True, it is assumed that the code to load is provided by the user.

ads.common.function.fn_util.get_function_config() → dict: Returns dictionary loaded from func_conf.yaml

ads.common.function.fn_util.prepare_fn_attributes(func_name: str, schema_version=20180708, version=None, python_runtime=None, entry_point=None, memory=None) → dict: Workaround for collections.namedtuples. The defaults are not supported.

ads.common.function.fn_util.write_score(path, **kwargs)

Module contents

Submodules

ads.common.analyzer module

ads.common.auth module

class ads.common.auth.APIKey(args: Optional[Dict] = None)

Bases: AuthSignerGenerator

Creates api keys auth instance. This signer is intended to be used when signing requests for a given user - it requires that user’s ID, their private key and certificate fingerprint. It prepares extra arguments necessary for creating clients for variety of OCI services.

Signer created based on args provided. If not provided current values of according arguments will be used from current global state from AuthState class.

Parameters:

args (dict) –

args that are required to create api key config and signer. Contains keys: oci_config, oci_config_location, oci_key_profile, client_kwargs.

oci_config is a configuration dict that can be used to create clients
oci_config_location - path to config file
oci_key_profile - the profile to load from config file
client_kwargs - optional parameters for OCI client creation in next steps

create_signer() → Dict

Creates api keys configuration and signer with extra arguments necessary for creating clients. Signer constructed from the oci_config provided. If not ‘oci_config’, configuration will be constructed from ‘oci_config_location’ and ‘oci_key_profile’ in place.

Resturns

dict

Contains keys - config, signer and client_kwargs.

config contains the configuration information
signer contains the signer object created. It is instantiated from signer_callable, or

signer provided in args used, or instantiated in place - client_kwargs contains the client_kwargs that was passed in as input parameter

Examples

>>> signer_args = dict(
>>>     client_kwargs=client_kwargs
>>> )
>>> signer_generator = AuthFactory().signerGenerator(AuthType.API_KEY)
>>> signer_generator(signer_args).create_signer()

class ads.common.auth.AuthContext(**kwargs)

Bases: object

AuthContext used in ‘with’ statement for properly managing global authentication type, signer, config and global configuration parameters.

Examples

>>> from ads import set_auth
>>> from ads.jobs import DataFlowRun
>>> with AuthContext(auth='resource_principal'):
>>>     df_run = DataFlowRun.from_ocid(run_id)

>>> from ads.model.framework.sklearn_model import SklearnModel
>>> model = SklearnModel.from_model_artifact(uri="model_artifact_path", artifact_dir="model_artifact_path")
>>> set_auth(auth='api_key', oci_config_location="~/.oci/config")
>>> with AuthContext(auth='api_key', oci_config_location="~/another_config_location/config"):
>>>     # upload model to Object Storage using config from another_config_location/config
>>>     model.upload_artifact(uri="oci://bucket@namespace/prefix/")
>>> # upload model to Object Storage using config from ~/.oci/config, which was set before 'with AuthContext():'
>>> model.upload_artifact(uri="oci://bucket@namespace/prefix/")

Initialize class AuthContext and saves global state of authentication type, signer, config and global configuration parameters.

Parameters:

**kwargs (optional, list of parameters passed to ads.set_auth() method, which can be:) –

auth: Optional[str], default ‘api_key’: ’api_key’, ‘resource_principal’ or ‘instance_principal’. Enable/disable resource principal identity, instance principal or keypair identity
oci_config_location: Optional[str], default oci.config.DEFAULT_LOCATION, which is ‘~/.oci/config’: config file location
profile: Optional[str], default is DEFAULT_PROFILE, which is ‘DEFAULT’: profile name for api keys config file
config: Optional[Dict], default {}: created config dictionary
signer: Optional[Any], default None: created signer, can be resource principals signer, instance principal signer or other
signer_callable: Optional[Callable], default None: a callable object that returns signer
signer_kwargs: Optional[Dict], default None: parameters accepted by the signer
client_kwargs: Optional[Dict], default None: Additional keyword arguments for initializing the OCI client. Example: client_kwargs = {“timeout”: 60}

class ads.common.auth.AuthFactory

Bases: object

AuthFactory class which contains list of registered signers and alllows to register new signers. Check documentation for more signers: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html.

Current signers:

APIKey
ResourcePrincipal
InstancePrincipal

classes = {'api_key': <class 'ads.common.auth.APIKey'>, 'instance_principal': <class 'ads.common.auth.InstancePrincipal'>, 'resource_principal': <class 'ads.common.auth.ResourcePrincipal'>}

classmethod register(signer_type: str, signer: Any) → None

Registers a new signer.

Parameters:

signer_type (str) – Singer type to be registers
signer (RecordParser) – A new Singer class to be registered.

Returns:

Nothing.

Return type:

None

signerGenerator(iam_type: Optional[str] = 'api_key')

Generates signer classes based of iam_type, which specify one of auth methods: ‘api_key’, ‘resource_principal’ or ‘instance_principal’.

Parameters:: iam_type (str, default 'api_key') – type of auth provided in IAM_TYPE environment variable or set in parameters in ads.set_auth() method.
Returns:: returns one of classes, which implements creation of signer of specified type
Return type:: APIKey or ResourcePrincipal or InstancePrincipal
Raises:: ValueError – If iam_type is not supported.

class ads.common.auth.AuthSignerGenerator

Bases: object

Abstract class for auth configuration and signer creation.

create_signer()

class ads.common.auth.AuthState(*args, **kwargs)

Bases: object

Class stores state of variables specified for auth method, configuration, configuration file location, profile name, signer or signer_callable, which set by use at any given time and can be provided by this class in any ADS module.

oci_cli_auth: str = None

oci_client_kwargs: Dict = None

oci_config: str = None

oci_config_path: str = None

oci_iam_type: str = None

oci_key_profile: str = None

oci_signer: Any = None

oci_signer_callable: Callable = None

oci_signer_kwargs: Dict = None

class ads.common.auth.AuthType

Bases: str

API_KEY = 'api_key'

INSTANCE_PRINCIPAL = 'instance_principal'

RESOURCE_PRINCIPAL = 'resource_principal'

class ads.common.auth.InstancePrincipal(args: Optional[Dict] = None)

Bases: AuthSignerGenerator

Creates Instance Principal signer - a SecurityTokenSigner which uses a security token for an instance principal. It prepares extra arguments necessary for creating clients for variety of OCI services.

Signer created based on args provided. If not provided current values of according arguments will be used from current global state from AuthState class.

Parameters:

args (dict) –

args that are required to create Instance Principal signer. Contains keys: signer_kwargs, client_kwargs.

signer_kwargs - optional parameters required to instantiate instance principal signer
client_kwargs - optional parameters for OCI client creation in next steps

create_signer() → Dict

Creates Instance Principal signer with extra arguments necessary for creating clients. Signer instantiated from the signer_callable or if the signer provided is will be return by this method. If signer_callable or signer not provided new signer will be created in place.

Resturns

dict

Contains keys - config, signer and client_kwargs.

config contains the configuration information
signer contains the signer object created. It is instantiated from signer_callable, or

signer provided in args used, or instantiated in place - client_kwargs contains the client_kwargs that was passed in as input parameter

Examples

>>> signer_args = dict(signer_kwargs={"log_requests": True})
>>> signer_generator = AuthFactory().signerGenerator(AuthType.INSTANCE_PRINCIPAL)
>>> signer_generator(signer_args).create_signer()

class ads.common.auth.OCIAuthContext(profile: str = None)

Bases: object

OCIAuthContext used in ‘with’ statement for properly managing global authentication type and global configuration profile parameters.

Examples

>>> from ads.jobs import DataFlowRun
>>> with OCIAuthContext(profile='TEST'):
>>>     df_run = DataFlowRun.from_ocid(run_id)

Initialize class OCIAuthContext and saves global state of authentication type and configuration profile.

Parameters:: profile (str, default is None) – profile name for api keys config file

class ads.common.auth.ResourcePrincipal(args: Optional[Dict] = None)

Bases: AuthSignerGenerator

Creates Resource Principal signer - a security token for a resource principal. It prepares extra arguments necessary for creating clients for variety of OCI services.

Signer created based on args provided. If not provided current values of according arguments will be used from current global state from AuthState class.

Parameters:

args (dict) –

args that are required to create Resource Principal signer. Contains keys: client_kwargs.

client_kwargs - optional parameters for OCI client creation in next steps

create_signer() → Dict

Creates Resource Principal signer with extra arguments necessary for creating clients.

Resturns

dict

Contains keys - config, signer and client_kwargs.

config contains the configuration information
signer contains the signer object created. It is instantiated from signer_callable, or

signer provided in args used, or instantiated in place - client_kwargs contains the client_kwargs that was passed in as input parameter

Examples

>>> signer_args = dict(
>>>     signer=oci.auth.signers.get_resource_principals_signer()
>>> )
>>> signer_generator = AuthFactory().signerGenerator(AuthType.RESOURCE_PRINCIPAL)
>>> signer_generator(signer_args).create_signer()

class ads.common.auth.SingletonMeta: Bases: type

ads.common.auth.api_keys(oci_config: str = '/home/docs/.oci/config', profile: str = 'DEFAULT', client_kwargs: Optional[Dict] = None) → Dict

Prepares authentication and extra arguments necessary for creating clients for different OCI services using API Keys.

Parameters:

oci_config (Optional[str], default is $HOME/.oci/config) – OCI authentication config file location.
profile (Optional[str], is DEFAULT_PROFILE, which is 'DEFAULT') – Profile name to select from the config file.
client_kwargs (Optional[Dict], default None) – kwargs that are required to instantiate the Client if we need to override the defaults.

Returns:

Contains keys - config, signer and client_kwargs.

The config contains the config loaded from the configuration loaded from oci_config.
The signer contains the signer object created from the api keys.
client_kwargs contains the client_kwargs that was passed in as input parameter.

Return type:

dict

Examples

>>> from ads.common import oci_client as oc
>>> auth = ads.auth.api_keys(oci_config="/home/datascience/.oci/config", profile="TEST", client_kwargs={"timeout": 6000})
>>> oc.OCIClientFactory(**auth).object_storage # Creates Object storage client with timeout set to 6000 using API Key authentication

ads.common.auth.create_signer(auth_type: Optional[str] = 'api_key', oci_config_location: Optional[str] = '~/.oci/config', profile: Optional[str] = 'DEFAULT', config: Optional[Dict] = {}, signer: Optional[Any] = None, signer_callable: Optional[Callable] = None, signer_kwargs: Optional[Dict] = {}, client_kwargs: Optional[Dict] = None) → Dict

Prepares authentication and extra arguments necessary for creating clients for different OCI services based on provided parameters. If signer or signer`_callable provided, authentication with that signer will be created. If config provided, api_key type of authentication will be created. Accepted values for auth_type: api_key (default), ‘instance_principal’, ‘resource_principal’.

Parameters:

auth_type (Optional[str], default 'api_key') –

‘api_key’, ‘resource_principal’ or ‘instance_principal’. Enable/disable resource principal identity,
instance principal or keypair identity in a notebook session
oci_config_location (Optional[str], default oci.config.DEFAULT_LOCATION, which is '~/.oci/config') – config file location
profile (Optional[str], default is DEFAULT_PROFILE, which is 'DEFAULT') – profile name for api keys config file
config (Optional[Dict], default {}) – created config dictionary
signer (Optional[Any], default None) – created signer, can be resource principals signer, instance principal signer or other. Check documentation for more signers: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html
signer_callable (Optional[Callable], default None) – a callable object that returns signer
signer_kwargs (Optional[Dict], default None) – parameters accepted by the signer. Check documentation: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html
client_kwargs (dict) – kwargs that are required to instantiate the Client if we need to override the defaults

Examples

>>> import ads
>>> auth = ads.auth.create_signer() # api_key type of authentication dictionary created with default config location and default profile

>>> config = oci.config.from_file("other_config_location", "OTHER_PROFILE")
>>> auth = ads.auth.create_signer(config=config) # api_key type of authentication dictionary created based on provided config

>>> singer = oci.auth.signers.get_resource_principals_signer()
>>> auth = ads.auth.create_signer(config={}, signer=signer) # resource principals authentication dictionary created

>>> auth = ads.auth.create_signer(auth_type='instance_principal') # instance principals authentication dictionary created

>>> signer_callable = oci.auth.signers.InstancePrincipalsSecurityTokenSigner
>>> signer_kwargs = dict(log_requests=True) # will log the request url and response data when retrieving
>>> auth = ads.auth.create_signer(signer_callable=signer_callable, signer_kwargs=signer_kwargs) # instance principals authentication dictionary created based on callable with kwargs parameters

ads.common.auth.default_signer(client_kwargs: Optional[Dict] = None) → Dict

Prepares authentication and extra arguments necessary for creating clients for different OCI services based on the default authentication setting for the session. Refer ads.set_auth API for further reference.

Parameters:

client_kwargs (dict) – kwargs that are required to instantiate the Client if we need to override the defaults. Example: client_kwargs = {“timeout”: 60}

Returns:

Contains keys - config, signer and client_kwargs.

The config contains the config loaded from the configuration loaded from the default location if the default auth mode is API keys, otherwise it is empty dictionary.
The signer contains the signer object created from default auth mode.
client_kwargs contains the client_kwargs that was passed in as input parameter.

Return type:

dict

Examples

>>> import ads
>>> from ads.common import oci_client as oc

>>> auth = ads.auth.default_signer()
>>> oc.OCIClientFactory(**auth).object_storage # Creates Object storage client

>>> ads.set_auth("resource_principal")
>>> auth = ads.auth.default_signer()
>>> oc.OCIClientFactory(**auth).object_storage # Creates Object storage client using resource principal authentication

>>> signer_callable = oci.auth.signers.InstancePrincipalsSecurityTokenSigner
>>> ads.set_auth(signer_callable=signer_callable) # Set instance principal callable
>>> auth = ads.auth.default_signer() # signer_callable instantiated
>>> oc.OCIClientFactory(**auth).object_storage # Creates Object storage client using instance principal authentication

ads.common.auth.get_signer(oci_config: Optional[str] = None, oci_profile: Optional[str] = None, **client_kwargs) → Dict

Provides config and signer based given parameters. If oci_config (api key config file location) and oci_profile specified new signer will ge generated. Else singer of a type specified in OCI_CLI_AUTH environment variable will be used to generate signer and return. If OCI_CLI_AUTH not set, resource principal signer will be provided. Accepted values for OCI_CLI_AUTH: ‘api_key’, ‘instance_principal’, ‘resource_principal’.

Parameters:

oci_config (Optional[str], default None) – Path to the config file
oci_profile (Optional[str], default None) – the profile to load from the config file
client_kwargs – kwargs that are required to instantiate the Client if we need to override the defaults

ads.common.auth.resource_principal(client_kwargs: Optional[Dict] = None) → Dict

Prepares authentication and extra arguments necessary for creating clients for different OCI services using Resource Principals.

Parameters:

client_kwargs (Dict, default None) – kwargs that are required to instantiate the Client if we need to override the defaults.

Returns:

Contains keys - config, signer and client_kwargs.

The config contains and empty dictionary.
The signer contains the signer object created from the resource principal.
client_kwargs contains the client_kwargs that was passed in as input parameter.

Return type:

dict

Examples

>>> from ads.common import oci_client as oc
>>> auth = ads.auth.resource_principal({"timeout": 6000})
>>> oc.OCIClientFactory(**auth).object_storage # Creates Object Storage client with timeout set to 6000 seconds using resource principal authentication

ads.common.auth.set_auth(auth: Optional[str] = 'api_key', oci_config_location: Optional[str] = '~/.oci/config', profile: Optional[str] = 'DEFAULT', config: Optional[Dict] = {}, signer: Optional[Any] = None, signer_callable: Optional[Callable] = None, signer_kwargs: Optional[Dict] = {}, client_kwargs: Optional[Dict] = {}) → None

Save type of authentication, profile, config location, config (keypair identity) or signer, which will be used when actual creation of config or signer happens.

Parameters:

auth (Optional[str], default 'api_key') –

‘api_key’, ‘resource_principal’ or ‘instance_principal’. Enable/disable resource principal identity,
instance principal or keypair identity in a notebook session
oci_config_location (Optional[str], default oci.config.DEFAULT_LOCATION, which is '~/.oci/config') – config file location
profile (Optional[str], default is DEFAULT_PROFILE, which is 'DEFAULT') – profile name for api keys config file
config (Optional[Dict], default {}) – created config dictionary
signer (Optional[Any], default None) – created signer, can be resource principals signer, instance principal signer or other. Check documentation for more signers: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html
signer_callable (Optional[Callable], default None) – a callable object that returns signer
signer_kwargs (Optional[Dict], default None) – parameters accepted by the signer. Check documentation: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html
client_kwargs (Optional[Dict], default None) – Additional keyword arguments for initializing the OCI client. Example: client_kwargs = {“timeout”: 60}

Examples

>>> ads.set_auth("api_key") # default signer is set to api keys

>>> ads.set_auth("api_key", profile = "TEST") # default signer is set to api keys and to use TEST profile

>>> ads.set_auth("api_key", oci_config_location = "other_config_location") # use non-default oci_config_location

>>> ads.set_auth("api_key", client_kwargs={"timeout": 60}) # default signer with connection and read timeouts set to 60 seconds for the client.

>>> other_config = oci.config.from_file("other_config_location", "OTHER_PROFILE") # Create non-default config
>>> ads.set_auth(config=other_config) # Set api keys type of authentication based on provided config

>>> ads.set_auth("resource_principal")  # Set resource principal authentication

>>> ads.set_auth("instance_principal")  # Set instance principal authentication

>>> singer = oci.auth.signers.get_resource_principals_signer()
>>> ads.auth.create_signer(config={}, singer=signer) # resource principals authentication dictionary created

>>> signer_callable = oci.auth.signers.ResourcePrincipalsFederationSigner
>>> ads.set_auth(signer_callable=signer_callable) # Set resource principal federation singer callable

>>> signer_callable = oci.auth.signers.InstancePrincipalsSecurityTokenSigner
>>> signer_kwargs = dict(log_requests=True) # will log the request url and response data when retrieving
>>> # instance principals authentication dictionary created based on callable with kwargs parameters:
>>> ads.set_auth(signer_callable=signer_callable, signer_kwargs=signer_kwargs)

ads.common.base_properties module

class ads.common.base_properties.BaseProperties

Bases: Serializable

Represents base properties class.

with_prop(name: str, value: Any) → BaseProperties: Sets property value.

with_dict(obj_dict: Dict) → BaseProperties: Populates properties values from dict.

with_env() → BaseProperties: Populates properties values from environment variables.

to_dict() → Dict: Serializes instance of class into a dictionary.

with_config(config: ads.config.ConfigSection) → BaseProperties: Sets properties values from the config profile.

from_dict(obj_dict: Dict[str, Any]) → 'BaseProperties': Creates an instance of the properties class from a dictionary.

from_config(uri: str, profile: str, auth: Optional[Dict] = None) → "BaseProperties":: Loads properties from the config file.

to_config(uri: str, profile: str, force_overwrite: Optional[bool] = False, auth: Optional[Dict] = None) → None: Saves properties to the config file.

classmethod from_config(uri: str, profile: str, auth: Optional[Dict] = None) → BaseProperties

Loads properties from the config file.

Parameters:

uri (str) – The URI of the config file. Can be local path or OCI object storage URI.
profile (str) – The config profile name.
auth ((Dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

Instance of the BaseProperties.

Return type:

BaseProperties

classmethod from_dict(obj_dict: Dict[str, Any]) → BaseProperties

Creates an instance of the properties class from a dictionary.

Parameters:: obj_dict (Dict[str, Any]) – List of properties and values in dictionary format.
Returns:: Instance of the BaseProperties.
Return type:: BaseProperties

to_config(uri: str, profile: str, force_overwrite: Optional[bool] = False, auth: Optional[Dict] = None) → None

Saves properties to the config file.

Parameters:

uri (str) – The URI of the config file. Can be local path or OCI object storage URI.
profile (str) – The config profile name.
force_overwrite ((bool, optional). Defaults to False.) – Whether to overwrite existing files or not.
auth ((Dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

Nothing

Return type:

None

to_dict(**kwargs)

Serializes instance of class into a dictionary.

Returns:: A dictionary.
Return type:: Dict

with_config(config: ConfigSection) → BaseProperties

Sets properties values from the config profile.

Returns:: Instance of the BaseProperties.
Return type:: BaseProperties

with_dict(obj_dict: Dict[str, Any]) → BaseProperties

Sets properties from a dict.

Parameters:: obj_dict (Dict[str, Any]) – List of properties and values in dictionary format.
Returns:: Instance of the BaseProperties.
Return type:: BaseProperties
Raises:: TypeError – If input object has a wrong type.

with_env() → BaseProperties

Sets properties values from environment variables.

Returns:: Instance of the BaseProperties.
Return type:: BaseProperties

with_prop(name: str, value: Any) → BaseProperties

Sets property value.

Parameters:

name (str) – Property name.
value – Property value.

Returns:

Instance of the BaseProperties.

Return type:

BaseProperties

ads.common.card_identifier module

credit card patterns refer to https://en.wikipedia.org/wiki/Payment_card_number#Issuer_identification_number_(IIN) Active and frequent card information American Express: 34, 37 Diners Club (US & Canada): 54,55 Discover Card: 6011, 622126 - 622925, 624000 - 626999, 628200 - 628899, 64, 65 Master Card: 2221-2720, 51–55 Visa: 4

class ads.common.card_identifier.card_identify

Bases: object

identify_issue_network(card_number)

Returns the type of credit card based on its digits

Parameters:: card_number (String) –
Returns:: String
Return type:: A string corresponding to the kind of credit card.

ads.common.config module

class ads.common.config.Config(uri: Optional[str] = '~/.ads/config', auth: Optional[Dict] = None)

Bases: object

The class representing a config.

Initializes a config instance.

Parameters:

uri ((str, optional). Defaults to ~/.ads/config.) – The path to the config file. Can be local or Object Storage file.
auth ((Dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

default() → ConfigSection

Gets default config section.

Returns:: A default config section.
Return type:: ConfigSection

keys() → List[str]

Gets the all registered config section keys.

Returns:: The list of the all registered config section keys.
Return type:: List[str]

load(uri: Optional[str] = None, auth: Optional[Dict] = None) → Config

Loads config from a config file.

Parameters:

uri ((str, optional). Defaults to ~/.ads/config.) – The path where the config file needs to be saved. Can be local or Object Storage file.
auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

A config object.

Return type:

Config

save(uri: Optional[str] = None, auth: Optional[Dict] = None, force_overwrite: Optional[bool] = False) → Config

Saves config to a config file.

Parameters:

uri ((str, optional). Defaults to ~/.ads/config.) – The path to the config file. Can be local or Object Storage file.
auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
force_overwrite ((bool, optional). Defaults to False.) – Overwrites the config if exists.

Returns:

Nothing

Return type:

None

section_exists(key: str) → bool

Checks if a config section exists.

Parameters:: key (str) – A key of a config section.
Returns:: True if a config section exists, Fasle otherwise.
Return type:: bool

section_get(key: str) → ConfigSection

Gets the config section by key.

Returns:: A config section object.
Return type:: ConfigSection
Raises:: KeyError – If a config section not exists.

section_remove(key: str) → Config

Removes config section form config.

Parameters:: key (str) – A key of a config section that needs to be removed.
Returns:: Nothing
Return type:: None

section_set(key: str, info: Union[dict, ConfigSection], replace: Optional[bool] = False) → ConfigSection

Sets a config section to config. The new config section will be added in case if it doesn’t exist. Otherwise the existing config section will be merged with the new fields.

Parameters:

key (str) – A key of a config section.
info (Union[dict, ConfigSection]) – The config section information in a dictionary or ConfigSection format.
replace ((bool, optional). Defaults to False.) – If set as True, overwrites config section with the new information.

Returns:

A config section object.

Return type:

ConfigSection

Raises:

ValueError – If section with given key is already exist and replace flag set to False.
TypeError – If input info has a wrong format.

to_dict() → Dict[str, Dict[str, Any]]

Converts config to a dictionary format.

Returns:: A config in a dictionary format.
Return type:: Dict[str, Dict[str, Any]]

with_dict(info: Dict[str, Union[Dict[str, Any], ConfigSection]], replace: Optional[bool] = False) → Config

Merging dictionary to config.

Parameters:

info (Dict[str, Union[Dict[str, Any], ConfigSection]]) – A dictionary that needs to be merged to config.
replace ((bool, optional). Defaults to False.) – If set as True, overwrites config section with the new information.

Returns:

A config object.

Return type:

Config

class ads.common.config.ConfigSection

Bases: object

The class representing a config section.

Initializes the config section instance.

clear() → None

Clears the config section values.

Returns:: Nothing
Return type:: None

copy() → ConfigSection

Makes a copy of a config section.

Returns:: The instance of a copied ConfigSection.
Return type:: ConfigSection

get(key: str) → str

Gets the config section value by key.

Returns:: A specific config section value.
Return type:: str

keys() → Tuple[str]

Gets the list of the keys of a config section.

Returns:: The list of config section keys.
Return type:: Tuple[str]

remove(key: str) → None

Removes the config section field by key.

Parameters:: key (str) – The config section field key.
Returns:: Nothing
Return type:: None

set(key: str, value: str, replace: Optional[bool] = False) → None

Sets the config section value by key.

Parameters:

key (str) – The config section field key.
value (str) – The config section field value.

Returns:

Nothing

Return type:

None

to_dict() → Dict[str, Any]

Converts config section to a dictionary.

Returns:: The config section in a dictionary format.
Return type:: Dict[str, Any]

with_dict(info: Dict[str, Any], replace: Optional[bool] = False) → ConfigSection

Populates the config section from a dictionary.

Parameters:

info (Dict[str, Any]) – The config section information in a dictionary format.
replace ((bool, optional). Defaults to False.) – If set as True, overwrites config section with the new information.

class ads.common.config.EventType(value)

Bases: Enum

An enumeration.

CHANGE = 'change'

class ads.common.config.Eventing

Bases: object

The class helper to register event handlers.

on(event_name: str, callback: Callable) → None: Registers a callback for the particular event.

trigger(event: str) → None: Triggers all the registered callbacks for the particular event.

class ads.common.config.ExtendedConfigParser(uri: Optional[str] = '~/.ads/config', auth: Optional[Dict] = None)

Bases: ConfigParser

Class helper to read/write information to the config file.

Initializes a config parser instance.

Parameters:

uri ((str, optional). Defaults to ~/.ads/config.) – The path to the config file. Can be local or Object Storage file.
auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

read(uri: Optional[str] = None, auth: Optional[Dict] = None) → ExtendedConfigParser

Reads config file.

uri: (str, optional). Defaults to ~/.ads/config.: The path to the config file. Can be local or Object Storage file.
auth: (Dict, optional). Defaults to None.: The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:: Config parser object.
Return type:: ExtendedConfigParser

save(uri: Optional[str] = None, auth: Optional[Dict] = None, force_overwrite: Optional[bool] = False) → None

Saves the config to the file.

Parameters:

uri ((str, optional). Defaults to ~/.ads/config.) – The path to the config file. Can be local or Object Storage file.
auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
force_overwrite ((bool, optional). Defaults to False.) – Overwrites the config if exists.

Return type:

None

Raises:

FileExistsError – In case if file exists and force_overwrite is false.

to_dict() → Dict[str, Any]

Converts config to a dictionary.

Returns:: Config in a dictionary format.
Return type:: Dict[str, Any]

with_dict(info: Dict[str, Dict[str, Any]]) → ExtendedConfigParser

Populates config with values from a dictionary.

Parameters:: info (Dict[str, Dict[str, Any]]) – Config in a dictionary format.
Returns:: Config parser object.
Return type:: ExtendedConfigParser

class ads.common.config.Mode

Bases: object

READ = 'r'

WRITE = 'w'

ads.common.data module

class ads.common.data.ADSData(X=None, y=None, name='', dataset_type=None)

Bases: object

This class wraps the input dataframe to various models, evaluation, and explanation frameworks. It’s primary purpose is to hold any metadata relevant to these tasks. This can include it’s:

X - the independent variables as some dataframe-like structure,
y - the dependent variable or target column as some array-like structure,
name - a string to name the data for user convenience,
dataset_type - the type of the X value.

As part of this initiative, ADSData knows how to turn itself into an onnxruntime compatible data structure with the method .to_onnxrt(), which takes and onnx session as input.

Parameters:

X (Union[pandas.DataFrame, dask.DataFrame, numpy.ndarray, scipy.sparse.csr.csr_matrix]) – If str, URI for the dataset. The dataset could be read from local or network file system, hdfs, s3 and gcs Should be none if X_train, y_train, X_test, Y_test are provided
y (Union[str, pandas.DataFrame, dask.DataFrame, pandas.Series, dask.Series, numpy.ndarray]) – If str, name of the target in X, otherwise series of labels corresponding to X
name (str, optional) – Name to identify this data
dataset_type (ADSDataset optional) – When this value is available, would be used to evaluate the ads task type
kwargs – Additional keyword arguments that would be passed to the underlying Pandas read API.

static build(X=None, y=None, name='', dataset_type=None, **kwargs)

Returns an ADSData object built from the (source, target) or (X,y)

Parameters:

X (Union[pandas.DataFrame, dask.DataFrame, numpy.ndarray, scipy.sparse.csr.csr_matrix]) – If str, URI for the dataset. The dataset could be read from local or network file system, hdfs, s3 and gcs Should be none if X_train, y_train, X_test, Y_test are provided
y (Union[str, pandas.DataFrame, dask.DataFrame, pandas.Series, dask.Series, numpy.ndarray]) – If str, name of the target in X, otherwise series of labels corresponding to X
name (str, optional) – Name to identify this data
dataset_type (ADSDataset, optional) – When this value is available, would be used to evaluate the ads task type
kwargs – Additional keyword arguments that would be passed to the underlying Pandas read API.

Returns:

ads_data – A built ADSData object

Return type:

ads.common.data.ADSData

Examples

>>> data = open_csv("my.csv")

>>> data_ads = ADSData(data, 'target').build(data, 'target')

to_onnxrt(sess, idx_range=None, model=None, impute_values={}, **kwargs)

Returns itself formatted as an input for the onnxruntime session inputs passed in.

Parameters:

sess (Session) – The session object
idx_range (Range) – The range of inputs to convert to onnx
model (SupportedModel) – A model that supports being serialized for the onnx runtime.
kwargs (additional keyword arguments) –
- sess_inputs - Pass in the output from onnxruntime.InferenceSession(“model.onnx”).get_inputs()
- input_dtypes (list) - If sess_inputs cannot be passed in, pass in the numpy dtypes of each input
- input_shapes (list) - If sess_inputs cannot be passed in, pass in the shape of each input
- input_names (list) -If sess_inputs cannot be passed in, pass in the name of each input

Returns:

ort – array of inputs formatted for the given session.

Return type:

Array

ads.common.data_serializer module

class ads.common.data_serializer.InputDataSerializer(data: Union[Dict, str, List, ndarray, Series, DataFrame], data_type=None)

Bases: object

[An internal class] Defines the contract for input data

Parameters:

data (Union[Dict, str, list, numpy.ndarray, pd.core.series.Series,) –
pd.core.frame.DataFrame] – Data expected by the model deployment predict API.
data_type (Any, defaults to None.) – Type of the data. If not provided, it will be checked against data.

property data

property data_type

is_bytes()

send(endpoint: str, dry_run: bool = False, **kwargs)

to_dict()

ads.common.error module

exception ads.common.error.ChangesNotCommitted(path): Bases: Exception

ads.common.extended_enum module

class ads.common.extended_enum.ExtendedEnumMeta(name, bases, namespace, **kwargs)

Bases: ABCMeta

The helper metaclass to extend functionality of a generic Enum.

values(cls) → list:: Gets the list of class attributes.

values() → list

Gets the list of class attributes.

Returns:: The list of class values.
Return type:: list

ads.common.ipython module

ads.common.ipython.configure_plotting()

ads.common.ipython.set_ipython_traceback()

ads.common.model module

class ads.common.model.ADSModel(est, target=None, transformer_pipeline=None, client=None, booster=None, classes=None, name=None)

Bases: object

Construct an ADSModel

Parameters:

est (fitted estimator object) – The estimator can be a standard sklearn estimator, a keras, lightgbm, or xgboost estimator, or any other object that implement methods from (BaseEstimator, RegressorMixin) for regression or (BaseEstimator, ClassifierMixin) for classification.
target (PandasSeries) – The target column you are using in your dataset, this is assigned as the “y” attribute.
transformer_pipeline (TransformerPipeline) – A custom trasnformer pipeline object.
client (Str) – Currently unused.
booster (Str) – Currently unused.
classes (list, optional) – List of target classes. Required for classification problem if the est does not contain classes attribute.
name (str, optional) – Name of the model.

static convert_dataframe_schema(df, drop=None)

feature_names(X=None)

static from_estimator(est, transformers=None, classes=None, name=None)

Build ADSModel from a fitted estimator

Parameters:

est (fitted estimator object) – The estimator can be a standard sklearn estimator or any object that implement methods from (BaseEstimator, RegressorMixin) for regression or (BaseEstimator, ClassifierMixin) for classification.
transformers (a scalar or an iterable of objects implementing transform function, optional) – The transform function would be applied on data before calling predict and predict_proba on estimator.
classes (list, optional) – List of target classes. Required for classification problem if the est does not contain classes attribute.
name (str, optional) – Name of the model.

Returns:

model

Return type:

ads.common.model.ADSModel

Examples

>>> model = MyModelClass.train()
>>> model_ads = from_estimator(model)

static get_init_types(df, underlying_model=None)

is_classifier()

Returns True if ADS believes that the model is a classifier

Returns:: Boolean
Return type:: True if the model is a classifier, False otherwise.

predict(X)

Runs the models predict function on some data

Parameters:: X (ADSData) – A ADSData object which holds the examples to be predicted on.
Returns:: Usually a list or PandasSeries of predictions
Return type:: Union[List, pandas.Series], depending on the estimator

predict_proba(X)

Runs the models predict probabilities function on some data

Parameters:: X (ADSData) – A ADSData object which holds the examples to be predicted on.
Returns:: Usually a list or PandasSeries of predictions
Return type:: Union[List, pandas.Series], depending on the estimator

prepare(target_dir=None, data_sample=None, X_sample=None, y_sample=None, include_data_sample=False, force_overwrite=False, fn_artifact_files_included=False, fn_name='model_api', inference_conda_env=None, data_science_env=False, ignore_deployment_error=False, use_case_type=None, inference_python_version=None, imputed_values={}, **kwargs)

Prepare model artifact directory to be published to model catalog

Parameters:

target_dir (str, default: model.name[:12]) – Target directory under which the model artifact files need to be added
data_sample (ADSData) – Note: This format is preferable to X_sample and y_sample. A sample of the test data that will be provided to predict() API of scoring script Used to generate schema_input.json and schema_output.json which defines the input and output formats
X_sample (pandas.DataFrame) – A sample of input data that will be provided to predict() API of scoring script Used to generate schema.json which defines the input formats
y_sample (pandas.Series) – A sample of output data that is expected to be returned by predict() API of scoring script, corresponding to X_sample Used to generate schema_output.json which defines the output formats
force_overwrite (bool, default: False) – If True, overwrites the target directory if exists already
fn_artifact_files_included (bool, default: True) – If True, generates artifacts to export a model as a function without ads dependency
fn_name (str, default: 'model_api') – Required parameter if fn_artifact_files_included parameter is setup.
inference_conda_env (str, default: None) – Conda environment to use within the model deployment service for inferencing
data_science_env (bool, default: False) – If set to True, datascience environment represented by the slug in the training conda environment will be used.
ignore_deployment_error (bool, default: False) – If set to True, the prepare will ignore all the errors that may impact model deployment
use_case_type (str) – The use case type of the model. Use it through UserCaseType class or string provided in UseCaseType. For example, use_case_type=UseCaseType.BINARY_CLASSIFICATION or use_case_type=”binary_classification”. Check with UseCaseType class to see all supported types.
inference_python_version (str, default:None.) – If provided will be added to the generated runtime yaml
**kwargs –
-------- –
max_col_num ((int, optional). Defaults to utils.DATA_SCHEMA_MAX_COL_NUM.) – The maximum column size of the data that allows to auto generate schema.

Returns:

model_artifact

Return type:

an instance of ModelArtifact that can be used to test the generated scoring script

rename(name)

Changes the name of a model

Parameters:: name (str) – A string which is supplied for naming a model.

score(X, y_true, score_fn=None)

Scores a model according to a custom score function

Parameters:

X (ADSData) – A ADSData object which holds the examples to be predicted on.
y_true (ADSData) – A ADSData object which holds ground truth labels for the examples which are being predicted on.
score_fn (Scorer (callable)) – A callable object that returns a score, usually created with sklearn.metrics.make_scorer().

Returns:

Almost always a scalar score (usually a float).

Return type:

float, depending on the estimator

show_in_notebook(): Describe the model by showing it’s properties

summary(): A summary of the ADSModel

transform(X)

Process some ADSData through the selected ADSModel transformers

Parameters:: X (ADSData) – A ADSData object which holds the examples to be transformed.

visualize_transforms(): A graph of the ADSModel transformer pipeline. It is only supported in JupyterLabs Notebooks.

ads.common.model_artifact module

class ads.common.model_artifact.ConflictStrategy

Bases: object

CREATE = 'CREATE'

IGNORE = 'IGNORE'

UPDATE = 'UPDATE'

exception ads.common.model_artifact.InvalidDataType

Bases: Exception

Invalid Data Type.

class ads.common.model_artifact.ModelArtifact(artifact_dir, conflict_strategy='IGNORE', install_libs=False, reload=True, create=False, progress=None, model_file_name='model.onnx', inference_conda_env=None, data_science_env=False, ignore_deployment_error=False, inference_python_version=None)

Bases: Introspectable

A class used to construct model artifacts.

…

artifact_dir

Path to the model artifacts.

Type:: str

conflict_strategy

How to handle version conflicts between the current environment and the requirements of model artifact.

Type:: ConflictStrategy, default: IGNORE

install_libs

Re-install the environment inwhich the model artifact were trained in.

Type:: bool

reload

Reload the model into the environment.

Type:: bool

create

Create the runtime.yaml file.

Type:: bool

progress: Show a progress bar.

model_file_name

Name of the model file.

Type:: str

inference_conda_env

The inference conda environment. If provided, the value will be set in the runtime.yaml. This is expected to be full oci URI format - oci://{bucket}@{namespace}/path/to/condapack.

Type:: str

data_science_env

Is the inference conda environment managed by the Oracle Data Science service?

Type:: bool

ignore_deployment_error

Determine whether to turn off logging for deployment error. If set to True, the .prepare() method will ignore errors that impact model deployment.

Type:: bool

inference_python_version

The version of Python to be used in inference. The value will be set in the runtime.yaml file

Type:: str Optional, default None

reload(self, model_file_name=None): Reload the files in the model artifact directory.

verify(self, input_data): Verifies a model artifact directory.

install_requirements(self, conflict_strategy=ConflictStrategy.IGNORE): Installs missing libraries listed in the model artifact.

populate_metadata(self, model=None, use_case_type=None): Extracts and populate taxonomy metadata from the model.

save(: self, display_name: str = None, description: str = None, project_id: str = None, compartment_id: str = None, training_script_path: str = None, ignore_pending_changes: bool = False, auth: dict = None, training_id: str = None, timeout: int = None, ignore_introspection=False,

): Saves this model artifact in model catalog.

populate_schema(: self, data_sample: ADSData = None, X_sample: Union[list, tuple, pd.DataFrame, pd.Series, np.ndarray] = None, y_sample: Union[list, tuple, pd.DataFrame, pd.Series, np.ndarray] = None,

): Populates input and output schema.

introspect(self) → pd.DataFrame: Runs model introspection.

classmethod from_model_catalog(model_id: str, artifact_dir: str, model_file_name: Optional[str] = 'model.onnx', auth: Optional[Dict] = None, force_overwrite: Optional[bool] = False, install_libs: Optional[bool] = False, conflict_strategy='IGNORE', bucket_uri: Optional[str] = None, remove_existing_artifact: Optional[bool] = True, **kwargs) → ModelArtifact

Download model artifacts from the model catalog to the target artifact directory.

Parameters:

model_id (str) – The model OCID.
artifact_dir (str) – The artifact directory to store the files needed for deployment. Will be created if not exists.
model_file_name ((str, optional). Defaults to "model.onnx".) – The name of the serialized model.
auth ((Dict, optional). Defaults to None.) – Default authetication is set using the ads.set_auth() method. Use the ads.common.auth.api_keys() or ads.common.auth.resource_principal() to create appropriate authentication signer and kwargs required to instantiate a IdentityClient object.
force_overwrite ((bool, optional). Defaults to False.) – Overwrite existing files.
install_libs (bool, default: False) – Install the libraries specified in ds-requirements.txt.
conflict_strategy (ConflictStrategy, default: IGNORE) – Determines how to handle version conflicts between the current environment and requirements of model artifact. Valid values: “IGNORE”, “UPDATE” or ConflictStrategy. IGNORE: Use the installed version in case of conflict UPDATE: Force update dependency to the version required by model artifact in case of conflict
bucket_uri ((str, optional). Defaults to None.) – The OCI Object Storage URI where model artifacts will be copied to. The bucket_uri is only necessary for downloading large artifacts with size is greater than 2GB. Example: oci://<bucket_name>@<namespace>/prefix/.
remove_existing_artifact ((bool, optional). Defaults to True.) – Whether artifacts uploaded to object storage bucket need to be removed or not.
kwargs –

compartment_id: (str, optional)
Compartment OCID. If not specified, the value will be taken from the environment variables.

timeout: (int, optional). Defaults to 10 seconds.
The connection timeout in seconds for the client.

Returns:

An instance of ModelArtifact class.

Return type:

ModelArtifact

install_requirements(conflict_strategy='IGNORE')

Installs missing libraries listed in the model artifacts.

Parameters:: conflict_strategy (ConflictStrategy, default: IGNORE) – Update the conflicting dependency to the version required by the model artifact. Valid values: “IGNORE” or ConflictStrategy.IGNORE, “UPDATE” or ConflictStrategy.UPDATE. IGNORE: Use the installed version in case of a conflict. UPDATE: Force update dependency to the version required by model artifact in case of conflict.

introspect() → DataFrame

Runs model introspection.

Returns:: The introspection result in a dataframe format.
Return type:: pd.DataFrame

populate_metadata(model=None, use_case_type=None)

Extracts and populate taxonomy metadata from given model.

Parameters:

model ((Any, optional). Defaults to None.) – The model object.
use_case_type ((str, optional). Default to None.) – The use case type of the model.
model – This is an optional model object which is only used to extract taxonomy metadata. Supported models: keras, lightgbm, pytorch, sklearn, tensorflow, and xgboost. If the model is not under supported frameworks, then extracting taxonomy metadata will be skipped.
use_case_type – The use case type of the model.

Returns:

Nothing.

Return type:

None

populate_schema(data_sample: Optional[ADSData] = None, X_sample: Optional[Union[list, tuple, DataFrame, Series, ndarray]] = None, y_sample: Optional[Union[list, tuple, DataFrame, Series, ndarray]] = None, max_col_num: int = 2000)

Populate the input and output schema. If the schema exceeds the limit of 32kb, save as json files to the artifact directory.

Parameters:

data_sample (ADSData) – A sample of the data that will be used to generate input_schema and output_schema.
X_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]) – A sample of input data that will be used to generate input schema.
y_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]) – A sample of output data that will be used to generate output schema.
max_col_num ((int, optional). Defaults to utils.DATA_SCHEMA_MAX_COL_NUM.) – The maximum column size of the data that allows to auto generate schema.

reload(model_file_name: Optional[str] = None)

Reloads files in model artifact directory.

Parameters:: model_file_name (str) – The model file name.

save(display_name: str = None, description: str = None, project_id: str = None, compartment_id: str = None, training_script_path: str = None, ignore_pending_changes: bool = False, auth: dict = None, training_id: str = None, timeout: int = None, ignore_introspection=True, freeform_tags=None, defined_tags=None, bucket_uri: Optional[str] = None, remove_existing_artifact: Optional[bool] = True, model_version_set: Optional[Union[str, ModelVersionSet]] = None, version_label: Optional[str] = None)

Saves the model artifact in the model catalog.

Parameters:

display_name (str, optional) – Model display name.
description (str, optional) – Description for the model.
project_id (str, optional) – Model’s project OCID. If None, the default project OCID config.PROJECT_OCID would be used.
compartment_id (str, optional) – Model’s compartment OCID. If None, the default compartment OCID config.NB_SESSION_COMPARTMENT_OCID would be used.
training_script_path (str, optional) – The training script path is either relative to the working directory, or an absolute path.
ignore_pending_changes (bool, default: False) – If True, ignore uncommitted changes and use the current git HEAD commit for provenance metadata. This argument is used only when the function is called from a script in git managed directory.
auth (dict) – Default is None. Default authetication is set using the ads.set_auth() method. Use the ads.common.auth.api_keys() or ads.common.auth.resource_principal() to create appropriate authentication signer and kwargs required to instantiate a DataScienceClient object.
training_id (str, optional) – The training OCID for the model.
timeout (int, default: 10) – The connection timeout in seconds.
ignore_introspection (bool, optional) – Ignore the result of model introspection . If set to True, the .save() will ignore all model introspection errors.
freeform_tags (dict(str, str), optional) – Freeform tags for the model.
defined_tags (dict(str, dict(str, object)), optional) – Defined tags for the model.
bucket_uri ((str, optional). Defaults to None.) – The OCI Object Storage URI where model artifacts will be copied to. The bucket_uri is only necessary for uploading large artifacts which size is greater than 2GB. Example: oci://<bucket_name>@<namespace>/prefix/
remove_existing_artifact ((bool, optional). Defaults to True.) – Whether artifacts uploaded to object storage bucket need to be removed or not.
model_version_set ((Union[str, ModelVersionSet], optional). Defaults to None.) – The Model version set OCID, or name, or ModelVersionSet instance.
version_label ((str, optional). Defaults to None.) – The model version label.

Examples

>>> from ads.common.model_artifact import ModelArtifact
>>> from ads.config import NB_SESSION_OCID

>>> # Getting auth details.
>>> # If you are using API keys
>>> auth=ads.common.auth.api_keys()

>>> # If you are using resource principal
>>> auth=ads.common.auth.resource_principal()

>>> # If you have set the auth type using ads.set_auth()
>>> auth=ads.common.auth.default_signer()

>>> # Preparing model artifacts
>>> model_artifact = prepare_generic_model(
...     "path_to_model_artifacts",
...     force_overwrite=True,
...     data_science_env=True,
...     model=gamma_reg_model,
... )

>>> # Saving model to the model catalog
>>> model_artifact.save(
...     project_id=PROJECT_ID,
...     compartment_id=COMPARTMENT,
...     display_name="RF Classifier 2",
...     description="A sample Random Forest classifier",
...     ignore_pending_changes=True,
...     auth=auth,
...     training_id=NB_SESSION_OCID,
...     timeout=6000,
...     ignore_introspection = True
... )

verify(input_data)

Verifies the contents of the model artifact directory.

Parameters:

input_data (str, dict, BytesIO stream) – Data to be passed into the deployed model. It can be of type json (str), a dict object, or a BytesIO stream. All types get converted into a UTF-8 encoded BytesIO stream and is then sent to the handler. Any data handling past there is done in func.py. By default it looks for data under the keyword “input”, and returns data under teh keyword “prediction”.

Returns:

output_data (the resulting prediction, formatted in the same way as input_data)
Example – ——– input_dict = {“input”: train.X[:3].to_dict()} model_artifact.verify(input_dict)
- returns {“prediction”: [30/4, 24.8, 30.7]} *

class ads.common.model_artifact.PACK_TYPE(value)

Bases: Enum

An enumeration.

SERVICE_PACK = 'data_science'

USER_CUSTOM_PACK = 'published'

class ads.common.model_artifact.VersionConflictWarning(version_conflicts): Bases: object

ads.common.model_artifact.execute(cmd)

ads.common.model_artifact.pip_install(package, options='-U')

ads.common.model_export_util module

ads.common.model_export_util.prepare_generic_model(model_path: str, fn_artifact_files_included: bool = False, fn_name: str = 'model_api', force_overwrite: bool = False, model: Any = None, data_sample: ADSData = None, use_case_type=None, X_sample: Union[list, tuple, Series, ndarray, DataFrame] = None, y_sample: Union[list, tuple, Series, ndarray, DataFrame] = None, **kwargs) → ModelArtifact

Generates template files to aid model deployment. The model could be accompanied by other artifacts all of which can be dumped at model_path. Following files are generated: * func.yaml * func.py * requirements.txt * score.py

Parameters:

model_path (str) – Path where the artifacts must be saved. The serialized model object and any other associated files/objects must be saved in the model_path directory
fn_artifact_files_included (bool) – Default is False, if turned off, function artifacts are not generated.
fn_name (str) – Opional parameter to specify the function name
force_overwrite (bool) – Opional parameter to specify if the model_artifact should overwrite the existing model_path (if it exists)
model ((Any, optional). Defaults to None.) – This is an optional model object which is only used to extract taxonomy metadata. Supported models: automl, keras, lightgbm, pytorch, sklearn, tensorflow, and xgboost. If the model is not under supported frameworks, then extracting taxonomy metadata will be skipped. The alternative way is using atifact.populate_metadata(model=model, usecase_type=UseCaseType.REGRESSION).
data_sample (ADSData) – A sample of the test data that will be provided to predict() API of scoring script Used to generate schema_input and schema_output
use_case_type (str) – The use case type of the model
X_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame, dask.dataframe.core.Series, dask.dataframe.core.DataFrame]) – A sample of input data that will be provided to predict() API of scoring script Used to generate input schema.
y_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame, dask.dataframe.core.Series, dask.dataframe.core.DataFrame]) – A sample of output data that is expected to be returned by predict() API of scoring script, corresponding to X_sample Used to generate output schema.
**kwargs –
________ –
data_science_env (bool, default: False) – If set to True, the datascience environment represented by the slug in the training conda environment will be used.
inference_conda_env (str, default: None) – Conda environment to use within the model deployment service for inferencing. For example, oci://bucketname@namespace/path/to/conda/env
ignore_deployment_error (bool, default: False) – If set to True, the prepare method will ignore all the errors that may impact model deployment.
underlying_model (str, default: 'UNKNOWN') – Underlying Model Type, could be “automl”, “sklearn”, “h2o”, “lightgbm”, “xgboost”, “torch”, “mxnet”, “tensorflow”, “keras”, “pyod” and etc.
model_libs (dict, default: {}) – Model required libraries where the key is the library names and the value is the library versions. For example, {numpy: 1.21.1}.
progress (int, default: None) – max number of progress.
inference_python_version (str, default:None.) – If provided will be added to the generated runtime yaml
max_col_num ((int, optional). Defaults to utils.DATA_SCHEMA_MAX_COL_NUM.) – The maximum column size of the data that allows to auto generate schema.

Examples

>>> import cloudpickle
>>> import os
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.datasets import make_classification
>>> import ads
>>> from ads.common.model_export_util import prepare_generic_model
>>> import yaml
>>> import oci
>>>
>>> ads.set_auth('api_key', oci_config_location=oci.config.DEFAULT_LOCATION, profile='DEFAULT')
>>> model_artifact_location = os.path.expanduser('~/myusecase/model/')
>>> inference_conda_env="oci://my-bucket@namespace/conda_environments/cpu/Data_Exploration_and_Manipulation_for_CPU_Python_3.7/2.0/dataexpl_p37_cpu_v2"
>>> inference_python_version = "3.7"
>>> if not os.path.exists(model_artifact_location):
...     os.makedirs(model_artifact_location)
>>> X, y = make_classification(n_samples=100, n_features=20, n_classes=2)
>>> lrmodel = LogisticRegression().fit(X, y)
>>> with open(os.path.join(model_artifact_location, 'model.pkl'), "wb") as mfile:
...     cloudpickle.dump(lrmodel, mfile)
>>> modelartifact = prepare_generic_model(
...     model_artifact_location,
...     model = lrmodel,
...     force_overwrite=True,
...     inference_conda_env=inference_conda_env,
...     ignore_deployment_error=True,
...     inference_python_version=inference_python_version
... )
>>> modelartifact.reload() # Call reload to update the ModelArtifact object with the generated score.py
>>> assert len(modelartifact.predict(X[:5])['prediction']) == 5 #Test the generated score.py works. This may require customization.
>>> with open(os.path.join(model_artifact_location, "runtime.yaml")) as rf:
...     content = yaml.load(rf, Loader=yaml.FullLoader)
...     assert content['MODEL_DEPLOYMENT']['INFERENCE_CONDA_ENV']['INFERENCE_ENV_PATH'] == inference_conda_env
...     assert content['MODEL_DEPLOYMENT']['INFERENCE_CONDA_ENV']['INFERENCE_PYTHON_VERSION'] == inference_python_version
>>> # Save Model to model artifact
>>> ocimodel = modelartifact.save(
...     project_id="oci1......", # OCID of the project to which the model to be associated
...     compartment_id="oci1......", # OCID of the compartment where the model will reside
...     display_name="LRModel_01",
...     description="My Logistic Regression Model",
...     ignore_pending_changes=True,
...     timeout=100,
...     ignore_introspection=True,
... )
>>> print(f"The OCID of the model is: {ocimodel.id}")

Returns:: model_artifact – A generic model artifact
Return type:: ads.model_artifact.model_artifact

ads.common.model_export_util.serialize_model(model=None, target_dir=None, X=None, y=None, model_type=None, **kwargs)

Parameters:

model (ads.Model) – A model to be serialized
target_dir (str, optional) – directory to output the serialized model
X (Union[pandas.DataFrame, pandas.Series]) – The X data
y (Union[list, pandas.DataFrame, pandas.Series]) – Tbe Y data
model_type (str, optional) – A string corresponding to the model type

Returns:

model_kwargs – A dictionary of model kwargs for the serialized model

Return type:

Dict

ads.common.model_introspect module

The module that helps to minimize the number of errors of the model post-deployment process. The model provides a simple testing harness to ensure that model artifacts are thoroughly tested before being saved to the model catalog.

Classes

ModelIntrospect: Class to introspect model artifacts.

Examples

>>> model_introspect = ModelIntrospect(artifact=model_artifact)
>>> model_introspect()
... Test key         Test name            Result              Message
... ----------------------------------------------------------------------------
... test_key_1       test_name_1          Passed              test passed
... test_key_2       test_name_2          Not passed          some error occured
>>> model_introspect.status
... Passed

class ads.common.model_introspect.Introspectable

Bases: ABC

Base class that represents an introspectable object.

exception ads.common.model_introspect.IntrospectionNotPassed: Bases: ValueError

class ads.common.model_introspect.ModelIntrospect(artifact: Introspectable)

Bases: object

Class to introspect model artifacts.

Parameters:

status (str) – Returns the current status of model introspection. The possible variants: Passed, Not passed, Not tested.
failures (int) – Returns the number of failures of introspection result.

run(self) → None: Invokes model artifacts introspection.

to_dataframe(self) → pd.DataFrame: Serializes model introspection result into a DataFrame.

Examples

>>> model_introspect = ModelIntrospect(artifact=model_artifact)
>>> result = model_introspect()
... Test key         Test name            Result              Message
... ----------------------------------------------------------------------------
... test_key_1       test_name_1          Passed              test passed
... test_key_2       test_name_2          Not passed          some error occured

Initializes the Model Introspect.

Parameters:

artifact (Introspectable) – The instance of ModelArtifact object.

Raises:

ValueError – If model artifact object not provided.:
TypeError – If provided input paramater not a ModelArtifact instance.:

property failures: int

Calculates the number of failures.

Returns:: The number of failures.
Return type:: int

run() → DataFrame

Invokes introspection.

Returns:: The introspection result in a DataFrame format.
Return type:: pd.DataFrame

property status: str: Gets the current status of model introspection.

to_dataframe() → DataFrame

Serializes model introspection result into a DataFrame.

Returns:: The model introspection result in a DataFrame representation.
Return type:: pandas.DataFrame

class ads.common.model_introspect.PrintItem(key: str = '', case: str = '', result: str = '', message: str = '')

Bases: object

Class represents the model introspection print item.

case: str = ''

key: str = ''

message: str = ''

result: str = ''

to_list() → List[str]

Converts instance to a list representation.

Returns:: The instance in a list representation.
Return type:: List[str]

class ads.common.model_introspect.TEST_STATUS

Bases: str

NOT_PASSED = 'Failed'

NOT_TESTED = 'Skipped'

PASSED = 'Passed'

ads.common.model_metadata module

The module created for the back compatability. The original model_metadata was moved to the ads.model package.

ads.common.model_metadata_mixin module

class ads.common.model_metadata_mixin.MetadataMixin

Bases: object

MetadataMixin class which populates the custom metadata, taxonomy metadata, input/output schema and provenance metadata.

populate_metadata(use_case_type: Optional[str] = None, data_sample: Optional[ADSData] = None, X_sample: Optional[Union[list, tuple, DataFrame, Series, ndarray]] = None, y_sample: Optional[Union[list, tuple, DataFrame, Series, ndarray]] = None, training_script_path: Optional[str] = None, training_id: Optional[str] = None, ignore_pending_changes: bool = True, max_col_num: int = 2000)

Populates input schema and output schema. If the schema exceeds the limit of 32kb, save as json files to the artifact directory.

Parameters:

use_case_type ((str, optional). Defaults to None.) – The use case type of the model.
data_sample ((ADSData, optional). Defaults to None.) – A sample of the data that will be used to generate intput_schema and output_schema.
X_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]. Defaults to None.) – A sample of input data that will be used to generate input schema.
y_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]. Defaults to None.) – A sample of output data that will be used to generate output schema.
training_script_path (str. Defaults to None.) – Training script path.
training_id ((str, optional). Defaults to None.) – The training model OCID.
ignore_pending_changes (bool. Defaults to False.) – Ignore the pending changes in git.
max_col_num ((int, optional). Defaults to utils.DATA_SCHEMA_MAX_COL_NUM.) – The maximum number of columns allowed in auto generated schema.

Returns:

Nothing.

Return type:

None

populate_schema(data_sample: Optional[ADSData] = None, X_sample: Optional[Union[List, Tuple, DataFrame, Series, ndarray]] = None, y_sample: Optional[Union[List, Tuple, DataFrame, Series, ndarray]] = None, max_col_num: int = 2000)

Populate input and output schemas. If the schema exceeds the limit of 32kb, save as json files to the artifact dir.

Parameters:

data_sample (ADSData) – A sample of the data that will be used to generate input_schema and output_schema.
X_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]) – A sample of input data that will be used to generate the input schema.
y_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]) – A sample of output data that will be used to generate the output schema.
max_col_num ((int, optional). Defaults to utils.DATA_SCHEMA_MAX_COL_NUM.) – The maximum number of columns allowed in auto generated schema.

ads.common.object_storage_details module

exception ads.common.object_storage_details.InvalidObjectStoragePath

Bases: Exception

Invalid Object Storage Path.

class ads.common.object_storage_details.ObjectStorageDetails(bucket: str, namespace: str = '', filepath: str = '', auth: Optional[Dict] = None)

Bases: object

Class that represents the Object Storage bucket URI details.

bucket

The Object Storage bucket name.

Type:: str

namespace

The Object Storage namespace. Will be extracted automatically if not provided.

Type:: (str, optional). Defaults to empty string.

filepath

The path to the object.

Type:: (str, optional). Defaults to empty string.

auth

ADS auth dictionary for OCI authentication. This can be generated by calling ads.common.auth.api_keys() or ads.common.auth.resource_principal() If this is None, ads.common.default_signer() will be used.

Type:: (Dict, optional). Defaults to None.

auth: Dict = None

bucket: str

fetch_metadata_of_object() → Dict

Fetches the manifest metadata from the object storage of a conda pack.

Returns:: The metadata in dictionary format.
Return type:: Dict

filepath: str = ''

classmethod from_path(env_path: str) → ObjectStorageDetails

Construct an ObjectStorageDetails instance from conda pack path.

Parameters:: env_path ((str)) – codna pack object storage path.
Raises:: Exception – OCI conda url path not properly configured.:
Returns:: An ObjectStorageDetails instance.
Return type:: ObjectStorageDetails

static is_valid_uri(uri: str) → bool: Validates the Object Storage URI.

namespace: str = ''

property path: Full object storage path of this file.

to_tuple(): Returns the values of the fields of ObjectStorageDetails class.

ads.common.oci_client module

class ads.common.oci_client.OCIClientFactory(config={}, signer=None, client_kwargs=None)

Bases: object

A factory class to create OCI client objects. The constructor takes in config, signer and client_kwargs. client_kwargs is passed to the client constructor as key word argutments.

Examples

from ads.common import auth as authutil from ads.common import oci_client as oc

auth = authutil.default_signer() oc.OCIClientFactory(**auth).object_storage # Creates Object storage client

auth = authutil.default_signer({“timeout”: 6000}) oc.OCIClientFactory(**auth).object_storage # Creates Object storage client with timeout set to 6000

auth = authutil.api_keys(config=”/home/datascience/.oci/config”, profile=”TEST”, {“timeout”: 6000}) oc.OCIClientFactory(**auth).object_storage # Creates Object storage client with timeout set to 6000 using API Key authentication

auth = authutil.resource_principal({“timeout”: 6000}) oc.OCIClientFactory(**auth).object_storage # Creates Object storage client with timeout set to 6000 using resource principal authentication

auth = authutil.create_signer(“instance_principal”) oc.OCIClientFactory(**auth).object_storage # Creates Object storage client using instance principal authentication

property ai_language

create_client(client_name)

property data_labeling_cp

property data_labeling_dp

property data_science

property dataflow

property identity

property object_storage

property secret

property vault

ads.common.oci_datascience module

class ads.common.oci_datascience.DSCNotebookSession(config: Optional[dict] = None, signer: Optional[Signer] = None, client_kwargs: Optional[dict] = None, **kwargs)

Bases: OCIDataScienceMixin, NotebookSession

Represents a data science notebook session

To get the information of an existing notebook session: >>> notebook = DSCNotebookSession.from_ocid(NOTEBOOK_OCID) Get the name of the notebook session >>> notebook.display_name Get the subnet ID of the notebook session >>> notebook.notebook_session_configuration_details.subnet_id

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

config (dict, optional) – OCI API key config dictionary, by default None.
signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.
client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

class ads.common.oci_datascience.OCIDataScienceMixin(config: Optional[dict] = None, signer: Optional[Signer] = None, client_kwargs: Optional[dict] = None, **kwargs)

Bases: OCIModelMixin

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

config (dict, optional) – OCI API key config dictionary, by default None.
signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.
client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

property client: DataScienceClient: OCI client

property client_composite: DataScienceClientCompositeOperations

classmethod init_client(**kwargs) → DataScienceClient

Initializes the OCI client specified in the “client” keyword argument Sub-class should override this method and call cls._init_client(client=OCI_CLIENT)

Parameters:: **kwargs – Additional keyword arguments for initializing the OCI client.
Return type:: An instance of OCI client.

ads.common.oci_logging module

class ads.common.oci_logging.ConsolidatedLog(*args)

Bases: object

Represents the Consolidated OCI Log resource.

Usage: Initialize consolidated log instance for oci_log_one and oci_log_two >>> oci_log_one = OCILog( >>> compartment_id=<compartment_id_one>, >>> id=<id_one>, >>> log_group_id=<log_group_id_one>, >>> annotation=<annotation_one> >>> ) >>> oci_log_two = OCILog( >>> compartment_id=<compartment_id_two>, >>> id=<id_two>, >>> log_group_id=<log_group_id_two>, >>> annotation=<annotation_two> >>> ) >>> consolidated_log = ConsolidatedLog(oci_log_one, oci_log_two) Stream, sort and annotate the logs from oci_log_one and oci_log_two >>> consolidated_log.stream() Get the most recent 20 consolidated logs from oci_log_one and oci_log_two >>> consolidated_log.tail(limit=20) Get the most recent 20 raw logs from oci_log_one and oci_log_two >>> consolidated_log.search(limit=20)

Initializes a consolidate log model instance.

Parameters:: args – A list of OCILog instance.

head(source: Optional[str] = None, limit: int = 100, time_start: Optional[datetime] = None) → None

Returns the preceding consolidated log records.

Parameters:

source (str, optional) – Expression or OCID to filter the “source” field of the OCI log record. Defaults to None.
limit (int, optional.) – Maximum number of records to be returned. If limit is set to None, all logs from time_start to now will be returned. Defaults to 100.
time_start (datetime.datetime, optional) – Starting time for the log query. Defaults to None.

search(source: Optional[str] = None, time_start: Optional[datetime] = None, time_end: Optional[datetime] = None, limit: Optional[int] = None, sort_by: str = 'datetime', sort_order: str = 'DESC', log_filter: Optional[str] = None) → List[SearchResult]

Searches raw logs.

Parameters:

source (str, optional) – Expression or OCID to filter the “source” field of the OCI log record. Defaults to None.
time_start (datetime.datetime, optional) – Starting time for the log query. Defaults to None.
time_end (datetime.datetime, optional.) – Ending time for the query. Defaults to None.
limit (int, optional.) – Maximum number of records to be returned. All logs will be returned. Defaults to None.
sort_by (str, optional.) – The field for sorting the logs. Defaults to “datetime”
sort_order (str, optional.) – The sort order for the log records. Can be “ASC” or “DESC”. Defaults to “DESC”.
log_filter (str, optional) – Expression for filtering the logs. This will be the WHERE clause of the query. Defaults to None.

Returns:

A list of oci.loggingsearch.models.SearchResult objects

Return type:

list

stream(source: Optional[str] = None, interval: int = 3, stop_condition: Optional[callable] = None, time_start: Optional[datetime] = None, log_filter: Optional[str] = None)

Streams consolidated logs to console/terminal until stop_condition() returns true.

Parameters:

source (str, optional) – Expression or OCID to filter the “source” field of the OCI log record. Defaults to None.
interval (int, optional) – The time interval between sending each request to pull logs from OCI logging service. Defaults to 3.
stop_condition (callable, optional) – A function to determine if the streaming should stop. The log streaming will stop if the function returns true. Defaults to None.
time_start (datetime.datetime, optional) – Starting time for the log query. Defaults to None.
log_filter (str, optional) – Expression for filtering the logs. This will be the WHERE clause of the query. Defaults to None.

tail(source: Optional[str] = None, limit: int = 100, time_start: Optional[datetime] = None, log_filter: Optional[str] = None) → None

Returns the most recent consolidated log records.

Parameters:

source (str, optional) – Expression or OCID to filter the “source” field of the OCI log record. Defaults to None.
limit (int, optional.) – Maximum number of records to be returned. If limit is set to None, all logs from time_start to now will be returned. Defaults to 100.
time_start (datetime.datetime, optional) – Starting time for the log query. Defaults to None.
log_filter (str, optional) – Expression for filtering the logs. This will be the WHERE clause of the query. Defaults to None.

class ads.common.oci_logging.OCILog(log_type: str = 'CUSTOM', **kwargs)

Bases: OCILoggingModelMixin, Log

Represents the OCI Log resource.

Usage: (OCI requires display_name to be unique and it cannot contain space) >>> log = OCILog(display_name=”My_Log”, log_group_id=LOG_GROUP_ID).create() Usually it is better to create a log using the create_log() method in OCILogGroup. >>> log.delete() # Delete the resource Get a log object from OCID >>> oci_log = OCILog.from_ocid(“LOG_OCID_HERE”) Stream the logs from an OCI Data Science Job run to stdout >>> oci_log.stream(source=”JOB_RUN_OCID_HERE”) Gets the most recent 10 logs >>> oci_log.tail(10)

Initializes an OCI log model locally. The resource is not created in OCI until the create() or create_async() method is called.

create_async(): Creates a new Log with OCI logging service

delete(): Delete the log

static format_datetime(dt: datetime) → str

Converts datetime object to RFC3339 date time format in string

Parameters:: dt (datetime.datetime) – Datetime object to be formated.
Returns:: A timestamp in RFC3339 format.
Return type:: str

head(source: Optional[str] = None, limit=100, time_start: Optional[datetime] = None) → List[dict]

Returns the preceding log records.

Parameters:

source ((str, optional). Defaults to None.) – The source field to filter the log records.
limit ((int, optional). Defaults to 100.) – Maximum number of records to be returned. If limit is set to None, all logs from time_start to now will be returned.
time_start ((datetime.datetime, optional)) – Starting time for the log query. Defaults to None. Logs up to 14 days from now will be returned.

Returns:

A list of log records. Each log record is a dictionary with the following keys: id, time, message.

Return type:

list

search(source: Optional[str] = None, time_start: Optional[datetime] = None, time_end: Optional[datetime] = None, limit: Optional[int] = None, sort_by: str = 'datetime', sort_order: str = 'DESC', log_filter: Optional[str] = None) → List[SearchResult]

Search logs

Parameters:

source (str, optional) – Expression or OCID to filter the “source” field of the OCI log record. Defaults to None. No filtering will be performed.
time_start (datetime.datetime, optional) – Starting UTC time for the query. Defaults to None. Logs from the past 24 hours will be returned.
time_end (datetime.datetime, optional) – Ending UTC time for the query. Defaults to None. The current time will be used.
limit (int, optional) – Maximum number of records to be returned. Defaults to None. All logs will be returned.
sort_by (str, optional) – The field for sorting the logs. Defaults to “datetime”
sort_order (str, optional) – Specify how the records should be sorted. Must be “ASC” or “DESC”. Defaults to “DESC”.
log_filter (str, optional) – Expression for filtering the logs. This will be the WHERE clause of the query. Defaults to None.

Returns:

A list of SearchResult objects

Return type:

List[oci.loggingsearch.models.SearchResult]

property search_client: OCI logging search client.

stream(source: Optional[str] = None, interval: int = 3, stop_condition: Optional[callable] = None, time_start: Optional[datetime] = None, log_filter: Optional[str] = None)

Streams logs to console/terminal until stop_condition() returns true.

Parameters:

source (str) –
interval (int) – The time interval between sending each request to pull logs from OCI logging service (Default value = 3)
stop_condition (callable) – A function to determine if the streaming should stop. (Default value = None) The log streaming will stop if the function returns true.
time_start (datetime.datetime) – Starting time for the log query. Defaults to None. Logs up to 14 days from now will be returned.
log_filter (str, optional) – Expression for filtering the logs. This will be the WHERE clause of the query. Defaults to None.

Returns:

The number of logs printed.

Return type:

int

sync(**kwargs) → None

Refreshes the properties of the Log model OCI requires both Log OCID and Log group OCID to get the Log model.

This method override the sync() method from OCIMixin to improve performance.

tail(source: Optional[str] = None, limit=100, time_start: Optional[datetime] = None, log_filter: Optional[str] = None) → List[dict]

Returns the most recent log records.

Parameters:

source ((str, optional). Defaults to None.) – The source field to filter the log records.
limit ((int, optional). Defaults to 100.) – Maximum number of records to be returned. If limit is set to None, all logs from time_start to now will be returned.
time_start ((datetime.datetime, optional)) – Starting time for the log query. Defaults to None. Logs up to 14 days from now will be returned.
log_filter ((str, optional). Defaults to None.) – Expression for filtering the logs. This will be the WHERE clause of the query.

Returns:

A list of log records. Each log record is a dictionary with the following keys: id, time, message.

Return type:

list

class ads.common.oci_logging.OCILogGroup(config: Optional[dict] = None, signer: Optional[Signer] = None, client_kwargs: Optional[dict] = None, **kwargs)

Bases: OCILoggingModelMixin, LogGroup

Represents the OCI Log Group resource.

Using OCILogGroup to create a new log group. OCI requires display_name to be unique and it cannot contain space. >>> log_group = OCILogGroup(display_name=”My_Log_Group”).create() Once created, access the OCID and other properties >>> log_group.id # The OCID is None before the log is created. >>> None Create a log resource within the log group >>> log_group.id # OCID will be available once the log group is created Access the property >>> log_group.display_name Create logs within the log group >>> log = log_group.create_log(“My custom access log”) >>> log_group.create_log(“My custom prediction log”) List the logs in a log group. The following line will return a list of OCILog objects. >>> logs = log_group.list_logs() Delete the resource >>> log_group.delete()

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

config (dict, optional) – OCI API key config dictionary, by default None.
signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.
client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

create_async(): Creates a new LogGroup asynchronously with OCI logging service

create_log(display_name: str, **kwargs)

Create a log (OCI resource) within the log group.

Parameters:

display_name (str) – The display name of the log
**kwargs – Keyword arguments to be passed into the OCI API for log properties.

Returns:

An instance of OCILog

Return type:

OCILog

delete(): Deletes the log group and the logs in the log group.

list_logs(**kwargs) → list

Lists all logs within the log group.

Parameters:: **kwargs – keyword arguments for filtering the results. They are passed into oci.logging.LoggingManagementClient.list_logs()
Returns:: A list of OCILog
Return type:: list

class ads.common.oci_logging.OCILoggingModelMixin(config: Optional[dict] = None, signer: Optional[Signer] = None, client_kwargs: Optional[dict] = None, **kwargs)

Bases: OCIModelMixin, OCIWorkRequestMixin

Base model for representing OCI logging resources managed through oci.logging.LoggingManagementClient. This class should not be initialized directly. Use a sub-class (OCILogGroup or OCILog) instead.

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

config (dict, optional) – OCI API key config dictionary, by default None.
signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.
client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

property client: LoggingManagementClient: OCI logging management client

create()

Creates a new resource with OCI service synchronously. This method will wait for the resource creation to be succeeded or failed.

Each sub-class should implement the create_async() method with the corresponding method in OCI SDK: to create the resource.

Raises:

NotImplementedError – when user called create but the create_async() method is not implemented.
oci.exceptions.RequestException – when there is an error creating the resource with OCI request.

create_async(): Creates the OCI logging resource asynchronously. Sub-class should implement this method with OCI Python SDK and return the response from the OCI PythonSDK.

classmethod from_name(display_name: str, **kwargs) → Union[OCILogGroup, OCILog]

Obtain an existing OCI logging resource by using its display name. OCI log group or log resource requires display name to be unique.

Parameters:: display_name (str) – Display name of the logging resource (e.g. log group)
Return type:: An instance of logging resource, e.g. OCILogGroup, or OCILog.

classmethod init_client(**kwargs) → LoggingManagementClient: Initialize OCI client

class ads.common.oci_logging.SortOrder

Bases: object

ASC = 'ASC'

DESC = 'DESC'

ads.common.oci_mixin module

Contains Mixins for integrating OCI data models

class ads.common.oci_mixin.MergeStrategy(value)

Bases: Enum

An enumeration.

MERGE = 'merge'

OVERRIDE = 'override'

class ads.common.oci_mixin.OCIClientMixin(config=None, signer=None, client_kwargs=None)

Bases: object

Mixin class for representing OCI resource/service with OCI client.

Most OCI requests requires a client for making the connection. Usually the same client will be used for the requests related the same resource/service type. This Mixin adds a “client” property to simplify accessing the client. The actual client will be initialize lazily so that it is not required for a sub-class To use the client, sub-class should override the init_client() method pass in the “client” keyword argument. For example:

@class_or_instance_method def init_client(cls, **kwargs) -> oci.logging.LoggingManagementClient:

return cls._init_client(client=oci.logging.LoggingManagementClient, **kwargs)

Instance methods in the sub-class can use self.client to access the client. The init_client() method is a class method used to create the client. Any class method using the client should use init_client() to create the client. The call to this method may come from an instance or a class. When the method is called from a class,

the default authentication configured at ADS level with ads.set_auth() will be used.

When the method is called from an instance,: the config, signer and kwargs specified when initializing the instance will be used.
The sub-class’s __init__ method should take config, signer and client_kwargs as argument,: then pass them to the __init__ method of this class.

This allows users to override the authentication and client initialization parameters. For example, log_group = OCILogGroup(config=config, signer=signer, client_kwargs=kwargs)

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

config (dict, optional) – OCI API key config dictionary, by default None.
signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.
client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

property auth: dict: The ADS authentication config used to initialize the client. This auth has the same format as those obtained by calling functions in ads.common.auth. The config is a dict containing the following key-value pairs: config: The config contains the config loaded from the configuration loaded from oci_config. signer: The signer contains the signer object created from the api keys. client_kwargs: client_kwargs contains the client_kwargs that was passed in as input parameter.

property client: OCI client

config = None

classmethod create_instance(*args, **kwargs): Creates an instance using the same authentication as the class or an existing instance. If this method is called by a class, the default ADS authentication method will be used. If this method is called by an instance, the authentication method set in the instance will be used.

classmethod init_client(**kwargs)

Initializes the OCI client specified in the “client” keyword argument Sub-class should override this method and call cls._init_client(client=OCI_CLIENT)

Parameters:: **kwargs – Additional keyword arguments for initializing the OCI client.
Return type:: An instance of OCI client.

kwargs = None

signer = None

class ads.common.oci_mixin.OCIModelMixin(config: Optional[dict] = None, signer: Optional[Signer] = None, client_kwargs: Optional[dict] = None, **kwargs)

Bases: OCISerializableMixin

Mixin class to operate OCI model. OCI resources are represented by models in the OCI Python SDK.

Unifying OCI models for the same resource

OCI SDK uses different models to represent the same resource for different operations. For example, CreateLogDetails is used when creating a log resource,

while LogSummary is returned when listing the log resources.

However, both CreateLogDetails and LogSummary have the same commonly used attribute like: compartment_id, display_name, log_type, etc.

In general, there is a class with a super set of all properties. For example, the Log class contains all properties of CreateLogDetails and LogSummary,

as well as other classes representing an OCI log resource.

A subclass can be implemented with this Mixin to unify the OCI models,: so that all properties are available to the user.
For example, if we define the Mixin model as class OCILog(OCIModelMixin, oci.logging.models.Log),: users will be able to access properties like OCILog().display_name

Since this sub-class contains all the properties, it can be converted to any related OCI model in an operation. For example, we can create CreateLogDetails from OCILog by extracting a subset of the properties. When listing the resources, properties from LogSummary can be used to update

the corresponding properties of OCILog.

Such convertion can be done be the generic methods provided by this Mixin. Although OCI SDK accepts dictionary (JSON) data instead of objects like CreateLogDetails when creating or

updating the resource, the structure and keys of the dictionary is not easy for a user to remember.

It is also unnecessary for the users to construct the entire dictionary if they only want to update a single value.

This Mixin class should be the first parent as the class from OCI SDK does not call super().__init__(): in its __init__() constructor.

Mixin properties may not be intialized correctly if super().__init__() is not called.

Provide generic methods for CRUDL operations

Since OCI SDK uses different models in CRUDL operations,: this Mixin provides the following method to convert between them.

An OCI model instance can be any OCI model of the resource containing some properties, e.g. LogSummary from_oci_model() static method can be used to initialize a new instance from an OCI model instance. update_from_oci_model() can be used to update the existing properties from an OCI model instance. to_oci_model() can be used to extract properties from the Mixin model to OCI model.

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

param config:: OCI API key config dictionary, by default None.
type config:: dict, optional
param signer:: OCI authentication signer, by default None.
type signer:: oci.signer.Signer, optional
param client_kwargs:: Additional keyword arguments for initializing the OCI client.
type client_kwargs:: dict, optional

CONS_COMPARTMENT_ID = 'compartment_id'

OCI_MODEL_PATTERN = 'oci.[^.]+\\.models[\\..*]?'

static check_compartment_id(compartment_id: Optional[str]) → str

Checks if a compartment ID has value and: return the value from NB_SESSION_COMPARTMENT_OCID environment variable if it is not specified.

Parameters:: compartment_id (str) – Compartment OCID or None
Returns:: str: Compartment OCID
Return type:: type
Raises:: ValueError – compartment_id is not specified and NB_SESSION_COMPARTMENT_OCID environment variable is not set

delete(): Deletes the resource

classmethod deserialize(data: dict, to_cls: Optional[str] = None)

Deserialize data

Parameters:

data (dict) – A dictionary containing the data to be deserialized.
to_cls (str) – The name of the OCI model class to be initialized using the data. The OCI model class must be from the same OCI service of the OCI client (self.client). Defaults to None, the parent OCI model class name will be used if current class is inherited from an OCI model. If parent OCI model class is not found or not from the same OCI service, the data will be returned as is.

static flatten(data: dict) → dict

Flattens a nested dictionary.

Parameters:: data (A nested dictionary) –
Returns:: The flattened dictionary.
Return type:: dict

classmethod from_dict(data)

Initialize an instance from a dictionary.

Parameters:: data (dict) – A dictionary containing the properties to initialize the class.

classmethod from_oci_model(oci_instance)

Initialize an instance from an instance of OCI model.

Parameters:: oci_instance – An instance of an OCI model.

classmethod from_ocid(ocid: str)

Initializes an object from OCID

Parameters:: ocid (str) – The OCID of the object

classmethod list_resource(compartment_id: Optional[str] = None, limit: int = 0, **kwargs) → list

Generic method to list OCI resources

Parameters:

compartment_id (str) – Compartment ID of the OCI resources. Defaults to None. If compartment_id is not specified, the value of NB_SESSION_COMPARTMENT_OCID in environment variable will be used.
limit (int) – The maximum number of items to return. Defaults to 0, All items will be returned
**kwargs – Additional keyword arguments to filter the resource. The kwargs are passed into OCI API.

Returns:

A list of OCI resources

Return type:

list

Raises:

NotImplementedError – List method is not supported or implemented.

load_properties_from_env(): Loads properties from the environment

property name: str: Gets the name of the object.

property status: Optional[str]

Status of the object.

Returns:: Status of the object.
Return type:: str

sync(merge_strategy: MergeStrategy = MergeStrategy.OVERRIDE): Refreshes the properties of the object from OCI

to_dict(flatten: bool = False) → dict

Converts the properties to a dictionary

Parameters:: flatten – (Default value = False)

to_oci_model(oci_model)

Converts the object into an instance of OCI data model.

Parameters:

oci_model (class or str) – The OCI model to be converted to. This can be a string of the model name.
type_mapping (dict) – A dictionary mapping the models. Returns: An instance of the oci_model

to_yaml() → str

Serializes the object into YAML string.

Returns:: YAML stored in a string.
Return type:: str

update_from_oci_model(oci_model_instance, merge_strategy: MergeStrategy = MergeStrategy.OVERRIDE)

Updates the properties from OCI model with the same properties.

Parameters:: oci_model_instance – An instance of OCI model, which should have the same properties of this class.

exception ads.common.oci_mixin.OCIModelNotExists: Bases: Exception

class ads.common.oci_mixin.OCIModelWithNameMixin

Bases: object

Mixin class to operate OCI model which contains name property.

classmethod from_name(name: str, compartment_id: Optional[str] = None)

Initializes an object from name.

Parameters:

name (str) – The name of the object.
compartment_id ((str, optional). Defaults to None.) – Compartment OCID of the OCI resources. If compartment_id is not specified, the value will be taken from environment variables.

class ads.common.oci_mixin.OCISerializableMixin(config=None, signer=None, client_kwargs=None)

Bases: OCIClientMixin

Mixin class containing OCI serialization/de-serialization methods. These methods are copied and modified from the OCI BaseClient.

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

config (dict, optional) – OCI API key config dictionary, by default None.
signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.
client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

classmethod deserialize(data, to_cls): De-serialize data from dictionary to an OCI model

serialize()

Serialize the model to a dictionary that is ready to be send to OCI API.

Returns:: A dictionary that is ready to be send to OCI API.
Return type:: dict

type_mappings = None

class ads.common.oci_mixin.OCIWorkRequestMixin

Bases: object

Mixin class containing methods related to OCI work request

get_work_request_response(response: str, wait_for_state: Union[str, tuple], success_state: str, max_wait_seconds: Optional[int] = None, wait_interval_seconds: Optional[int] = None, error_msg: str = '')

wait_for_work_request(work_request_id: str, wait_for_state: Union[str, tuple], max_wait_seconds: Optional[int] = None, wait_interval_seconds: Optional[int] = None)

Wait for a work request to be completed.

Parameters:

work_request_id (str) – OCI work request ID
wait_for_state (str or tuple) – The state to wait for. Must be a tuple for multiple states.
max_wait_seconds (int) – Max wait seconds for the work request. Defaults to None (Default value from OCI SDK will be used).
wait_interval_seconds (int) – Interval in seconds between each status check. Defaults to None (Default value from OCI SDK will be used).

Returns:

OCI API Response

Return type:

Response

ads.common.oci_resource module

Contains class wrapping OCI resource search service

class ads.common.oci_resource.OCIResource(config=None, signer=None, client_kwargs=None)

Bases: OCIClientMixin

Contains helper methods for getting information from OCIResourceSearch service.

Usage: Find the compartment ID of an OCI resource: >>> OCIResource.get_compartment_id(“YOUR_OCID”) Search for OCI resource matching free text (Similar to the search box in OCI console): >>> OCIResource.search(“YOUR_FREE_TEXT”) Search for OCI resource matching structured text: >>> OCIResource.search(“STRUCTURED_TEXT”, type=”Structured”)

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

config (dict, optional) – OCI API key config dictionary, by default None.
signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.
client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

classmethod get_compartment_id(ocid) → str

Gets the compartment OCID of an OCI resource, given the resource’s OCID.

Parameters:: ocid (str) – OCID of a resource
Returns:: Compartment OCID of the resource
Return type:: str

classmethod init_client(**kwargs) → ResourceSearchClient

Initializes the OCI client specified in the “client” keyword argument Sub-class should override this method and call cls._init_client(client=OCI_CLIENT)

Parameters:: **kwargs – Additional keyword arguments for initializing the OCI client.
Return type:: An instance of OCI client.

classmethod search(query: str, type: str = 'FreeText', config: Optional[dict] = None, tenant_id: Optional[str] = None, limit: int = 500, page: Optional[str] = None, **kwargs) → list

Search OCI resource by free text.

Parameters:

query (str) – The content to search for.
type (str (optional)) – The type of SearchDetails, whether “FreeText” or “Structured”. Defaults to “FreeText”.
config (dict (optional)) – Configuration keys and values as per SDK and Tool Configuration. The from_file() method can be used to load configuration from a file. Alternatively, a dict can be passed. You can validate_config the dict using validate_config(). Defaults to None.
tenant_id (str (optional)) – The tenancy ID, which can be used to specify a different tenancy (for cross-tenancy authorization) when searching for resources in a different tenancy. Defaults to None.
limit (int (optional)) – The maximum number of items to return. The value must be between 1 and 1000. Defaults to 500.
page (str (optional)) – The page at which to start retrieving results.

Returns:

A list of search results

Return type:

list

exception ads.common.oci_resource.ResourceNotFoundError

Bases: Exception

Exception when an OCI resource is not found or user does not have permission to access it. This could mean the resource does not exist, or there is not enough permission to find/access the resource.

class ads.common.oci_resource.SEARCH_TYPE

Bases: str

FREETEXT = 'FreeText'

STRUCTURED = 'Structured'

ads.common.serializer module

class ads.common.serializer.DataClassSerializable

Bases: Serializable

Wrapper class that inherit from Serializable class.

to_dict(self) → dict: Serializes the object into a dictionary.

from_dict(cls, obj_dict) → cls: Returns an instance of the class instantiated from the dictionary provided.

classmethod from_dict(obj_dict: dict, side_effect: Optional[SideEffect] = 'lower') → DataClassSerializable

Returns an instance of the class instantiated by the dictionary provided.

Parameters:

obj_dict ((dict)) – Dictionary representation of the object
side_effect (Optional[SideEffect]) – side effect to take on the dictionary. The side effect can be either convert the dictionary keys to “lower” (SideEffect.CONVERT_KEYS_TO_LOWER.value) or “upper”(SideEffect.CONVERT_KEYS_TO_UPPER.value) cases.

Returns:

A DataClassSerializable instance.

Return type:

DataClassSerializable

to_dict(**kwargs) → Dict

Serializes instance of class into a dictionary

kwargs

side_effect: Optional[SideEffect]: side effect to take on the dictionary. The side effect can be either convert the dictionary keys to “lower” (SideEffect.CONVERT_KEYS_TO_LOWER.value) or “upper”(SideEffect.CONVERT_KEYS_TO_UPPER.value) cases.

returns:: A dictionary.
rtype:: Dict

class ads.common.serializer.Serializable

Bases: ABC

Base class that represents a serializable item.

to_dict(self) → dict: Serializes the object into a dictionary.

from_dict(cls, obj_dict) → cls: Returns an instance of the class instantiated from the dictionary provided.

_write_to_file(s, uri, \*\*kwargs): Write string s into location specified by uri

_read_from_file(uri, \*\*kwargs): Returns contents from location specified by URI

to_json(self, uri=None, \*\*kwargs): Returns object serialized as a JSON string

from_json(cls, json_string=None, uri=None, \*\*kwargs): Creates an object from JSON string provided or from URI location containing JSON string

to_yaml(self, uri=None, \*\*kwargs): Returns object serialized as a YAML string

from_yaml(cls, yaml_string=None, uri=None, \*\*kwargs): Creates an object from YAML string provided or from URI location containing YAML string

from_string(cls, obj_string=None: str, uri=None, \*\*kwargs): Creates an object from string provided or from URI location containing string

abstract classmethod from_dict(obj_dict: dict) → Serializable

Returns an instance of the class instantiated by the dictionary provided.

Parameters:: obj_dict ((dict)) – Dictionary representation of the object.
Returns:: A Serializable instance.
Return type:: Serializable

classmethod from_json(json_string: ~typing.Optional[str] = None, uri: ~typing.Optional[str] = None, decoder: callable = <class 'json.decoder.JSONDecoder'>, **kwargs)

Creates an object from JSON string provided or from URI location containing JSON string

Parameters:

json_string ((string, optional)) – JSON string. Defaults to None.
uri ((string, optional)) – URI location of file containing JSON string. Defaults to None.
decoder ((callable, optional)) – Custom decoder. Defaults to simple JSON decoder.
kwargs –
------ –
storage (keyword arguments to be passed into fsspec.open(). For OCI object) – For other storage connections consider e.g. host, port, username, password, etc.
config="path/to/.oci/config". (this should be) – For other storage connections consider e.g. host, port, username, password, etc.

Raises:

ValueError – Raised if neither string nor uri is provided

Returns:

Returns instance of the class

Return type:

cls

classmethod from_string(obj_string: ~typing.Optional[str] = None, uri: ~typing.Optional[str] = None, loader: callable = <class 'yaml.cyaml.CSafeLoader'>, **kwargs) → Serializable

Creates an object from string provided or from URI location containing string

Parameters:

obj_string ((str, optional)) – String representing the object
uri ((str, optional)) – URI location of file containing string. Defaults to None.
loader ((callable, optional)) – Custom YAML loader. Defaults to CLoader/SafeLoader.
kwargs ((dict)) – keyword arguments to be passed into fsspec.open(). For OCI object storage, this should be config=”path/to/.oci/config”. For other storage connections consider e.g. host, port, username, password, etc.

Returns:

A Serializable instance

Return type:

Serializable

classmethod from_yaml(yaml_string: ~typing.Optional[str] = None, uri: ~typing.Optional[str] = None, loader: callable = <class 'yaml.cyaml.CSafeLoader'>, **kwargs)

Creates an object from YAML string provided or from URI location containing YAML string

Parameters:

(string (uri) –
optional) (Custom YAML loader. Defaults to CLoader/SafeLoader.) –
(string –
optional) –
(callable (loader) –
optional) –
(dict) (kwargs) – For other storage connections consider e.g. host, port, username, password, etc.

Raises:

ValueError – Raised if neither string nor uri is provided

Returns:

Returns instance of the class

Return type:

cls

abstract to_dict(**kwargs) → dict

Serializes instance of class into a dictionary.

Returns:: A dictionary.
Return type:: Dict

to_json(uri: ~typing.Optional[str] = None, encoder: callable = <class 'json.encoder.JSONEncoder'>, **kwargs) → str

Returns object serialized as a JSON string

Parameters:

uri ((string, optional)) – URI location to save the JSON string. Defaults to None.
encoder ((callable, optional)) – Encoder for custom data structures. Defaults to JSONEncoder.
kwargs –
------ –
storage (keyword arguments to be passed into fsspec.open(). For OCI object) – For other storage connections consider e.g. host, port, username, password, etc.
config="path/to/.oci/config". (this should be) – For other storage connections consider e.g. host, port, username, password, etc.
Returns – string: Serialized version of object

to_yaml(uri: ~typing.Optional[str] = None, dumper: callable = <class 'yaml.cyaml.CSafeDumper'>, **kwargs) → str

Returns object serialized as a YAML string

Parameters:

uri ((string, optional)) – URI location to save the YAML string. Defaults to None.
dumper ((callable, optional)) – Custom YAML Dumper. Defaults to CDumper/SafeDumper.
kwargs –
------ –
side_effect (Optional[SideEffect]) – side effect to take on the dictionary. The side effect can be either convert the dictionary keys to “lower” (SideEffect.CONVERT_KEYS_TO_LOWER.value) or “upper”(SideEffect.CONVERT_KEYS_TO_UPPER.value) cases.
storage (keyword arguments to be passed into fsspec.open(). For OCI object) – For other storage connections consider e.g. host, port, username, password, etc.
config="path/to/.oci/config". (this should be) – For other storage connections consider e.g. host, port, username, password, etc.
Returns –

Union[str, None]
Serialized version of object. None in case when uri provided.

class ads.common.serializer.SideEffect(value)

Bases: Enum

An enumeration.

CONVERT_KEYS_TO_LOWER = 'lower'

CONVERT_KEYS_TO_UPPER = 'upper'

ads.common.utils module

exception ads.common.utils.FileOverwriteError: Bases: Exception

class ads.common.utils.JsonConverter(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: JSONEncoder

Constructor for JSONEncoder, with sensible defaults.

If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, float or None. If skipkeys is True, such items are simply skipped.

If ensure_ascii is true, the output is guaranteed to be str objects with all incoming non-ASCII characters escaped. If ensure_ascii is false, the output can contain non-ASCII characters.

If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an OverflowError). Otherwise, no such check takes place.

If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.

If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.

If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation.

If specified, separators should be an (item_separator, key_separator) tuple. The default is (’, ‘, ‘: ‘) if indent is None and (‘,’, ‘: ‘) otherwise. To get the most compact JSON representation, you should specify (‘,’, ‘:’) to eliminate whitespace.

If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a TypeError.

default(obj)

Converts an object to JSON based on its type

Parameters:: obj (Object) – An object which is being converted to Json, supported types are pandas Timestamp, series, dataframe, or categorical or numpy ndarrays.
Returns:: Json
Return type:: A json repersentation of the object.

ads.common.utils.batch_convert_case(spec: dict, to_fmt: str) → Dict

Convert the case of a dictionary of spec from camel to snake or vice versa.

Parameters:

spec (dict) – dictionary of spec to convert
to_fmt (str) – format to convert to, can be “camel” or “snake”

Returns:

dictionary of converted spec

Return type:

dict

ads.common.utils.camel_to_snake(name: str) → str

Converts the camel case string to the snake representation.

Parameters:: name (str) – The name to convert.
Returns:: str
Return type:: The name converted to the snake representation.

ads.common.utils.copy_file(uri_src: str, uri_dst: str, force_overwrite: Optional[bool] = False, auth: Optional[Dict] = None, chunk_size: Optional[int] = 8192, progressbar_description: Optional[str] = 'Copying `{uri_src}` to `{uri_dst}`') → str

Copies file from uri_src to uri_dst. If uri_dst specifies a directory, the file will be copied into uri_dst using the base filename from uri_src. Returns the path to the newly created file.

Parameters:

uri_src (str) – The URI of the source file, which can be local path or OCI object storage URI.
uri_dst (str) – The URI of the destination file, which can be local path or OCI object storage URI.
force_overwrite ((bool, optional). Defaults to False.) – Whether to overwrite existing files or not.
auth ((Dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
chunk_size ((int, optinal). Defaults to DEFAULT_BUFFER_SIZE) – How much data can be copied in one iteration.

Returns:

The path to the newly created file.

Return type:

str

Raises:

FileExistsError – If a destination file exists and force_overwrite set to False.

ads.common.utils.copy_from_uri(uri: str, to_path: str, unpack: Optional[bool] = False, force_overwrite: Optional[bool] = False, auth: Optional[Dict] = None) → None

Copies file(s) to local path. Can be a folder, archived folder or a separate file. The source files can be located in a local folder or in OCI Object Storage.

Parameters:

uri (str) – The URI of the source file or directory, which can be local path or OCI object storage URI.
to_path (str) – The local destination path. If this is a directory, the source files will be placed under it.
unpack ((bool, optional). Defaults to False.) – Indicate if zip or tar.gz file specified by the uri should be unpacked. This option has no effect on other files.
force_overwrite ((bool, optional). Defaults to False.) – Whether to overwrite existing files or not.
auth ((Dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

Nothing

Return type:

None

Raises:

ValueError – If destination path is already exist and force_overwrite is set to False.

ads.common.utils.download_from_web(url: str, to_path: str) → None

Downloads a single file from http/https/ftp.

Parameters:

url (str) – The URL of the source file.
to_path (path-like object) – Local destination path.

Returns:

Nothing

Return type:

None

ads.common.utils.ellipsis_strings(raw, n=24): takes a sequence (<string>, list(<string>), tuple(<string>), pd.Series(<string>) and Ellipsis’ize them at position n

ads.common.utils.extract_lib_dependencies_from_model(model) → dict

Extract a dictionary of library dependencies for a model

Parameters:: model –
Returns:: Dict
Return type:: A dictionary of library dependencies.

ads.common.utils.extract_region(auth: Optional[Dict] = None) → Optional[str]

Extracts region information from the environment variables and signer.

Parameters:: auth (Dict) – The ADS authentication config used to initialize the client. Contains keys - config, signer and client_kwargs.
Returns:: The region identifier. For example: us-ashburn-1. Returns None if region cannot be extracted.
Return type:: Union[str, None]

ads.common.utils.first_not_none(itr): Returns the first non-none result from an iterable, similar to any() but return value not true/false

ads.common.utils.flatten(d, parent_key='')

Flattens nested dictionaries to a single layer dictionary

Parameters:

d (dict) – The dictionary that needs to be flattened
parent_key (str) – Keys in the dictionary that are nested

Returns:

a_dict – a single layer dictionary

Return type:

dict

ads.common.utils.folder_size(path: str) → int

Recursively calculating a size of the path folder.

Parameters:: path (str) – Path to the folder.
Returns:: The size fo the folder in bytes.
Return type:: int

ads.common.utils.generate_requirement_file(requirements: dict, file_path: str, file_name: str = 'requirements.txt')

Generate requirements file at file_path.

Parameters:

requirements (dict) – Key is the library name and value is the version
file_path (str) – Directory to save requirements.txt
file_name (str) – Opional parameter to specify the file name

ads.common.utils.get_base_modules(model): Get the base modules from an ADS model

ads.common.utils.get_bootstrap_styles(): Returns HTML bootstrap style information

ads.common.utils.get_compute_accelerator_ncores()

ads.common.utils.get_cpu_count(): Returns the number of CPUs available on this machine

ads.common.utils.get_dataframe_styles(max_width=75)

Styles used for dataframe, example usage:

df.style .set_table_styles(utils.get_dataframe_styles()) .set_table_attributes(‘class=table’) .render())

Returns:: styles – A list of dataframe table styler styles.
Return type:: array

ads.common.utils.get_files(directory: str)

List out all the file names under this directory.

Parameters:: directory (str) – The directory to list out all the files from.
Returns:: List of the files in the directory.
Return type:: List

ads.common.utils.get_oci_config(): Returns the OCI config location, and the OCI config profile.

ads.common.utils.get_progress_bar(max_progress: int, description: str = 'Initializing', verbose: bool = False) → TqdmProgressBar

Returns an instance of the TqdmProgressBar class.

Parameters:

max_progress (int) – The number of steps for the progressbar.
description ((str, optional). Defaults to "Initializing".) – The first step description.
verbose ((bool, optional). Defaults to False) – If the progress should show the debug information.

Returns:

An instance of the TqdmProgressBar.

Return type:

TqdmProgressBar

ads.common.utils.get_random_name_for_resource() → str

Returns randomly generated easy to remember name. It consists from 1 adjective and 1 animal word, tailed by UTC timestamp (joined with ‘-‘). This is an ADS default resource name generated for models, jobs, jobruns, model deployments, pipelines.

Returns:: Randomly generated easy to remember name for oci resources - models, jobs, jobruns, model deployments, pipelines. Example: polite-panther-2022-08-17-21:15.46; strange-spider-2022-08-17-23:55.02
Return type:: str

ads.common.utils.get_sqlalchemy_engine(connection_url, *args, **kwargs)

The SqlAlchemny docs say to use a single engine per connection_url, this class will take care of that.

Parameters:: connection_url (string) – The URL to connect to
Returns:: engine – The engine from which SqlAlchemny commands can be ran on
Return type:: SqlAlchemny engine

ads.common.utils.get_value(obj, attr, default=None)

Gets a copy of the value from a nested dictionary of an object with nested attributes.

Parameters:

obj – An object or a dictionary
attr – Attributes as a string seprated by dot(.)
default – Default value to be returned if attribute is not found.

Returns:

A copy of the attribute value. For dict or list, a deepcopy will be returned.

Return type:

Any

ads.common.utils.highlight_text(text)

Returns text with html highlights. :param text: The text to be highlighted. :type text: String

Returns:: ht – The text with html highlight information.
Return type:: String

ads.common.utils.horizontal_scrollable_div(html)

Wrap html with the necessary html to make horizontal scrolling possible.

Examples

display(HTML(utils.horizontal_scrollable_div(my_html)))

Parameters:: html (str) – Your HTML to wrap.
Returns:: Wrapped HTML.
Return type:: type

ads.common.utils.human_size(num_bytes: int, precision: Optional[int] = 2) → str

Converts bytes size to a string representing its value in B, KB, MB and GB.

Parameters:

num_bytes (int) – The size in bytes.
precision ((int, optional). Defaults to 2.) – The precision of converting the bytes value.

Returns:

A string representing the size in B, KB, MB and GB.

Return type:

str

ads.common.utils.inject_and_copy_kwargs(kwargs, **args)

Takes in a dictionary and returns a copy with the args injected

Examples

>>> foo(arg1, args, utils.inject_and_copy_kwargs(kwargs, arg3=12, arg4=42))

Parameters:

kwargs (dict) – The original kwargs.
**args (type) – A series of arguments, foo=42, bar=12 etc

Returns:

d – new dictionary object that you can use in place of kwargs

Return type:

dict

ads.common.utils.is_data_too_wide(data: Union[list, tuple, Series, ndarray, DataFrame], max_col_num: int) → bool

Returns true if the data has too many columns.

Parameters:

data (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]) – A sample of data that will be used to generate schema.
max_col_num (int.) – The maximum column size of the data that allows to auto generate schema.

ads.common.utils.is_debug_mode(): Returns true if ADS is in debug mode.

ads.common.utils.is_documentation_mode(): Returns true if ADS is in documentation mode.

ads.common.utils.is_notebook(): Returns true if the environment is a jupyter notebook.

ads.common.utils.is_resource_principal_mode(): Returns true if ADS is in resource principal mode.

ads.common.utils.is_same_class(obj, cls): checks to see if object is the same class as cls

ads.common.utils.is_test(): Returns true if ADS is in test mode.

class ads.common.utils.ml_task_types(value)

Bases: Enum

An enumeration.

BINARY_CLASSIFICATION = 2

BINARY_TEXT_CLASSIFICATION = 4

MULTI_CLASS_CLASSIFICATION = 3

MULTI_CLASS_TEXT_CLASSIFICATION = 5

REGRESSION = 1

UNSUPPORTED = 6

ads.common.utils.numeric_pandas_dtypes(): Returns a list of the “numeric” pandas data types

ads.common.utils.oci_config_file(): Returns the OCI config file location

ads.common.utils.oci_config_location(): Returns oci configuration file location.

ads.common.utils.oci_config_profile(): Returns the OCI config profile location.

ads.common.utils.oci_key_location(): Returns the OCI key location

ads.common.utils.oci_key_profile(): Returns key profile value specified in oci configuration file.

ads.common.utils.print_user_message(msg, display_type='tip', see_also_links=None, title='Tip')

This method is deprecated and will be removed in future releases. Prints in html formatted block one of tip|info|warn type.

Parameters:

msg (str or list) – The actual message to display. display_type is “module’, msg can be a list of [module name, module package name], i.e. [“automl”, “ads[ml]”]
display_type (str (default 'tip')) – The type of user message.
see_also_links (list of tuples in the form of [('display_name', 'url')]) –
title (str (default 'tip')) – The title of user message.

ads.common.utils.random_valid_ocid(prefix='ocid1.dataflowapplication.oc1.iad')

Generates a random valid ocid.

Parameters:: prefix (str) – A prefix, corresponding to a region location.
Returns:: ocid – a valid ocid with the given prefix.
Return type:: str

ads.common.utils.remove_file(file_path: str, auth: Optional[Dict] = None) → None

Reoves file.

Parameters:

file_path (str) – The path of the source file, which can be local path or OCI object storage URI.
auth ((Dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

Nothing.

Return type:

None

ads.common.utils.replace_spaces(lst)

Replace all spaces with underscores for strings in the list.

Requires that the list contains strings for each element.

lst: list of strings

ads.common.utils.set_oci_config(oci_config_location, oci_config_profile)

Parameters:

oci_config_location – location of the config file, for example, ~/.oci/config
oci_config_profile – The profile to load from the config file. Defaults to “DEFAULT”

ads.common.utils.snake_to_camel(name: str, capitalized_first_token: Optional[bool] = False) → str

Converts the snake case string to the camel representation.

Parameters:

name (str) – The name to convert.
capitalized_first_token ((bool, optional). Defaults to False.) – Wether the first token needs to be capitalized or not.

Returns:

str

Return type:

The name converted to the camel representation.

ads.common.utils.split_data(X, y, random_state=42, test_size=0.3)

Splits data using Sklearn based on the input type of the data.

Parameters:

X (a Pandas Dataframe) – The data points.
y (a Pandas Dataframe) – The labels.
random_state (int) – A random state for reproducability.
test_size (int) – The number of elements that should be included in the test dataset.

ads.common.utils.to_dataframe(data: Union[list, tuple, Series, ndarray, DataFrame])

Convert to pandas DataFrame.

Parameters:: data (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]) – Convert data to pandas DataFrame.
Returns:: pandas DataFrame.
Return type:: pd.DataFrame

ads.common.utils.truncate_series_top_n(series, n=24): take a series which can be interpreted as a dict, index=key, this function sorts by the values and takes the top-n values, and returns a new series

ads.common.utils.wrap_lines(li, heading=''): Wraps the elements of iterable into multi line string of fixed width

ads.common.word_lists module

Module contents

ads.data_labeling package

Subpackages

ads.data_labeling.interface package

Submodules

ads.data_labeling.interface.loader module

class ads.data_labeling.interface.loader.Loader

Bases: ABC

Data Loader Interface.

abstract load(**kwargs) → Any

ads.data_labeling.interface.parser module

class ads.data_labeling.interface.parser.Parser

Bases: ABC

Data Parser Interface.

abstract parse() → Any

ads.data_labeling.interface.reader module

class ads.data_labeling.interface.reader.Reader

Bases: ABC

Data Reader Interface.

info() → Serializable

abstract read() → Any

Module contents

ads.data_labeling.loader package

Submodules

ads.data_labeling.loader.file_loader module

class ads.data_labeling.loader.file_loader.FileLoader(auth: Optional[Dict] = None)

Bases: object

FileLoader Base Class.

Attributes:

auth: (dict, optional). Defaults to None.: The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Examples

>>> from ads.data_labeling.loader.file_loader import FileLoader
>>> import oci
>>> import os
>>> from ads.common import auth as authutil
>>> path = "path/to/your_text_file.txt"
>>> file_content = FileLoader(auth=authutil.api_keys()).load(path)

Initiates a FileLoader instance.

param auth:: The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
type auth:: (dict, optional). Defaults to None.

bulk_load(paths: List[str], **kwargs) → Dict[str, Any]

Loads the files content from the list of paths. The ThreadPoolExecutor is used to load the files in parallel threads.

Parameters:: paths (List[str]) – The list of file paths, can be local or object storage paths.
Returns:: The map between file path and file content.
Return type:: Dict[str, Any]

load(path: str, **kwargs) → BytesIO

Loads the file content from the path.

Parameters:

path (str) – The file path, can be local or object storage path.
kwargs – Nothing.

Returns:

The data in BytesIO format.

Return type:

BytesIO

class ads.data_labeling.loader.file_loader.FileLoaderFactory

Bases: object

FileLoaderFactory class to create/register FileLoaders.

static loader(dataset_type: str, auth: Optional[Dict] = None) → FileLoader

Gets the loader based on the dataset_type.

Parameters:

dataset_type (str) – Dataset type. Currently supports TEXT, IMAGE and DOCUMENT.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

A FileLoader instance corresponding to the dataset_type.

Return type:

FileLoader

classmethod register(dataset_type: str, loader: Loader) → None

Registers a new loader for a given dataset_type.

Parameters:

dataset_type (str) – Dataset type. Currently supports TEXT and IMAGE.
loader (Loader) – A Loader class which supports loading content of the given dataset_type.

Returns:

Nothing.

Return type:

None

class ads.data_labeling.loader.file_loader.ImageFileLoader(auth: Optional[Dict] = None)

Bases: FileLoader

ImageFileLoader class which loads image files.

Examples

>>> from ads.data_labeling import ImageFileLoader
>>> import oci
>>> import os
>>> from ads.common import auth as authutil
>>> path = "path/to/image.png"
>>> image = ImageFileLoader(auth=authutil.api_keys()).load(path)

Initiates a FileLoader instance.

Parameters:: auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

load(path: str, **kwargs) → ImageFile

Loads the image from the path.

Parameters:

path (str) – Image file path, can be local or object storage path.
kwargs – Nothing.

Returns:

Image opened by Pillow.

Return type:

PIL.ImageFile.ImageFile

class ads.data_labeling.loader.file_loader.TextFileLoader(auth: Optional[Dict] = None)

Bases: FileLoader

TextFileLoader class which loads text files.

Examples

>>> from ads.data_labeling import TextFileLoader
>>> import oci
>>> import os
>>> from ads.common import auth as authutil
>>> path = "path/to/your_text_file.txt"
>>> file_content = TextFileLoader(auth=authutil.api_keys()).load(path)

Initiates a FileLoader instance.

Parameters:: auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

load(path: str, backend: Union[str, Base] = 'default', **kwargs) → str

Loads the content from the path.

Parameters:

path (str) – Text file path, can be local or object storage path.
backend (Union[str, backends.Base]) – Default to “default”. Valid options are “default” and “tika” or ads.text_dataset.backends.Base, ads.text_dataset.backends.Tika
kwargs –

encoding: (str, optional). Defaults to ‘utf-8’.
Encoding for text files. Used only to extract the content of the text dataset contents.

Returns:

Content of the text file.

Return type:

str

Module contents

ads.data_labeling.mixin package

Submodules

ads.data_labeling.mixin.data_labeling module

class ads.data_labeling.mixin.data_labeling.DataLabelingAccessMixin

Bases: object

Mixin class for labeled text data.

static read_labeled_data(path: Optional[str] = None, dataset_id: Optional[str] = None, compartment_id: Optional[str] = None, auth: Optional[Dict] = None, materialize: bool = False, encoding: str = 'utf-8', include_unlabeled: bool = False, format: Optional[str] = None, chunksize: Optional[int] = None)

Loads the dataset generated by data labeling service from either the export file or the Data Labeling Service.

Parameters:

path ((str, optional). Defaults to None) – The export file path, can be either local or object storage path.
dataset_id ((str, optional). Defaults to None) – The dataset OCID.
compartment_id (str. Defaults to the compartment_id from the env variable.) – The compartment OCID of the dataset.
auth ((dict, optional). Defaults to None) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
materialize ((bool, optional). Defaults to False) – Whether the content of the dataset file should be loaded or it should return the file path to the content. By default the content will not be loaded.
encoding ((str, optional). Defaults to 'utf-8') – Encoding of files. Only used for “TEXT” dataset.
include_unlabeled ((bool, optional). Default to False) – Whether to load the unlabeled records or not.
format ((str, optional). Defaults to None) –
Output format of annotations. Can be None, “spacy” for dataset Entity Extraction type or “yolo for Object Detection type.
- When None, it outputs List[NERItem] or List[BoundingBoxItem],
- When “spacy”, it outputs List[Tuple],
- When “yolo”, it outputs List[List[Tuple]].
chunksize ((int, optional). Defaults to None) – The amount of records that should be read in one iteration. The result will be returned in a generator format.

Returns:

pd.Dataframe if chunksize is not specified. Generator[pd.Dataframe] if chunksize is specified.

Return type:

Union[Generator[pd.DataFrame, Any, Any], pd.DataFrame]

Examples

>>> import pandas as pd
>>> import ads
>>> from ads.common import auth as authutil
>>> df = pd.DataFrame.ads.read_labeled_data(path="path_to_your_metadata.jsonl",
...                                         auth=authutil.api_keys(),
...                                         materialize=False)
                            Path       Content               Annotations
    --------------------------------------------------------------------
    0   path/to/the/content/file                                     yes
    1   path/to/the/content/file                                      no

>>> df = pd.DataFrame.ads.read_labeled_data_from_dls(dataset_id="your_dataset_ocid",
...                                                  compartment_id="your_compartment_id",
...                                                  auth=authutil.api_keys(),
...                                                  materialize=False)
                            Path       Content               Annotations
    --------------------------------------------------------------------
    0   path/to/the/content/file                                     yes
    1   path/to/the/content/file                                      no

render_bounding_box(options: Optional[Dict] = None, content_column: str = 'Content', annotations_column: str = 'Annotations', categories: Optional[List[str]] = None, limit: int = 50, path: Optional[str] = None) → None

Renders bounding box dataset. Displays only first 50 rows.

Parameters:

options (dict) – The colors options specified for rendering.
content_column (Optional[str]) – The column name with the content data.
annotations_column (Optional[str]) – The column name for the annotations list.
categories (Optional List[str]) – The list of object categories in proper order for model training. Only used when bounding box annotations are in YOLO format. Example: [‘cat’,’dog’,’horse’]
limit (Optional[int]. Defaults to 50) – The maximum amount of records to display.
path (Optional[str]) – Path to save the image with annotations to local directory.

Returns:

Nothing

Return type:

None

Examples

>>> import pandas as pd
>>> import ads
>>> from ads.common import auth as authutil
>>> df = pd.DataFrame.ads.read_labeled_data(path="path_to_your_metadata.jsonl",
...                                         auth=authutil.api_keys(),
...                                         materialize=True)
>>> df.ads.render_bounding_box(content_column="Content", annotations_column="Annotations")

render_ner(options: Dict = None, content_column: str = 'Content', annotations_column: str = 'Annotations', limit: int = 50) → None

Renders NER dataset. Displays only first 50 rows.

Parameters:

options (dict) – The colors options specified for rendering.
content_column (Optional[str]) – The column name with the content data.
annotations_column (Optional[str]) – The column name for the annotations list.
limit (Optional[int]. Defaults to 50) – The maximum amount of records to display.

Returns:

Nothing

Return type:

None

Examples

>>> import pandas as pd
>>> import ads
>>> from ads.common import auth as authutil
>>> df = pd.DataFrame.ads.read_labeled_data(path="path_to_your_metadata.jsonl",
...                                         auth=authutil.api_keys(),
...                                         materialize=True)
>>> df.ads.render_ner(content_column="Content", annotations_column="Annotations")

Module contents

ads.data_labeling.parser package

Submodules

ads.data_labeling.parser.dls_record_parser module

exception ads.data_labeling.parser.dls_record_parser.AnnotationNotFoundError(id: str): Bases: Exception

class ads.data_labeling.parser.dls_record_parser.DLSBoundingBoxRecordParser(dataset_source_path: str, auth: Optional[dict] = None, format: Optional[str] = None, categories: Optional[List[str]] = None)

BoundingBoxRecordParser class which parses the label of BoundingBox label data.

Initiates a DLSRecordParser instance.

Parameters:

dataset_source_path (str) – Dataset source path.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
format ((str, optional). Defaults to None.) – Output format of annotations.
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

DLSRecordParser instance.

Return type:

class ads.data_labeling.parser.dls_record_parser.DLSMultiLabelRecordParser(dataset_source_path: str, auth: Optional[dict] = None, format: Optional[str] = None, categories: Optional[List[str]] = None)

MultiLabelRecordParser class which parses the label of Multiple label data.

Initiates a DLSRecordParser instance.

Parameters:

dataset_source_path (str) – Dataset source path.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
format ((str, optional). Defaults to None.) – Output format of annotations.
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

DLSRecordParser instance.

Return type:

class ads.data_labeling.parser.dls_record_parser.DLSNERRecordParser(dataset_source_path: str, auth: Optional[dict] = None, format: Optional[str] = None, categories: Optional[List[str]] = None)

NERRecordParser class which parses the label of NER label data.

Initiates a DLSRecordParser instance.

Parameters:

dataset_source_path (str) – Dataset source path.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
format ((str, optional). Defaults to None.) – Output format of annotations.
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

DLSRecordParser instance.

Return type:

class ads.data_labeling.parser.dls_record_parser.DLSRecordParser(dataset_source_path: str, auth: Optional[dict] = None, format: Optional[str] = None, categories: Optional[List[str]] = None)

Bases: Parser

DLSRecordParser class which parses the labels from the record.

Initiates a DLSRecordParser instance.

Parameters:

dataset_source_path (str) – Dataset source path.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
format ((str, optional). Defaults to None.) – Output format of annotations.
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

DLSRecordParser instance.

Return type:

parse(oci_record_summary: OCIRecordSummary) → Record

Extracts the annotations. Constructs and returns a Record instance which contains the file path and the labels.

Parameters:: oci_record_summary (OCIRecordSummary) – The summary information about the record.
Returns:: Record instance which contains the file path and the annotations.
Return type:: Record

class ads.data_labeling.parser.dls_record_parser.DLSRecordParserFactory

Bases: object

DLSRecordParserFactory class which contains a list of registered parsers and allows to register new DLSRecordParsers.

Current parsers include:

DLSSingleLabelRecordParser
DLSMultiLabelRecordParser
DLSNERRecordParser
DLSBoundingBoxRecordParser

static parser(annotation_type: str, dataset_source_path: str, format: Optional[str] = None, categories: Optional[List[str]] = None, auth: Optional[dict] = None) → DLSRecordParser

Gets the parser based on the annotation_type.

Parameters:

annotation_type (str) – Annotation type which can be SINGLE_LABEL, MULTI_LABEL, ENTITY_EXTRACTION and BOUNDING_BOX.
dataset_source_path (str) – Dataset source path.
format ((str, optional). Defaults to None.) – Output format of annotations. Can be None, “spacy” for dataset Entity Extraction type or “yolo” for Object Detection type. When None, it outputs List[NERItem] or List[BoundingBoxItem]. When “spacy”, it outputs List[Tuple]. When “yolo”, it outputs List[List[Tuple]].
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

DLSRecordParser corresponding to the annotation type.

Return type:

Raises:

ValueError – If annotation_type is not supported.

classmethod register(annotation_type: str, parser: DLSRecordParser) → None

Registers a new parser.

Parameters:

annotation_type (str) – Annotation type which can be SINGLE_LABEL, MULTI_LABEL, ENTITY_EXTRACTION and BOUNDING_BOX.
parser (DLSRecordParser) – A new Parser class to be registered.

Returns:

Nothing.

Return type:

None

class ads.data_labeling.parser.dls_record_parser.DLSSingleLabelRecordParser(dataset_source_path: str, auth: Optional[dict] = None, format: Optional[str] = None, categories: Optional[List[str]] = None)

SingleLabelRecordParser class which parses the label of Single label data.

Initiates a DLSRecordParser instance.

Parameters:

dataset_source_path (str) – Dataset source path.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
format ((str, optional). Defaults to None.) – Output format of annotations.
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

DLSRecordParser instance.

Return type:

exception ads.data_labeling.parser.dls_record_parser.ReadAnnotationError(id: str): Bases: Exception

ads.data_labeling.parser.export_metadata_parser module

class ads.data_labeling.parser.export_metadata_parser.MetadataParser

Bases: Parser

MetadataParser class which parses the metadata from the record.

EXPECTED_KEYS = ['id', 'compartmentId', 'displayName', 'labelsSet', 'annotationFormat', 'datasetSourceDetails', 'datasetFormatDetails']

static parse(json_data: Dict[Any, Any]) → Metadata

Parses the metadata jsonl file.

Parameters:: json_data (dict) – dictionary format of the metadata jsonl file content.
Returns:: Metadata object which contains the useful fields from the metadata jsonl file
Return type:: Metadata

ads.data_labeling.parser.export_record_parser module

class ads.data_labeling.parser.export_record_parser.BoundingBoxRecordParser(dataset_source_path: str, format: Optional[str] = None, categories: Optional[List[str]] = None)

Bases: RecordParser

BoundingBoxRecordParser class which parses the label of BoundingBox label data.

Initiates a RecordParser instance.

Parameters:

dataset_source_path (str) – Dataset source path.
format ((str, optional). Defaults to None.) – Output format of annotations.
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

RecordParser instance.

Return type:

class ads.data_labeling.parser.export_record_parser.EntityType

Bases: object

Entity type class for supporting multiple types of entities.

GENERIC = 'GENERIC'

IMAGEOBJECTSELECTION = 'IMAGEOBJECTSELECTION'

TEXTSELECTION = 'TEXTSELECTION'

class ads.data_labeling.parser.export_record_parser.MultiLabelRecordParser(dataset_source_path: str, format: Optional[str] = None, categories: Optional[List[str]] = None)

Bases: RecordParser

MultiLabelRecordParser class which parses the label of Multiple label data.

Initiates a RecordParser instance.

Parameters:

dataset_source_path (str) – Dataset source path.
format ((str, optional). Defaults to None.) – Output format of annotations.
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

RecordParser instance.

Return type:

class ads.data_labeling.parser.export_record_parser.NERRecordParser(dataset_source_path: str, format: Optional[str] = None, categories: Optional[List[str]] = None)

Bases: RecordParser

NERRecordParser class which parses the label of NER label data.

Initiates a RecordParser instance.

Parameters:

dataset_source_path (str) – Dataset source path.
format ((str, optional). Defaults to None.) – Output format of annotations.
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

RecordParser instance.

Return type:

class ads.data_labeling.parser.export_record_parser.RecordParser(dataset_source_path: str, format: Optional[str] = None, categories: Optional[List[str]] = None)

Bases: Parser

RecordParser class which parses the labels from the record.

Examples

>>> from ads.data_labeling.parser.export_record_parser import SingleLabelRecordParser
>>> from ads.data_labeling.parser.export_record_parser import MultiLabelRecordParser
>>> from ads.data_labeling.parser.export_record_parser import NERRecordParser
>>> from ads.data_labeling.parser.export_record_parser import BoundingBoxRecordParser
>>> import fsspec
>>> import json
>>> from ads.common import auth as authutil
>>> labels = []
>>> with fsspec.open("/path/to/records_file.jsonl", **authutil.api_keys()) as f:
>>>     for line in f:
>>>         bounding_box_labels = BoundingBoxRecordParser("source_data_path").parse(json.loads(line))
>>>         labels.append(bounding_box_labels)

Initiates a RecordParser instance.

Parameters:

dataset_source_path (str) – Dataset source path.
format ((str, optional). Defaults to None.) – Output format of annotations.
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

RecordParser instance.

Return type:

parse(record: Dict) → Record

Extracts the annotations from the record content. Constructs and returns a Record instance containing the file path and the labels.

Parameters:: record (Dict) – Content of the record from the record file.
Returns:: Record instance which contains the file path as well as the annotations.
Return type:: Record

class ads.data_labeling.parser.export_record_parser.RecordParserFactory

Bases: object

RecordParserFactory class which contains a list of registered parsers and allows to register new RecordParsers.

Current parsers include:

SingleLabelRecordParser
MultiLabelRecordParser
NERRecordParser
BoundingBoxRecordParser

static parser(annotation_type: str, dataset_source_path: str, format: Optional[str] = None, categories: Optional[List[str]] = None) → RecordParser

Gets the parser based on the annotation_type.

Parameters:

annotation_type (str) – Annotation type which can be SINGLE_LABEL, MULTI_LABEL, ENTITY_EXTRACTION and BOUNDING_BOX.
dataset_source_path (str) – Dataset source path.
format ((str, optional). Defaults to None.) – Output format of annotations. Can be None, “spacy” for dataset Entity Extraction type or “yolo” for Object Detection type. When None, it outputs List[NERItem] or List[BoundingBoxItem]. When “spacy”, it outputs List[Tuple]. When “yolo”, it outputs List[List[Tuple]].
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

RecordParser corresponding to the annotation type.

Return type:

Raises:

ValueError – If annotation_type is not supported.

classmethod register(annotation_type: str, parser) → None

Registers a new parser.

Parameters:

annotation_type (str) – Annotation type which can be SINGLE_LABEL, MULTI_LABEL, ENTITY_EXTRACTION and BOUNDING_BOX.
parser (RecordParser) – A new Parser class to be registered.

Returns:

Nothing.

Return type:

None

class ads.data_labeling.parser.export_record_parser.SingleLabelRecordParser(dataset_source_path: str, format: Optional[str] = None, categories: Optional[List[str]] = None)

Bases: RecordParser

SingleLabelRecordParser class which parses the label of Single label data.

Initiates a RecordParser instance.

Parameters:

dataset_source_path (str) – Dataset source path.
format ((str, optional). Defaults to None.) – Output format of annotations.
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

RecordParser instance.

Return type:

Module contents

ads.data_labeling.reader package

Submodules

ads.data_labeling.reader.dataset_reader module

The module containing classes to read labeled datasets. Allows to read labeled datasets from exports or from the cloud.

Classes

LabeledDatasetReader
The LabeledDatasetReader class to read labeled dataset.

ExportReader
The ExportReader class to read labeled dataset from the export.

DLSDatasetReader
The DLSDatasetReader class to read labeled dataset from the cloud.

Examples

>>> from ads.common import auth as authutil
>>> from ads.data_labeling import LabeledDatasetReader
>>> ds_reader = LabeledDatasetReader.from_export(
...    path="oci://bucket_name@namespace/dataset_metadata.jsonl",
...    auth=authutil.api_keys(),
...    materialize=True
... )
>>> ds_reader.info()
    ------------------------------------------------------------------------
    annotation_type                                             SINGLE_LABEL
    compartment_id                                          TEST_COMPARTMENT
    dataset_id                                                  TEST_DATASET
    dataset_name                                           test_dataset_name
    dataset_type                                                        TEXT
    labels                                                     ['yes', 'no']
    records_path                                             path/to/records
    source_path                                              path/to/dataset

>>> ds_reader.read()
                             Path            Content            Annotations
    -----------------------------------------------------------------------
    0   path/to/the/content/file1       file content                    yes
    1   path/to/the/content/file2       file content                     no
    2   path/to/the/content/file3       file content                     no

>>> next(ds_reader.read(iterator=True))
    ("path/to/the/content/file1", "file content", "yes")

>>> next(ds_reader.read(iterator=True, chunksize=2))
    [("path/to/the/content/file1", "file content", "yes"),
    ("path/to/the/content/file2", "file content", "no")]

>>> next(ds_reader.read(chunksize=2))
                            Path            Content            Annotations
    ----------------------------------------------------------------------
    0   path/to/the/content/file1       file content                    yes
    1   path/to/the/content/file2       file content                     no

>>> ds_reader = LabeledDatasetReader.from_DLS(
...    dataset_id="dataset_OCID",
...    compartment_id="compartment_OCID",
...    auth=authutil.api_keys(),
...    materialize=True
... )

class ads.data_labeling.reader.dataset_reader.DLSDatasetReader(dataset_id: str, compartment_id: str, auth: Dict, encoding='utf-8', materialize: bool = False, include_unlabeled: bool = False)

Bases: Reader

The DLSDatasetReader class to read labeled dataset from the cloud.

info(self) → Metadata: Gets the labeled dataset metadata.

read(self) → Generator[Tuple, Any, Any]: Reads the labeled dataset.

Initializes the DLS dataset reader instance.

Parameters:

dataset_id (str) – The dataset OCID.
compartment_id (str) – The compartment OCID of the dataset.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
encoding ((str, optional). Defaults to 'utf-8'.) – Encoding for files. The encoding is used to extract the metadata information of the labeled dataset and also to extract the content of the text dataset records.
materialize ((bool, optional). Defaults to False.) – Whether the content of dataset files should be loaded/materialized or not. By default the content will not be materialized.
include_unlabeled ((bool, optional). Defaults to False.) – Whether to load the unlabeled records or not.

Raises:

ValueError – When dataset_id is empty or not a string.:
TypeError – When dataset_id not a string.:

info() → Metadata

Gets the labeled dataset metadata.

Returns:: The labeled dataset metadata.
Return type:: Metadata

read(format: Optional[str] = None) → Generator[Tuple, Any, Any]

Reads the labeled dataset records.

Parameters:: format ((str, optional). Defaults to None.) – Output format of annotations. Can be None, “spacy” for dataset Entity Extraction type or “yolo” for Object Detection type. When None, it outputs List[NERItem] or List[BoundingBoxItem]. When “spacy”, it outputs List[Tuple]. When “yolo”, it outputs List[List[Tuple]].
Returns:: The labeled dataset records.
Return type:: Generator[Tuple, Any, Any]

class ads.data_labeling.reader.dataset_reader.ExportReader(path: str, auth: Optional[Dict] = None, encoding='utf-8', materialize: bool = False, include_unlabeled: bool = False)

Bases: Reader

The ExportReader class to read labeled dataset from the export.

info(self) → Metadata: Gets the labeled dataset metadata.

read(self) → Generator[Tuple, Any, Any]: Reads the labeled dataset.

Initializes the labeled dataset export reader instance.

Parameters:

path (str) – The metadata file path, can be either local or object storage path.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
encoding ((str, optional). Defaults to 'utf-8'.) – Encoding for files. The encoding is used to extract the metadata information of the labeled dataset and also to extract the content of the text dataset records.
materialize ((bool, optional). Defaults to False.) – Whether the content of dataset files should be loaded/materialized or not. By default the content will not be materialized.
include_unlabeled ((bool, optional). Defaults to False.) – Whether to load the unlabeled records or not.

Raises:

ValueError – When path is empty or not a string.:
TypeError – When path not a string.:

info() → Metadata

Gets the labeled dataset metadata.

Returns:: The labeled dataset metadata.
Return type:: Metadata

read(format: Optional[str] = None) → Generator[Tuple, Any, Any]

Reads the labeled dataset records.

Parameters:: format ((str, optional). Defaults to None.) – Output format of annotations. Can be None, “spacy” for dataset Entity Extraction type or “yolo” for Object Detection type. When None, it outputs List[NERItem] or List[BoundingBoxItem]. When “spacy”, it outputs List[Tuple]. When “yolo”, it outputs List[List[Tuple]].
Returns:: The labeled dataset records.
Return type:: Generator[Tuple, Any, Any]

class ads.data_labeling.reader.dataset_reader.LabeledDatasetReader(reader: Reader)

Bases: object

The labeled dataset reader class.

info(self) → Metadata: Gets labeled dataset metadata.

read(self, iterator: bool = False) → Union[Generator[Any, Any, Any], pd.DataFrame]: Reads labeled dataset.

from_export(cls, path: str, auth: Dict = None, encoding='utf-8', materialize: bool = False) → 'LabeledDatasetReader': Constructs a Labeled Dataset Reader instance.

Examples

>>> from ads.common import auth as authutil
>>> from ads.data_labeling import LabeledDatasetReader

>>> ds_reader = LabeledDatasetReader.from_export(
...    path="oci://bucket_name@namespace/dataset_metadata.jsonl",
...    auth=authutil.api_keys(),
...    materialize=True
... )

>>> ds_reader = LabeledDatasetReader.from_DLS(
...    dataset_id="dataset_OCID",
...    compartment_id="compartment_OCID",
...    auth=authutil.api_keys(),
...    materialize=True
... )

>>> ds_reader.info()
    ------------------------------------------------------------------------
    annotation_type                                             SINGLE_LABEL
    compartment_id                                          TEST_COMPARTMENT
    dataset_id                                                  TEST_DATASET
    dataset_name                                           test_dataset_name
    dataset_type                                                        TEXT
    labels                                                     ['yes', 'no']
    records_path                                             path/to/records
    source_path                                              path/to/dataset

>>> ds_reader.read()
                             Path            Content            Annotations
    -----------------------------------------------------------------------
    0   path/to/the/content/file1       file content                    yes
    1   path/to/the/content/file2       file content                     no
    2   path/to/the/content/file3       file content                     no

>>> next(ds_reader.read(iterator=True))
    ("path/to/the/content/file1", "file content", "yes")

>>> next(ds_reader.read(iterator=True, chunksize=2))
    [("path/to/the/content/file1", "file content", "yes"),
    ("path/to/the/content/file2", "file content", "no")]

>>> next(ds_reader.read(chunksize=2))
                            Path            Content            Annotations
    ----------------------------------------------------------------------
    0   path/to/the/content/file1       file content                    yes
    1   path/to/the/content/file2       file content                     no

Initializes the labeled dataset reader instance.

Parameters:: reader (Reader) – The Reader instance which reads and extracts the labeled dataset.

classmethod from_DLS(dataset_id: str, compartment_id: Optional[str] = None, auth: Optional[dict] = None, encoding: str = 'utf-8', materialize: bool = False, include_unlabeled: bool = False) → LabeledDatasetReader

Constructs Labeled Dataset Reader instance.

Parameters:

dataset_id (str) – The dataset OCID.
compartment_id (str. Defaults to the compartment_id from the env variable.) – The compartment OCID of the dataset.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
encoding ((str, optional). Defaults to 'utf-8'.) – Encoding for files.
materialize ((bool, optional). Defaults to False.) – Whether the content of the dataset file should be loaded or it should return the file path to the content. By default the content will not be loaded.

Returns:

The LabeledDatasetReader instance.

Return type:

LabeledDatasetReader

classmethod from_export(path: str, auth: Optional[dict] = None, encoding: str = 'utf-8', materialize: bool = False, include_unlabeled: bool = False) → LabeledDatasetReader

Constructs Labeled Dataset Reader instance.

Parameters:

path (str) – The metadata file path, can be either local or object storage path.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
encoding ((str, optional). Defaults to 'utf-8'.) – Encoding for files.
materialize ((bool, optional). Defaults to False.) – Whether the content of the dataset file should be loaded or it should return the file path to the content. By default the content will not be loaded.

Returns:

The LabeledDatasetReader instance.

Return type:

LabeledDatasetReader

info() → Serializable

Gets the labeled dataset metadata.

Returns:: The labeled dataset metadata.
Return type:: Metadata

read(iterator: bool = False, format: Optional[str] = None, chunksize: Optional[int] = None) → Union[Generator[Any, Any, Any], DataFrame]

Reads the labeled dataset records.

Parameters:

iterator ((bool, optional). Defaults to False.) – True if the result should be represented as a Generator. Fasle if the result should be represented as a Pandas DataFrame.
format ((str, optional). Defaults to None.) – Output format of annotations. Can be None, “spacy” or “yolo”.
chunksize ((int, optional). Defaults to None.) – The number of records that should be read in one iteration. The result will be returned in a generator format.

Returns:

Union[ – Generator[Tuple[str, str, Any], Any, Any], Generator[List[Tuple[str, str, Any]], Any, Any], Generator[pd.DataFrame, Any, Any], pd.DataFrame
] – pd.Dataframe if iterator and chunksize are not specified. Generator[pd.Dataframe] ` if `iterator equal to False and chunksize is specified. Generator[List[Tuple[str, str, Any]]] if iterator equal to True and chunksize is specified. Generator[Tuple[str, str, Any]] if iterator equal to True and chunksize is not specified.

ads.data_labeling.reader.dls_record_reader module

class ads.data_labeling.reader.dls_record_reader.DLSRecordReader(dataset_id: str, compartment_id: str, auth: Optional[dict] = None)

Bases: Reader

DLS Record Reader Class that reads records from the cloud into ADS format.

Initiates a DLSRecordReader instance.

Parameters:

dataset_id (str) – The dataset OCID.
compartment_id (str) – The compartment OCID of the dataset.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

read() → Generator[OCIRecordSummary, Any, Any]

Reads OCI records.

Yields:: OCIRecordSummary – The OCIRecordSummary instance.

class ads.data_labeling.reader.dls_record_reader.OCIRecordSummary(record: Optional[RecordSummary] = None, annotation: Optional[List[AnnotationSummary]] = None)

Bases: object

The class that representing the labeled record in ADS format.

record

OCI RecordSummary.

Type:: RecordSummary

annotations

List of OCI AnnotationSummary.

Type:: List[AnnotationSummary]

annotation: List[AnnotationSummary] = None

record: RecordSummary = None

exception ads.data_labeling.reader.dls_record_reader.ReadAnnotationsError(dataset_id: str): Bases: Exception

exception ads.data_labeling.reader.dls_record_reader.ReadRecordsError(dataset_id: str): Bases: Exception

ads.data_labeling.reader.export_record_reader module

class ads.data_labeling.reader.export_record_reader.ExportRecordReader(path: str, auth: Optional[Dict] = None, encoding='utf-8', includes_metadata: bool = False)

Bases: JsonlReader

The ExportRecordReader class to read labeled dataset records from the export.

read(self) → Generator[Dict, Any, Any]: Reads labeled dataset records.

Initiates an ExportRecordReader instance.

Parameters:

path (str) – object storage path or local path for a file.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
encoding ((str, optional). Defaults to 'utf-8'.) – Encoding of files. Only used for “TEXT” dataset.
includes_metadata ((bool, optional). Defaults to False.) – Determines whether the export file includes metadata or not.

Examples

>>> from ads.data_labeling.reader.export_record_reader import ExportRecordReader
>>> path = "your/path/to/jsonl/file.jsonl"
>>> from ads.common import auth as authutil
>>> reader = ExportRecordReader(path=path, auth=authutil.api_keys(), encoding="utf-8")
>>> next(reader.read())

read() → Generator[Dict, Any, Any]

Reads labeled dataset records.

Returns:: The labeled dataset records.
Return type:: Generator[Dict, Any, Any]

ads.data_labeling.reader.jsonl_reader module

class ads.data_labeling.reader.jsonl_reader.JsonlReader(path: str, auth: Optional[Dict] = None, encoding='utf-8')

Bases: Reader

JsonlReader class which reads the file.

Initiates a JsonlReader object.

Parameters:

path (str) – object storage path or local path for a file.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
encoding ((str, optional). Defaults to 'utf-8'.) – Encoding of files. Only used for “TEXT” dataset.

Examples

>>> from ads.data_labeling.reader.jsonl_reader import JsonlReader
>>> path = "your/path/to/jsonl/file.jsonl"
>>> from ads.common import auth as authutil
>>> reader = JsonlReader(path=path, auth=authutil.api_keys(), encoding="utf-8")
>>> next(reader.read())

read(skip: Optional[int] = None) → Generator[Dict, Any, Any]

Reads and yields the content of the file.

Parameters:

skip ((int, optional). Defaults to None.) – The number of records that should be skipped.

Returns:

The content of the file.

Return type:

Generator[Dict, Any, Any]

Raises:

ValueError – If skip not empty and not a positive integer.
FileNotFoundError – When file not found.

ads.data_labeling.reader.metadata_reader module

class ads.data_labeling.reader.metadata_reader.DLSMetadataReader(dataset_id: str, compartment_id: str, auth: dict)

Bases: Reader

DLSMetadataReader class which reads the metadata jsonl file from the cloud.

Initializes the DLS metadata reader instance.

Parameters:

dataset_id (str) – The dataset OCID.
compartment_id (str) – The compartment OCID of the dataset.
auth (dict) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Raises:

ValueError – When dataset_id is empty or not a string.:
TypeError – When dataset_id not a string.:

read() → Metadata

Reads the content from the metadata file.

Returns:

The metadata of the labeled dataset.

Return type:

Metadata

Raises:

DatasetNotFoundError – If dataset not found.
ReadDatasetError – If any error occured in attempt to read dataset.

exception ads.data_labeling.reader.metadata_reader.DatasetNotFoundError(id: str): Bases: Exception

exception ads.data_labeling.reader.metadata_reader.EmptyMetadata

Bases: Exception

Empty Metadata.

class ads.data_labeling.reader.metadata_reader.ExportMetadataReader(path: str, auth: Optional[Dict] = None, encoding='utf-8')

Bases: JsonlReader

ExportMetadataReader class which reads the metadata jsonl file from local/object storage path.

Initiates a JsonlReader object.

Parameters:

path (str) – object storage path or local path for a file.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
encoding ((str, optional). Defaults to 'utf-8'.) – Encoding of files. Only used for “TEXT” dataset.

Examples

>>> from ads.data_labeling.reader.jsonl_reader import JsonlReader
>>> path = "your/path/to/jsonl/file.jsonl"
>>> from ads.common import auth as authutil
>>> reader = JsonlReader(path=path, auth=authutil.api_keys(), encoding="utf-8")
>>> next(reader.read())

read() → Metadata

Reads the content from the metadata file.

Returns:: The metadata of the labeled dataset.
Return type:: Metadata

class ads.data_labeling.reader.metadata_reader.MetadataReader(reader: Reader)

Bases: object

MetadataReader class which reads and extracts the labeled dataset metadata.

Examples

>>> from ads.data_labeling import MetadataReader
>>> import oci
>>> import os
>>> from ads.common import auth as authutil
>>> reader = MetadataReader.from_export_file("metadata_export_file_path",
...                                 auth=authutil.api_keys())
>>> reader.read()

Initiate a MetadataReader instance.

Parameters:: reader (Reader) – Reader instance which reads and extracts the labeled dataset metadata.

classmethod from_DLS(dataset_id: str, compartment_id: Optional[str] = None, auth: Optional[dict] = None) → MetadataReader

Contructs a MetadataReader instance.

Parameters:

dataset_id (str) – The dataset OCID.
compartment_id ((str, optional). Default None) – The compartment OCID of the dataset.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

The MetadataReader instance whose reader is a DLSMetadataReader instance.

Return type:

MetadataReader

classmethod from_export_file(path: str, auth: Optional[Dict] = None) → MetadataReader

Contructs a MetadataReader instance.

Parameters:

path (str) – metadata file path, can be either local or object storage path.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

The MetadataReader instance whose reader is a ExportMetadataReader instance.

Return type:

MetadataReader

read() → Metadata

Reads the content from the metadata file.

Returns:: The metadata of the labeled dataset.
Return type:: Metadata

exception ads.data_labeling.reader.metadata_reader.ReadDatasetError(id: str): Bases: Exception

ads.data_labeling.reader.record_reader module

class ads.data_labeling.reader.record_reader.RecordReader(reader: Reader, parser: Parser, loader: Optional[Loader] = None, include_unlabeled: bool = False, encoding: str = 'utf-8', materialize: bool = False)

Bases: object

Record Reader Class consists of parser, reader and loader. Reader reads the the content from the record file. Parser parses the label for each record. And Loader loads the content of the file path in that record.

Examples

>>> import os
>>> import oci
>>> from ads.data_labeling import RecordReader
>>> from ads.common import auth as authutil
>>> file_path = "/path/to/your_record.jsonl"
>>> dataset_type = "IMAGE"
>>> annotation_type = "BOUNDING_BOX"
>>> record_reader = RecordReader.from_export_file(file_path, dataset_type, annotation_type, "image_file_path", authutil.api_keys())
>>> next(record_reader.read())

Initiates a RecordReader instance.

Parameters:

reader (Reader) – Reader instance to read content from the record file.
parser (Parser) – Parser instance to parse the labels from record file.
loader (Loader. Defaults to None.) – Loader instance to load the content from the file path in the record.
materialize (bool, optional. Defaults to False.) – Whether to materialize the content using loader.
include_unlabeled ((bool, optional). Default to False.) – Whether to load the unlabeled records or not.
encoding (str, optional) – Encoding for text files. Used only to extract the content of the text dataset contents.

Raises:

ValueError – If the record reader and record parser must be specified. If the loader is not specified when materialize if True.

classmethod from_DLS(dataset_id: str, dataset_type: str, annotation_type: str, dataset_source_path: str, compartment_id: Optional[str] = None, auth: Optional[Dict] = None, include_unlabeled: bool = False, encoding: str = 'utf-8', materialize: bool = False, format: Optional[str] = None, categories: Optional[List[str]] = None) → RecordReader

Constructs Record Reader instance.

Parameters:

dataset_id (str) – The dataset OCID.
dataset_type (str) – Dataset type. Currently supports TEXT, IMAGE and DOCUMENT.
annotation_type (str) – Annotation Type. Currently TEXT supports SINGLE_LABEL, MULTI_LABEL, ENTITY_EXTRACTION. IMAGE supports SINGLE_LABEL, MULTI_LABEL and BOUNDING_BOX. DOCUMENT supports SINGLE_LABEL and MULTI_LABEL.
dataset_source_path (str) – Dataset source path.
compartment_id ((str, optional). Defaults to None.) – The compartment OCID of the dataset.
auth ((dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
encoding ((str, optional). Defaults to 'utf-8'.) – Encoding for files.
materialize ((bool, optional). Defaults to False.) – Whether the content of the dataset file should be loaded or it should return the file path to the content. By default the content will not be loaded.
format ((str, optional). Defaults to None.) – Output format of annotations. Can be None, “spacy” for dataset Entity Extraction type or “yolo” for Object Detection type. When None, it outputs List[NERItem] or List[BoundingBoxItem]. When “spacy”, it outputs List[Tuple]. When “yolo”, it outputs List[List[Tuple]].
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

The RecordReader instance.

Return type:

RecordReader

classmethod from_export_file(path: str, dataset_type: str, annotation_type: str, dataset_source_path: str, auth: Optional[Dict] = None, include_unlabeled: bool = False, encoding: str = 'utf-8', materialize: bool = False, format: Optional[str] = None, categories: Optional[List[str]] = None, includes_metadata=False) → RecordReader

Initiates a RecordReader instance.

Parameters:

path (str) – Record file path.
dataset_type (str) – Dataset type. Currently supports TEXT, IMAGE and DOCUMENT.
annotation_type (str) – Annotation Type. Currently TEXT supports SINGLE_LABEL, MULTI_LABEL, ENTITY_EXTRACTION. IMAGE supports SINGLE_LABEL, MULTI_LABEL and BOUNDING_BOX. DOCUMENT supports SINGLE_LABEL and MULTI_LABEL.
dataset_source_path (str) – Dataset source path.
auth ((dict, optional). Default None) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
include_unlabeled ((bool, optional). Default to False.) – Whether to load the unlabeled records or not.
encoding ((str, optional). Defaults to "utf-8".) – Encoding for text files. Used only to extract the content of the text dataset contents.
materialize ((bool, optional). Defaults to False.) – Whether to materialize the content by loader.
format ((str, optional). Defaults to None.) – Output format of annotations. Can be None, “spacy” for dataset Entity Extraction type or “yolo” for Object Detection type. When None, it outputs List[NERItem] or List[BoundingBoxItem]. When “spacy”, it outputs List[Tuple]. When “yolo”, it outputs List[List[Tuple]].
categories ((List[str], optional). Defaults to None.) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]
includes_metadata ((bool, optional). Defaults to False.) – Determines whether the export file includes metadata or not.

Returns:

A RecordReader instance.

Return type:

RecordReader

read() → Generator[Tuple[str, Union[List, str]], Any, Any]

Reads the record.

Yields:: Generator[Tuple[str, Union[List, str]], Any, Any] – File path, content and labels in a tuple.

Module contents

ads.data_labeling.visualizer package

Submodules

ads.data_labeling.visualizer.image_visualizer module

The module that helps to visualize Image Dataset.

ads.data_labeling.visualizer.image_visualizer.render(items: List[LabeledImageItem], options: Dict = None): Renders Labeled Image dataset.

Examples

>>> bbox1 = BoundingBoxItem(bottom_left=(0.3, 0.4),
>>>                        top_left=(0.3, 0.09),
>>>                        top_right=(0.86, 0.09),
>>>                        bottom_right=(0.86, 0.4),
>>>                        labels=['dolphin', 'fish'])

>>> record1 = LabeledImageItem(img_obj1, [bbox1])

>>> bbox2 = BoundingBoxItem(bottom_left=(0.2, 0.4),
>>>                        top_left=(0.2, 0.2),
>>>                        top_right=(0.8, 0.2),
>>>                        bottom_right=(0.8, 0.4),
>>>                        labels=['dolphin'])
>>> bbox3 = BoundingBoxItem(bottom_left=(0.5, 1.0),
>>>                        top_left=(0.5, 0.8),
>>>                        top_right=(0.8, 0.8),
>>>                        bottom_right=(0.8, 1.0),
>>>                        labels=['shark'])

>>> record2 = LabeledImageItem(img_obj2, [bbox2, bbox3])
>>> render(items = [record1, record2], options={"default_color":"blue", "colors": {"dolphin":"blue", "whale":"red"}})

class ads.data_labeling.visualizer.image_visualizer.ImageLabeledDataFormatter

Bases: object

The ImageRender class to render Image items in a notebook session.

static render_item(item: LabeledImageItem, options: Optional[Dict] = None, path: Optional[str] = None) → None

Renders image dataset.

Parameters:

item (LabeledImageItem) – Item to render.
options (Optional[dict]) – Render options.
path (str) – Path to save the image with annotations to local directory.

Returns:

Nothing.

Return type:

None

Raises:

ValueError – If items not provided. If path is not valid.
TypeError – If items provided in a wrong format.

class ads.data_labeling.visualizer.image_visualizer.LabeledImageItem(img: ImageFile, boxes: List[BoundingBoxItem])

Bases: object

Data class representing Image Item.

img

the labeled image object.

Type:: ImageFile

boxes

a list of BoundingBoxItem

Type:: List[BoundingBoxItem]

boxes: List[BoundingBoxItem]

img: ImageFile

class ads.data_labeling.visualizer.image_visualizer.RenderOptions(default_color: str, colors: Optional[dict])

Bases: object

Data class representing render options.

default_color

The specified default color.

Type:: str

colors

The multiple specified colors.

Type:: Optional[dict]

colors: Optional[dict]

default_color: str

classmethod from_dict(options: dict) → RenderOptions

Constructs an instance of RenderOptions from a dictionary.

Parameters:: options (dict) – Render options in dictionary format.
Returns:: The instance of RenderOptions.
Return type:: RenderOptions

to_dict()

Converts RenderOptions instance to dictionary format.

Returns:: The render options in dictionary format.
Return type:: dict

exception ads.data_labeling.visualizer.image_visualizer.WrongEntityFormat: Bases: ValueError

ads.data_labeling.visualizer.image_visualizer.render(items: List[LabeledImageItem], options: Optional[Dict] = None, path: Optional[str] = None) → None

Render image dataset.

Parameters:

items (List[LabeledImageItem]) – The list of LabeledImageItem to render.
options (dict, optional) – The options for rendering.
path (str) – Path to save the images with annotations to local directory.

Returns:

Nothing.

Return type:

None

Raises:

ValueError – If items not provided. If path is not valid.
TypeError – If items provided in a wrong format.

Examples

>>> bbox1 = BoundingBoxItem(bottom_left=(0.3, 0.4),
>>>                        top_left=(0.3, 0.09),
>>>                        top_right=(0.86, 0.09),
>>>                        bottom_right=(0.86, 0.4),
>>>                        labels=['dolphin', 'fish'])

>>> record1 = LabeledImageItem(img_obj1, [bbox1])
>>> render(items = [record1])

ads.data_labeling.visualizer.text_visualizer module

The module that helps to visualize NER Text Dataset.

ads.data_labeling.visualizer.text_visualizer.render(items: List[LabeledTextItem], options: Dict = None) → str: Renders NER dataset to Html format.

Examples

>>> record1 = LabeledTextItem("London is the capital of the United Kingdom", [NERItem('city', 0, 6), NERItem("country", 29, 14)])
>>> record2 = LabeledTextItem("Houston area contractor seeking a Sheet Metal Superintendent.", [NERItem("city", 0, 6)])
>>> result = render(items = [record1, record2], options={"default_color":"#DDEECC", "colors": {"city":"#DDEECC", "country":"#FFAAAA"}})
>>> display(HTML(result))

class ads.data_labeling.visualizer.text_visualizer.LabeledTextItem(txt: str, ents: List[NERItem])

Bases: object

Data class representing NER Item.

txt

The labeled sentence.

Type:: str

ents

The list of entities.

Type:: List[NERItem]

ents: List[NERItem]

txt: str

class ads.data_labeling.visualizer.text_visualizer.RenderOptions(default_color: str, colors: Optional[dict])

Bases: object

Data class representing render options.

default_color

The specified default color.

Type:: str

colors

The multiple specified colors.

Type:: Optional[dict]

colors: Optional[dict]

default_color: str

classmethod from_dict(options: dict) → RenderOptions

Constructs an instance of RenderOptions from a dictionary.

Parameters:: options (dict) – Render options in dictionary format.
Returns:: The instance of RenderOptions.
Return type:: RenderOptions

to_dict()

Converts RenderOptions instance to dictionary format.

Returns:: The render options in dictionary format.
Return type:: dict

class ads.data_labeling.visualizer.text_visualizer.TextLabeledDataFormatter

Bases: object

The TextLabeledDataFormatter class to render NER items into Html format.

static render(items: List[LabeledTextItem], options: Optional[Dict] = None) → str

Renders NER dataset to Html format.

Parameters:

items (List[LabeledTextItem]) – Items to render.
options (Optional[dict]) – Render options.

Returns:

Html representation of rendered NER dataset.

Return type:

str

Raises:

ValueError – If items not provided.
TypeError – If items provided in a wrong format.

ads.data_labeling.visualizer.text_visualizer.render(items: List[LabeledTextItem], options: Optional[Dict] = None) → str

Renders NER dataset to Html format.

Parameters:

items (List[LabeledTextItem]) – The list of NER items to render.
options (dict, optional) – The options for rendering.

Returns:

Html string.

Return type:

str

Examples

>>> record = LabeledTextItem("London is the capital of the United Kingdom", [NERItem('city', 0, 6), NERItem("country", 29, 14)])
>>> result = render(items = [record], options={"default_color":"#DDEECC", "colors": {"city":"#DDEECC", "country":"#FFAAAA"}})
>>> display(HTML(result))

Module contents

Submodules

ads.data_labeling.boundingbox module

class ads.data_labeling.boundingbox.BoundingBoxItem(top_left: ~typing.Tuple[float, float], bottom_left: ~typing.Tuple[float, float], bottom_right: ~typing.Tuple[float, float], top_right: ~typing.Tuple[float, float], labels: ~typing.List[str] = <factory>)

Bases: object

BoundingBoxItem class representing bounding box label.

labels

List of labels for this bounding box.

Type:: List[str]

top_left

Top left corner of this bounding box.

Type:: Tuple[float, float]

bottom_left

Bottom left corner of this bounding box.

Type:: Tuple[float, float]

bottom_right

Bottom right corner of this bounding box.

Type:: Tuple[float, float]

top_right

Top right corner of this bounding box.

Type:: Tuple[float, float]

Examples

>>> item = BoundingBoxItem(
...     labels = ['cat','dog']
...     bottom_left=(0.2, 0.4),
...     top_left=(0.2, 0.2),
...     top_right=(0.8, 0.2),
...     bottom_right=(0.8, 0.4))
>>> item.to_yolo(categories = ['cat','dog', 'horse'])

bottom_left: Tuple[float, float]

bottom_right: Tuple[float, float]

classmethod from_yolo(bbox: List[Tuple], categories: Optional[List[str]] = None) → BoundingBoxItem

Converts the YOLO formated annotations to BoundingBoxItem.

Parameters:

bboxes (List[Tuple]) – The list of bounding box annotations in YOLO format. Example: [(0, 0.511560675, 0.50234826, 0.47013485, 0.57803468)]
categories (List[str]) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

The BoundingBoxItem.

Return type:

BoundingBoxItem

Raises:

TypeError – When categories list has a wrong format.

labels: List[str]

to_yolo(categories: List[str]) → List[Tuple[int, float, float, float, float]]

Converts BoundingBoxItem to the YOLO format.

Parameters:

categories (List[str]) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

The list of YOLO formatted bounding boxes.

Return type:

List[Tuple[int, float, float, float, float]]

Raises:

ValueError – When categories list not provided. When categories list not matched with the labels.
TypeError – When categories list has a wrong format.

top_left: Tuple[float, float]

top_right: Tuple[float, float]

class ads.data_labeling.boundingbox.BoundingBoxItems(items: ~typing.List[~ads.data_labeling.boundingbox.BoundingBoxItem] = <factory>)

Bases: object

BoundingBoxItems class which consists of a list of BoundingBoxItem.

items

List of BoundingBoxItem.

Type:: List[BoundingBoxItem]

Examples

>>> item = BoundingBoxItem(
...     labels = ['cat','dog']
...     bottom_left=(0.2, 0.4),
...     top_left=(0.2, 0.2),
...     top_right=(0.8, 0.2),
...     bottom_right=(0.8, 0.4))
>>> items = BoundingBoxItems(items = [item])
>>> items.to_yolo(categories = ['cat','dog', 'horse'])

items: List[BoundingBoxItem]

to_yolo(categories: List[str]) → List[Tuple[int, float, float, float, float]]

Converts BoundingBoxItems to the YOLO format.

Parameters:

categories (List[str]) – The list of object categories in proper order for model training. Example: [‘cat’,’dog’,’horse’]

Returns:

The list of YOLO formatted bounding boxes.

Return type:

List[Tuple[int, float, float, float, float]]

Raises:

ValueError – When categories list not provided. When categories list not matched with the labels.
TypeError – When categories list has a wrong format.

ads.data_labeling.constants module

class ads.data_labeling.constants.AnnotationType

Bases: object

AnnotationType class which contains all the annotation types that data labeling service supports.

BOUNDING_BOX = 'BOUNDING_BOX'

ENTITY_EXTRACTION = 'ENTITY_EXTRACTION'

MULTI_LABEL = 'MULTI_LABEL'

SINGLE_LABEL = 'SINGLE_LABEL'

class ads.data_labeling.constants.DatasetType

Bases: object

DatasetType class which contains all the dataset types that data labeling service supports.

DOCUMENT = 'DOCUMENT'

IMAGE = 'IMAGE'

TEXT = 'TEXT'

class ads.data_labeling.constants.Formats

Bases: object

Common formats class which contains all the common formats that are supported to convert to.

SPACY = 'spacy'

YOLO = 'yolo'

ads.data_labeling.data_labeling_service module

class ads.data_labeling.data_labeling_service.DataLabeling(compartment_id: Optional[str] = None, dls_cp_client_auth: Optional[dict] = None, dls_dp_client_auth: Optional[dict] = None)

Bases: OCIWorkRequestMixin

Class for data labeling service. Integrate the data labeling service APIs.

Examples

>>> import ads
>>> import pandas
>>> from ads.data_labeling.data_labeling_service import DataLabeling
>>> ads.set_auth("api_key")
>>> dls = DataLabeling()
>>> dls.list_dataset()
>>> metadata_path = dls.export(dataset_id="your dataset id",
...     path="oci://<bucket_name>@<namespace>/folder")
>>> df = pd.DataFrame.ads.read_labeled_data(metadata_path)

Initialize a DataLabeling class.

Parameters:

compartment_id (str, optional) – OCID of data labeling datasets’ compartment
dls_cp_client_auth (dict, optional) – Data Labeling control plane client auth. Default is None. The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.
dls_dp_client_auth (dict, optional) – Data Labeling data plane client auth. Default is None. The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

Nothing.

Return type:

None

export(dataset_id: str, path: str, include_unlabeled=False) → str

Export dataset based on the dataset_id and save the jsonl files under the path (metadata jsonl file and the records jsonl file) to the object storage path provided by the user and return the metadata jsonl path.

Parameters:

dataset_id (str) – The dataset id of which the snapshot will be generated.
path (str) – The object storage path to store the generated snapshot. “oci://<bucket_name>@<namespace>/prefix”
include_unlabeled (bool, Optional. Defaults to False.) – Whether to include unlabeled records or not.

Returns:

oci path of the metadata jsonl file.

Return type:

str

list_dataset(**kwargs) → DataFrame

List all the datasets created from the data labeling service under a given compartment.

Parameters:: kwargs (dict, optional) – Additional keyword arguments will be passed to oci.data_labeling_serviceDataLabelingManagementClient.list_datasets method.
Returns:: pandas dataframe which contains the dataset information.
Return type:: pandas.DataFrame
Raises:: Exception – If pagination.list_call_get_all_results() fails

ads.data_labeling.metadata module

class ads.data_labeling.metadata.Metadata(source_path: str = '', records_path: str = '', labels: ~typing.List[str] = <factory>, dataset_name: str = '', compartment_id: str = '', dataset_id: str = '', annotation_type: str = '', dataset_type: str = '')

ads.dataflow.dataflow.DataFlowApp

The class that representing the labeled dataset metadata.

source_path

Contains information on where all the source data(image/text/document) stores.

Type:: str

records_path

Contains information on where records jsonl file stores.

Type:: str

labels

List of classes/labels for the dataset.

Type:: List

dataset_name

Dataset display name on the Data Labeling Service console.

Type:: str

compartment_id

Compartment id of the labeled dataset.

Type:: str

dataset_id

Dataset id.

Type:: str

annotation_type

Type of the labeling/annotation task. Currently supports SINGLE_LABEL, MULTI_LABEL, ENTITY_EXTRACTION, BOUNDING_BOX.

Type:: str

dataset_type

Type of the dataset. Currently supports Text, Image, DOCUMENT.

Type:: str

annotation_type: str = ''

compartment_id: str = ''

dataset_id: str = ''

dataset_name: str = ''

dataset_type: str = ''

classmethod from_dls_dataset(dataset: Dataset) → Metadata

Contructs a Metadata instance from OCI DLS dataset.

Parameters:: dataset (OCIDLSDataset) – OCIDLSDataset object.
Returns:: The ads labeled dataset metadata instance.
Return type:: Metadata

labels: List[str]

records_path: str = ''

source_path: str = ''

to_dataframe() → DataFrame

Converts the metadata to dataframe format.

Returns:: The metadata in Pandas dataframe format.
Return type:: pandas.DataFrame

to_dict() → Dict

Converts to dictionary representation.

Returns:: The metadata in dictionary type.
Return type:: Dict

ads.data_labeling.ner module

class ads.data_labeling.ner.NERItem(label: str = '', offset: int = 0, length: int = 0)

Bases: object

NERItem class which is a representation of a token span.

label

Entity name.

Type:: str

offset

The token span’s entity start index position in the text.

Type:: int

length

Length of the token span.

Type:: int

classmethod from_spacy(token) → NERItem

label: str = ''

length: int = 0

offset: int = 0

to_spacy() → tuple

Converts one NERItem to the spacy format.

Returns:: NERItem in the spacy format
Return type:: Tuple

class ads.data_labeling.ner.NERItems(items: ~typing.List[~ads.data_labeling.ner.NERItem] = <factory>)

Bases: object

NERItems class consists of a list of NERItem.

items

List of NERItem.

Type:: List[NERItem]

items: List[NERItem]

to_spacy() → List[tuple]

Converts NERItems to the spacy format.

Returns:: List of NERItems in the Spacy format.
Return type:: List[tuple]

exception ads.data_labeling.ner.WrongEntityFormatLabelIsEmpty: Bases: ValueError

exception ads.data_labeling.ner.WrongEntityFormatLabelNotString: Bases: ValueError

exception ads.data_labeling.ner.WrongEntityFormatLengthIsNegative: Bases: ValueError

exception ads.data_labeling.ner.WrongEntityFormatLengthNotInteger: Bases: ValueError

exception ads.data_labeling.ner.WrongEntityFormatOffsetIsNegative: Bases: ValueError

exception ads.data_labeling.ner.WrongEntityFormatOffsetNotInteger: Bases: ValueError

ads.data_labeling.record module

class ads.data_labeling.record.Record(path: str = '', content: Optional[Any] = None, annotation: Optional[Union[Tuple, str, List[BoundingBoxItem], List[NERItem]]] = None)

Bases: object

Class representing Record.

path

File path.

Type:: str

content

Content of the record.

Type:: Any

annotation

Annotation/label of the record.

Type:: Union[Tuple, str, List[BoundingBoxItem], List[NERItem]]

annotation: Union[Tuple, str, List[BoundingBoxItem], List[NERItem]] = None

content: Any = None

path: str = ''

to_dict() → Dict

Convert the Record instance to a dictionary.

Returns:: Dictionary representation of the Record instance.
Return type:: Dict

to_tuple() → Tuple[str, Any, Union[Tuple, str, List[BoundingBoxItem], List[NERItem]]]

Convert the Record instance to a tuple.

Returns:: Tuple representation of the Record instance.
Return type:: Tuple

Module contents

ads.database package

Submodules

ads.database.connection module

class ads.database.connection.Connector(secret_id: Optional[str] = None, key: Optional[str] = None, repository_path: Optional[str] = None, **kwargs)

Bases: object

Validate that a connection could be made for the given set of connection parameters, and contruct a Connector object provided that the validation is successful.

Parameters:

secret_id (str, optional) – The ocid of the secret to retrieve from Oracle Cloud Infrastructure Vault.
key (str, optional) – The key to find the database directory.
repository_path (str, optional) – The local database information store, default to ~/.database unless specified otherwise.
kwargs (dict, optional) – Name-value pairs that are to be added to the list of connection parameters. For example, database_name=”mydb”, database_type=”oracle”, username = “root”, password = “pwd”.

Return type:

A Connector object.

connect()

class ads.database.connection.OracleConnector(oracle_connection_config): Bases: object

ads.database.connection.get_repository(key: str, repository_path: Optional[str] = None) → dict

Get all values from local database store.

Parameters:

key (str) – The key to find the database directory.
repository_path (str, optional) – The path to local database store, default to ~/.database unless specified otherwise.

Return type:

A dictionary of all values in the store.

ads.database.connection.import_wallet(wallet_path: str, key: str, repository_path: Optional[str] = None) → None

Saves wallet to local database store. Unzip the wallet zip file, update sqlnet.ora and store wallet files.

Parameters:

wallet_path (str) – The local path to the downloaded wallet zip file.
key (str) – The key to find the database directory.
repository_path (str, optional) – The local database store, default to ~/.database unless specified otherwise.

ads.database.connection.update_repository(value: dict, key: str, replace: bool = True, repository_path: Optional[str] = None) → dict

Saves value into local database store.

Parameters:

value (dict) – The values to store locally.
key (str) – The key to find the local database directory.
replace (bool, default to True) – If set to false, updates the stored value.
repository_path (str: str, optional) – The local database store, default to ~/.database unless specified otherwise.

Return type:

A dictionary of all values in the repository for the given key.

Module contents

ads.dataflow package

Submodules

ads.dataflow.dataflow module

class ads.dataflow.dataflow.DataFlow(compartment_id=None, dataflow_base_folder='/home/datascience/dataflow', os_auth=None, df_auth=None)

Bases: object

create_app(app_config: dict, overwrite_script=False, overwrite_archive=False) → object

Create a new dataflow application with the supplied app config. app_config contains parameters needed to create a new application, according to oci.data_flow.models.CreateApplicationDetails.

Parameters:

app_config (dict) – the config file that contains all necessary parameters used to create a dataflow app
overwrite_script (bool) – whether to overwrite the existing pyscript script on Object Storage
overwrite_archive (bool) – whether to overwrite the existing archive file on Object Storage

Returns:

df_app – New dataflow application.

Return type:

oci.dataflow.models.Application

get_app(app_id: str)

Get the Project based on app_id.

Parameters:: app_id (str, required) – The OCID of the dataflow app to get.
Returns:: app – The oci.dataflow.models.Application with the matching ID.
Return type:: oci.dataflow.models.Application

list_apps(include_deleted: bool = False, compartment_id: Optional[str] = None, datetime_format: str = '%Y-%m-%d %H:%M:%S', **kwargs) → object

List all apps in a given compartment, or in the current notebook session’s compartment.

Parameters:

include_deleted (bool, optional, default=False) – Whether to include deleted apps in the returned list.
compartment_id (str, optional, default: NB_SESSION_COMPARTMENT_OCID) – The compartment specified to list apps.
datetime_format (str, optional, default: '%Y-%m-%d %H:%M:%S') – Change format for date time fields.

Returns:

dsl – List of Dataflow applications.

Return type:

List

load_app(app_id: str, target_folder: Optional[str] = None) → object

Load an existing dataflow application based on application id. The existing dataflow application can be created either from dataflow service or the dataflow integration of ADS.

Parameters:

app_id (str, required) – The OCID of the dataflow app to load.
target_folder (str, optional,) – the folder to store the local artifacts of this application. If not specified, the target_folder will use the dataflow_base_folder by default.

Returns:

dfa – A dataflow application of type ads.dataflow.dataflow.DataFlowApp

Return type:

prepare_app(display_name: str, script_bucket: str, pyspark_file_path: str, spark_version: str = '2.4.4', compartment_id: Optional[str] = None, archive_path: Optional[str] = None, archive_bucket: Optional[str] = None, logs_bucket: str = 'dataflow-logs', driver_shape: str = 'VM.Standard2.4', executor_shape: str = 'VM.Standard2.4', num_executors: int = 1, arguments: list = [], script_parameters: dict = []) → dict

Check if the parameters provided by users to create an application are valid and then prepare app_configuration for creating an app or saving for future reuse.

Parameters:

display_name (str, required) – A user-friendly name. This name is not necessarily unique.
script_bucket (str, required) – bucket in object storage to upload the pyspark file
pyspark_file_path (str, required) – path to the pyspark file
spark_version (str) – Allowed values are “2.4.4”, “3.0.2”.
compartment_id (str) – OCID of the compartment to create a dataflow app. If not provided, compartment_id will use the same as the notebook session.
archive_path (str, optional) – path to the archive file
archive_bucket (str, optional) – bucket in object storage to upload the archive file
logs_bucket (str, default is 'dataflow-logs') – bucket in object storage to put run logs
driver_shape (str) – The value to assign to the driver_shape property of this CreateApplicationDetails. Allowed values for this property are: “VM.Standard2.1”, “VM.Standard2.2”, “VM.Standard2.4”, “VM.Standard2.8”, “VM.Standard2.16”, “VM.Standard2.24”.
executor_shape (str) – The value to assign to the executor_shape property of this CreateApplicationDetails. Allowed values for this property are: “VM.Standard2.1”, “VM.Standard2.2”, “VM.Standard2.4”, “VM.Standard2.8”, “VM.Standard2.16”, “VM.Standard2.24”.
num_executors (int) – The number of executor VMs requested.
arguments (list of str) – The values passed into the command line string to run the application
script_parameters (dict) – The value of the parameters passed to the running application as command line arguments for the pyspark script.

Returns:

app_configuration

Return type:

dictionary containing all the validated params for CreateApplicationDetails.

template(job_type: str = 'standard_pyspark', script_str: str = '', file_dir: Optional[str] = None, file_name: Optional[str] = None) → str

Populate a prewritten pyspark or sparksql python script with user’s choice to write additional lines and save in local directory.

Parameters:

job_type (str, default is 'standard_pyspark') – Currently supports two types, ‘standard_pyspark’ or ‘sparksql’
script_str (str, optional, default is '') – code provided by user to write in the python script
file_dir (str, optional) – Directory to save the python script in local directory
file_name (str, optional) – name of the python script to save to the local directory

Returns:

script_path – Path to the template generated python file in local directory

Return type:

str

class ads.dataflow.dataflow.DataFlowApp(app_config, app_response, app_dir, oci_link, **kwargs)

Bases: DataFlow

property config: dict

Retrieve the app_config file used to create the data flow app

Returns:: app_config – dictionary containing all the validated params for this DataFlowApp
Return type:: Dict

get_run(run_id: str)

Get the Run based on run_id

Parameters:: run_id (str, required) – The OCID of the dataflow run to get.
Returns:: df_run – The oci.dataflow.models.Run with the matching ID.
Return type:: oci.dataflow.models.Run

list_runs(include_failed: bool = False, datetime_format: str = '%Y-%m-%d %H:%M:%S', **kwargs) → object

List all run of a dataflow app

Parameters:

include_failed (bool, optional, default=False) – Whether to include failed runs in the returned list
datetime_format (str, optional, default: '%Y-%m-%d %H:%M:%S') – Change format for date time fields

Returns:

df_runs – List of Data flow runs.

Return type:

List

property oci_link: object

Retrieve the oci link of the data flow app

Returns:: oci_link – a link to the app page in an oci console.
Return type:: str

prepare_run(run_display_name: str, compartment_id: Optional[str] = None, logs_bucket: str = '', driver_shape: str = 'VM.Standard2.4', executor_shape: str = 'VM.Standard2.4', num_executors: int = 1, **kwargs) → dict

Check if the parameters provided by users to create a run are valid and then prepare run_config for creating run details.

Parameters:

run_display_name (str) – A user-friendly name. This name is not necessarily unique.
compartment_id (str) – OCID of the compartment to create a dataflow run. If not provided, compartment_id will use the same as the dataflow app.
logs_bucket (str) – bucket in object storage to put run logs, if not provided, will use the same logs_bucket as defined in app_config
driver_shape (str) – The value to assign to the driver_shape property of this CreateApplicationDetails. Allowed values for this property are: “VM.Standard2.1”, “VM.Standard2.2”, “VM.Standard2.4”, “VM.Standard2.8”, “VM.Standard2.16”, “VM.Standard2.24”.
executor_shape (str) – The value to assign to the executor_shape property of this CreateApplicationDetails. Allowed values for this property are: “VM.Standard2.1”, “VM.Standard2.2”, “VM.Standard2.4”, “VM.Standard2.8”, “VM.Standard2.16”, “VM.Standard2.24”.
num_executors (int) – The number of executor VMs requested.

Returns:

run_config – Dictionary containing all the validated params for CreateRunDetails.

Return type:

Dict

run(run_config: dict, save_log_to_local: bool = False, copy_script_to_object_storage: bool = True, copy_archive_to_object_storage: bool = True, pyspark_file_path: Optional[str] = None, archive_path: Optional[str] = None, wait: bool = True) → object

Create a new dataflow run with the supplied run config. run_config contains parameters needed to create a new run, according to oci.data_flow.models.CreateRunDetails.

Parameters:

run_config (dict, required) – The config file that contains all necessary parameters used to create a dataflow run
save_log_to_local (bool, optional) – A boolean value that defaults to false. If set to true, it saves the log files to local dir
copy_script_to_object_storage (bool, optional) – A boolean value that defaults to true. Local script will be copied to object storage
copy_archive_to_object_storage (bool, optional) – A boolean value that defaults to true. Local archive file will be copied to object storage
pyspark_file_path (str, optional) – The pyspark file path used for creating the dataflow app. if pyspark_file_path isn’t specified then reuse the path that the app was created with.
archive_path (str, optional) – The archive file path used for creating the dataflow app. if archive_path isn’t specified then reuse the path that the app was created with.
wait (bool, optional) – A boolean value that defaults to true. When True, the return will be ads.dataflow.dataflow.DataFlowRun in terminal state. When False, the return will be a ads.dataflow.dataflow.RunObserver.

Returns:

df_run – Either a new Data Flow run or a run observer.

Return type:

Variable

class ads.dataflow.dataflow.DataFlowLog(text, oci_path, log_local_dir)

Bases: object

head(n: int = 10)

Show the first n lines of the log as the output of the notebook cell

Parameters:: n (int, default is 10) – the number of lines from head of the log file
Return type:: None

property local_dir

Get the local directory where the log file is saved.

Returns:: local_dir – Path to the local directory where the log file is saved.
Return type:: str

property local_path

Get the path of the log file in local directory

Returns:: local_path – Path of the log file in local directory
Return type:: str

property oci_path

Get the path of the log file in object storage

Returns:: oci_path – Path of the log file in object storage
Return type:: str

save(log_dir=None)

save the log file to a local directory.

Parameters:

log_dir (str,) – The path to the local directory to save log file, if not
set –
default. (log will be saved to the _local_dir by) –

Return type:

None

show_all()

Show all content of the log as the output of the notebook cell

Return type:: None

tail(n: int = 10)

Show the last n lines of the log as the output of the notebook cell

Parameters:: n (int, default is 10) – the number of lines from tail of the log file
Return type:: None

class ads.dataflow.dataflow.DataFlowRun(run_config, run_response, save_log_to_local, local_dir, **kwargs)

Bases: DataFlow

LOG_OUTPUTS = ['stdout', 'stderr']

property config: dict

Retrieve the run_config file used to create the Data Flow run

Returns:: run_config – dictionary containing all the validated params for this DataFlowRun
Return type:: Dict

fetch_log(log_type: str) → object

Fetch the log information of a run

Parameters:: log_type (str, have two values, 'stdout' or 'stderr') –
Returns:: dfl – a Data Flow log object
Return type:: DataFlowLog

property local_dir: str

Retrieve the local directory of the data flow run

Returns:: local_dir – the local path to the Data Flow run
Return type:: str

property log_stderr: object

Retrieve the stderr of the data flow run

Returns:: log_error – a clickable link that opens the stderror log in another tab in jupyter notebook environment
Return type:: ads.dataflow.dataflow.DataFlowLog

property log_stdout: object

Retrieve the stdout of the data flow run

Returns:: log_out – a clickable link that opens the stdout log in another tab in a JupyterLab notebook environment
Return type:: ads.dataflow.dataflow.DataFlowLog

property oci_link: object

Retrieve the oci link of the data flow run

Returns:: oci_link – link to the run page in an oci console
Return type:: str

property status: str

Retrieve the status of the data flow run

Returns:: status – String that describes the status of the run
Return type:: str

update_config(param_dict) → None

Modify the run_config file used to create the data flow run

Parameters:: param_dict (Dict) – Dictionary containing the key value pairs of the run_config parameters and the updated values.
Return type:: None

class ads.dataflow.dataflow.RunObserver(app, run_config, save_log_to_local)

Bases: object

property config: dict

Retrieve the run_config file used to create the data flow run

Returns:: run_config – Dictionary containing all the validated parameters for this Data Flow run
Return type:: Dict

property local_dir: str

Retrieve the local directory of the data flow run

Returns:: local_dir – the local path to the Data Flow run
Return type:: str

property oci_link: object

Retrieve the oci link of the data flow run

Returns:: oci_link – link to the run page in an oci console
Return type:: str

property status: str: Returns the lifecycle state of the Data Flow run

update_config(param_dict) → None

Modify the run_config file used to create the data flow run

Parameters:: param_dict (Dict) – dictionary containing the key value pairs of the run_config parameters and the updated values.
Return type:: None

wait()

Wait and monitor the run creation process.

Parameters:: None –
Returns:: df_run – The oci.dataflow.models.Run after monitoring is done.
Return type:: oci.dataflow.models.Run

class ads.dataflow.dataflow.SPARK_VERSION

Bases: str

v2_4_4 = '2.4.4'

v3_0_2 = '3.0.2'

ads.dataflow.dataflowsummary module

class ads.dataflow.dataflowsummary.SummaryList(entity_list, datetime_format='%Y-%m-%d %H:%M:%S')

Bases: list

abstract filter(selection, instance=None): Abstract filter method for dataflow summary.

abstract sort_by(columns, reverse=False): Abstract sort method for dataflow summary.

to_dataframe(datetime_format=None): Abstract to_dataframe method for dataflow summary.

Module contents

ads.dataset package

Submodules

ads.dataset.classification_dataset module

class ads.dataset.classification_dataset.BinaryClassificationDataset(df, sampled_df, target, target_type, shape, positive_class=None, **kwargs)

Bases: ClassificationDataset

Dataset for binary classification

set_positive_class(positive_class, missing_value=False)

Return new dataset with values in target column mapped to True or False in accordance with the specified positive label.

Parameters:

positive_class (same dtype as target) – The target label which should be identified as positive outcome from model.
missing_value (bool) – missing values will be converted to this

Returns:

dataset

Return type:

same type as the caller

Raises:

ValidationError – if the positive_class is not present in target

Examples

>>> ds = DatasetFactory.open("iris.csv")
>>> ds_with_target = ds.set_target('class')
>>> ds_with_pos_class = ds.set_positive_class('setosa')

class ads.dataset.classification_dataset.BinaryTextClassificationDataset(df, sampled_df, target, target_type, shape, **kwargs)

Bases: BinaryClassificationDataset

Dataset for binary text classification

auto_transform(): Automatically chooses the most effective dataset transformation

select_best_features(score_func=None, k=12): Automatically chooses the best features and removes the rest

class ads.dataset.classification_dataset.ClassificationDataset(df, sampled_df, target, target_type, shape, **kwargs)

Bases: ADSDatasetWithTarget

Dataset for classification task

auto_transform(fix_imbalance: bool = True, correlation_threshold: float = 0.7, frac: float = 1.0, correlation_methods: str = 'pearson')

Return transformed dataset with several optimizations applied automatically. The optimizations include:

Dropping constant and primary key columns, which has no predictive quality,
Imputation, to fill in missing values in noisy data:
- For continuous variables, fill with mean if less than 40% is missing, else drop,
- For categorical variables, fill with most frequent if less than 40% is missing, else drop,
Dropping strongly co-correlated columns that tend to produce less generalizable models,
Balancing dataset using up or down sampling.

Parameters:

fix_imbalance (bool, defaults to True.) – Fix imbalance between classes in dataset. Used only for classification datasets.
correlation_threshold (float, defaults to 0.7. It must be between 0 and 1, inclusive.) – The correlation threshold where columns with correlation higher than the threshold will be considered as strongly co-correlated and recommended to be taken care of.
frac (float, defaults to 1.0. Range -> (0, 1].) – What fraction of the data should be used in the calculation?
correlation_methods (Union[list, str], defaults to 'pearson'.) –
- ‘pearson’: Use Pearson’s Correlation between continuous features,
- ’cramers v’: Use Cramer’s V correlations between categorical features,
- ’correlation ratio’: Use Correlation Ratio Correlation between categorical and continuous features,
- ’all’: Is equivalent to [‘pearson’, ‘cramers v’, ‘correlation ratio’].
Or a list containing any combination of these methods, for example, [‘pearson’, ‘cramers v’].

Returns:

transformed_dataset – The dataset after transformation

Return type:

ADSDatasetWithTarget

Examples

>>> ds_clean = ds.auto_transform(correlation_threshold=0.6)

convert_to_text_classification(text_column: str)

Builds a new dataset with the given text column as the only feature besides target.

Parameters:: text_column (str) – Feature name to use for text classification task
Returns:: ds – Dataset with one text feature and a classification target
Return type:: TextClassificationDataset

Examples

>>> review_ds = DatasetFactory.open("review_data.csv")
>>> ds_text_class = review_ds.convert_to_text_classification('reviews')

down_sample(sampler=None)

Fixes an imbalanced dataset by down-sampling.

Parameters:: sampler (An instance of SamplerMixin) – Should implement fit_resample(X,y) method. If None, does random down sampling.
Returns:: down_sampled_ds – A down-sampled dataset.
Return type:: ClassificationDataset

Examples

>>> ds = DatasetFactory.open("some_data.csv")
>>> ds_balanced_small = ds.down_sample()

up_sample(sampler='default')

Fixes imbalanced dataset by up-sampling

Parameters:

sampler (An instance of SamplerMixin) – Should implement fit_resample(X,y) method. If ‘default’, either SMOTE or random sampler will be used
fill_missing_type (a string) – Can either be ‘mean’, ‘mode’ or ‘median’.

Returns:

up_sampled_ds – an up-sampled dataset

Return type:

ClassificationDataset

Examples

>>> ds = DatasetFactory.open("some_data.csv")
>>> ds_balanced_large = ds.up_sample()

class ads.dataset.classification_dataset.MultiClassClassificationDataset(df, sampled_df, target, target_type, shape, **kwargs)

Bases: ClassificationDataset

Dataset for multi-class classification

class ads.dataset.classification_dataset.MultiClassTextClassificationDataset(df, sampled_df, target, target_type, shape, **kwargs)

Bases: MultiClassClassificationDataset

Dataset for multi-class text classification

auto_transform(): Automatically chooses the most effective dataset transformation

select_best_features(score_func=None, k=12): Automatically chooses the best features and removes the rest

ads.dataset.correlation module

ads.dataset.correlation_plot module

class ads.dataset.correlation_plot.BokehHeatMap(ds)

Bases: object

Generate a HeatMap or horizontal bar plot to compare features.

debug(): Return True if in debug mode, otherwise False.

flatten_corr_matrix(corr_matrix)

Flatten a correlation matrix into a pandas Dataframe.

Parameters:: corr_matrix (Pandas Dataframe) – The correlation matrix to be flattened.
Returns:: corr_flatten – The flattened correlation matrix.
Return type:: Pandas DataFrame

generate_heatmap(corr_matrix, title: str, msg: str, correlation_threshold: float)

Generate a heatmap from a correlation matrix.

Parameters:

corr_matrix (Pandas Dataframe) – The dataframe to be used for heatmap generation.
title (str) – title of the heatmap.
msg (str) – An additional msg to include in the plot.
correlation_threshold (float) – A float between 0 and 1 which is used for excluding correlations which are not intense enough from the plot.

Returns:

tab – A matplotlib Panel object which includes a plotted heatmap

Return type:

matplotlib Panel

generate_target_heatmap(corr_matrix, title: str, correlation_target: str, msg: str, correlation_threshold: float)

Generate a heatmap from a correlation matrix and its targets.

Parameters:

corr_matrix (Pandas Dataframe) – The dataframe to be used for heatmap generation.
title (str) – title of the heatmap.
correlation_target (str) – The target column name for computing correlations against.
msg (str) – An additional msg to include in the plot.
correlation_threshold (float) – A float between 0 and 1 which is used for excluding correlations which are not intense enough from the plot.

Returns:

tab – A matplotlib Panel object which includes a plotted heatmap.

Return type:

matplotlib Panel

plot_correlation_heatmap(ds, plot_type: str = 'heatmap', correlation_target: str = None, correlation_threshold=-1, correlation_methods: str = 'pearson', **kwargs)

Plots a correlation heatmap.

Parameters:

ds (Pandas Slice) – A data slice or file
plot_type (str Defaults to "heatmap") – The type of plot - “bar” is another option.
correlation_target (str, Defaults to None) – the target column for correlation calculations.
correlation_threshold (float, Defaults to -1) – the threshold for computing correlation heatmap elements.
correlation_methods (str, Defaults to "pearson") – the way to compute correlations, other options are “cramers v” and “correlation ratio”

plot_hbar(matrix, low: float = 1, high=1, title: str = None, tool_tips: list = None, column_name: str = None)

Plots a histogram bar-graph.

Parameters:

matrix (Pandas Dataframe) – The dataframe to be plotted.
low (float, Defaults to 1) – The color mapping value for “low” points.
high (float, Defaults to 1) – The color mapping value for “high” points.
title (str, Defaults to None) – The optional title of the heat map.
tool_tips (list of str, Defaults to None) – An optional list of tool tips to include with the plot.
column_name (str, Defaults to None) – The name of the column which is being plotted.

Returns:

fig – A matplotlib heatmap figure object.

Return type:

matplotlib Figure

plot_heat_map(matrix, xrange: list, yrange: list, low: float = 1, high=1, title: str = None, tool_tips: list = None)

Plots a matrix as a heatmap.

Parameters:

matrix (Pandas Dataframe) – The dataframe to be plotted.
xrange (List of floats) – The range of x values to plot.
yrange (List of floats) – The range of y values to plot.
low (float, Defaults to 1) – The color mapping value for “low” points.
high (float, Defaults to 1) – The color mapping value for “high” points.
title (str, Defaults to None) – The optional title of the heat map.
tool_tips (list of str, Defaults to None) – An optional list of tool tips to include with the plot.

Returns:

fig – A matplotlib heatmap figure object.

Return type:

matplotlib Figure

ads.dataset.correlation_plot.plot_correlation_heatmap(ds=None, **kwargs) → None

Plots a correlation heatmap.

Parameters:: ds (Pandas Slice) – A data slice or file

ads.dataset.dask_series module

ads.dataset.dataframe_transformer module

class ads.dataset.dataframe_transformer.DataFrameTransformer(func_name, target_name, target_sample_val, args=None, kw_args=None)

Bases: TransformerMixin

A DataFrameTransformer object.

fit(df): Takes in a DF and returns a fitted model

transform(df): Takes in a DF and returns a transformed DF

ads.dataset.dataframe_transformer.expand_lambda_function(lambda_func): Returns a lambda function after expansion.

ads.dataset.dataset module

class ads.dataset.dataset.ADSDataset(df, sampled_df, shape, name='', description=None, type_discovery=True, types={}, metadata=None, progress=<ads.dataset.progress.DummyProgressBar object>, transformer_pipeline=None, interactive=False, **kwargs)

Bases: PandasDataset

An ADSDataset Object.

The ADSDataset object cannot be used for classification or regression problems until a target has been set using set_target. To see some rows in the data use any of the usual Pandas functions like head(). There are also a variety of converters, to_dask, to_pandas, to_h2o, to_xgb, to_csv, to_parquet, to_json & to_hdf .

assign_column(column, arg)

Return new dataset with new column or values of the existing column mapped according to input correspondence.

Used for adding a new column or substituting each value in a column with another value, that may be derived from a function, a pandas.Series or a pandas.DataFrame.

Parameters:

column (str) – Name of the feature to update.
arg (function, dict, Series or DataFrame) – Mapping correspondence.

Returns:

dataset – a dataset with the specified column assigned.

Return type:

same type as the caller

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds_same_size = ds.assign_column('target',lambda x:  x>15 if x not None)
>>> ds_bigger = ds.assign_column('new_col', np.arange(ds.shape[0]))

astype(types)

Convert data type of features.

Parameters:: types (dict) – key is the existing feature name value is the data type to which the values of the feature should be converted. Valid data types: All numpy datatypes (Example: np.float64, np.int64, …) or one of categorical, continuous, ordinal or datetime.
Returns:: updated_dataset – an ADSDataset with new data types
Return type:: ADSDataset

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds_reformatted = ds.astype({"target": "categorical"})

call(func, *args, sample_size=None, **kwargs)

Runs a custom function on dataframe

func will receive the pandas dataframe (which represents the dataset) as an argument named ‘df’ by default. This can be overridden by specifying the dataframe argument name in a tuple (func, dataframe_name).

Parameters:

func (Union[callable, tuple]) – Custom function that takes pandas dataframe as input Alternatively a (callable, data) tuple where data is a string indicating the keyword of callable that expects the dataframe name
args (iterable, optional) – Positional arguments passed into func
sample_size (int, Optional) – To use a sampled dataframe
kwargs (mapping, optional) – A dictionary of keyword arguments passed into func

Returns:

func – a plotting function that contains *args and **kwargs

Return type:

function

Examples

>>> ds = DatasetFactory.open("classfication_data.csv")
>>> def f1(df):
...  return(sum(df), axis=0)
>>> sum_ds = ds.call(f1)

compute()

corr(correlation_methods: Union[list, str] = 'pearson', frac: float = 1.0, sample_size: float = 1.0, nan_threshold: float = 0.8, overwrite: Optional[bool] = None, force_recompute: bool = False)

Compute pairwise correlation of numeric and categorical columns, output a matrix or a list of matrices computed using the correlation methods passed in.

Parameters:

correlation_methods (Union[list, str], default to 'pearson') –
- ‘pearson’: Use Pearson’s Correlation between continuous features,
- ’cramers v’: Use Cramer’s V correlations between categorical features,
- ’correlation ratio’: Use Correlation Ratio Correlation between categorical and continuous features,
- ’all’: Is equivalent to [‘pearson’, ‘cramers v’, ‘correlation ratio’].
Or a list containing any combination of these methods, for example, [‘pearson’, ‘cramers v’].
frac – Is deprecated and replaced by sample_size.
sample_size (float, defaults to 1.0. Float, Range -> (0, 1]) – What fraction of the data should be used in the calculation?
nan_threshold (float, default to 0.8, Range -> [0, 1]) – Only compute a correlation when the proportion of the values, in a column, is less than or equal to nan_threshold.
overwrite – Is deprecated and replaced by force_recompute.
force_recompute (bool, default to be False) –
- If False, it calculates the correlation matrix if there is no cached correlation matrix. Otherwise, it returns the cached correlation matrix.
- If True, it calculates the correlation matrix regardless whether there is cached result or not.

Returns:

correlation – The pairwise correlations as a matrix (DataFrame) or list of matrices

Return type:

Union[list, pandas.DataFrame]

property ddf

df_read_functions = ['head', 'describe', '_get_numeric_data']

drop_columns(columns)

Return new dataset with specified columns removed.

Parameters:: columns (str or list) – columns to drop.
Returns:: dataset – a dataset with specified columns dropped.
Return type:: same type as the caller
Raises:: ValidationError – If any of the feature names is not found in the dataset.

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds_smaller = ds.drop_columns(['col1', 'col2'])

merge(data, **kwargs)

Merges this dataset with another ADSDataset or pandas dataframe.

Parameters:

data (Union[ADSDataset, pandas.DataFrame]) – Data to merge.
kwargs (dict, optional) – additional keyword arguments that would be passed to underlying dataframe’s merge API.

Examples

>>> ds1 = DatasetFactory.open("data1.csv")
>>> ds2 = DatasetFactory.open("data2.csv")
>>> ds_12 = ds1.merge(ds2)

rename_columns(columns)

Returns a new dataset with altered column names.

dict values must be unique (1-to-1). Labels not contained in a dict will be left as-is. Extra labels listed don’t throw an error.

Parameters:: columns (dict-like or function or list of str) – dict to rename columns selectively, or list of names to rename all columns, or a function like str.upper
Returns:: dataset – A dataset with specified columns renamed.
Return type:: same type as the caller

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds_renamed = ds.rename_columns({'col1': 'target'})

sample(frac=None, random_state=42)

Returns random sample of dataset.

Parameters:

frac (float, optional) – Fraction of axis items to return.
random_state (int or np.random.RandomState) – If int we create a new RandomState with this as the seed Otherwise we draw from the passed RandomState

Returns:

sampled_dataset – An ADSDataset which was randomly sampled.

Return type:

ADSDataset

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds_sample = ds.sample()

set_description(description)

Sets description for the dataset.

Give your dataset a description.

Parameters:: description (str) – Description of the dataset.

Examples

>>> ds = DatasetFactory.open("data1.csv")
>>> ds_renamed = ds.set_description("dataset1 is from "data1.csv"")

set_name(name)

Sets name for the dataset.

This name will be used to filter the datasets returned by ds.list() API. Calling this API is optional. By default name of the dataset is set to empty.

Parameters:: name (str) – Name of the dataset.

Examples

>>> ds = DatasetFactory.open("data1.csv")
>>> ds_renamed = ds.set_name("dataset1")

set_target(target, type_discovery=True, target_type=None)

Returns a dataset tagged based on the type of target.

Parameters:

target (str) – name of the feature to use as target.
type_discovery (bool) – This is set as True by default.
target_type (type) – If provided, then the target will be typed with the provided value.

Returns:

ds – tagged according to the type of the target column.

Return type:

ADSDataset

Examples

>>> ds = DatasetFactory.open("classfication_data.csv")
>>> ds_with_target= ds.set_target("target_class")

show_corr(frac: float = 1.0, sample_size: float = 1.0, nan_threshold: float = 0.8, overwrite: Optional[bool] = None, force_recompute: bool = False, correlation_target: Optional[str] = None, plot_type: str = 'heatmap', correlation_threshold: float = -1, correlation_methods='pearson', **kwargs)

Show heatmap or barplot of pairwise correlation of numeric and categorical columns, output three tabs which are heatmap or barplot of correlation matrix of numeric columns vs numeric columns using pearson correlation method, categorical columns vs categorical columns using Cramer’s V method, and numeric vs categorical columns, excluding NA/null values and columns which have more than 80% of NA/null values. By default, only ‘pearson’ correlation is calculated and shown in the first tab. Set correlation_methods=’all’ to show all correlation charts.

Parameters:

frac (Is superseded by sample_size) –
sample_size (float, defaults to 1.0. Float, Range -> (0, 1]) – What fraction of the data should be used in the calculation?
nan_threshold (float, defaults to 0.8, Range -> [0, 1]) – In the default case, it will only calculate the correlation of the columns which has less than or equal to 80% of missing values.
overwrite – Is deprecated and replaced by force_recompute.
force_recompute (bool, default to be False.) –
- If False, it calculates the correlation matrix if there is no cached correlation matrix. Otherwise, it returns the cached correlation matrix.
- If True, it calculates the correlation matrix regardless whether there is cached result or not.
plot_type (str, default to "heatmap") – It can only be “heatmap” or “bar”. Note that if “bar” is chosen, correlation_target also has to be set and the bar chart will only show the correlation values of the pairs which have the target in them.
correlation_target (str, default to Non) – It can be any columns of type continuous, ordinal, categorical or zipcode. When correlation_target is set, only pairs that contains correlation_target will show.
correlation_threshold (float, default to -1) – It can be any number between -1 and 1.
correlation_methods (Union[list, str], defaults to 'pearson') –
- ‘pearson’: Use Pearson’s Correlation between continuous features,
- ’cramers v’: Use Cramer’s V correlations between categorical features,
- ’correlation ratio’: Use Correlation Ratio Correlation between categorical and continuous features,
- ’all’: Is equivalent to [‘pearson’, ‘cramers v’, ‘correlation ratio’].
Or a list containing any combination of these methods, for example, [‘pearson’, ‘cramers v’].

Return type:

None

show_in_notebook(correlation_threshold=-1, selected_index=0, sample_size=0, visualize_features=True, correlation_methods='pearson', **kwargs)

Provide visualization of dataset.

Display feature distribution. The data table display will show a maximum of 8 digits,
Plot the correlation between the dataset features (as a heatmap) only when all the features are continuous or ordinal,
Display data head.

Parameters:

correlation_threshold (int, default -1) – The correlation threshold to select, which only show features that have larger or equal correlation values than the threshold.
selected_index (int, str, default 0) – The displayed output is stacked into an accordion widget, use selected_index to force the display to open a specific element, use the (zero offset) index or any prefix string of the name (eg, ‘corr’ for correlations)
sample_size (int, default 0) – The size (in rows) to sample for visualizations
visualize_features (bool default False) – For the “Features” section control if feature visualizations are shown or not. If not only a summary of the numeric statistics is shown. The numeric statistics are also always shows for wide (>64 features) datasets
correlation_methods (Union[list, str], default to 'pearson') –
- ‘pearson’: Use Pearson’s Correlation between continuous features,
- ’cramers v’: Use Cramer’s V correlations between categorical features,
- ’correlation ratio’: Use Correlation Ratio Correlation between categorical and continuous features,
- ’all’: Is equivalent to [‘pearson’, ‘cramers v’, ‘correlation ratio’].
Or a list containing any combination of these methods, for example, [‘pearson’, ‘cramers v’].

snapshot(snapshot_dir=None, name='', storage_options=None)

Snapshot the dataset with modifications made so far.

Optionally caller can invoke ds.set_name() before saving to identify the dataset uniquely at the time of using ds.list().

The snapshot can be reloaded by providing the URI returned by this API to DatasetFactory.open()

Parameters:

snapshot_dir (str, optional) – Directory path under which dataset snapshot will be created. Defaults to snapshots_dir set using DatasetFactory.set_default_storage().
name (str, optional, default: "") – Name to uniquely identify the snapshot using DatasetFactory.list_snapshots(). If not provided, an auto-generated name is used.
storage_options (dict, optional) – Parameters passed on to the backend filesystem class. Defaults to storage_options set using DatasetFactory.set_default_storage().

Returns:

p_str – the URI to access the snapshotted dataset.

Return type:

str

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds_uri = ds.snapshot()

to_avro(path, schema=None, storage_options=None, **kwargs)

Save data to Avro files. Avro is a remote procedure call and data serialization framework developed within Apache’s Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format.

Parameters:

path (string) – Path to a target filename. May contain a * to denote many filenames.
schema (dict) – Avro schema dictionary, see below.
storage_options (dict, optional) – Parameters passed to the backend filesystem class. Defaults to storage_options set using DatasetFactory.set_default_storage().
kwargs (dict, optional) – See https://fastavro.readthedocs.io/en/latest/writer.html

Notes

Avro schema is a complex dictionary describing the data, see https://avro.apache.org/docs/1.8.2/gettingstartedpython.html#Defining+a+schema and https://fastavro.readthedocs.io/en/latest/writer.html. Its structure is as follows:

{'name': 'Test',
'namespace': 'Test',
'doc': 'Descriptive text',
'type': 'record',
'fields': [
    {'name': 'a', 'type': 'int'},
]}

where the “name” field is required, but “namespace” and “doc” are optional descriptors; “type” must always be “record”. The list of fields should have an entry for every key of the input records, and the types are like the primitive, complex or logical types of the Avro spec (https://avro.apache.org/docs/1.8.2/spec.html).

Examples

>>> ds = DatasetFactory.open("data.avro")
>>> ds.to_avro("my/path.avro")

to_csv(path, storage_options=None, **kwargs)

Save the materialized dataframe to csv file.

Parameters:

path (str) – Location to write to. If there are more than one partitions in df, should include a glob character to expand into a set of file names, or provide a name_function=parameter. Supports protocol specifications such as “oci://”, “s3://”.
storage_options (dict, optional) – Parameters passed on to the backend filesystem class. Defaults to storage_options set using DatasetFactory.set_default_storage().
kwargs (dict, optional) –

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> [ds_link] = ds.to_csv("my/path.csv")

to_dask(filter=None, frac=None, npartitions=None, include_transformer_pipeline=False)

Returns a copy of the data as dask.dataframe.core.DataFrame, and a sklearn pipeline optionally that holds the transformations run so far on the data.

The pipeline returned can be updated with the transformations done offline and passed along with the dataframe to Dataset.open API if the transformations need to be reproduced at the time of scoring.

Parameters:

filter (str, optional) – The query string to filter the dataframe, for example ds.to_dask(filter=”age > 50 and location == ‘san francisco”) See also https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html
frac (float, optional) – fraction of original data to return.
include_transformer_pipeline (bool, default: False) – If True, (dataframe, transformer_pipeline) is returned as a tuple.

Returns:

dataframe (dask.dataframe.core.DataFrame) – if include_transformer_pipeline is False.
(data, transformer_pipeline) (tuple of dask.dataframe.core.DataFrame and dataset.pipeline.TransformerPipeline) – if include_transformer_pipeline is True.

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds_dask = ds.to_dask()

Notes

to_dask_dataframe(filter=None, frac=None, npartitions=None, include_transformer_pipeline=False)

to_h2o(filter=None, frac=None, include_transformer_pipeline=False)

Returns a copy of the data as h2o.H2OFrame, and a sklearn pipeline optionally that holds the transformations run so far on the data.

The pipeline returned can be updated with the transformations done offline and passed along with the dataframe to Dataset.open API if the transformations need to be reproduced at the time of scoring.

Parameters:

filter (str, optional) – The query string to filter the dataframe, for example ds.to_h2o(filter=”age > 50 and location == ‘san francisco”) See also https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html
frac (float, optional) – fraction of original data to return.
include_transformer_pipeline (bool, default: False) – If True, (dataframe, transformer_pipeline) is returned as a tuple.

Returns:

dataframe (h2o.H2OFrame) – if include_transformer_pipeline is False.
(data, transformer_pipeline) (tuple of h2o.H2OFrame and dataset.pipeline.TransformerPipeline) – if include_transformer_pipeline is True.

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds_as_h2o = ds.to_h2o()

Notes

to_h2o_dataframe(filter=None, frac=None, include_transformer_pipeline=False)

to_hdf(path: str, key: str, storage_options: Optional[dict] = None, **kwargs) → str

Save data to Hierarchical Data Format (HDF) files.

Parameters:

path (string) – Path to a target filename.
key (string) – Datapath within the files.
storage_options (dict, optional) – Parameters passed to the backend filesystem class. Defaults to storage_options set using DatasetFactory.set_default_storage().
kwargs (dict, optional) –

Returns:

The filename of the HDF5 file created.

Return type:

str

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds.to_hdf(path="my/path.h5", key="df")

to_json(path, storage_options=None, **kwargs)

Save data to JSON files.

Parameters:

path (str) – Location to write to. If there are more than one partitions in df, should include a glob character to expand into a set of file names, or provide a name_function=parameter. Supports protocol specifications such as “oci://”, “s3://”.
storage_options (dict, optional) – Parameters passed on to the backend filesystem class. Defaults to storage_options set using DatasetFactory.set_default_storage().
kwargs (dict, optional) –

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds.to_json("my/path.json")

to_pandas(filter=None, frac=None, include_transformer_pipeline=False)

Returns a copy of the data as pandas.DataFrame, and a sklearn pipeline optionally that holds the transformations run so far on the data.

The pipeline returned can be updated with the transformations done offline and passed along with the dataframe to Dataset.open API if the transformations need to be reproduced at the time of scoring.

Parameters:

filter (str, optional) – The query string to filter the dataframe, for example ds.to_pandas(filter=”age > 50 and location == ‘san francisco”) See also https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html
frac (float, optional) – fraction of original data to return.
include_transformer_pipeline (bool, default: False) – If True, (dataframe, transformer_pipeline) is returned as a tuple

Returns:

dataframe (pandas.DataFrame) – if include_transformer_pipeline is False.
(data, transformer_pipeline) (tuple of pandas.DataFrame and dataset.pipeline.TransformerPipeline) – if include_transformer_pipeline is True.

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds_as_df = ds.to_pandas()

Notes

to_pandas_dataframe(filter=None, frac=None, include_transformer_pipeline=False)

to_parquet(path, storage_options=None, **kwargs)

Save data to parquet file.

Parameters:

path (str) – Location to write to. If there are more than one partitions in df, should include a glob character to expand into a set of file names, or provide a name_function=parameter. Supports protocol specifications such as “oci://”, “s3://”.
storage_options (dict, optional) – Parameters passed on to the backend filesystem class. Defaults to storage_options set using DatasetFactory.set_default_storage().
kwargs (dict, optional) –

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> ds.to_parquet("my/path")

to_xgb(filter=None, frac=None, include_transformer_pipeline=False)

Returns a copy of the data as xgboost.DMatrix, and a sklearn pipeline optionally that holds the transformations run so far on the data.

The pipeline returned can be updated with the transformations done offline and passed along with the dataframe to Dataset.open API if the transformations need to be reproduced at the time of scoring.

Parameters:

filter (str, optional) – The query string to filter the dataframe, for example ds.to_xgb(filter=”age > 50 and location == ‘san francisco”) See also https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html
frac (float, optional) – fraction of original data to return.
include_transformer_pipeline (bool, default: False) – If True, (dataframe, transformer_pipeline) is returned as a tuple.

Returns:

dataframe (xgboost.DMatrix) – if include_transformer_pipeline is False.
(data, transformer_pipeline) (tuple of xgboost.DMatrix and dataset.pipeline.TransformerPipeline) – if include_transformer_pipeline is True.

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> xgb_dmat = ds.to_xgb()

Notes

to_xgb_dmatrix(filter=None, frac=None, include_transformer_pipeline=False)

ads.dataset.dataset_browser module

class ads.dataset.dataset_browser.DatasetBrowser

Bases: ABC

static GitHub(user: str, repo: str, branch: str = 'master'): Returns a GitHubDataset

static filesystem(folder: str): Returns a LocalFilesystemDataset.

filter_list(L, filter_pattern) → List[str]: Filters a list of dataset names.

static list(filter_pattern='*') → List[str]: Return a list of dataset browser strings.

abstract open(**kwargs)

Return new dataset for the given name.

Parameters:: name (str) – the name of the dataset to open.
Returns:: ds
Return type:: Dataset

Examples

ds_browser = DatasetBrowser(“sklearn”)

ds = ds_browser.open(“iris”)

static seaborn(): Returns a SeabornDataset.

static sklearn(): Returns a SklearnDataset.

static web(index_url: str): Returns a WebDataset.

class ads.dataset.dataset_browser.GitHubDatasets(user: str, repo: str, branch: str)

Bases: DatasetBrowser

list(filter_pattern: str = '.*') → List[str]: Return a list of dataset browser strings.

open(name: str, **kwargs)

Return new dataset for the given name.

Parameters:: name (str) – the name of the dataset to open.
Returns:: ds
Return type:: Dataset

Examples

ds_browser = DatasetBrowser(“sklearn”)

ds = ds_browser.open(“iris”)

class ads.dataset.dataset_browser.LocalFilesystemDatasets(folder: str)

Bases: DatasetBrowser

list(filter_pattern: str = '.*') → List[str]: Return a list of dataset browser strings.

open(name: str, **kwargs)

Return new dataset for the given name.

Parameters:: name (str) – the name of the dataset to open.
Returns:: ds
Return type:: Dataset

Examples

ds_browser = DatasetBrowser(“sklearn”)

ds = ds_browser.open(“iris”)

class ads.dataset.dataset_browser.SeabornDatasets

Bases: DatasetBrowser

list(filter_pattern: str = '.*') → List[str]: Return a list of dataset browser strings.

open(name: str, **kwargs)

Return new dataset for the given name.

Parameters:: name (str) – the name of the dataset to open.
Returns:: ds
Return type:: Dataset

Examples

ds_browser = DatasetBrowser(“sklearn”)

ds = ds_browser.open(“iris”)

class ads.dataset.dataset_browser.SklearnDatasets

Bases: DatasetBrowser

list(filter_pattern: str = '.*') → List[str]: Return a list of dataset browser strings.

open(name: str, **kwargs)

Return new dataset for the given name.

Parameters:: name (str) – the name of the dataset to open.
Returns:: ds
Return type:: Dataset

Examples

ds_browser = DatasetBrowser(“sklearn”)

ds = ds_browser.open(“iris”)

sklearn_datasets = ['breast_cancer', 'diabetes', 'iris', 'wine', 'digits']

class ads.dataset.dataset_browser.WebDatasets(index_url: str)

Bases: DatasetBrowser

list(filter_pattern: str = '.*') → List[str]: Return a list of dataset browser strings.

open(name: str, **kwargs)

Return new dataset for the given name.

Parameters:: name (str) – the name of the dataset to open.
Returns:: ds
Return type:: Dataset

Examples

ds_browser = DatasetBrowser(“sklearn”)

ds = ds_browser.open(“iris”)

ads.dataset.dataset_with_target module

class ads.dataset.dataset_with_target.ADSDatasetWithTarget(df, sampled_df, target, target_type, shape, sample_max_rows=-1, type_discovery=True, types={}, parent=None, name='', metadata=None, transformer_pipeline=None, description=None, progress=<ads.dataset.progress.DummyProgressBar object>, **kwargs)

Bases: ADSDataset

This class provides APIs for preparing dataset for modeling.

auto_transform(correlation_threshold: float = 0.7, frac: float = 1.0, sample_size=1.0, correlation_methods: Union[str, list] = 'pearson')

Return transformed dataset with several optimizations applied automatically. The optimizations include:

Dropping constant and primary key columns, which has no predictive quality,
Imputation, to fill in missing values in noisy data:
- For continuous variables, fill with mean if less than 40% is missing, else drop,
- For categorical variables, fill with most frequent if less than 40% is missing, else drop,
Dropping strongly co-correlated columns that tend to produce less generalizable models.

Parameters:

correlation_threshold (float, defaults to 0.7. It must be between 0 and 1, inclusive) – the correlation threshold where columns with correlation higher than the threshold will be considered as strongly co-correlated and recommended to be taken care of.
frac (Is superseded by sample_size) –
sample_size (float, defaults to 1.0. Float, Range -> (0, 1]) – What fraction of the data should be used in the calculation?
correlation_methods (Union[list, str], defaults to 'pearson') –
- ‘pearson’: Use Pearson’s Correlation between continuous features,
- ’cramers v’: Use Cramer’s V correlations between categorical features,
- ’correlation ratio’: Use Correlation Ratio Correlation between categorical and continuous features,
- ’all’: Is equivalent to [‘pearson’, ‘cramers v’, ‘correlation ratio’].
Or a list containing any combination of these methods, for example, [‘pearson’, ‘cramers v’].

Returns:

transformed_dataset

Return type:

ADSDatasetWithTarget

Examples

>>> ds_clean = ds.auto_transform()

get_recommendations(correlation_methods: str = 'pearson', correlation_threshold: float = 0.7, frac: float = 1.0, sample_size: float = 1.0, overwrite: bool = None, force_recompute: bool = False, display_format: str = 'widget')

Generate recommendations for dataset optimization. This includes:

Identifying constant and primary key columns, which has no predictive quality,
Imputation, to fill in missing values in noisy data:
- For continuous variables, fill with mean if less than 40% is missing, else drop,
- For categorical variables, fill with most frequent if less than 40% is missing, else drop,
Identifying strongly co-correlated columns that tend to produce less generalizable models,
Automatically balancing dataset for classification problems using up or down sampling.

Parameters:

correlation_methods (Union[list, str], default to 'pearson') –
- ‘pearson’: Use Pearson’s Correlation between continuous features,
- ’cramers v’: Use Cramer’s V correlations between categorical features,
- ’correlation ratio’: Use Correlation Ratio Correlation between categorical and continuous features,
- ’all’: Is equivalent to [‘pearson’, ‘cramers v’, ‘correlation ratio’].
Or a list containing any combination of these methods, for example, [‘pearson’, ‘cramers v’].
correlation_threshold (float, defaults to 0.7. It must be between 0 and 1, inclusive) – The correlation threshold where columns with correlation higher than the threshold will be considered as strongly co-correlated and recommended to be taken care of.
frac (Is superseded by sample_size) –
sample_size (float, defaults to 1.0. Float, Range -> (0, 1]) – What fraction of the data should be used in the calculation?
overwrite – Is deprecated and replaced by force_recompute.
force_recompute (bool, default to be False) –
- If False, it calculates the correlation matrix if there is no cached correlation matrix. Otherwise, it returns the cached correlation matrix.
- If True, it calculates the correlation matrix regardless whether there is cached result or not.
display_format (string, defaults to 'widget'.) – Should be either ‘widget’ or ‘table’. If ‘widget’, a GUI style interface is popped out; if ‘table’, a table of suggestions is shown.

get_transformed_dataset()

Return the transformed dataset with the recommendations applied.

This method should be called after applying the recommendations using the Recommendation#show_in_notebook() API.

rename_columns(columns): Returns a dataset with columns renamed.

select_best_features(score_func=None, k=12)

Return new dataset containing only the top k features.

Parameters:

k (int, default 12) – The top ‘k’ features to select.
score_func (function) – Scoring function to use to rank the features. This scoring function should take a 2d array X(features) and an array like y(target) and return a numeric score for each feature in the same order as X.

Notes

Examples

>>> ds = DatasetBrowser("sklearn").open("iris")
>>> ds_small = ds.select_best_features(k=2)

suggest_recommendations(correlation_methods: Union[str, list] = 'pearson', print_code: bool = True, correlation_threshold: float = 0.7, overwrite: Optional[bool] = None, force_recompute: bool = False, frac: float = 1.0, sample_size: float = 1.0, **kwargs)

Returns a pandas dataframe with suggestions for dataset optimization. This includes:

Identifying constant and primary key columns, which has no predictive quality,
Imputation, to fill in missing values in noisy data:
- For continuous variables, fill with mean if less than 40% is missing, else drop,
- For categorical variables, fill with most frequent if less than 40% is missing, else drop,
Identifying strongly co-correlated columns that tend to produce less generalizable models,
Automatically balancing dataset for classification problems using up or down sampling.

Parameters:

correlation_methods (Union[list, str], default to 'pearson') –
- ‘pearson’: Use Pearson’s Correlation between continuous features,
- ’cramers v’: Use Cramer’s V correlations between categorical features,
- ’correlation ratio’: Use Correlation Ratio Correlation between categorical and continuous features,
- ’all’: Is equivalent to [‘pearson’, ‘cramers v’, ‘correlation ratio’].
Or a list containing any combination of these methods, for example, [‘pearson’, ‘cramers v’]
print_code (bool, Defaults to True) – Print Python code for the suggested actions.
correlation_threshold (float. Defaults to 0.7. It must be between 0 and 1, inclusive) – the correlation threshold where columns with correlation higher than the threshold will be considered as strongly co-correated and recommended to be taken care of.
frac (Is superseded by sample_size) –
sample_size (float, defaults to 1.0. Float, Range -> (0, 1]) – What fraction of the data should be used in the calculation?
overwrite – Is deprecated and replaced by force_recompute.
force_recompute (bool, default to be False) –
- If False, it calculates the correlation matrix if there is no cached correlation matrix. Otherwise, it returns the cached correlation matrix.
- If True, it calculates the correlation matrix regardless whether there is cached result or not.

Returns:

suggestion dataframe

Return type:

pandas.DataFrame

Examples

>>> suggestion_df = ds.suggest_recommendations(correlation_threshold=0.7)

train_test_split(test_size=0.1, random_state=42)

Splits dataset to train and test data.

Parameters:

test_size (Union[float, int], optional, default=0.1) –
random_state (Union[int, RandomState], optional, default=None) –
- If int, random_state is the seed used by the random number generator;
- If RandomState instance, random_state is the random number generator;
- If None, the random number generator is the RandomState instance used by np.random.

Returns:

train_data, test_data – tuple of ADSData instances

Return type:

tuple

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> train, test = ds.train_test_split()

train_validation_test_split(test_size=0.1, validation_size=0.1, random_state=42)

Splits dataset to train, validation and test data.

Parameters:

test_size (Union[float, int], optional, default=0.1) –
validation_size (Union[float, int], optional, default=0.1) –
random_state (Union[int, RandomState], optional, default=None) –
- If int, random_state is the seed used by the random number generator;
- If RandomState instance, random_state is the random number generator;
- If None, the random number generator is the RandomState instance used by np.random.

Returns:

train_data, validation_data, test_data – tuple of ADSData instances

Return type:

tuple

Examples

>>> ds = DatasetFactory.open("data.csv")
>>> train, valid, test = ds.train_validation_test_split()

type_of_target()

Return the target type for the dataset.

Returns:: target_type – an object of TypedFeature
Return type:: TypedFeature

Examples

>>> ds = ds.set_target('target_class')
>>> assert(ds.type_of_target() == 'categorical')

visualize_transforms(): Render a representation of the dataset’s transform DAG.

ads.dataset.exception module

exception ads.dataset.exception.DatasetError(*args, **kwargs)

Bases: BaseException

Base class for dataset errors.

exception ads.dataset.exception.ValidationError(msg)

Bases: DatasetError

Handles validation errors in dataset.

ads.dataset.factory module

class ads.dataset.factory.CustomFormatReaders

Bases: object

DEFAULT_SQL_ARRAYSIZE = 50000

DEFAULT_SQL_CHUNKSIZE = 12007

DEFAULT_SQL_CTU = False

DEFAULT_SQL_MIL = 128

static read_arff(path, **kwargs)

static read_avro(path: str, **kwargs) → DataFrame

static read_html(path, html_table_index: Optional[int] = None, **kwargs)

static read_json(path: str, **kwargs) → DataFrame

static read_libsvm(path: str, **kwargs) → DataFrame

static read_log(path, **kwargs)

classmethod read_sql(path: str, table: Optional[str] = None, **kwargs) → DataFrame

Parameters:

path – str This is the connection URL that gets passed to sqlalchemy’s create_engine method
table – str This is either the name of a table to select * from or a sql query to be run
kwargs –

Returns:

pd.DataFrame

static read_tsv(path: str, **kwargs) → DataFrame

static read_xml(path: str, **kwargs) → DataFrame

Load data from xml file.

Parameters:

path (str) – Path to XML file
storage_options (dict, optional) – Storage options passed to Pandas to read the file.

Returns:

dataframe

Return type:

pandas.DataFrame

class ads.dataset.factory.DatasetFactory

Bases: object

static download(remote_path, local_path, storage=None, overwrite=False)

Download a remote file or directory to local storage.

Parameters:

remote_path (str) – Supports protocols like oci, s3, also supports glob expressions
local_path (str) – Supports glob expressions
storage (dict) – Parameters passed on to the backend remote filesystem class.
overwrite (bool, default False) – If True, the method will overwrite any existing files in the local_path

Examples

>>> DatasetFactory.download("oci://Bucket/prefix/to/data/*.csv",
...         "/home/datascience/data/")

static from_dataframe(df, target: Optional[str] = None, **kwargs)

Returns an object of ADSDatasetWithTarget or ADSDataset given a pandas.DataFrame

Parameters:

df (pandas.DataFrame) –
target (str) –
kwargs (dict) – See DatasetFactory.open() for supported kwargs

Returns:

dataset – according to the type of target

Return type:

an object of ADSDataset target is not specified, otherwise an object of ADSDatasetWithTarget tagged

Examples

>>> df = pd.DataFrame(data)
>>> ds = from_dataframe(df)

classmethod infer_target_type(target, target_series, discover_target_type=True)

static list_snapshots(snapshot_dir=None, name='', storage_options=None, **kwargs)

Displays the URIs for dataset snapshots under the given directory path.

Parameters:

snapshot_dir (str) – Return all dataset snapshots created using ADSDataset.snapshot() within this directory. The path can contain protocols such as oci, s3.
name (str, optional) – The list of snapshots in the directory gets filtered by the name. Accepts glob expressions. default = “ads_”
storage_options (dict) – Parameters passed on to the backend filesystem class.

Example

>>> DatasetFactory.list_snapshots(snapshot_dir="oci://my_bucket/snapshots_dir",
...             name="ads_iris_")

Returns a list of all snapshots (recursively) saved to obj storage bucket “my_bucket” with prefix “/snapshots_dir/ads_iris_**” sorted by time created.

static open(source, target=None, format='infer', reader_fn: Callable = None, name: str = None, description='', npartitions: int = None, type_discovery=True, html_table_index=None, column_names='infer', sample_max_rows=10000, positive_class=None, transformer_pipeline=None, types={}, **kwargs)

Returns an object of ADSDataset or ADSDatasetWithTarget read from the given path

Deprecated since version 2.6.6: “Deprecated in favor of using Pandas. Pandas supports reading from object storage directly. Check https://accelerated-data-science.readthedocs.io/en/latest/user_guide/loading_data/connect.html”,

Parameters:

source (Union[str, pandas.DataFrame, h2o.DataFrame, pyspark.sql.dataframe.DataFrame]) – If str, URI for the dataset. The dataset could be read from local or network file system, hdfs, s3, gcs and optionally pyspark in pyspark conda env
target (str, optional) – Name of the target in dataset. If set an ADSDatasetWithTarget object is returned, otherwise an ADSDataset object is returned which can be used to understand the dataset through visualizations
format (str, default: infer) – Format of the dataset. Supported formats: CSV, TSV, Parquet, libsvm, JSON, XLS/XLSX (Excel), HDF5, SQL, XML, Apache server log files (clf, log), ARFF. By default, the format would be inferred from the ending of the dataset file path.
reader_fn (Callable, default: None) – The user may pass in their own custom reader function. It must accept (path, **kwarg) and return a pandas DataFrame
name (str, optional default: "") –
description (str, optional default: "") – Text describing the dataset
npartitions (int, deprecated) – Number of partitions to split the data By default this is set to the max number of cores supported by the backend compute accelerator
type_discovery (bool, default: True) – If false, the data types of the dataframe are used as such. By default, the dataframe columns are associated with the best suited data types. Associating the features with the disovered datatypes would impact visualizations and model prediction.
html_table_index (int, optional) – The index of the dataframe table in html content. This is used when the format of dataset is html
column_names ('infer', list of str or None, default: 'infer') – Supported only for CSV and TSV. List of column names to use. By default, column names are inferred from the first line of the file. If set to None, column names would be auto-generated instead of inferring from file. If the file already contains a column header, specify header=0 to ignore the existing column names.
sample_max_rows (int, default: 10000, use -1 auto calculate sample size, use 0 (zero) for no sampling) – Sample size of the dataframe to use for visualization and optimization.
positive_class (Any, optional) – Label in target for binary classification problems which should be identified as positive for modeling. By default, the first unique value is considered as the positive label.
types (dict, optional) – Dictionary of <feature_name> : <data_type> to override the data type of features.
transformer_pipeline (datasets.pipeline.TransformerPipeline, optional) – A pipeline of transformations done outside the sdk and need to be applied at the time of scoring
storage_options (dict, default: varies by source type) – Parameters passed on to the backend filesystem class.
sep (str) – Delimiting character for parsing the input file.
kwargs (additional keyword arguments that would be passed to underlying dataframe read API) – based on the format of the dataset

Returns:

dataset (An instance of ADSDataset)
(or)
dataset_with_target (An instance of ADSDatasetWithTarget)

Examples

>>> ds = DatasetFactory.open("/path/to/data.data", format='csv', delimiter=" ",
...          na_values="n/a", skipinitialspace=True)

>>> ds = DatasetFactory.open("/path/to/data.csv", target="col_1", prefix="col_",
...           skiprows=1, encoding="ISO-8859-1")

>>> ds = DatasetFactory.open("oci://bucket@namespace/path/to/data.tsv",
...         column_names=["col1", "col2", "col3"], header=0)

>>> ds = DatasetFactory.open("oci://bucket@namespace/path/to/data.csv",
...         storage_options={"config": "~/.oci/config",
...         "profile": "USER_2"}, delimiter = ';')

>>> ds = DatasetFactory.open("/path/to/data.parquet", engine='pyarrow',
...         types={"col1": "ordinal",
...                "col2": "categorical",
...                "col3" : "continuous",
...                "col4" : "float64"})

>>> ds = DatasetFactory.open(df, target="class", sample_max_rows=5000,
...          positive_class="yes")

>>> ds = DatasetFactory.open("s3://path/to/data.json.gz", format="json",
...         compression="gzip", orient="records")

static open_to_pandas(source: str, format: Optional[str] = None, reader_fn: Optional[Callable] = None, **kwargs) → DataFrame

static set_default_storage(snapshots_dir=None, storage_options=None)

Set default storage directory and options.

Both snapshots_dir and storage_options can be overridden at the API scope.

Parameters:

snapshots_dir (str) – Path for the snapshots directory. Can contain protocols such as oci, s3
storage_options (dict, optional) – Parameters passed on to the backend filesystem class.

static upload(local_file_or_dir, remote_file_or_dir, storage_options=None)

Upload local file or directory to remote storage

Parameters:

local_file_or_dir (str) – Supports glob expressions
remote_file_or_dir (str) – Supports protocols like oci, s3, also supports glob expressions
storage_options (dict) – Parameters passed on to the backend remote filesystem class.

ads.dataset.factory.get_format_reader(path: ElaboratedPath, **kwargs) → Callable

ads.dataset.factory.load_dataset(path: ElaboratedPath, reader_fn: Callable, **kwargs) → DataFrame

ads.dataset.feature_engineering_transformer module

class ads.dataset.feature_engineering_transformer.FeatureEngineeringTransformer(feature_metadata=None)

Bases: TransformerMixin

fit(X, y=None)

fit_transform(X, y=None, **fit_params)

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters:

X (array-like of shape (n_samples, n_features)) – Input samples.
y (array-like of shape (n_samples,) or (n_samples, n_outputs), default=None) – Target values (None for unsupervised transformations).
**fit_params (dict) – Additional fit parameters.

Returns:

X_new – Transformed array.

Return type:

ndarray array of shape (n_samples, n_features_new)

transform(df, progress=<ads.dataset.progress.DummyProgressBar object>, fit_transform=False)

ads.dataset.feature_selection module

class ads.dataset.feature_selection.FeatureImportance(ds, score_func=None, n=None)

Bases: object

show_in_notebook(fig_size=(10, 10)): Shows selected features in the notebook with matplotlib.

ads.dataset.forecasting_dataset module

class ads.dataset.forecasting_dataset.ForecastingDataset(df, sampled_df, target, target_type, shape, **kwargs)

Bases: ADSDatasetWithTarget

select_best_features(score_func=None, k=12): Not yet implemented

ads.dataset.helper module

class ads.dataset.helper.DatasetDefaults

Bases: object

sampling_confidence_interval = 1.0

sampling_confidence_level = 95

exception ads.dataset.helper.DatasetLoadException(exc_msg): Bases: BaseException

class ads.dataset.helper.ElaboratedPath(source: Union[str, List[str]], format: Optional[str] = None, name: Optional[str] = None, **kwargs)

Bases: object

The Elaborated Path class unifies all of the operations and information related to a path or pathlist. Whether the user wants to An Elaborated path can accept any of the following as a valid source: * A single path * A glob pattern path * A directory * A list of paths (Note: all of these paths must be from the same filesystem AND have the same format) * A sqlalchemy connection url

Parameters:

source –
format –
kwargs –

By the end of this method, this class needs to have paths, format, and name ready

property format: str

property name: str

property num_paths: int: This method will return the number of paths found with the associated original glob, folder, or path. If this returns 0, :return:

property paths: List[str]

a list of str Each element will be a valid path

Type:: return

ads.dataset.helper.calculate_sample_size(population_size, min_size_to_sample, confidence_level=95, confidence_interval=1.0)

Find sample size for a population using Cochran’s Sample Size Formula.: With default values for confidence_level (percentage, default: 95%) and confidence_interval (margin of error, percentage, default: 1%)

SUPPORTED CONFIDENCE LEVELS: 50%, 68%, 90%, 95%, and 99% ONLY - this is because the Z-score is table based, and I’m only providing Z for common confidence levels.

ads.dataset.helper.concatenate(X, y)

ads.dataset.helper.convert_columns(df, feature_metadata=None, dtypes=None)

ads.dataset.helper.convert_to_html(plot)

ads.dataset.helper.deprecate_default_value(var, old_value, new_value, warning_msg, warning_type)

ads.dataset.helper.deprecate_variable(old_var, new_var, warning_msg, warning_type)

ads.dataset.helper.down_sample(df, target)

Fixes imbalanced dataset by down-sampling

Parameters:

df (pandas.DataFrame) –
target (name of the target column in df) –

Returns:

downsampled_df

Return type:

pandas.DataFrame

ads.dataset.helper.fix_column_names(X)

ads.dataset.helper.generate_sample(df: DataFrame, n: int, confidence_level: int = 95, confidence_interval: float = 1.0, **kwargs)

ads.dataset.helper.get_dtype(feature_type, dtype)

ads.dataset.helper.get_feature_type(name, series)

ads.dataset.helper.get_fill_val(feature_types, column, action, constant='constant')

ads.dataset.helper.is_text_data(df, target=None)

ads.dataset.helper.map_types(types)

ads.dataset.helper.parse_apache_log_datetime(x)

Parses datetime with timezone formatted as:: [day/month/year:hour:minute:second zone]

Source: https://mmas.github.io/read-apache-access-log-pandas .. rubric:: Example

>>> parse_datetime(‘13/Nov/2015:11:45:42 +0000’) datetime.datetime(2015, 11, 3, 11, 45, 4, tzinfo=<UTC>)

Due to problems parsing the timezone (%z) with datetime.strptime, the timezone will be obtained using the pytz library.

ads.dataset.helper.parse_apache_log_str(x)

Returns the string delimited by two characters.

Source: https://mmas.github.io/read-apache-access-log-pandas .. rubric:: Example

>>> parse_str(‘[my string]’) ‘my string’

ads.dataset.helper.rename_duplicate_cols(original_cols)

ads.dataset.helper.up_sample(df, target, sampler='default', feature_types=None)

Fixes imbalanced dataset by up-sampling

Parameters:

df (Union[pandas.DataFrame, dask.dataframe.core.DataFrame]) –
target (name of the target column in df) –
sampler (Should implement fit_resample(X,y) method) –
fillna (a dictionary contains the column name as well as the fill value,) – only needed when the column has missing values

Returns:

upsampled_df

Return type:

Union[pandas.DataFrame, dask.dataframe.core.DataFrame]

ads.dataset.helper.visualize_transformation(transformer_pipeline, text=None)

ads.dataset.helper.write_parquet(path, data, engine='fastparquet', metadata_dict=None, compression=None, storage_options=None)

Uses fast parquet to write dask dataframe and custom metadata in parquet format

Parameters:

path (str) – Path to write to
data (pandas.DataFrame) –
engine (string) – “auto” by default
metadata_dict (Deprecated, will not pass through) –
compression ({{'snappy', 'gzip', 'brotli', None}}, default 'snappy') – Name of the compression to use
storage_options (dict, optional) – storage arguments required to read the path

Returns:

str

Return type:

the file path the parquet was written to

ads.dataset.label_encoder module

class ads.dataset.label_encoder.DataFrameLabelEncoder

Bases: TransformerMixin

Label encoder for pandas.dataframe. dask.dataframe.core.DataFrame

fit(X): Fits a DataFrameLAbelEncoder.

transform(X): Transforms a dataset using the DataFrameLAbelEncoder.

ads.dataset.pipeline module

class ads.dataset.pipeline.TransformerPipeline(steps)

Bases: Pipeline

add(transformer)

Add transformer to data transformation pipeline

Parameters:: transformer (Union[TransformerMixin, tuple(str, TransformerMixin)]) – if tuple, (name, transformer implementing transform)

steps: List[Any]

visualize()

ads.dataset.plot module

class ads.dataset.plot.Plotting(df, feature_types, x, y=None, plot_type='infer', yscale=None)

Bases: object

select_best_plot(): Returns the best plot for a given dataset

show_in_notebook(**kwargs)

Visualizes the dataset by plotting the distribution of a feature or relationship between two features.

Parameters:

figsize (tuple) – defines the size of the fig
------- –

ads.dataset.progress module

class ads.dataset.progress.DummyProgressBar(*args, **kwargs)

Bases: ProgressBar

update(*args, **kwargs): Updates the progress bar

class ads.dataset.progress.ProgressBar

Bases: object

abstract update(description)

class ads.dataset.progress.TqdmProgressBar(max_progress=100, description='Running', verbose=False)

Bases: ProgressBar

update(description=None): Updates the progress bar

ads.dataset.recommendation module

class ads.dataset.recommendation.Recommendation(ds, recommendation_transformer)

Bases: object

recommendation_type_labels = ['Constant Columns', 'Potential Primary Key Columns', 'Imputation', 'Multicollinear Columns', 'Identify positive label for target', 'Fix imbalance in dataset']

recommendation_types = ['constant_column', 'primary_key', 'imputation', 'strong_correlation', 'positive_class', 'fix_imbalance']

show_in_notebook()

ads.dataset.recommendation_transformer module

class ads.dataset.recommendation_transformer.RecommendationTransformer(feature_metadata=None, correlation=None, target=None, is_balanced=False, target_type=None, feature_ranking=None, len=0, fix_imbalance=True, auto_transform=True, correlation_threshold=0.7)

Bases: TransformerMixin

fit(X)

fit_transform(X, y=None, **fit_params)

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters:

X (array-like of shape (n_samples, n_features)) – Input samples.
y (array-like of shape (n_samples,) or (n_samples, n_outputs), default=None) – Target values (None for unsupervised transformations).
**fit_params (dict) – Additional fit parameters.

Returns:

X_new – Transformed array.

Return type:

ndarray array of shape (n_samples, n_features_new)

transform(X, progress=<ads.dataset.progress.DummyProgressBar object>, fit_transform=False, update_transformer_log=False)

transformer_log(action): local wrapper to both log and record in the actions_performed array

ads.dataset.regression_dataset module

class ads.dataset.regression_dataset.RegressionDataset(df, sampled_df, target, target_type, shape, **kwargs): Bases: ADSDatasetWithTarget

ads.dataset.sampled_dataset module

class ads.dataset.sampled_dataset.PandasDataset(sampled_df, type_discovery=True, types={}, metadata=None, progress=<ads.dataset.progress.DummyProgressBar object>)

Bases: object

This class provides APIs that can work on a sampled dataset.

plot(x, y=None, plot_type='infer', yscale=None, verbose=True, sample_size=0)

Supports plotting feature distribution, and relationship between features.

Parameters:

x (str) – The name of the feature to plot
y (str, optional) – Name of the feature to plot against x
plot_type (str, default: infer) –
Override the inferred plot type for certain combinations of the data types of x and y. By default, the best plot type is inferred based on x and y data types. Valid values:
- box_plot - discrete feature vs continuous feature. Draw a box plot to show distributions with respect to categories,
- scatter - continuous feature vs continuous feature. Draw a scatter plot with possibility of several semantic groupings.
yscale (str, optional) – One of {“linear”, “log”, “symlog”, “logit”}. The y axis scale type to apply. Can be used when either x or y is an ordinal feature.
verbose (bool, default True) – Displays Note/Tips if True

plot_gis_scatter(lon='longitude', lat='latitude', ax=None)

Supports plotting Choropleth maps

Parameters:

df (pandas dataframe) – The dataframe to plot
x (str) – The name of the feature to plot, usually the longitude
y (str) – THe name of the feature to plot, usually the latitude

summary(feature_name=None)

Display list of features & their datatypes. Shows the column name and the feature’s meta_data if given a specific feature name.

Parameters:: date_col (str) – The name of the feature
Returns:: a dictionary that contains requested information
Return type:: dict

timeseries(date_col)

Supports any plotting operations where x=datetime.

Parameters:: date_col (str) – The name of the feature to plot
Returns:: a plotting object that contains a date column and dataframe
Return type:: func

ads.dataset.target module

class ads.dataset.target.TargetVariable(sampled_ds, target, target_type)

Bases: object

This class provides target specific APIs.

is_balanced(skewness_threshold=0.5, class_imbalance_threshold=0.5)

Returns True if the target is balanced, False otherwise.

Returns:: is_balanced
Return type:: bool

show_in_notebook(feature_names=None)

Plot target distribution or target versus feature relation.

Parameters:: feature_names (list, Optional) – Plot target against a list of features. Display target distribution if feature_names is not provided.

ads.dataset.timeseries module

class ads.dataset.timeseries.Timeseries(col_name, df, date_range=None, min=None, max=None)

Bases: object

plot(**kwargs)

Module contents

ads.dbmixin package

Submodules

ads.dbmixin.db_pandas_accessor module

class ads.dbmixin.db_pandas_accessor.ConnectionFactory

Bases: object

connectionprovider = {'hive': <class 'ads.bds.big_data_service.ADSHiveConnection'>}

classmethod get(engine='oracle')

class ads.dbmixin.db_pandas_accessor.DBAccessMixin

Bases: object

static read_sql(sql: str, connection_parameters: dict, bind_variables: Dict = {}, chunksize: Optional[int] = None, engine='oracle') → Union[DataFrame, Iterator[DataFrame]]

Read SQL query from oracle database into a DataFrame.

Parameters:

sql (str) – SQL query to be executed.
connection_parameters (dict) – A dictionary of connection_parameters - {“user_name”:””, “password”:””, “service_name”:””, “wallet_location”:””}
bind_variables (Optional[Dict]) – Key value of pair of bind variables and corresponding values
chunksize (Optional[int], default None) – If specified, return an iterator where chunksize is the number of rows to include in each chunk.
engine ({'oracle', 'mysql', 'hive'}, default 'oracle') – Select the database type - MySQL/Oracle/Hive to store the data

Returns:

DataFrame or Iterator[DataFrame].

Return type:

DataFrame or Iterator[DataFrame]

Examples

>>> connection_parameters = {
        "user_name": "<username>",
        "password": "<password>",
        "service_name": "{service_name}_{high|med|low}",
        "wallet_location": "/full/path/to/my_wallet.zip",
    }
>>> import pandas as pd
>>> import ads
>>> df = pd.DataFrame.ads.read_sql("SELECT * from Employee", connection_parameters=connection_parameters)
>>> df_with_bind = pd.DataFrame.ads.read_sql("SELECT * from EMPLOYEE WHERE EMPLOYEE_ID = :ID", bind_variables={"ID":"121212", connection_parameters=connection_parameters)

to_sql(table_name: str, connection_parameters: dict, if_exists: str = 'fail', batch_size=100000, engine='oracle', encoding='utf-8')

To save the dataframe df to database.

Parameters:

table_name (str) – Name of SQL table.
connection_parameters (dict) – A dictionary of connection_parameters - {“user_name”:””, “password”:””, “service_name”:””, “wallet_location”:””}
if_exists (: {'fail', 'replace', 'append'}, default 'fail') – How to behave if the table already exists. * fail: Raise a ValueError. If table exists, do nothing * replace: Drop the table before inserting new values. If table exists, drop it, recreate it, and insert data. * append: Insert new values to the existing table. If table exists, insert data. Create if does not exist.
batch_size (int, default 100000) – Inserting in batches improves insertion performance. Choose this value based on available memore and network bandwidth.
engine ({'oracle', 'mysql'}, default 'oracle') – Select the database type - MySQL or Oracle to store the data
encoding (str, default is "utf-8") – Encoding provided will be used for ecoding all columns, when inserting into table

Returns:

Nothing.

Return type:

None

Examples

>>> connection_parameters = {
        "user_name": "<username>",
        "password": "<password>",
        "service_name": "{service_name}_{high|med|low}",
        "wallet_location": "/full/path/to/my_wallet.zip",
    }
>>> import pandas as pd
>>> import ads
>>> df2 = pd.read_csv("my/data/csv")
>>> df2.ads.to_sql("MY_DATA_CSV", connection_parameters=connection_parameters)

Module contents

ads.environment package

Submodules

ads.environment.ml_runtime module

Module contents

ads.evaluations package

Submodules

ads.evaluations.evaluation_plot module

class ads.evaluations.evaluation_plot.EvaluationPlot

Bases: object

EvaluationPlot holds data and methods for plots and it used to output them

baseline(bool): whether to plot the null model or zero information model

baseline_kwargs(dict): keyword arguments for the baseline plot

color_wheel(dict): color information used by the plot

font_sz(dict): dictionary of plot methods

perfect(bool): determines whether a “perfect” classifier curve is displayed

perfect_kwargs(dict): parameters for the perfect classifier for precision/recall curves

prob_type(str): model type, i.e. classification or regression

get_legend_labels(legend_labels): Renders the legend labels on the plot

plot(evaluation, plots, num_classes, perfect, baseline, legend_labels): Generates the evalation plot

baseline = None

baseline_kwargs = {'c': '.2', 'ls': '--'}

color_wheel = ['teal', 'blueviolet', 'forestgreen', 'peru', 'y', 'dodgerblue', 'r']

double_overlay_plots = ['pr_and_roc_curve', 'lift_and_gain_chart']

font_sz = {'l': 14, 'm': 12, 's': 10, 'xl': 16, 'xs': 8}

classmethod get_legend_labels(legend_labels)

Gets the legend labels, resolves any conflicts such as length, and renders the labels for the plot

Parameters:: (dict) (legend_labels) – key/value dictionary containing legend label data
Return type:: Nothing

Examples

EvaluationPlot.get_legend_labels({‘class_0’: ‘green’, ‘class_1’: ‘yellow’, ‘class_2’: ‘red’})

perfect = None

perfect_kwargs = {'color': 'gold', 'label': 'Perfect Classifier', 'ls': '--'}

classmethod plot(evaluation, plots, num_classes, perfect=False, baseline=True, legend_labels=None)

Generates the evaluation plot

Parameters:

(DataFrame) (evaluation) – DataFrame with models as columns and metrics as rows.
(str) (plots) – The plot type based on class attribute prob_type.
(int) (num_classes) – The number of classes for the model.
(bool (baseline) – Whether to display the curve of a perfect classifier. Default value is False.
optional) – Whether to display the curve of a perfect classifier. Default value is False.
(bool – Whether to display the curve of the baseline, featureless model. Default value is True.
optional) – Whether to display the curve of the baseline, featureless model. Default value is True.
(dict (legend_labels) – Legend labels dictionary. Default value is None. If legend_labels not specified class names will be used for plots.
optional) – Legend labels dictionary. Default value is None. If legend_labels not specified class names will be used for plots.

Return type:

Nothing

prob_type = None

single_overlay_plots = ['lift_chart', 'gain_chart', 'roc_curve', 'pr_curve']

ads.evaluations.evaluator module

class ads.evaluations.evaluator.ADSEvaluator(test_data, models, training_data=None, positive_class=None, legend_labels=None, show_full_name=False, classes=None, classification_threshold=50)

Bases: object

ADS Evaluator class. This class holds field and methods for creating and using ADS evaluator objects.

evaluations

list of evaluations.

Type:: list[DataFrame]

is_classifier

Whether the dataset looks like a classification problem (versus regression).

Type:: bool

legend_labels

List of legend labels. Defaults to None.

Type:: dict

metrics_to_show

Names of metrics to show.

Type:: list[str]

models

The object built using ADSModel.from_estimator().

Type:: list[ads.common.model.ADSModel]

positive_class

The class to report metrics for binary dataset, assumed to be true.

Type:: str or int

show_full_name

Whether to show the name of the evaluator in relevant contexts.

Type:: bool

test_data

Test data to evaluate model on.

Type:: ads.common.data.ADSData

training_data

Training data to evaluate model.

Type:: ads.common.data.ADSData

Positive_Class_names

Class attribute listing the ways to represent positive classes

Type:: list

add_metrics(func, names): Adds the listed metics to the evaluator it is called on

del_metrics(names): Removes listed metrics from the evaluator object it is called on

add_models(models, show_full_name): Adds the listed models to the evaluator object

del_models(names): Removes the listed models from the evaluator object

show_in_notebook(plots, use_training_data, perfect, baseline, legend_labels): Visualize evalutation plots in the notebook

calculate_cost(tn_weight, fp_weight, fn_weight, tp_weight, use_training_data): Returns a cost associated with the input weights

Creates an ads evaluator object.

Parameters:

test_data (ads.common.data.ADSData instance) – Test data to evaluate model on. The object can be built using ADSData.build().
models (list[ads.common.model.ADSModel]) – The object can be built using ADSModel.from_estimator(). Maximum length of the list is 3
training_data (ads.common.data.ADSData instance, optional) – Training data to evaluate model on and compare metrics against test data. The object can be built using ADSData.build()
positive_class (str or int, optional) – The class to report metrics for binary dataset. If the target classes is True or False, positive_class will be set to True by default. If the dataset is multiclass or multilabel, this will be ignored.
legend_labels (dict, optional) – List of legend labels. Defaults to None. If legend_labels not specified class names will be used for plots.
show_full_name (bool, optional) – Show the name of the evaluator object. Defaults to False.
classes (List or None, optional) – A List of the possible labels for y, when evaluating a classification use case
classification_threshold (int, defaults to 50) – The maximum number of unique values that y must have to qualify as classification. If this threshold is exceeded, Evaluator assumes the model is regression.

Examples

>>> train, test = ds.train_test_split()
>>> model1 = MyModelClass1.train(train)
>>> model2 = MyModelClass2.train(train)
>>> evaluator = ADSEvaluator(test, [model1, model2])

>>> legend_labels={'class_0': 'one', 'class_1': 'two', 'class_2': 'three'}
>>> multi_evaluator = ADSEvaluator(test, models=[model1, model2],
...             legend_labels=legend_labels)

class EvaluationMetrics(ev_test, ev_train, use_training=False, less_is_more=None, precision=4)

Bases: object

Class holding evaluation metrics.

ev_test

evaluation test metrics

Type:: list

ev_train

evaluation training metrics

Type:: list

use_training

use training data

Type:: bool

less_is_more

metrics list

Type:: list

show_in_notebook(): Shows visualization metrics as a color coded table

DEFAULT_LABELS_MAP = {'accuracy': 'Accuracy', 'auc': 'ROC AUC', 'f1': 'F1', 'hamming_loss': 'Hamming distance', 'kappa_score_': "Cohen's kappa coefficient", 'precision': 'Precision', 'recall': 'Recall'}

property precision

show_in_notebook(labels={'accuracy': 'Accuracy', 'auc': 'ROC AUC', 'f1': 'F1', 'hamming_loss': 'Hamming distance', 'kappa_score_': "Cohen's kappa coefficient", 'precision': 'Precision', 'recall': 'Recall'})

Visualizes evaluation metrics as a color coded table.

Parameters:: labels (dictionary) – map printing specific labels for metrics display
Return type:: Nothing

Positive_Class_Names = ['yes', 'y', 't', 'true', '1']

add_metrics(funcs, names)

Adds the listed metrics to the evaluator object it is called on.

Parameters:

funcs (list) – The list of metrics to be added. This function will be provided y_true and y_pred, the true and predicted values for each model.
names (list[str])) – The list of metric names corresponding to the functions.

Return type:

Nothing

Examples

>>> def f1(y_true, y_pred):
...    return np.max(y_true - y_pred)
>>> evaluator = ADSEvaluator(test, [model1, model2])
>>> evaluator.add_metrics([f1], ['Max Residual'])
>>> evaluator.metrics
Output table will include the desired metric

add_models(models, show_full_name=False)

Adds the listed models to the evaluator object it is called on.

Parameters:

models (list[ADSModel]) – The list of models to be added
show_full_name (bool, optional) – Whether to show the full model name. Defaults to False. ** NOT USED **

Return type:

Nothing

Examples

>>> evaluator = ADSEvaluator(test, [model1, model2])
>>> evaluator.add_models("model3])

calculate_cost(tn_weight, fp_weight, fn_weight, tp_weight, use_training_data=False)

Returns a cost associated with the input weights.

Parameters:

tn_weight (int, float) – The weight to assign true negatives in calculating the cost
fp_weight (int, float) – The weight to assign false positives in calculating the cost
fn_weight (int, float) – The weight to assign false negatives in calculating the cost
tp_weight (int, float) – The weight to assign true positives in calculating the cost
use_training_data (bool, optional) – Use training data to pull the metrics. Defaults to False

Returns:

DataFrame with the cost calculated for each model

Return type:

pandas.DataFrame

Examples

>>> evaluator = ADSEvaluator(test, [model1, model2])
>>> costs_table = evaluator.calculate_cost(0, 10, 1000, 0)

del_metrics(names)

Removes the listed metrics from the evaluator object it is called on.

Parameters:: names (list[str]) – The list of names of metrics to be deleted. Names can be found by calling evaluator.test_evaluations.index.
Returns:: None
Return type:: None

Examples

>>> evaluator = ADSEvaluator(test, [model1, model2])
>>> evaluator.del_metrics(['mse])
>>> evaluator.metrics
Output table will exclude the desired metric

del_models(names)

Removes the listed models from the evaluator object it is called on.

Parameters:: names (list[str]) – the list of models to be delete. Names are the model names by default, and assigned internally when conflicts exist. Actual names can be found using evaluator.test_evaluations.columns
Return type:: Nothing

Examples

>>> model3.rename("model3")
>>> evaluator = ADSEvaluator(test, [model1, model2, model3])
>>> evaluator.del_models([model3])

property metrics

Returns evaluation metrics

Returns:: HTML representation of a table comparing relevant metrics.
Return type:: metrics

Examples

>>> evaluator = ADSEvaluator(test, [model1, model2])
>>> evaluator.metrics
Outputs table displaying metrics.

property raw_metrics

Returns the raw metric numbers

Parameters:

metrics (list, optional) – Request metrics to pull. Defaults to all.
use_training_data (bool, optional) – Use training data to pull metrics. Defaults to False

Returns:

The requested raw metrics for each model. If metrics is None return all.

Return type:

dict

Examples

>>> evaluator = ADSEvaluator(test, [model1, model2])
>>> raw_metrics_dictionary = evaluator.raw_metrics()

show_in_notebook(plots=None, use_training_data=False, perfect=False, baseline=True, legend_labels=None)

Visualize evaluation plots.

Parameters:

plots (list, optional) –
Filter the plots that are displayed. Defaults to None. The name of the plots are as below:
- regression - residuals_qq, residuals_vs_fitted
- binary classification - normalized_confusion_matrix, roc_curve, pr_curve
- multi class classification - normalized_confusion_matrix, precision_by_label, recall_by_label, f1_by_label
use_training_data (bool, optional) – Use training data to generate plots. Defaults to False. By default, this method uses test data to generate plots
legend_labels (dict, optional) – Rename legend labels, that used for multi class classification plots. Defaults to None. legend_labels dict keys are the same as class names. legend_labels dict values are strings. If legend_labels not specified class names will be used for plots.

Returns:

Nothing. Outputs several evaluation plots as specified by plots.

Return type:

None

Examples

>>> evaluator = ADSEvaluator(test, [model1, model2])
>>> evaluator.show_in_notebook()

>>> legend_labels={'class_0': 'green', 'class_1': 'yellow', 'class_2': 'red'}
>>> multi_evaluator = ADSEvaluator(test, [model1, model2],
...             legend_labels=legend_labels)
>>> multi_evaluator.show_in_notebook(plots=["normalized_confusion_matrix",
...             "precision_by_label", "recall_by_label", "f1_by_label"])

class ads.evaluations.evaluator.Evaluator(models: List[GenericModel], X: Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]], y: Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]], y_preds: Optional[List[Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]]]] = None, y_scores: Optional[List[Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]]]] = None, X_train: Optional[Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]]] = None, y_train: Optional[Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]]] = None, classes: Optional[List] = None, positive_class: Optional[str] = None, legend_labels: Optional[dict] = None, use_case_type: Optional[UseCaseType] = None)

Bases: object

BETA FEATURE Evaluator is the new and preferred way to evaluate a model of list of models. It contains a superset of the features of the soon-to-be-deprecated ADSEvaluator.

display(): Shows all plots and metrics within the jupyter notebook.

html(): Returns the raw string of the html report

save(filename): Saves the html report to the provided file location.

add_model(model): Adds a model to the existsing report. See documentation for more details.

add_metric(metric_fn): Adds a metric to the existsing report. See documentation for more details.

add_plot(plotting_fn): Adds a plot to the existing report. See documentation for more details.

Creates an ads evaluator object.

Parameters:

models (ads.model.GenericModel instance) – Test data to evaluate model on. The object can be built using from one of the framworks supported in ads.model.framework
X (DataFrame-like) – The data used to make a prediction. Can be set to None if y_preds is given. (And y_scores for more thorough analysis).
y (array-like) – The true values corresponding to the input data
y_preds (list of array-like, optional) – The predictions from each model in the same order as the models
y_scores (list of array-like, optional) – The predict_probas from each model in the same order as the models
X_train (DataFrame-like, optional) – The data used to train the model
y_train (array-like, optional) – The true values corresponding to the input training data
positive_class (str or int, optional) – The class to report metrics for binary dataset. If the target classes is True or False, positive_class will be set to True by default. If the dataset is multiclass or multilabel, this will be ignored.
legend_labels (dict, optional) – List of legend labels. Defaults to None. If legend_labels not specified class names will be used for plots.
classes (List or None, optional) – A List of the possible labels for y, when evaluating a classification use case
use_case_type (str, optional) – The type of problem this model is solving. This can be set during prepare(). Examples: “binary_classification”, “regression”, “multinomial_classification” Full list of supported types can be found here: ads.common.model_metadata.UseCaseType

Examples

>>> import tempfile
>>> from ads.evaluations.evaluator import Evaluator
>>> from sklearn.tree import DecisionTreeClassifier
>>> from sklearn.datasets import make_classification
>>> from sklearn.model_selection import train_test_split
>>> from ads.model.framework.sklearn_model import SklearnModel
>>> from ads.common.model_metadata import UseCaseType
>>>
>>> X, y = make_classification(n_samples=1000)
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
>>> est = DecisionTreeClassifier().fit(X_train, y_train)
>>> model = SklearnModel(estimator=est, artifact_dir=tempfile.mkdtemp())
>>> model.prepare(
        inference_conda_env="generalml_p38_cpu_v1",
        training_conda_env="generalml_p38_cpu_v1",
        X_sample=X_test,
        y_sample=y_test,
        use_case_type=UseCaseType.BINARY_CLASSIFICATION,
    )
>>> report = Evaluator([my_model], X=X_test, y=y_test)
>>> report.display()

add_models(models: List[GenericModel], y_preds: Optional[List[Any]] = None, y_scores: Optional[List[Any]] = None)

Add a model to an existing Evaluator to avoid re-calculating the values.

Parameters:

models (List[ads.model.GenericModel]) – Test data to evaluate model on. The object can be built using from one of the framworks supported in ads.model.framework
y_preds (list of array-like, optional) – The predictions from each model in the same order as the models
y_scores (list of array-like, optional) – The predict_probas from each model in the same order as the models

Return type:

self

Examples

>>> evaluator = Evaluator(models = [model1, model2], X=X, y=y)
>>> evaluator.add_models(models = [model3])

display(plots=None, perfect=False, baseline=True, legend_labels=None, precision=4, metrics_labels=None)

Visualize evaluation report.

Parameters:

plots (list, optional) –
Filter the plots that are displayed. Defaults to None. The name of the plots are as below:
- regression - residuals_qq, residuals_vs_fitted
- binary classification - normalized_confusion_matrix, roc_curve, pr_curve
- multi class classification - normalized_confusion_matrix, precision_by_label, recall_by_label, f1_by_label
perfect (bool, optional (default False)) – If True, will show how a perfect classifier would perform.
baseline (bool, optional (default True)) – If True, will show how a random classifier would perform.
legend_labels (dict, optional) – Rename legend labels, that used for multi class classification plots. Defaults to None. legend_labels dict keys are the same as class names. legend_labels dict values are strings. If legend_labels not specified class names will be used for plots.
precision (int, optional (default 4)) – The number of decimal points to show for each score/loss value
metrics_labels (List, optional) – The metrics that should be included in the html table.

Returns:

Nothing. Outputs several evaluation plots as specified by plots.

Return type:

None

Examples

>>> evaluator = Evaluator(models=[model1, model2], X=X, y=y)
>>> evaluator.display()

>>> legend_labels={'class_0': 'green', 'class_1': 'yellow', 'class_2': 'red'}
>>> multi_evaluator = Evaluator(models=[model1, model2], X=X, y=y, legend_labels=legend_labels)
>>> multi_evaluator.display(plots=["normalized_confusion_matrix",
...             "precision_by_label", "recall_by_label", "f1_by_label"])

html(plots=None, perfect=False, baseline=True, legend_labels=None, precision=4, metrics_labels=None)

Get raw HTML report.

Parameters:

plots (list, optional) –
Filter the plots that are displayed. Defaults to None. The name of the plots are as below:
- regression - residuals_qq, residuals_vs_fitted
- binary classification - normalized_confusion_matrix, roc_curve, pr_curve
- multi class classification - normalized_confusion_matrix, precision_by_label, recall_by_label, f1_by_label
perfect (bool, optional (default False)) – If True, will show how a perfect classifier would perform.
baseline (bool, optional (default True)) – If True, will show how a random classifier would perform.
legend_labels (dict, optional) – Rename legend labels, that used for multi class classification plots. Defaults to None. legend_labels dict keys are the same as class names. legend_labels dict values are strings. If legend_labels not specified class names will be used for plots.
precision (int, optional (default 4)) – The number of decimal points to show for each score/loss value
metrics_labels (List, optional) – The metrics that should be included in the html table.

Returns:

Nothing. Outputs several evaluation plots as specified by plots.

Return type:

None

Examples

>>> evaluator = Evaluator(models=[model1, model2], X=X, y=y)
>>> raw_html = evaluator.html()

save(filename: str, **kwargs)

Save HTML report.

Parameters:

filename (str) – The name and path of where to save the html report.
plots (list, optional) –
Filter the plots that are displayed. Defaults to None. The name of the plots are as below:
- regression - residuals_qq, residuals_vs_fitted
- binary classification - normalized_confusion_matrix, roc_curve, pr_curve
- multi class classification - normalized_confusion_matrix, precision_by_label, recall_by_label, f1_by_label
perfect (bool, optional (default False)) – If True, will show how a perfect classifier would perform.
baseline (bool, optional (default True)) – If True, will show how a random classifier would perform.
legend_labels (dict, optional) – Rename legend labels, that used for multi class classification plots. Defaults to None. legend_labels dict keys are the same as class names. legend_labels dict values are strings. If legend_labels not specified class names will be used for plots.
precision (int, optional (default 4)) – The number of decimal points to show for each score/loss value
metrics_labels (List, optional) – The metrics that should be included in the html table.

Returns:

Nothing. Outputs several evaluation plots as specified by plots.

Return type:

None

Examples

>>> evaluator = Evaluator(models=[model1, model2], X=X, y=y)
>>> evaluator.save("report.html")

ads.evaluations.statistical_metrics module

class ads.evaluations.statistical_metrics.ModelEvaluator(y_true, y_pred, model_name, classes=None, positive_class=None, y_score=None)

Bases: object

ModelEvaluator takes in the true and predicted values and returns a pandas dataframe

y_true

Type:: array-like object holding the true values for the model

y_pred

Type:: array-like object holding the predicted values for the model

model_name(str)

Type:: the name of the model

classes(list)

Type:: list of target classes

positive_class(str)

Type:: label for positive outcome from model

y_score

Type:: array-like object holding the scores for true values for the model

metrics(dict)

Type:: dictionary object holding model data

get_metrics(): Gets the metrics information in a dataframe based on the number of classes

safe_metrics_call(scoring_functions, \*args): Applies sklearn scoring functions to parameters in args

get_metrics()

Gets the metrics information in a dataframe based on the number of classes

Parameters:: self ((ModelEvaluator instance)) – The ModelEvaluator instance with the metrics.
Returns:: Pandas dataframe containing the metrics
Return type:: pandas.DataFrame

safe_metrics_call(scoring_functions, *args, **kwargs)

Applies the sklearn function in scoring_functions to parameters in args.

Parameters:

scoring_functions ((dict)) – Scoring functions dictionary
args ((keyword arguments)) – Arguments passed to the sklearn function from metrics

Returns:

Nothing

Raises:

Exception – If an error is enountered applying the sklearn function fn to arguments.

Module contents

class ads.evaluations.EvaluatorMixin

Bases: object

evaluate(X: Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]], y: Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]], y_pred: Optional[Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]]] = None, y_score: Optional[Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]]] = None, X_train: Optional[Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]]] = None, y_train: Optional[Union[Sequence[Sequence[Sequence[Sequence[Sequence[Any]]]]], _SupportsArray[dtype], Sequence[_SupportsArray[dtype]], Sequence[Sequence[_SupportsArray[dtype]]], Sequence[Sequence[Sequence[_SupportsArray[dtype]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype]]]]], bool, int, float, complex, str, bytes, Sequence[Union[bool, int, float, complex, str, bytes]], Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]], Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]], Sequence[Sequence[Sequence[Sequence[Union[bool, int, float, complex, str, bytes]]]]]]] = None, classes: Optional[List] = None, positive_class: Optional[str] = None, legend_labels: Optional[dict] = None, perfect: bool = True, filename: Optional[str] = None, use_case_type: Optional[str] = None)

Creates an ads evaluation report.

Parameters:

X (DataFrame-like) – The data used to make a prediction. Can be set to None if y_preds is given. (And y_scores for more thorough analysis).
y (array-like) – The true values corresponding to the input data
y_pred (array-like, optional) – The predictions from each model in the same order as the models
y_score (array-like, optional) – The predict_probas from each model in the same order as the models
X_train (DataFrame-like, optional) – The data used to train the model
y_train (array-like, optional) – The true values corresponding to the input training data
classes (List or None, optional) – A List of the possible labels for y, when evaluating a classification use case
positive_class (str or int, optional) – The class to report metrics for binary dataset. If the target classes is True or False, positive_class will be set to True by default. If the dataset is multiclass or multilabel, this will be ignored.
legend_labels (dict, optional) – List of legend labels. Defaults to None. If legend_labels not specified class names will be used for plots.
use_case_type (str, optional) – The type of problem this model is solving. This can be set during prepare(). Examples: “binary_classification”, “regression”, “multinomial_classification” Full list of supported types can be found here: ads.common.model_metadata.UseCaseType
filename (str, optional) – If filename is given, the html report will be saved to the location specified.

Examples

>>> import tempfile
>>> from ads.evaluations.evaluator import Evaluator
>>> from sklearn.tree import DecisionTreeClassifier
>>> from sklearn.datasets import make_classification
>>> from sklearn.model_selection import train_test_split
>>> from ads.model.framework.sklearn_model import SklearnModel
>>> from ads.common.model_metadata import UseCaseType
>>>
>>> X, y = make_classification(n_samples=1000)
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
>>> est = DecisionTreeClassifier().fit(X_train, y_train)
>>> model = SklearnModel(estimator=est, artifact_dir=tempfile.mkdtemp())
>>> model.prepare(
        inference_conda_env="generalml_p38_cpu_v1",
        training_conda_env="generalml_p38_cpu_v1",
        X_sample=X_test,
        y_sample=y_test,
        use_case_type=UseCaseType.BINARY_CLASSIFICATION,
    )
>>> model.evaluate(X_test, y_test, filename="report.html")

ads.experiments package

Module contents

ads.feature_engineering package

Subpackages

ads.feature_engineering.accessor package

Subpackages

ads.feature_engineering.accessor.mixin package

Submodules

ads.feature_engineering.accessor.mixin.correlation module

ads.feature_engineering.accessor.mixin.correlation.cat_vs_cat(df: DataFrame, normal_form: bool = True) → DataFrame: Calculates the correlation of all pairs of categorical features and categorical features.

ads.feature_engineering.accessor.mixin.correlation.cat_vs_cont(df: DataFrame, categorical_columns, continuous_columns, normal_form: bool = True) → DataFrame: Calculates the correlation of all pairs of categorical features and continuous features.

ads.feature_engineering.accessor.mixin.correlation.cont_vs_cont(df: DataFrame, normal_form: bool = True) → DataFrame: Calculates the Pearson correlation between two columns of the DataFrame.

ads.feature_engineering.accessor.mixin.eda_mixin module

This exploratory data analysis (EDA) Mixin is used in the ADS accessor for the Pandas Dataframe. The series of purpose-driven methods enable the data scientist to complete analysis on the dataframe.

From the accessor we have access to the pandas object the user is interacting with as well as corresponding lists of feature types per column.

class ads.feature_engineering.accessor.mixin.eda_mixin.EDAMixin

Bases: object

correlation_ratio() → DataFrame

Generate a Correlation Ratio data frame for all categorical-continuous variable pairs.

Returns:

pandas.DataFrame
Correlation Ratio correlation data frame with the following 3 columns –
1. Column 1 (name of the first categorical/continuous column)
2. Column 2 (name of the second categorical/continuous column)
3. Value (correlation value)

Note

Pairs will be replicated. For example for variables x and y, we would have (x,y), (y,x) both with same correlation value. We will also have (x,x) and (y,y) with value 1.0.

correlation_ratio_plot() → Axes

Generate a heatmap of the Correlation Ratio correlation for all categorical-continuous variable pairs.

Returns:: Correlation Ratio correlation plot object that can be updated by the customer
Return type:: Plot object

cramersv() → DataFrame

Generate a Cramer’s V correlation data frame for all categorical variable pairs.

Gives a warning for dropped non-categorical columns.

Returns:

Cramer’s V correlation data frame with the following 3 columns:

Column 1 (name of the first categorical column)
Column 2 (name of the second categorical column)
Value (correlation value)

Return type:

pandas.DataFrame

Note

Pairs will be replicated. For example for variables x and y, we would have (x,y), (y,x) both with same correlation value. We will also have (x,x) and (y,y) with value 1.0.

cramersv_plot() → Axes

Generate a heatmap of the Cramer’s V correlation for all categorical variable pairs.

Gives a warning for dropped non-categorical columns.

Returns:: Cramer’s V correlation plot object that can be updated by the customer
Return type:: Plot object

feature_count() → DataFrame

Counts the number of columns for each feature type and each primary feature. The column of primary is the number of primary feature types that is assigned to the column.

Returns:: The number of columns for each feature type The number of columns for each primary feature
Return type:: Dataframe with

Examples

>>> df.ads.feature_type
{'PassengerId': ['ordinal', 'category'],
'Survived': ['ordinal'],
'Pclass': ['ordinal'],
'Name': ['category'],
'Sex': ['category']}
>>> df.ads.feature_count()
    Feature Type        Count       Primary
0       category            3             2
1        ordinal            3             3

feature_plot() → DataFrame

For every column in the dataframe plot generate a list of summary plots based on the most relevant feature type.

Returns:: Dataframe with 2 columns: 1. Column - feature name 2. Plot - plot object
Return type:: pandas.DataFrame

feature_stat() → DataFrame

Summary statistics Dataframe provided.

This returns feature stats on each column using FeatureType summary method.

Examples

>>> df = pd.read_csv('~/advanced-ds/tests/vor_datasets/vor_titanic.csv')
>>> df.ads.feature_stat().head()
         Column    Metric                       Value
0       PassengerId         count                       891.000
1       PassengerId         mean                        446.000
2       PassengerId         standard deviation      257.354
3       PassengerId         sample minimum          1.000
4       PassengerId         lower quartile              223.500

Returns:: Dataframe with 3 columns: name, metric, value
Return type:: pandas.DataFrame

pearson() → DataFrame

Generate a Pearson correlation data frame for all continuous variable pairs.

Gives a warning for dropped non-numerical columns.

Returns:

pandas.DataFrame
Pearson correlation data frame with the following 3 columns –
1. Column 1 (name of the first continuous column)
2. Column 2 (name of the second continuous column)
3. Value (correlation value)

Note

Pairs will be replicated. For example for variables x and y, we’d have (x,y), (y,x) both with same correlation value. We’ll also have (x,x) and (y,y) with value 1.0.

pearson_plot() → Axes

Generate a heatmap of the Pearson correlation for all continuous variable pairs.

Returns:: Pearson correlation plot object that can be updated by the customer
Return type:: Plot object

warning() → DataFrame

Generates a data frame that lists feature specific warnings.

Returns:: The list of feature specific warnings.
Return type:: pandas.DataFrame

Examples

>>> df.ads.warning()
    Column    Feature Type         Warning               Message       Metric    Value
--------------------------------------------------------------------------------------
0      Age      continuous           Zeros      Age has 38 zeros        Count       38
1      Age      continuous           Zeros   Age has 12.2% zeros   Percentage    12.2%

ads.feature_engineering.accessor.mixin.eda_mixin_series module

This exploratory data analysis (EDA) Mixin is used in the ADS accessor for the Pandas Series. The series of purpose-driven methods enable the data scientist to complete univariate analysis.

From the accessor we have access to the pandas object the user is interacting with as well as corresponding list of feature types.

class ads.feature_engineering.accessor.mixin.eda_mixin_series.EDAMixinSeries

Bases: object

feature_plot() → Axes

For the series generate a summary plot based on the most relevant feature type.

Returns:: Plot object for the series based on the most relevant feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

feature_stat() → DataFrame

Summary statistics Dataframe provided.

This returns feature stats on series using FeatureType summary method.

Examples

>>> df = pd.read_csv('~/advanced-ds/tests/vor_datasets/vor_titanic.csv')
>>> df['Cabin'].ads.feature_stat()
    Metric      Value
0       count       891
1       unqiue      147
2       missing     687

Returns:: Dataframe with 2 columns and rows for different metric values
Return type:: pandas.DataFrame

warning() → DataFrame

Generates a data frame that lists feature specific warnings.

Returns:: The list of feature specific warnings.
Return type:: pandas.DataFrame

Examples

>>> df["Age"].ads.warning()
  Feature Type       Warning               Message         Metric      Value
 ---------------------------------------------------------------------------
0   continuous         Zeros      Age has 38 zeros          Count         38
1   continuous         Zeros   Age has 12.2% zeros     Percentage      12.2%

ads.feature_engineering.accessor.mixin.feature_types_mixin module

The module that represents the ADS Feature Types Mixin class that extends Pandas Series and Dataframe accessors.

Classes

ADSFeatureTypesMixin
ADS Feature Types Mixin class that extends Pandas Series and Dataframe accessors.

class ads.feature_engineering.accessor.mixin.feature_types_mixin.ADSFeatureTypesMixin

Bases: object

ADS Feature Types Mixin class that extends Pandas Series and DataFrame accessors.

warning_registered(cls) → pd.DataFrame: Lists registered warnings for registered feature types.

validator_registered(cls) → pd.DataFrame: Lists registered validators for registered feature types.

help(self, prop: str = None) → None: Help method that prints either a table of available properties or, given a property, returns its docstring.

help(prop: Optional[str] = None) → None

Help method that prints either a table of available properties or, given an individual property, returns its docstring.

Parameters:: prop (str) – The Name of property.
Returns:: Nothing.
Return type:: None

validator_registered() → DataFrame

Lists registered validators for registered feature types.

Returns:: The list of registered validators for registered feature types
Return type:: pandas.DataFrame

Examples

>>> df.ads.validator_registered()
         Column     Feature Type        Validator                 Condition                    Handler
------------------------------------------------------------------------------------------------------
0   PhoneNumber    phone_number   is_phone_number                        ()            default_handler
1   PhoneNumber    phone_number   is_phone_number    {'country_code': '+7'}   specific_country_handler
2    CreditCard    credit_card     is_credit_card                        ()            default_handler

>>> df['PhoneNumber'].ads.validator_registered()
    Feature Type            Validator                 Condition                     Handler
-------------------------------------------------------------------------------------------
0   phone_number      is_phone_number                        ()             default_handler
1   phone_number      is_phone_number    {'country_code': '+7'}    specific_country_handler

warning_registered() → DataFrame

Lists registered warnings for all registered feature types.

Returns:: The list of registered warnings for registered feature types.
Return type:: pandas.DataFrame

Examples

>>> df.ads.warning_registered()
       Column    Feature Type             Warning                    Handler
   -------------------------------------------------------------------------
   0      Age      continuous               zeros              zeros_handler
   1      Age      continuous    high_cardinality   high_cardinality_handler

>>> df["Age"].ads.warning_registered()
       Feature Type             Warning                    Handler
   ---------------------------------------------------------------
   0     continuous               zeros              zeros_handler
   1     continuous    high_cardinality   high_cardinality_handler

ads.feature_engineering.accessor.mixin.utils module

Module contents

Submodules

ads.feature_engineering.accessor.dataframe_accessor module

The ADS accessor for the Pandas DataFrame. The accessor will be initialized with the pandas object the user is interacting with.

Examples

>>> from ads.feature_engineering.accessor.dataframe_accessor import ADSDataFrameAccessor
    >>> from ads.feature_engineering.feature_type.continuous import Continuous
    >>> from ads.feature_engineering.feature_type.creditcard import CreditCard
    >>> from ads.feature_engineering.feature_type.string import String
    >>> from ads.feature_engineering.feature_type.base import Tag
>>> df = pd.DataFrame({'Name': ['Alex'], 'CreditCard': ["4532640527811543"]})
>>> df.ads.feature_type
{'Name': ['string'], 'Credit Card': ['string']}
>>> df.ads.feature_type_description
          Column   Feature Type                        Description
------------------------------------------------------------------
0           Name         string    Type representing string values.
1    Credit Card         string    Type representing string values.
>>> df.ads.default_type
{'Name': 'string', 'Credit Card': 'string'}
>>> df.ads.feature_type = {'Name':['string', Tag('abc')]}
>>> df.ads.tags
{'Name': ['abc']}
>>> df.ads.feature_type = {'Credit Card':['credit_card']}
>>> df.ads.feature_select(include=['credit_card'])
                    Credit Card
-------------------------------
0                 4532640527811543

class ads.feature_engineering.accessor.dataframe_accessor.ADSDataFrameAccessor(pandas_obj)

Bases: ADSFeatureTypesMixin, EDAMixin, DBAccessMixin, DataLabelingAccessMixin

ADS accessor for the Pandas DataFrame.

columns

The column labels of the DataFrame.

Type:: List[str]

tags(self) → Dict[str, str]: Gets the dictionary of user defined tags for the dataframe.

default_type(self) → Dict[str, str]: Gets the map of columns and associated default feature type names.

feature_type(self) → Dict[str, List[str]]: Gets the list of registered feature types.

feature_type_description(self) → pd.DataFrame: Gets the list of registered feature types in a DataFrame format.

sync(self, src: Union[pd.DataFrame, pd.Series]) → pd.DataFrame: Syncs feature types of current DataFrame with that from src.

feature_select(self, include: List[Union[FeatureType, str]] = None, exclude: List[Union[FeatureType, str]] = None) → pd.DataFrame: Gets the list of registered feature types in a DataFrame format.

help(self, prop: str = None) → None: Provids docstring for affordable methods and properties.

Examples

>>> from ads.feature_engineering.accessor.dataframe_accessor import ADSDataFrameAccessor
>>> from ads.feature_engineering.feature_type.continuous import Continuous
>>> from ads.feature_engineering.feature_type.creditcard import CreditCard
>>> from ads.feature_engineering.feature_type.string import String
>>> from ads.feature_engineering.feature_type.base import Tag
df = pd.DataFrame({'Name': ['Alex'], 'CreditCard': ["4532640527811543"]})
>>> df.ads.feature_type
{'Name': ['string'], 'Credit Card': ['string']}
>>> df.ads.feature_type_description
          Column   Feature Type                        Description
-------------------------------------------------------------------
0           Name         string    Type representing string values.
1    Credit Card         string    Type representing string values.
>>> df.ads.default_type
{'Name': 'string', 'Credit Card': 'string'}
>>> df.ads.feature_type = {'Name':['string', Tag('abc')]}
>>> df.ads.tags
{'Name': ['abc']}
>>> df.ads.feature_type = {'Credit Card':['credit_card']}
>>> df.ads.feature_select(include=['credit_card'])
                   Credit Card
------------------------------
0             4532640527811543

Initializes ADS Pandas DataFrame Accessor.

Parameters:: pandas_obj (pandas.DataFrame) – Pandas dataframe
Raises:: ValueError – If provided DataFrame has duplicate columns.

property default_type: Dict[str, str]

Gets the map of columns and associated default feature type names.

Returns:: The dictionary where key is column name and value is the name of default feature type.
Return type:: Dict[str, str]

feature_select(include: Optional[List[Union[FeatureType, str]]] = None, exclude: Optional[List[Union[FeatureType, str]]] = None) → DataFrame

Returns a subset of the DataFrame’s columns based on the column feature_types.

Parameters:

include (List[Union[FeatureType, str]], optional) – Defaults to None. A list of FeatureType subclass or str to be included.
exclude (List[Union[FeatureType, str]], optional) – Defaults to None. A list of FeatureType subclass or str to be excluded.

Raises:

ValueError – If both of include and exclude are empty
ValueError – If include and exclude are used simultaneously

Returns:

The subset of the frame including the feature types in include and excluding the feature types in exclude.

Return type:

pandas.DataFrame

property feature_type: Dict[str, List[str]]

Gets the list of registered feature types.

Returns:: The dictionary where key is column name and value is list of associated feature type names.
Return type:: Dict[str, List[str]]

property feature_type_description: DataFrame

Gets the list of registered feature types in a DataFrame format.

Return type:: pandas.DataFrame

Examples

>>> df.ads.feature_type_description()
          Column   Feature Type                         Description
-------------------------------------------------------------------
0           City         string    Type representing string values.
1   Phone Number         string    Type representing string values.

info() → Any

Gets information about the dataframe.

Returns:: The information about the dataframe.
Return type:: Any

model_schema(max_col_num: int = 2000)

Generates schema from the dataframe.

Parameters:: max_col_num (int, optional. Defaults to 1000) – The maximum column size of the data that allows to auto generate schema.

Examples

>>> df = pd.read_csv('./orcl_attrition.csv', usecols=['Age', 'Attrition'])
>>> schema = df.ads.model_schema()
>>> schema
Schema:
    - description: Attrition
    domain:
        constraints: []
        stats:
        count: 1470
        unique: 2
        values: String
    dtype: object
    feature_type: String
    name: Attrition
    required: true
    - description: Age
    domain:
        constraints: []
        stats:
        25%: 31.0
        50%: 37.0
        75%: 44.0
        count: 1470.0
        max: 61.0
        mean: 37.923809523809524
        min: 19.0
        std: 9.135373489136732
        values: Integer
    dtype: int64
    feature_type: Integer
    name: Age
    required: true
>>> schema.to_dict()
{'Schema': [{'dtype': 'object',
    'feature_type': 'String',
    'name': 'Attrition',
    'domain': {'values': 'String',
        'stats': {'count': 1470, 'unique': 2},
        'constraints': []},
    'required': True,
    'description': 'Attrition'},
    {'dtype': 'int64',
    'feature_type': 'Integer',
    'name': 'Age',
    'domain': {'values': 'Integer',
        'stats': {'count': 1470.0,
        'mean': 37.923809523809524,
        'std': 9.135373489136732,
        'min': 19.0,
        '25%': 31.0,
        '50%': 37.0,
        '75%': 44.0,
        'max': 61.0},
        'constraints': []},
    'required': True,
    'description': 'Age'}]}

Returns:: data schema.
Return type:: ads.feature_engineering.schema.Schema
Raises:: ads.feature_engineering.schema.DataSizeTooWide – If the number of columns of input data exceeds max_col_num.

sync(src: Union[DataFrame, Series]) → DataFrame

Syncs feature types of current DataFrame with that from src.

Syncs feature types of current dataframe with that from src, where src can be a dataframe or a series. In either case, only columns with matched names are synced.

Parameters:: src (pd.DataFrame | pd.Series) – The source to sync from.
Returns:: Synced dataframe.
Return type:: pandas.DataFrame

property tags: Dict[str, List[str]]

Gets the dictionary of user defined tags for the dataframe. Key is column name and value is list of tag names.

Returns:: The map of columns and associated default tags.
Return type:: Dict[str, List[str]]

ads.feature_engineering.accessor.series_accessor module

The ADS accessor for the Pandas Series. The accessor will be initialized with the pandas object the user is interacting with.

Examples

>>> from ads.feature_engineering.accessor.series_accessor import ADSSeriesAccessor
>>> from ads.feature_engineering.feature_type.string import String
>>> from ads.feature_engineering.feature_type.ordinal import Ordinal
>>> from ads.feature_engineering.feature_type.base import Tag
>>> series = pd.Series(['name1', 'name2', 'name3'])
>>> series.ads.default_type
'string'
>>> series.ads.feature_type
['string']
>>> series.ads.feature_type_description
    Feature Type                         Description
----------------------------------------------------
0         string    Type representing string values.
>>> series.ads.feature_type = ['string', Ordinal, Tag('abc')]
>>> series.ads.feature_type
['string', 'ordinal', 'abc']
>>> series1 = series.dropna()
>>> series1.ads.sync(series)
>>> series1.ads.feature_type
['string', 'ordinal', 'abc']

class ads.feature_engineering.accessor.series_accessor.ADSSeriesAccessor(pandas_obj: Series)

Bases: ADSFeatureTypesMixin, EDAMixinSeries

ADS accessor for Pandas Series.

name

The name of Series.

Type:: str

tags

The list of tags for the Series.

Type:: List[str]

help(self, prop: str = None) → None: Provids docstring for affordable methods and properties.

sync(self, src: Union[pd.DataFrame, pd.Series]) → None: Syncs feature types of current series with that from src.

default_type(self) → str: Gets the name of default feature type for the series.

feature_type(self) → List[str]: Gets the list of registered feature types for the series.

feature_type_description(self) → pd.DataFrame: Gets the list of registered feature types in a DataFrame format.

Examples

>>> from ads.feature_engineering.accessor.series_accessor import ADSSeriesAccessor
>>> from ads.feature_engineering.feature_type.string import String
>>> from ads.feature_engineering.feature_type.ordinal import Ordinal
>>> from ads.feature_engineering.feature_type.base import Tag
>>> series = pd.Series(['name1', 'name2', 'name3'])
>>> series.ads.default_type
'string'
>>> series.ads.feature_type
['string']
>>> series.ads.feature_type_description
    Feature Type                         Description
----------------------------------------------------
0         string    Type representing string values.
>>> series.ads.feature_type = ['string', Ordinal, Tag('abc')]
>>> series.ads.feature_type
['string', 'ordinal', 'abc']
>>> series1 = series.dropna()
>>> series1.ads.sync(series)
>>> series1.ads.feature_type
['string', 'ordinal', 'abc']

Initializes ADS Pandas Series Accessor.

Parameters:: pandas_obj (pd.Series) – The pandas series

property default_type: str

Gets the name of default feature type for the series.

Returns:: The name of default feature type.
Return type:: str

property feature_type: List[str]

Gets the list of registered feature types for the series.

Returns:: Names of feature types.
Return type:: List[str]

Examples

>>> series = pd.Series(['name1'])
>>> series.ads.feature_type = ['name', 'string', Tag('tag for name')]
>>> series.ads.feature_type
['name', 'string', 'tag for name']

property feature_type_description: DataFrame

Gets the list of registered feature types in a DataFrame format.

Returns:: The DataFrame with feature types for this series.
Return type:: pd.DataFrame

Examples

>>> series = pd.Series(['name1'])
>>> series.ads.feature_type = ['name', 'string', Tag('Name tag')]
>>> series.ads.feature_type_description
        Feature Type                               Description
    ----------------------------------------------------------
    0           name            Type representing name values.
    1         string          Type representing string values.
    2        Name tag                                     Tag.

sync(src: Union[DataFrame, Series]) → None

Syncs feature types of current series with that from src.

The src could be a dataframe or a series. In either case, only columns with matched names are synced.

Parameters:: src ((pd.DataFrame | pd.Series)) – The source to sync from.
Returns:: Nothing.
Return type:: None

Examples

>>> series = pd.Series(['name1', 'name2', 'name3', None])
>>> series.ads.feature_type = ['name']
>>> series.ads.feature_type
['name', string]
>>> series.dropna().ads.feature_type
['string']
>>> series1 = series.dropna()
>>> series1.ads.sync(series)
>>> series1.ads.feature_type
['name', 'string']

class ads.feature_engineering.accessor.series_accessor.ADSSeriesValidator(feature_type_list: List[FeatureType], series: Series)

Bases: object

Class helper to invoke registerred validator on a series level.

Initializes ADS series validator.

Parameters:

feature_type_list (List[FeatureType]) – The list of feature types.
series (pd.Series) – The pandas series.

Module contents

ads.feature_engineering.adsimage package

Subpackages

ads.feature_engineering.adsimage.interface package

Submodules

ads.feature_engineering.adsimage.interface.reader module

class ads.feature_engineering.adsimage.interface.reader.Reader

Bases: ABC

The Data Reader Interface.

abstract read() → Generator[Any, Any, Any]

The abstract method to read data.

Yields:: Generator[Any, Any, Any]

Module contents

Submodules

ads.feature_engineering.adsimage.image module

The module helping to load/save images from/to the local path or OCI object storage bucket.

Classes

ADSImage: Work with image files that are stored in Oracle Cloud Infrastructure Object Storage.

Examples

>>> from ads.feature_engineering import ADSImage
>>> from IPython.core.display import display
>>> img = ADSImage.open("1.jpg")
>>> display(img)
>>> img.save("oci://<bucket_name>@<namespace>/1.jpg")
>>> img1 = ADSImage.open("oci://<bucket_name>@<namespace>/1.jpg")
>>> display(img1)

class ads.feature_engineering.adsimage.image.ADSImage(img: Image, filename: Optional[str] = None)

Bases: object

Work with image files that are stored in Oracle Cloud Infrastructure Object Storage. PIL (Python Imaging Library) is used as a backend to represent and manipulate images. The PIL adds support for opening, manipulating, processing and saving various image file formats.

img

A PIL Image object.

Type:: Image.Image

filename

The image filename.

Type:: str

save(self, path: str, ...) → None: Saves the image under the given filename.

open(cls, path: str, ...) → ADSImage: Opens the given image file.

Examples

>>> from ads.feature_engineering import ADSImage
>>> from IPython.core.display import display
>>> img = ADSImage.open("1.jpg")
>>> img.save("oci://<bucket_name>@<namespace>/1.jpg")
>>> img1 = ADSImage.open("oci://<bucket_name>@<namespace>/1.jpg")
>>> display(img1)

Initializes ADSImage object.

Parameters:

img (PIL.Image.Image) – The PIL Image object.
filename ((str, optional). Defaults to None.) – The image filename.

Returns:

Nothing.

Return type:

None

Raises:

TypeError – If img is not an instance of PIL.Image.Image.
ValueError – If img is not provided.

classmethod open(path: str, storage_options: Optional[Dict] = None) → ADSImage

Opens the given image file.

Parameters:

path (str) – The file path to open image. Can be local path or OCI object storage URI. Example: oci://<bucket_name>@<namespace>/1.jpg
storage_options ((Dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

The instance of ADSImage.

Return type:

ADSImage

save(path: str, format: Optional[str] = None, auth: Optional[Dict] = None, **kwargs: Optional[Dict]) → None

Save the image under the given filename. If no format is specified, the format to use is determined from the image object or filename extension, if possible.

Parameters:

path (str) – The file path to save image. It can be a local path or an Oracle Cloud Infrastructure Object Storage URI. Example: oci://<bucket_name>@<namespace>/1.jpg
format ((str, optional). Defaults to None.) – If omitted and path has a file extension the format of the image will be based on the extension. Can be any format supported by PIL Image. The available options are described in the PIL image format documentation: https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html
auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth() API. To override the default behavior, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create an authentication signer and provide an IdentityClient object.
kwargs – Additional keyword arguments that would be passed to the PIL Image save() method.

Returns:

Nothing.

Return type:

None

to_bytes(**kwargs: Optional[Dict]) → BytesIO

Converts image to bytes. If no format is specified, the format to use is determined from the image object if it possible.

Parameters:

kwargs –

format: (str, optional). Defaults to None.: If omitted and path has a file extension the format of the image will be based on the extension. Can be any format supported by PIL Image. The available options are described in the PIL image format documentation: https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html

Additional keyword arguments that would be passed to the PIL Image save() method.

Returns:

The image converted to bytes.

Return type:

BytesIO

ads.feature_engineering.adsimage.image_reader module

The module loads and saves images from/to a local path or Oracle Object Storage.

Classes

ADSImageReader: The class reads image files from a local path or an Oracle Cloud Infrastructure Object Storage bucket.

Examples

>>> from ads.feature_engineering import ADSImageReader
>>> image_reader = ADSImageReader.from_uri(path="oci://<bucket_name>@<namespace>/*.jpg")
>>> index=1
>>> for image in image_reader.read():
...     image.save(f"{index}.jpg")
...     index+=1

class ads.feature_engineering.adsimage.image_reader.ADSImageReader(reader: Reader)

Bases: object

The class reads image files from a local path or an Oracle Cloud Infrastructure Object Storage bucket.

from_uri(cls, path: Union[str, List[str]], ...) → ADSImageReader: Constructs the ADSImageReader object.

read(self) → Generator[ADSImage, Any, Any]: Reads images.

Examples

>>> from ads.feature_engineering import ADSImageReader
>>> image_reader = ADSImageReader.from_uri(path="oci://<bucket_name>@<namespace>/*.jpg")
>>> index=1
>>> for image in image_reader.read():
...     image.save(f"{index}.jpg")
...     index+=1

Initializes ADSImageReader instance.

Parameters:: reader (Reader) – Can be any reader implementing the `Reader` interface.
Returns:: Nothing.
Return type:: None

classmethod from_uri(path: Union[str, List[str]], auth: Optional[Dict] = None) → ADSImageReader

Constructs the ADSImageReader object.

Parameters:

path (Union[str, List[str]]) –
The path where the images located. Can be local path, OCI object storage URI or a list of paths. It is also support globbing. Example:

oci://<bucket_name>@<namespace>/1.jpg [oci://<bucket_name>@<namespace>/1.jpg, oci://<bucket_name>@<namespace>/2.jpg] oci://<bucket_name>@<namespace>/*.jpg
auth ((Dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

The ADSImageReader object.

Return type:

ADSImageReader

read() → Generator[ADSImage, Any, Any]

Read image files.

Yields:: Generator[ADSImage, Any, Any]

class ads.feature_engineering.adsimage.image_reader.ImageFileReader(path: Union[str, List[str]], auth: Optional[Dict] = None)

Bases: object

The class reads image files from a local path or an Oracle Cloud Infrastructure Object Storage bucket.

Initializes ImageFileReader instance.

Parameters:

path (Union[str, List[str]]) –
The path where the images located. Can be local path, OCI object storage URI or a list of paths. It is also support globbing. Example:

oci://<bucket_name>@<namespace>/1.jpg [oci://<bucket_name>@<namespace>/1.jpg, oci://<bucket_name>@<namespace>/2.jpg] oci://<bucket_name>@<namespace>/*.jpg
auth ((Dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

Nothing.

Return type:

None

Raises:

TypeError – If the path is not an instance of str or List[str].
ValueError – If the path is not provided.

read() → Generator[ADSImage, Any, Any]

Read image files.

Yields:: Generator[ADSImage, Any, Any]

Module contents

ads.feature_engineering.adsstring package

Subpackages

ads.feature_engineering.adsstring.oci_language package

Module contents

ads.feature_engineering.adsstring.parsers package

Submodules

ads.feature_engineering.adsstring.parsers.base module

class ads.feature_engineering.adsstring.parsers.base.Parser

Bases: object

property adjective

property adverb

property bigram

property noun

property pos

property sentence

property trigram

property verb

property word

property word_count

ads.feature_engineering.adsstring.parsers.nltk_parser module

ads.feature_engineering.adsstring.parsers.spacy_parser module

class ads.feature_engineering.adsstring.parsers.spacy_parser.SpacyParser

Bases: Parser

property adjective: List[str]

property adverb: List[str]

property bigram: DataFrame

property entity_artwork: List[str]

property entity_extract: DataFrame

property entity_location: List[str]

property entity_organization: List[str]

property entity_people: List[str]

property entity_product: List[str]

property lemma: List[str]

property noun: List[str]

property noun_phrase: List[str]

property pos: DataFrame

property sentence: List[str]

property token: List[str]

property trigram: DataFrame

property verb: List[str]

property word: List[str]

property word_count: DataFrame

Module contents

ads.feature_engineering.adsstring.string package

Module contents

Submodules

ads.feature_engineering.adsstring.common_regex_mixin module

class ads.feature_engineering.adsstring.common_regex_mixin.CommonRegex(text='')

Bases: object

class regex(obj, regex): Bases: object

regexes = {'address': re.compile('\\d{1,5} [\\w\\s]{1,30}(?:street|st|crescent|avenue|ave|road|rd|highway|hwy|square|sq|trail|trl|drive|dr|court|ct|park|parkway|pkwy|circle|cir|boulevard|blvd)\\W?(?=\\s|$)', re.IGNORECASE), 'address_with_zip': re.compile('\\d{1,5} [\\w\\s]{1,30}(?:street|st(?:\\s|\\.)+|avenue|ave(?:\\s|\\.)+|road|rd(?:\\s|\\.)+|highway|hwy(?:\\s|\\.)+|square|sq(?:\\s|\\.)+|trail|trl(?:\\s|\\.)+|drive|dr(?:\\s|\\.)+|court|ct(?:\\s|\\.), re.IGNORECASE), 'credit_card': re.compile('((?:(?:\\d{4}[- ]?){3}\\d{4}|\\d{15,16}))(?![\\d])'), 'date': re.compile('(?:(?<!\\:)(?<!\\:\\d)[0-3]?\\d(?:st|nd|rd|th)?\\s+(?:of\\s+)?(?:jan\\.?|january|feb\\.?|february|mar\\.?|march|apr\\.?|april|may|jun\\.?|june|jul\\.?|july|aug\\.?|august|sep\\.?|september|oct\\.?|oc, re.IGNORECASE), 'email': re.compile("([a-z0-9!#$%&\\'*+\\/=?^_`{|.}~-]+@(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?)", re.IGNORECASE), 'ip': re.compile('(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)', re.IGNORECASE), 'ipv6': re.compile('\\s*(?!.*::.*::)(?:(?!:)|:(?=:))(?:[0-9a-f]{0,4}(?:(?<=::)|(?<!::):)){6}(?:[0-9a-f]{0,4}(?:(?<=::)|(?<!::):)[0-9a-f]{0,4}(?:(?<=::)|(?<!:)|(?<=:)(?<!::):)|(?:25[0-4]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(?:\\, re.IGNORECASE|re.DOTALL|re.VERBOSE), 'link': re.compile('(?i)((?:https?://|www\\d{0,3}[.])?[a-z0-9.\\-]+[.](?:(?:international)|(?:construction)|(?:contractors)|(?:enterprises)|(?:photography)|(?:immobilien)|(?:management)|(?:technology)|(?:directory)|(?:e, re.IGNORECASE), 'phone_number_US': re.compile('((?:(?<![\\d-])(?:\\+?\\d{1,3}[-.\\s*]?)?(?:\$?\\d{3}\$?[-.\\s*]?)?\\d{3}[-.\\s*]?\\d{4}(?![\\d-]))|(?:(?<![\\d-])(?:(?:\$\\+?\\d{2}\$)|(?:\\+?\\d{2}))\\s*\\d{2}\\s*\\d{3}\\s*\\d{4}(?![\\d-])))'), 'phone_number_US_with_ext': re.compile('((?:(?:\\+?1\\s*(?:[.-]\\s*)?)?(?:\$\\s*(?:[2-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9])\\s*\$|(?:[2-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9]))\\s*(?:[.-]\\s*)?)?(?:[2-9]1[02-9]|[2-9][02-9]1|[2-9][0, re.IGNORECASE), 'po_box': re.compile('P\\.? ?O\\.? Box \\d+', re.IGNORECASE), 'price': re.compile('[$]\\s?[+-]?[0-9]{1,3}(?:(?:,?[0-9]{3}))*(?:\\.[0-9]{1,2})?'), 'ssn': re.compile('(?!666|000|9\\d{2})\\d{3}[- ](?!00)\\d{2}[- ](?!0{4})\\d{4}'), 'time': re.compile('\\d{1,2}:\\d{2} ?(?:[ap]\\.?m\\.?)?|\\d[ap]\\.?m\\.?', re.IGNORECASE), 'zip_code': re.compile('\\b\\d{5}(?:[-\\s]\\d{4})?\\b')}

class ads.feature_engineering.adsstring.common_regex_mixin.CommonRegexMixin

Bases: object

property address

property credit_card

property date

property email

property ip

property link

property phone_number_US

property price

redact(fields: Union[List[str], Dict[str, str]]) → str

Remove personal information in a string. For example, “Jane’s phone number is 123-456-7890” is turned into “Jane’s phone number is [phone_number_US].”

Parameters:: fields ((list(str) | dict)) – either a list of fields to redact, e.g. [‘email’, ‘phone_number_US’], in which case the redacted text is replaced with capitalized word like [EMAIL] or [PHONE_NUMBER_US_WITH_EXT], or a dictionary where key is a field to redact and value is the replacement text, e.g., {‘email’: ‘HIDDEN_EMAIL’}.
Returns:: redacted string
Return type:: str

redact_map = {'address': '[ADDRESS]', 'address_with_zip': '[ADDRESS_WITH_ZIP]', 'credit_card': '[CREDIT_CARD]', 'date': '[DATE]', 'email': '[EMAIL]', 'ip': '[IP]', 'ipv6': '[IPV6]', 'link': '[LINK]', 'phone_number_US': '[PHONE_NUMBER_US]', 'phone_number_US_with_ext': '[PHONE_NUMBER_US_WITH_EXT]', 'po_box': '[PO_BOX]', 'price': '[PRICE]', 'ssn': '[SSN]', 'time': '[TIME]', 'zip_code': '[ZIP_CODE]'}

property ssn

property time

property zip_code

ads.feature_engineering.adsstring.oci_language module

ads.feature_engineering.adsstring.string module

Module contents

ads.feature_engineering.dataset package

Submodules

ads.feature_engineering.dataset.zip_code_data module

The module save the gis data corresponding to zip code.

Module contents

ads.feature_engineering.feature_type package

Subpackages

ads.feature_engineering.feature_type.adsstring package

Subpackages

ads.feature_engineering.feature_type.adsstring.parsers package

Submodules

ads.feature_engineering.feature_type.adsstring.parsers.base module

class ads.feature_engineering.feature_type.adsstring.parsers.base.Parser

Bases: object

property adjective

property adverb

property bigram

property noun

property pos

property sentence

property trigram

property verb

property word

property word_count

ads.feature_engineering.feature_type.adsstring.parsers.nltk_parser module

ads.feature_engineering.feature_type.adsstring.parsers.spacy_parser module

class ads.feature_engineering.feature_type.adsstring.parsers.spacy_parser.SpacyParser

Bases: Parser

property adjective: List[str]

property adverb: List[str]

property bigram: DataFrame

property entity_artwork: List[str]

property entity_extract: DataFrame

property entity_location: List[str]

property entity_organization: List[str]

property entity_people: List[str]

property entity_product: List[str]

property lemma: List[str]

property noun: List[str]

property noun_phrase: List[str]

property pos: DataFrame

property sentence: List[str]

property token: List[str]

property trigram: DataFrame

property verb: List[str]

property word: List[str]

property word_count: DataFrame

Module contents

Submodules

ads.feature_engineering.feature_type.adsstring.common_regex_mixin module

class ads.feature_engineering.feature_type.adsstring.common_regex_mixin.CommonRegex(text='')

Bases: object

class regex(obj, regex): Bases: object

regexes = {'address': re.compile('\\d{1,5} [\\w\\s]{1,30}(?:street|st|crescent|avenue|ave|road|rd|highway|hwy|square|sq|trail|trl|drive|dr|court|ct|park|parkway|pkwy|circle|cir|boulevard|blvd)\\W?(?=\\s|$)', re.IGNORECASE), 'address_with_zip': re.compile('\\d{1,5} [\\w\\s]{1,30}(?:street|st(?:\\s|\\.)+|avenue|ave(?:\\s|\\.)+|road|rd(?:\\s|\\.)+|highway|hwy(?:\\s|\\.)+|square|sq(?:\\s|\\.)+|trail|trl(?:\\s|\\.)+|drive|dr(?:\\s|\\.)+|court|ct(?:\\s|\\.), re.IGNORECASE), 'credit_card': re.compile('((?:(?:\\d{4}[- ]?){3}\\d{4}|\\d{15,16}))(?![\\d])'), 'date': re.compile('(?:(?<!\\:)(?<!\\:\\d)[0-3]?\\d(?:st|nd|rd|th)?\\s+(?:of\\s+)?(?:jan\\.?|january|feb\\.?|february|mar\\.?|march|apr\\.?|april|may|jun\\.?|june|jul\\.?|july|aug\\.?|august|sep\\.?|september|oct\\.?|oc, re.IGNORECASE), 'email': re.compile("([a-z0-9!#$%&\\'*+\\/=?^_`{|.}~-]+@(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?)", re.IGNORECASE), 'ip': re.compile('(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)', re.IGNORECASE), 'ipv6': re.compile('\\s*(?!.*::.*::)(?:(?!:)|:(?=:))(?:[0-9a-f]{0,4}(?:(?<=::)|(?<!::):)){6}(?:[0-9a-f]{0,4}(?:(?<=::)|(?<!::):)[0-9a-f]{0,4}(?:(?<=::)|(?<!:)|(?<=:)(?<!::):)|(?:25[0-4]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(?:\\, re.IGNORECASE|re.DOTALL|re.VERBOSE), 'link': re.compile('(?i)((?:https?://|www\\d{0,3}[.])?[a-z0-9.\\-]+[.](?:(?:international)|(?:construction)|(?:contractors)|(?:enterprises)|(?:photography)|(?:immobilien)|(?:management)|(?:technology)|(?:directory)|(?:e, re.IGNORECASE), 'phone_number_US': re.compile('((?:(?<![\\d-])(?:\\+?\\d{1,3}[-.\\s*]?)?(?:\$?\\d{3}\$?[-.\\s*]?)?\\d{3}[-.\\s*]?\\d{4}(?![\\d-]))|(?:(?<![\\d-])(?:(?:\$\\+?\\d{2}\$)|(?:\\+?\\d{2}))\\s*\\d{2}\\s*\\d{3}\\s*\\d{4}(?![\\d-])))'), 'phone_number_US_with_ext': re.compile('((?:(?:\\+?1\\s*(?:[.-]\\s*)?)?(?:\$\\s*(?:[2-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9])\\s*\$|(?:[2-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9]))\\s*(?:[.-]\\s*)?)?(?:[2-9]1[02-9]|[2-9][02-9]1|[2-9][0, re.IGNORECASE), 'po_box': re.compile('P\\.? ?O\\.? Box \\d+', re.IGNORECASE), 'price': re.compile('[$]\\s?[+-]?[0-9]{1,3}(?:(?:,?[0-9]{3}))*(?:\\.[0-9]{1,2})?'), 'ssn': re.compile('(?!666|000|9\\d{2})\\d{3}[- ](?!00)\\d{2}[- ](?!0{4})\\d{4}'), 'time': re.compile('\\d{1,2}:\\d{2} ?(?:[ap]\\.?m\\.?)?|\\d[ap]\\.?m\\.?', re.IGNORECASE), 'zip_code': re.compile('\\b\\d{5}(?:[-\\s]\\d{4})?\\b')}

class ads.feature_engineering.feature_type.adsstring.common_regex_mixin.CommonRegexMixin

Bases: object

property address

property credit_card

property date

property email

property ip

property link

property phone_number_US

property price

redact(fields: Union[List[str], Dict[str, str]]) → str

Remove personal information in a string. For example, “Jane’s phone number is 123-456-7890” is turned into “Jane’s phone number is [phone_number_US].”

Parameters:: fields ((list(str) | dict)) – either a list of fields to redact, e.g. [‘email’, ‘phone_number_US’], in which case the redacted text is replaced with capitalized word like [EMAIL] or [PHONE_NUMBER_US_WITH_EXT], or a dictionary where key is a field to redact and value is the replacement text, e.g., {‘email’: ‘HIDDEN_EMAIL’}.
Returns:: redacted string
Return type:: str

redact_map = {'address': '[ADDRESS]', 'address_with_zip': '[ADDRESS_WITH_ZIP]', 'credit_card': '[CREDIT_CARD]', 'date': '[DATE]', 'email': '[EMAIL]', 'ip': '[IP]', 'ipv6': '[IPV6]', 'link': '[LINK]', 'phone_number_US': '[PHONE_NUMBER_US]', 'phone_number_US_with_ext': '[PHONE_NUMBER_US_WITH_EXT]', 'po_box': '[PO_BOX]', 'price': '[PRICE]', 'ssn': '[SSN]', 'time': '[TIME]', 'zip_code': '[ZIP_CODE]'}

property ssn

property time

property zip_code

ads.feature_engineering.feature_type.adsstring.oci_language module

class ads.feature_engineering.feature_type.adsstring.oci_language.OCILanguage(auth=None)

Bases: object

Defines the OCILanguage plugin for ADSString built on top of the OCI Language Services.

Example

>>> ADSString.plugin_register(OCILanguage)
>>> s = ADSString("This movie is awesome.")
>>> s.absa
>>> s.text_classification
>>> s.ner
>>> s.language_dominant

property absa: List[Dict]: Runs aspect-based sentiment analysis on the text to gauge teh mood or the tone of the text.

property key_phrase: List[Dict]: Extracts the most relevant words from the ADSString object and assigns them a score.

property language_dominant: List[Dict]: Determines the language of the text along with ISO 639-1 language code and a probability score.

property ner: List[Dict]: Detects named entites in the text.

property text_classification: List[Dict]: Analyses the text and identifies categories for the content with a confidence score.

ads.feature_engineering.feature_type.adsstring.string module

class ads.feature_engineering.feature_type.adsstring.string.ADSString(text: str, language='english')

Bases: str, FeatureType, CommonRegexMixin

Defines an enhanced string class for the purpose of performing NLP tasks. Its functionalities can be extended by registering plugins.

plugins

list of plugins that add functionalities to the class.

Type:: List

string

plain string

Type:: str

Example

>>> ADSString.nlp_backend('nltk')
>>> s = ADSString("Walking my dog on a breezy day is the best.")
>>> s.lower() # regular string methods still work
>>> s.replace("a", "e")
>>> s.noun
>>> s.pos #parts of speech
>>> s = ADSString("get in touch with my associate at john.smith@gmail.com to schedule")
>>> s.email
>>> ADSString.plugin_register(OCILanguage)
>>> s = ADSString("This movie is awesome.")
>>> s.absa

Initialze the class and register plugins.

Parameters:

text (str) – input text
language (str, optional) – language of the text, by default “english”.

Raises:

TypeError – input text is not a string.

capitalize()

Return a capitalized version of the string.

More specifically, make the first character have upper case and the rest lower case.

casefold(): Return a version of the string suitable for caseless comparisons.

center(width, fillchar=' ', /)

Return a centered string of length width.

Padding is done using the specified fill character (default is a space).

count(sub[, start[, end]]) → int: Return the number of non-overlapping occurrences of substring sub in string S[start:end]. Optional arguments start and end are interpreted as in slice notation.

description = 'Type representing enhanced string class.'

encode(encoding='utf-8', errors='strict')

Encode the string using the codec registered for encoding.

encoding: The encoding in which to encode the string.
errors: The error handling scheme to use for encoding errors. The default is ‘strict’ meaning that encoding errors raise a UnicodeEncodeError. Other possible values are ‘ignore’, ‘replace’ and ‘xmlcharrefreplace’ as well as any other name registered with codecs.register_error that can handle UnicodeEncodeErrors.

endswith(suffix[, start[, end]]) → bool: Return True if S ends with the specified suffix, False otherwise. With optional start, test S beginning at that position. With optional end, stop comparing S at that position. suffix can also be a tuple of strings to try.

expandtabs(tabsize=8)

Return a copy where all tab characters are expanded using spaces.

If tabsize is not given, a tab size of 8 characters is assumed.

find(sub[, start[, end]]) → int

Return the lowest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation.

Return -1 on failure.

format(*args, **kwargs) → str: Return a formatted version of S, using substitutions from args and kwargs. The substitutions are identified by braces (‘{’ and ‘}’).

format_map(mapping) → str: Return a formatted version of S, using substitutions from mapping. The substitutions are identified by braces (‘{’ and ‘}’).

help() → None

List available properties.

Parameters:: plugin (Any) – registered plugin
Return type:: None

index(sub[, start[, end]]) → int

Return the lowest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation.

Raises ValueError when the substring is not found.

isalnum()

Return True if the string is an alpha-numeric string, False otherwise.

A string is alpha-numeric if all characters in the string are alpha-numeric and there is at least one character in the string.

isalpha()

Return True if the string is an alphabetic string, False otherwise.

A string is alphabetic if all characters in the string are alphabetic and there is at least one character in the string.

isascii()

Return True if all characters in the string are ASCII, False otherwise.

ASCII characters have code points in the range U+0000-U+007F. Empty string is ASCII too.

isdecimal()

Return True if the string is a decimal string, False otherwise.

A string is a decimal string if all characters in the string are decimal and there is at least one character in the string.

isdigit()

Return True if the string is a digit string, False otherwise.

A string is a digit string if all characters in the string are digits and there is at least one character in the string.

isidentifier()

Return True if the string is a valid Python identifier, False otherwise.

Use keyword.iskeyword() to test for reserved identifiers such as “def” and “class”.

islower()

Return True if the string is a lowercase string, False otherwise.

A string is lowercase if all cased characters in the string are lowercase and there is at least one cased character in the string.

isnumeric()

Return True if the string is a numeric string, False otherwise.

A string is numeric if all characters in the string are numeric and there is at least one character in the string.

isprintable()

Return True if the string is printable, False otherwise.

A string is printable if all of its characters are considered printable in repr() or if it is empty.

isspace()

Return True if the string is a whitespace string, False otherwise.

A string is whitespace if all characters in the string are whitespace and there is at least one character in the string.

istitle()

Return True if the string is a title-cased string, False otherwise.

In a title-cased string, upper- and title-case characters may only follow uncased characters and lowercase characters only cased ones.

isupper()

Return True if the string is an uppercase string, False otherwise.

A string is uppercase if all cased characters in the string are uppercase and there is at least one cased character in the string.

join(iterable, /)

Concatenate any number of strings.

The string whose method is called is inserted in between each given string. The result is returned as a new string.

Example: ‘.’.join([‘ab’, ‘pq’, ‘rs’]) -> ‘ab.pq.rs’

language_model_cache = {}

ljust(width, fillchar=' ', /)

Return a left-justified string of length width.

Padding is done using the specified fill character (default is a space).

lower(): Return a copy of the string converted to lowercase.

lstrip(chars=None, /)

Return a copy of the string with leading whitespace removed.

If chars is given and not None, remove characters in chars instead.

maketrans(y=None, z=None, /)

Return a translation table usable for str.translate().

If there is only one argument, it must be a dictionary mapping Unicode ordinals (integers) or characters to Unicode ordinals, strings or None. Character keys will be then converted to ordinals. If there are two arguments, they must be strings of equal length, and in the resulting dictionary, each character in x will be mapped to the character at the same position in y. If there is a third argument, it must be a string, whose characters will be mapped to None in the result.

nlp_backend() → None

Set backend for extracting NLP related properties.

Parameters:

backend (str, optional) – name of backend, by default ‘nltk’.

Raises:

ModuleNotFoundError – module corresponding to backend is not found.
ValueError – input backend is invalid.

Return type:

None

partition(sep, /)

Partition the string into three parts using the given separator.

This will search for the separator in the string. If the separator is found, returns a 3-tuple containing the part before the separator, the separator itself, and the part after it.

If the separator is not found, returns a 3-tuple containing the original string and two empty strings.

plugin_clear() → None: Clears plugins.

plugin_list() → None: List registered plugins.

plugin_register() → None

Register a plugin

Parameters:: plugin (Any) – plugin to register
Return type:: None

plugins = []

redact(fields: Union[List[str], Dict[str, str]]) → str

Remove personal information in a string. For example, “Jane’s phone number is 123-456-7890” is turned into “Jane’s phone number is [phone_number_US].”

Parameters:: fields ((list(str) | dict)) – either a list of fields to redact, e.g. [‘email’, ‘phone_number_US’], in which case the redacted text is replaced with capitalized word like [EMAIL] or [PHONE_NUMBER_US_WITH_EXT], or a dictionary where key is a field to redact and value is the replacement text, e.g., {‘email’: ‘HIDDEN_EMAIL’}.
Returns:: redacted string
Return type:: str

replace(old, new, count=-1, /)

Return a copy with all occurrences of substring old replaced by new.

count
Maximum number of occurrences to replace. -1 (the default value) means replace all occurrences.

If the optional argument count is given, only the first count occurrences are replaced.

rfind(sub[, start[, end]]) → int

Return the highest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation.

Return -1 on failure.

rindex(sub[, start[, end]]) → int

Return the highest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation.

Raises ValueError when the substring is not found.

rjust(width, fillchar=' ', /)

Return a right-justified string of length width.

Padding is done using the specified fill character (default is a space).

rpartition(sep, /)

Partition the string into three parts using the given separator.

This will search for the separator in the string, starting at the end. If the separator is found, returns a 3-tuple containing the part before the separator, the separator itself, and the part after it.

If the separator is not found, returns a 3-tuple containing two empty strings and the original string.

rsplit(sep=None, maxsplit=-1)

Return a list of the words in the string, using sep as the delimiter string.

sep
The delimiter according which to split the string. None (the default value) means split according to any whitespace, and discard empty strings from the result.

maxsplit
Maximum number of splits to do. -1 (the default value) means no limit.

Splits are done starting at the end of the string and working to the front.

rstrip(chars=None, /)

Return a copy of the string with trailing whitespace removed.

If chars is given and not None, remove characters in chars instead.

split(sep=None, maxsplit=-1)

Return a list of the words in the string, using sep as the delimiter string.

sep: The delimiter according which to split the string. None (the default value) means split according to any whitespace, and discard empty strings from the result.
maxsplit: Maximum number of splits to do. -1 (the default value) means no limit.

splitlines(keepends=False)

Return a list of the lines in the string, breaking at line boundaries.

Line breaks are not included in the resulting list unless keepends is given and true.

startswith(prefix[, start[, end]]) → bool: Return True if S starts with the specified prefix, False otherwise. With optional start, test S beginning at that position. With optional end, stop comparing S at that position. prefix can also be a tuple of strings to try.

property string

strip(chars=None, /)

Return a copy of the string with leading and trailing whitespace removed.

If chars is given and not None, remove characters in chars instead.

swapcase(): Convert uppercase characters to lowercase and lowercase characters to uppercase.

title()

Return a version of the string where each word is titlecased.

More specifically, words start with uppercased characters and all remaining cased characters have lower case.

translate(table, /)

Replace each character in the string using the given translation table.

table
Translation table, which must be a mapping of Unicode ordinals to Unicode ordinals, strings, or None.

The table must implement lookup/indexing via __getitem__, for instance a dictionary or list. If this operation raises LookupError, the character is left untouched. Characters mapped to None are deleted.

upper(): Return a copy of the string converted to uppercase.

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning

The Feature Warning class.

Provides functionality to register warning handlers and invoke them.

register(self, name: str, handler: Callable) → None: Registers a new warning for the feature type.

unregister(self, name: str) → None: Unregisters warning.

registered(self) → pd.DataFrame: Gets the list of registered warnings.

Examples

>>> warning = FeatureWarning()
>>> def warning_handler_zeros_count(data):
...    return pd.DataFrame(
...        [['Zeros', 'Age has 38 zeros', 'Count', 38]],
...        columns=['Warning', 'Message', 'Metric', 'Value'])
>>> def warning_handler_zeros_percentage(data):
...    return pd.DataFrame(
...        [['Zeros', 'Age has 12.2% zeros', 'Percentage', '12.2%']],
...        columns=['Warning', 'Message', 'Metric', 'Value'])
>>> warning.register(name="zeros_count", handler=warning_handler_zeros_count)
>>> warning.register(name="zeros_percentage", handler=warning_handler_percentage)
>>> warning.registered()
                  Warning                              Handler
    ----------------------------------------------------------
    0         zeros_count          warning_handler_zeros_count
    1    zeros_percentage     warning_handler_zeros_percentage

>>> warning.zeros_percentage(data_series)
             Warning               Message         Metric      Value
    ----------------------------------------------------------------
    0          Zeros      Age has 38 zeros          Count         38

>>> warning.zeros_count(data_series)
              Warning              Message         Metric      Value
    ----------------------------------------------------------------
    1          Zeros   Age has 12.2% zeros     Percentage      12.2%

>>> warning(data_series)
             Warning               Message         Metric      Value
    ----------------------------------------------------------------
    0          Zeros      Age has 38 zeros          Count         38
    1          Zeros   Age has 12.2% zeros     Percentage      12.2%

>>> warning.unregister('zeros_count')
>>> warning(data_series)
             Warning               Message         Metric      Value
    ----------------------------------------------------------------
    0          Zeros   Age has 12.2% zeros     Percentage      12.2%

zfill(width, /)

Pad a numeric string with zeros on the left, to fill a field of the given width.

The string is never truncated.

ads.feature_engineering.feature_type.adsstring.string.to_adsstring(func: Callable) → Callable

Decorator that converts output of a function to ADSString if it returns a string.

Parameters:: func (Callable) – function to decorate
Returns:: decorated function
Return type:: Callable

ads.feature_engineering.feature_type.adsstring.string.wrap_output_string(decorator: Callable) → Callable

Class decorator that applies a decorator to all methods of a class.

Parameters:: decorator (Callable) – decorator to apply
Returns:: class decorator
Return type:: Callable

Module contents

ads.feature_engineering.feature_type.handler package

Submodules

ads.feature_engineering.feature_type.handler.feature_validator module

The module that helps to register custom validators for the feature types and extending registered validators with dispatching based on the specific arguments.

Classes

FeatureValidator
The Feature Validator class to manage custom validators.

FeatureValidatorMethod
The Feature Validator Method class. Extends methods which requires dispatching based on the specific arguments.

class ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator

Bases: object

The Feature Validator class to manage custom validators.

register(self, name: str, handler: Callable, condition: Union[Tuple, Dict[str, Any]] = None, replace: bool = False) → None: Registers new validator.

unregister(self, name: str, condition: Union[Tuple, Dict[str, Any]] = None) → None: Unregisters validator.

registered(self) → pd.DataFrame: Gets the list of registered validators.

Examples

>>> series = pd.Series(['+1-202-555-0141', '+1-202-555-0142'], name='Phone Number')

>>> def phone_number_validator(data: pd.Series) -> pd.Series:
...    print("phone_number_validator")
...    return data

>>> def universal_phone_number_validator(data: pd.Series, country_code) -> pd.Series:
...    print("universal_phone_number_validator")
...    return data

>>> def us_phone_number_validator(data: pd.Series, country_code) -> pd.Series:
...    print("us_phone_number_validator")
...    return data

>>> PhoneNumber.validator.register(name="is_phone_number", handler=phone_number_validator, replace=True)
>>> PhoneNumber.validator.register(name="is_phone_number", handler=universal_phone_number_validator, condition = ('country_code',))
>>> PhoneNumber.validator.register(name="is_phone_number", handler=us_phone_number_validator, condition = {'country_code':'+1'})

>>> PhoneNumber.validator.is_phone_number(series)
    phone_number_validator
    0     +1-202-555-0141
    1     +1-202-555-0142

>>> PhoneNumber.validator.is_phone_number(series, country_code = '+7')
    universal_phone_number_validator
    0     +1-202-555-0141
    1     +1-202-555-0142

>>> PhoneNumber.validator.is_phone_number(series, country_code = '+1')
    us_phone_number_validator
    0     +1-202-555-0141
    1     +1-202-555-0142

>>> PhoneNumber.validator.registered()
               Validator                 Condition                            Handler
    ---------------------------------------------------------------------------------
    0    is_phone_number                        ()             phone_number_validator
    1    is_phone_number          ('country_code')   universal_phone_number_validator
    2    is_phone_number    {'country_code': '+1'}          us_phone_number_validator

>>> series.ads.validator.is_phone_number()
    phone_number_validator
        0     +1-202-555-0141
        1     +1-202-555-0142

>>> series.ads.validator.is_phone_number(country_code = '+7')
    universal_phone_number_validator
        0     +1-202-555-0141
        1     +1-202-555-0142

>>> series.ads.validator.is_phone_number(country_code = '+1')
    us_phone_number_validator
    0     +1-202-555-0141
    1     +1-202-555-0142

Initializes the FeatureValidator.

register(name: str, handler: Callable, condition: Optional[Union[Tuple, Dict[str, Any]]] = None, replace: bool = False) → None

Registers new validator.

Parameters:

name (str) – The validator name.
handler (callable) – The handler.
condition (Union[Tuple, Dict[str, Any]]) – The condition for the validator.
replace (bool) – The flag indicating if the registered validator should be replaced with the new one.

Returns:

Nothing.

Return type:

None

Raises:

ValueError – The name is empty or handler is not provided.
TypeError – The handler is not callable. The name of the validator is not a string.
ValidatorAlreadyExists – The validator is already registered.

registered() → DataFrame

Gets the list of registered validators.

Returns:: The list of registerd validators.
Return type:: pd.DataFrame

unregister(name: str, condition: Optional[Union[Tuple, Dict[str, Any]]] = None) → None

Unregisters validator.

Parameters:

name (str) – The name of the validator to be unregistered.
condition (Union[Tuple, Dict[str, Any]]) – The condition for the validator to be unregistered.

Returns:

Nothing.

Return type:

None

Raises:

TypeError – The name of the validator is not a string.
ValidatorNotFound – The validator not found.
ValidatorWIthConditionNotFound – The validator with provided condition not found.

class ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidatorMethod(handler: Callable)

Bases: object

The Feature Validator Method class.

Extends methods which requires dispatching based on the specific arguments.

register(self, condition: Union[Tuple, Dict[str, Any]], handler: Callable) → None: Registers new handler.

unregister(self, condition: Union[Tuple, Dict[str, Any]]) → None: Unregisters existing handler.

registered(self) → pd.DataFrame: Gets the list of registered handlers.

Initializes the Feature Validator Method.

Parameters:: handler (Callable) – The handler that will be called by default if suitable one not found.

register(condition: Union[Tuple, Dict[str, Any]], handler: Callable) → None

Registers new handler.

Parameters:

condition (Union[Tuple, Dict[str, Any]]) – The condition which will be used to register a new handler.
handler (Callable) – The handler to be registered.

Returns:

Nothing.

Return type:

None

Raises:

ValueError – If condition not provided or provided in the wrong format. If handler not provided or has wrong format.

registered() → DataFrame

Gets the list of registered handlers.

Returns:: The list of registerd handlers.
Return type:: pd.DataFrame

unregister(condition: Union[Tuple, Dict[str, Any]]) → None

Unregisters existing handler.

Parameters:: condition (Union[Tuple, Dict[str, Any]]) – The condition which will be used to unregister a handler.
Returns:: Nothing.
Return type:: None
Raises:: ValueError – If condition not provided or provided in the wrong format. If condition not registered.

exception ads.feature_engineering.feature_type.handler.feature_validator.ValidatorAlreadyExists(name: str): Bases: ValueError

exception ads.feature_engineering.feature_type.handler.feature_validator.ValidatorNotFound(name: str): Bases: ValueError

exception ads.feature_engineering.feature_type.handler.feature_validator.ValidatorWithConditionAlreadyExists(name: str): Bases: ValueError

exception ads.feature_engineering.feature_type.handler.feature_validator.ValidatorWithConditionNotFound(name: str): Bases: ValueError

exception ads.feature_engineering.feature_type.handler.feature_validator.WrongHandlerMethodSignature(handler_name: str, condition: str, handler_signature: str): Bases: ValueError

ads.feature_engineering.feature_type.handler.feature_warning module

The module that helps to register custom warnings for the feature types.

Classes

FeatureWarning
The Feature Warning class. Provides functionality to register warning handlers and invoke them.

Examples

>>> warning = FeatureWarning()
>>> def warning_handler_zeros_count(data):
...    return pd.DataFrame(
...        [['Zeros', 'Age has 38 zeros', 'Count', 38]],
...        columns=['Warning', 'Message', 'Metric', 'Value'])
>>> def warning_handler_zeros_percentage(data):
...    return pd.DataFrame(
...        [['Zeros', 'Age has 12.2% zeros', 'Percentage', '12.2%']],
...        columns=['Warning', 'Message', 'Metric', 'Value'])
>>> warning.register(name="zeros_count", handler=warning_handler_zeros_count)
>>> warning.register(name="zeros_percentage", handler=warning_handler_percentage)
>>> warning.registered()
                    Name                               Handler
    ----------------------------------------------------------
    0         zeros_count          warning_handler_zeros_count
    1    zeros_percentage     warning_handler_zeros_percentage

>>> warning.zeros_percentage(data_series)
             Warning               Message         Metric      Value
    ----------------------------------------------------------------
    0          Zeros      Age has 38 zeros          Count         38

>>> warning.zeros_count(data_series)
             Warning               Message         Metric      Value
    ----------------------------------------------------------------
    1          Zeros   Age has 12.2% zeros     Percentage      12.2%

>>> warning(data_series)
        Warning                    Message         Metric      Value
    ----------------------------------------------------------------
    0          Zeros      Age has 38 zeros          Count         38
    1          Zeros   Age has 12.2% zeros     Percentage      12.2%

>>> warning.unregister('zeros_count')
>>> warning(data_series)
             Warning               Message         Metric      Value
    ----------------------------------------------------------------
    0          Zeros   Age has 12.2% zeros     Percentage      12.2%

class ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning

Bases: object

The Feature Warning class.

Provides functionality to register warning handlers and invoke them.

register(self, name: str, handler: Callable) → None: Registers a new warning for the feature type.

unregister(self, name: str) → None: Unregisters warning.

registered(self) → pd.DataFrame: Gets the list of registered warnings.

Examples

>>> warning = FeatureWarning()
>>> def warning_handler_zeros_count(data):
...    return pd.DataFrame(
...        [['Zeros', 'Age has 38 zeros', 'Count', 38]],
...        columns=['Warning', 'Message', 'Metric', 'Value'])
>>> def warning_handler_zeros_percentage(data):
...    return pd.DataFrame(
...        [['Zeros', 'Age has 12.2% zeros', 'Percentage', '12.2%']],
...        columns=['Warning', 'Message', 'Metric', 'Value'])
>>> warning.register(name="zeros_count", handler=warning_handler_zeros_count)
>>> warning.register(name="zeros_percentage", handler=warning_handler_percentage)
>>> warning.registered()
                  Warning                              Handler
    ----------------------------------------------------------
    0         zeros_count          warning_handler_zeros_count
    1    zeros_percentage     warning_handler_zeros_percentage

>>> warning.zeros_percentage(data_series)
             Warning               Message         Metric      Value
    ----------------------------------------------------------------
    0          Zeros      Age has 38 zeros          Count         38

>>> warning.zeros_count(data_series)
              Warning              Message         Metric      Value
    ----------------------------------------------------------------
    1          Zeros   Age has 12.2% zeros     Percentage      12.2%

>>> warning(data_series)
             Warning               Message         Metric      Value
    ----------------------------------------------------------------
    0          Zeros      Age has 38 zeros          Count         38
    1          Zeros   Age has 12.2% zeros     Percentage      12.2%

>>> warning.unregister('zeros_count')
>>> warning(data_series)
             Warning               Message         Metric      Value
    ----------------------------------------------------------------
    0          Zeros   Age has 12.2% zeros     Percentage      12.2%

Initializes the FeatureWarning.

register(name: str, handler: Callable, replace: bool = False) → None

Registers a new warning.

Parameters:

name (str) – The warning name.
handler (callable) – The handler associated with the warning.
replace (bool) – The flag indicating if the registered warning should be replaced with the new one.

Returns:

Nothing

Return type:

None

Raises:

ValueError – If warning name is empty or handler not defined.
TypeError – If handler is not callable.
WarningAlreadyExists – If warning is already registered.

registered() → DataFrame

Gets the list of registered warnings.

Return type:: pd.DataFrame

Examples

>>>    The list of registerd warnings in DataFrame format.
                     Name                               Handler
    -----------------------------------------------------------
    0         zeros_count           warning_handler_zeros_count
    1    zeros_percentage      warning_handler_zeros_percentage

unregister(name: str) → None

Unregisters warning.

Parameters:

name (str) – The name of warning to be unregistered.

Returns:

Nothing.

Return type:

None

Raises:

ValueError – If warning name is not provided or empty.
WarningNotFound – If warning not found.

ads.feature_engineering.feature_type.handler.warnings module

The module with all default warnings provided to user. These are registered to relevant feature types directly in the feature type files themselves.

ads.feature_engineering.feature_type.handler.warnings.high_cardinality_handler(s: Series) → DataFrame

Warning if number of unique values (including Nan) in series is greater than or equal to 15.

Parameters:: s (pd.Series) – Pandas series - column of some feature type.
Returns:: Dataframe with 4 columns ‘Warning’, ‘Message’, ‘Metric’, ‘Value’ and 1 rows, which lists count of unique values.
Return type:: pd.Dataframe

ads.feature_engineering.feature_type.handler.warnings.missing_values_handler(s: Series) → DataFrame

Warning for > 5 percent missing values (Nans) in series.

Parameters:: s (pd.Series) – Pandas series - column of some feature type.
Returns:: Dataframe with 4 columns ‘Warning’, ‘Message’, ‘Metric’, ‘Value’ and 2 rows, where first row is count of missing values and second is percentage of missing values.
Return type:: pd.Dataframe

ads.feature_engineering.feature_type.handler.warnings.skew_handler(s: Series) → DataFrame

Warning if absolute value of skew is greater than 1.

Parameters:: s (pd.Series) – Pandas series - column of some feature type, expects continuous values.
Returns:: Dataframe with 4 columns ‘Warning’, ‘Message’, ‘Metric’, ‘Value’ and 1 rows, which lists skew value of that column.
Return type:: pd.Dataframe

ads.feature_engineering.feature_type.handler.warnings.zeros_handler(s: Series) → DataFrame

Warning for greater than 10 percent zeros in series.

Parameters:: s (pd.Series) – Pandas series - column of some feature type.
Returns:: Dataframe with 4 columns ‘Warning’, ‘Message’, ‘Metric’, ‘Value’ and 2 rows, where first row is count of zero values and second is percentage of zero values.
Return type:: pd.Dataframe

Module contents

Submodules

ads.feature_engineering.feature_type.address module

The module that represents an Address feature type.

Classes:

Address: The Address feature type.

class ads.feature_engineering.feature_type.address.Address

Bases: String

Type representing address.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows the location of given address on map base on zip code.

Example

>>> from ads.feature_engineering.feature_type.address import Address
>>> import pandas as pd
>>> address = pd.Series(['1 Miller Drive, New York, NY 12345',
                        '1 Berkeley Street, Boston, MA 67891',
                        '54305 Oxford Street, Seattle, WA 95132',
                        ''])
>>> Address.validator.is_address(address)
0     True
1     True
2     True
3    False
dtype: bool

description = 'Type representing address.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> address = pd.Series(['1 Miller Drive, New York, NY 12345',
              '1 Berkeley Street, Boston, MA 67891',
              '54305 Oxford Street, Seattle, WA 95132',
              ''],
           name='address')
>>> address.ads.feature_type = ['address']
>>> address.ads.feature_domain()
constraints: []
stats:
    count: 4
    missing: 1
    unique: 3
values: Address

Returns:: Domain based on the Address feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows the location of given address on map base on zip code.

Examples

>>> address = pd.Series(['1 Miller Drive, New York, NY 12345',
              '1 Berkeley Street, Boston, MA 67891',
              '54305 Oxford Street, Seattle, WA 95132',
              ''],
           name='address')
>>> address.ads.feature_type = ['address']
>>> address.ads.feature_plot()

Returns:: Plot object for the series based on the Address feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count).

Examples

>>> address = pd.Series(['1 Miller Drive, New York, NY 12345',
              '1 Berkeley Street, Boston, MA 67891',
              '54305 Oxford Street, Seattle, WA 95132',
              ''],
           name='address')
>>> address.ads.feature_type = ['address']
>>> address.ads.feature_stat()
    Metric  Value
0       count   4
1       unique  3
2       missing 1

Returns:: Summary statistics of the Series provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.address.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pd.Series) – The data to process.
Returns:: The logical list indicating if the data matches requirements.
Return type:: pandas.Series

ads.feature_engineering.feature_type.base module

class ads.feature_engineering.feature_type.base.FeatureBaseType(classname, bases, dictionary)

Bases: type

The helper metaclass to extend fucntionality of FeatureType class.

class ads.feature_engineering.feature_type.base.FeatureBaseTypeMeta(classname, bases, dictionary)

Bases: FeatureBaseType, ABCMeta

The class to provide compatibility between ABC and FeatureBaseType metaclass.

class ads.feature_engineering.feature_type.base.FeatureType

Bases: ABC

Abstract case for feature types. Default class attribute include name and description. Name is auto generated using camel to snake conversion unless specified.

description = 'Base feature type.'

name = 'feature_type'

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

class ads.feature_engineering.feature_type.base.Name: Bases: object

class ads.feature_engineering.feature_type.base.Tag(name: str)

Bases: object

Class for free form tags. Name must be specified.

Initialize a tag instance.

Parameters:: name (str) – The name of the tag.

ads.feature_engineering.feature_type.boolean module

The module that represents a Boolean feature type.

Classes:

Boolean: The feature type that represents binary values True/False.

Functions:

default_handler(data: pd.Series) -> pd.Series: Processes given data and indicates if the data matches requirements.

class ads.feature_engineering.feature_type.boolean.Boolean

Bases: FeatureType

Type representing binary values True/False.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Show the counts of observations in True/False using bars.

Examples

>>> from ads.feature_engineering.feature_type.boolean import Boolean
>>> import pandas as pd
>>> import numpy as np
>>> s = pd.Series([True, False, True, False, np.NaN, None], name='bool')
>>> s.ads.feature_type = ['boolean']
>>> Boolean.validator.is_boolean(s)
0     True
1     True
2     True
3     True
4    False
5    False
dtype: bool

description = 'Type representing binary values True/False.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> s = pd.Series([True, False, True, False, np.NaN, None], name='bool')
>>> s.ads.feature_type = ['boolean']
>>> s.ads.feature_domain()
constraints:
- expression: $x in [True, False]
    language: python
stats:
    count: 6
    missing: 2
    unique: 2
values: Boolean

Returns:: Domain based on the Boolean feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows the counts of observations in True/False using bars.

Parameters:: x (pandas.Series) – The feature being evaluated.
Returns:: Plot object for the series based on the Boolean feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

Examples

>>> s = pd.Series([True, False, True, False, np.NaN, None], name='bool')
>>> s.ads.feature_type = ['boolean']
>>> s.ads.feature_plot()

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count).

Parameters:: x (pandas.Series) – The feature being evaluated.
Returns:: Summary statistics of the Series or Dataframe provided.
Return type:: pandas.DataFrame

Examples

>>> s = pd.Series([True, False, True, False, np.NaN, None], name='bool')
>>> s.ads.feature_type = ['boolean']
>>> s.ads.feature_stat()
    Metric  Value
0       count   6
1       unique  2
2       missing 2

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.boolean.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pandas.Series) – The data to process.
Returns:: The logical list indicating if the data matches requirements.
Return type:: pandas.Series

ads.feature_engineering.feature_type.category module

The module that represents a Category feature type.

Classes:

Category: The Category feature type.

class ads.feature_engineering.feature_type.category.Category

Bases: FeatureType

Type representing discrete unordered values.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows the counts of observations in each categorical bin using bar chart.

description = 'Type representing discrete unordered values.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> cat = pd.Series(['S', 'C', 'S', 'S', 'S', 'Q', 'S', 'S', 'S', 'C', 'S', 'S', 'S',
            'S', 'S', 'S', 'Q', 'S', 'S', '', np.NaN, None], name='category')
>>> cat.ads.feature_type = ['category']
>>> cat.ads.feature_domain()
constraints:
- expression: $x in ['S', 'C', 'Q', '']
    language: python
stats:
    count: 22
    missing: 3
    unique: 3
values: Category

Returns:: Domain based on the Category feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows the counts of observations in each categorical bin using bar chart.

Parameters:: x (pandas.Series) – The feature being evaluated.
Returns:: Plot object for the series based on the Category feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

Examples

>>> cat = pd.Series(['S', 'C', 'S', 'S', 'S', 'Q', 'S', 'S', 'S', 'C', 'S', 'S', 'S',
            'S', 'S', 'S', 'Q', 'S', 'S', '', np.NaN, None], name='сategory')
>>> cat.ads.feature_type = ['сategory']
>>> cat.ads.feature_plot()

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count) if there are any.

Parameters:: x (pandas.Series) – The feature being evaluated.
Returns:: Summary statistics of the Series or Dataframe provided.
Return type:: pandas.DataFrame

Examples

>>> cat = pd.Series(['S', 'C', 'S', 'S', 'S', 'Q', 'S', 'S', 'S', 'C', 'S', 'S', 'S',
            'S', 'S', 'S', 'Q', 'S', 'S', '', np.NaN, None], name='сategory')
>>> cat.ads.feature_type = ['сategory']
>>> cat.ads.feature_stat()
    Metric  Value
0       count   22
1       unique  3
2       missing 3

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.constant module

The module that represents a Constant feature type.

Classes:

Constant: The Constant feature type.

class ads.feature_engineering.feature_type.constant.Constant

Bases: FeatureType

Type representing constant values.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows the counts of observations in bars.

description = 'Type representing constant values.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type. .. rubric:: Example

>>> s = pd.Series([1, 1, 1, 1, 1], name='constant')
>>> s.ads.feature_type = ['constant']
>>> s.ads.feature_domain()
constraints: []
stats:
    count: 5
    unique: 1
values: Constant

Returns:: Domain based on the Constant feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows the counts of observations in bars.

Parameters:: x (pandas.Series) – The feature being shown.

Examples

>>> s = pd.Series([1, 1, 1, 1, 1], name='constant')
>>> s.ads.feature_type = ['constant']
>>> s.ads.feature_plot()

Returns:: Plot object for the series based on the Constant feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count).

Parameters:: x (pandas.Series) – The feature being evaluated.
Returns:: Summary statistics of the Series provided.
Return type:: pandas.DataFrame

Examples

>>> s = pd.Series([1, 1, 1, 1, 1], name='constant')
>>> s.ads.feature_type = ['constant']
>>> s.ads.feature_stat()
    Metric  Value
0       count   5
1       unique  1

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.continuous module

The module that represents a Continuous feature type.

Classes:

Continuous: The Continuous feature type.

class ads.feature_engineering.feature_type.continuous.Continuous

Bases: FeatureType

Type representing continuous values.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows distributions of datasets using box plot.

description = 'Type representing continuous values.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> cts = pd.Series([13.32, 3.32, 4.3, 2.45, 6.34, 2.25,
                    4.43, 3.26, np.NaN, None], name='continuous')
>>> cts.ads.feature_type = ['continuous']
>>> cts.ads.feature_domain()
constraints: []
stats:
    count: 10.0
    lower quartile: 3.058
    mean: 4.959
    median: 3.81
    missing: 2.0
    sample maximum: 13.32
    sample minimum: 2.25
    skew: 2.175
    standard deviation: 3.62
    upper quartile: 4.908
values: Continuous

Returns:: Domain based on the Continuous feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows distributions of datasets using box plot.

Examples

>>> cts = pd.Series([13.32, 3.32, 4.3, 2.45, 6.34, 2.25,
                    4.43, 3.26, np.NaN, None], name='continuous')
>>> cts.ads.feature_type = ['continuous']
>>> cts.ads.feture_plot()

Returns:: Plot object for the series based on the Continuous feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, mean, standard deviation, sample minimum, lower quartile, median, 75%, upper quartile, skew and missing(count).

Examples

>>> cts = pd.Series([13.32, 3.32, 4.3, 2.45, 6.34, 2.25,
                    4.43, 3.26, np.NaN, None], name='continuous')
>>> cts.ads.feature_type = ['continuous']
>>> cts.ads.feature_stat()
    Metric                  Value
0       count                   10.000
1       mean                    4.959
2       standard deviation          3.620
3       sample minimum          2.250
4       lower quartile          3.058
5       median                  3.810
6       upper quartile          4.908
7       sample maximum          13.320
8       skew                    2.175
9       missing                 2.000

Returns:: Summary statistics of the Series or Dataframe provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.creditcard module

The module that represents a CreditCard feature type.

Classes:

CreditCard: The CreditCard feature type.

Functions:

default_handler(data: pd.Series) -> pd.Series: Processes given data and indicates if the data matches requirements.
_luhn_checksum(card_number: str) -> float: Implements Luhn algorithm to validate a credit card number.

class ads.feature_engineering.feature_type.creditcard.CreditCard

Bases: String

Type representing credit card numbers.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows the counts of observations in each credit card type using bar chart.

Examples

>>> from ads.feature_engineering.feature_type.creditcard import CreditCard
>>> import pandas as pd
>>> s = pd.Series(["4532640527811543", None, "4556929308150929", "4539944650919740", "4485348152450846", "4556593717607190"], name='credit_card')
>>> s.ads.feature_type = ['credit_card']
>>> CreditCard.validator.is_credit_card(s)
0     True
1    False
2     True
3     True
4     True
5     True
Name: credit_card, dtype: bool

description = 'Type representing credit card numbers.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> visa = [
    "4532640527811543",
    None,
    "4556929308150929",
    "4539944650919740",
    "4485348152450846",
    "4556593717607190",
    ]
>>> mastercard = [
    "5334180299390324",
    "5111466404826446",
    "5273114895302717",
    "5430972152222336",
    "5536426859893306",
    ]
>>> amex = [
    "371025944923273",
    "374745112042294",
    "340984902710890",
    "375767928645325",
    "370720852891659",
    ]
>>> creditcard_list = visa + mastercard + amex
>>> creditcard_series = pd.Series(creditcard_list,name='card')
>>> creditcard_series.ads.feature_type = ['credit_card']
>>> creditcard_series.ads.feature_domain()
constraints: []
stats:
    count: 16
    count_Amex: 5
    count_Diners Club: 2
    count_MasterCard: 3
    count_Visa: 5
    count_missing: 1
    missing: 1
    unique: 15
values: CreditCard

Returns:: Domain based on the CreditCard feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows the counts of observations in each credit card type using bar chart.

Examples

>>> visa = [
    "4532640527811543",
    None,
    "4556929308150929",
    "4539944650919740",
    "4485348152450846",
    "4556593717607190",
    ]
>>> mastercard = [
    "5334180299390324",
    "5111466404826446",
    "5273114895302717",
    "5430972152222336",
    "5536426859893306",
    ]
>>> amex = [
    "371025944923273",
    "374745112042294",
    "340984902710890",
    "375767928645325",
    "370720852891659",
    ]
>>> creditcard_list = visa + mastercard + amex
>>> creditcard_series = pd.Series(creditcard_list,name='card')
>>> creditcard_series.ads.feature_type = ['credit_card']
>>> creditcard_series.ads.feature_plot()

Returns:: Plot object for the series based on the CreditCard feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series)

Generates feature statistics.

Feature statistics include (total)count, unique(count), missing(count) and: count of each credit card type.

Examples

>>> visa = [
    "4532640527811543",
    None,
    "4556929308150929",
    "4539944650919740",
    "4485348152450846",
    "4556593717607190",
    ]
>>> mastercard = [
    "5334180299390324",
    "5111466404826446",
    "5273114895302717",
    "5430972152222336",
    "5536426859893306",
    ]
>>> amex = [
    "371025944923273",
    "374745112042294",
    "340984902710890",
    "375767928645325",
    "370720852891659",
    ]
>>> creditcard_list = visa + mastercard + amex
>>> creditcard_series = pd.Series(creditcard_list,name='card')
>>> creditcard_series.ads.feature_type = ['credit_card']
>>> creditcard_series.ads.feature_stat()
    Metric              Value
0       count               16
1       unique              15
2       missing             1
3       count_Amex              5
4       count_Visa              5
5       count_MasterCard        3
6       count_Diners Club       2
7       count_missing       1

Returns:: Summary statistics of the Series or Dataframe provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.creditcard.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pandas.Series) – The data to process.
Returns:: The logical list indicating if the data matches requirements.
Return type:: pandas.Series

ads.feature_engineering.feature_type.datetime module

The module that represents a DateTime feature type.

Classes:

DateTime: The DateTime feature type.

class ads.feature_engineering.feature_type.datetime.DateTime

Bases: FeatureType

Type representing date and/or time.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows distributions of datetime datasets using histograms.

Example

>>> from ads.feature_engineering.feature_type.datetime import DateTime
>>> import pandas as pd
>>> s = pd.Series(["12/12/12", "12/12/13", None, "12/12/14"], name='datetime')
>>> s.ads.feature_type = ['date_time']
>>> DateTime.validator.is_datetime(s)
0     True
1     True
2    False
3     True
Name: datetime, dtype: bool

description = 'Type representing date and/or time.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> s = pd.Series(['3/11/2000', '3/12/2000', '3/13/2000', '', None, np.nan, 'April/13/2011', 'April/15/11'], name='datetime')
>>> s.ads.feature_type = ['date_time']
>>> s.ads.feature_domain()
constraints: []
stats:
    count: 8
    missing: 3
    sample maximum: April/15/11
    sample minimum: 3/11/2000
values: DateTime

Returns:: Domain based on the DateTime feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows distributions of datetime datasets using histograms.

Examples

>>> x = pd.Series(['3/11/2000', '3/12/2000', '3/13/2000', '', None, np.nan, 'April/13/2011', 'April/15/11'], name='datetime')
>>> x.ads.feature_type = ['date_time']
>>> x.ads.feature_plot()

Returns:: Plot object for the series based on the DateTime feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, sample maximum, sample minimum, and missing(count) if there is any.

Examples

>>> x = pd.Series(['3/11/2000', '3/12/2000', '3/13/2000', '', None, np.nan, 'April/13/2011', 'April/15/11'], name='datetime')
>>> x.ads.feature_type = ['date_time']
>>> x.ads.feature_stat()
    Metric              Value
0       count               8
1       sample maximum      April/15/11
2       sample minimum      3/11/2000
3       missing             3

Returns:: Summary statistics of the Series or Dataframe provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.datetime.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pandas.Series) – The data to process.
Returns:: The logical list indicating if the data matches requirements.
Return type:: pandas.Series

ads.feature_engineering.feature_type.discrete module

The module that represents a Discrete feature type.

Classes:

Discrete: The Discrete feature type.

class ads.feature_engineering.feature_type.discrete.Discrete

Bases: FeatureType

Type representing discrete values.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows distributions of datasets using box plot.

description = 'Type representing discrete values.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> discrete_numbers = pd.Series([35, 25, 13, 42],
           name='discrete')
>>> discrete_numbers.ads.feature_type = ['discrete']
>>> discrete_numbers.ads.feature_domain()
constraints: []
stats:
    count: 4
    unique: 4
values: Discrete

Returns:: Domain based on the Discrete feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows distributions of datasets using box plot.

Examples

>>> discrete_numbers = pd.Series([35, 25, 13, 42],
           name='discrete')
>>> discrete_numbers.ads.feature_type = ['discrete']
>>> discrete_numbers.ads.feature_stat()
    Metric  Value
0       count   4
1       unique  4

Returns:: Plot object for the series based on the Discrete feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count).

Examples

>>> discrete_numbers = pd.Series([35, 25, 13, 42],
           name='discrete')
>>> discrete_numbers.ads.feature_type = ['discrete']
>>> discrete_numbers.ads.feature_stat()
            discrete
count   4
unique  4

Returns:: Summary statistics of the Series provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.document module

The module that represents a Document feature type.

Classes:

Document: The Document feature type.

class ads.feature_engineering.feature_type.document.Document

Bases: FeatureType

Type representing document values.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

description = 'Type representing document values.'

classmethod feature_domain()

Returns:: Nothing.
Return type:: None

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.gis module

The module that represents a GIS feature type.

Classes:

GIS: The GIS feature type.

class ads.feature_engineering.feature_type.gis.GIS

Bases: FeatureType

Type representing geographic information.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows the location of given address on map base on longitude and latitute.

Example

>>> from ads.feature_engineering.feature_type.gis import GIS
>>> import pandas as pd
>>> s = pd.Series(["-18.2193965, -93.587285",
                    "-21.0255305, -122.478584",
                    "85.103913, 19.405744",
                    "82.913736, 178.225672",
                    "62.9795085,-66.989705",
                    "54.5604395,95.235090",
                    "24.2811855,-162.380403",
                    "-1.818319,-80.681214",
                    None,
                    "(51.816119, 175.979008)",
                    "(54.3392995,-11.801615)"],
                    name='gis')
>>> s.ads.feature_type = ['gis']
>>> GIS.validator.is_gis(s)
0      True
1      True
2      True
3      True
4      True
5      True
6      True
7      True
8     False
9      True
10     True
Name: gis, dtype: bool

description = 'Type representing geographic information.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> gis = pd.Series([
    "69.196241,-125.017615",
    "5.2272595,-143.465712",
    "-33.9855425,-153.445155",
    "43.340610,86.460554",
    "24.2811855,-162.380403",
    "2.7849025,-7.328156",
    "45.033805,157.490179",
    "-1.818319,-80.681214",
    "-44.510428,-169.269477",
    "-56.3344375,-166.407038",
    "",
    np.NaN,
    None
    ],
    name='gis'
)
>>> gis.ads.feature_type = ['gis']
>>> gis.ads.feature_domain()
constraints: []
stats:
    count: 13
    missing: 3
    unique: 10
values: GIS

Returns:: Domain based on the GIS feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows the location of given address on map base on longitude and latitute.

Examples

>>> gis = pd.Series([
    "69.196241,-125.017615",
    "5.2272595,-143.465712",
    "-33.9855425,-153.445155",
    "43.340610,86.460554",
    "24.2811855,-162.380403",
    "2.7849025,-7.328156",
    "45.033805,157.490179",
    "-1.818319,-80.681214",
    "-44.510428,-169.269477",
    "-56.3344375,-166.407038",
    "",
    np.NaN,
    None
    ],
    name='gis'
)
>>> gis.ads.feature_type = ['gis']
>>> gis.ads.feature_plot()

Returns:: Plot object for the series based on the GIS feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count).

Examples

>>> gis = pd.Series([
    "69.196241,-125.017615",
    "5.2272595,-143.465712",
    "-33.9855425,-153.445155",
    "43.340610,86.460554",
    "24.2811855,-162.380403",
    "2.7849025,-7.328156",
    "45.033805,157.490179",
    "-1.818319,-80.681214",
    "-44.510428,-169.269477",
    "-56.3344375,-166.407038",
    "",
    np.NaN,
    None
    ],
    name='gis'
)
>>> gis.ads.feature_type = ['gis']
>>> gis.ads.feature_stat()
        gis
count   13
unique  10
missing 3

Returns:: Summary statistics of the Series provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.gis.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pandas.Series) – The data to process.
Returns:: The logical list indicating if the data matches requirements.
Return type:: pandas.Series

ads.feature_engineering.feature_type.integer module

The module that represents an Integer feature type.

Classes:

Integer: The Integer feature type.

class ads.feature_engineering.feature_type.integer.Integer

Bases: FeatureType

Type representing integer values.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows distributions of datasets using box plot.

description = 'Type representing integer values.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> s = pd.Series([True, False, True, False, np.NaN, None], name='integer')
>>> s.ads.feature_type = ['integer']
>>> s.ads.feature_domain()
constraints: []
stats:
    count: 6
    freq: 2
    missing: 2
    top: true
    unique: 2
values: Integer

Returns:: Domain based on the Integer feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows distributions of datasets using box plot.

Examples

>>> x = pd.Series([1, 0, 1, 2, 3, 4, np.nan], name='integer')
>>> x.ads.feature_type = ['integer']
>>> x.ads.feature_plot()

Returns:: Plot object for the series based on the Integer feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, mean, standard deviation, sample minimum, lower quartile, median, 75%, upper quartile, max and missing(count) if there is any.

Examples

>>> x = pd.Series([1, 0, 1, 2, 3, 4, np.nan], name='integer')
>>> x.ads.feature_type = ['integer']
>>> x.ads.feature_stat()
    Metric                  Value
0       count                   7
1       mean                    1
2       standard deviation          1
3       sample minimum          0
4       lower quartile          1
5       median                  1
6       upper quartile          2
7       sample maximum          4
8       missing                 1

Returns:: Summary statistics of the Series or Dataframe provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.ip_address module

The module that represents an IpAddress feature type.

Classes:

IpAddress: The IpAddress feature type.

class ads.feature_engineering.feature_type.ip_address.IpAddress

Bases: FeatureType

Type representing IP Address.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

Example

>>> from ads.feature_engineering.feature_type.ip_address import IpAddress
>>> import pandas as pd
>>> import numpy as np
>>> s = pd.Series(['192.168.0.1', '2001:db8::', '', np.NaN, None], name='ip_address')
>>> s.ads.feature_type = ['ip_address']
>>> IpAddress.validator.is_ip_address(s)
0     True
1     True
2    False
3    False
4    False
Name: ip_address, dtype: bool

description = 'Type representing IP Address.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> s = pd.Series(['2002:db8::', '192.168.0.1', '2001:db8::', '2002:db8::', np.NaN, None], name='ip_address')
>>> s.ads.feature_type = ['ip_address']
>>> s.ads.feature_domain()
constraints: []
stats:
    count: 6
    missing: 2
    unique: 3
values: IpAddress

Returns:: Domain based on the IpAddress feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count).

Examples

>>> s = pd.Series(['2002:db8::', '192.168.0.1', '2001:db8::', '2002:db8::', np.NaN, None], name='ip_address')
>>> s.ads.feature_type = ['ip_address']
>>> s.ads.feature_stat()
    Metric  Value
0       count   6
1       unique  2
2       missing 2

Returns:: Summary statistics of the Series provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.ip_address.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pandas.Series) – The data to process.
Returns:: The logical list indicating if the data matches requirements.
Return type:: pandas.Series

ads.feature_engineering.feature_type.ip_address_v4 module

The module that represents an IpAddressV4 feature type.

Classes:

IpAddressV4: The IpAddressV4 feature type.

class ads.feature_engineering.feature_type.ip_address_v4.IpAddressV4

Bases: FeatureType

Type representing IP Address V4.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

Example

>>> from ads.feature_engineering.feature_type.ip_address_v4 import IpAddressV4
>>> import pandas as pd
>>> import numpy as np
>>> s = pd.Series(['192.168.0.1', '2001:db8::', '', np.NaN, None], name='ip_address')
>>> s.ads.feature_type = ['ip_address_v4']
>>> IpAddressV4.validator.is_ip_address_v4(s)
0     True
1    False
2    False
3    False
4    False
Name: ip_address, dtype: bool

description = 'Type representing IP Address V4.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> s = pd.Series(['192.168.0.1', '192.168.0.2', '192.168.0.3', '192.168.0.4', np.NaN, None], name='ip_address_v4')
>>> s.ads.feature_type = ['ip_address_v4']
>>> s.ads.feature_domain()
constraints: []
stats:
    count: 6
    missing: 2
    unique: 4
values: IpAddressV4

Returns:: Domain based on the IpAddressV4 feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count).

Examples

>>> s = pd.Series(['192.168.0.1', '192.168.0.2', '192.168.0.3', '192.168.0.4', np.NaN, None], name='ip_address')
>>> s.ads.feature_type = ['ip_address_v4']
>>> s.ads.feature_stat()
    Metric  Value
0       count   6
1       unique  4
2       missing 2

Returns:: Summary statistics of the Series provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.ip_address_v4.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pandas.Series) – The data to process.
Returns:: The logical list indicating if the data matches requirements.
Return type:: pandas.Series

ads.feature_engineering.feature_type.ip_address_v6 module

The module that represents an IpAddressV6 feature type.

Classes:

IpAddressV6: The IpAddressV6 feature type.

class ads.feature_engineering.feature_type.ip_address_v6.IpAddressV6

Bases: FeatureType

Type representing IP Address V6.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

Example

>>> from ads.feature_engineering.feature_type.ip_address_v6 import IpAddressV6
>>> import pandas as pd
>>> import numpy as np
>>> s = pd.Series(['192.168.0.1', '2001:db8::', '', np.NaN, None], name='ip_address')
>>> s.ads.feature_type = ['ip_address_v6']
>>> IpAddressV6.validator.is_ip_address_v6(s)
0    False
1     True
2    False
3    False
4    False
Name: ip_address, dtype: bool

description = 'Type representing IP Address V6.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> s = pd.Series(['2002:db8::', '2001:db8::', '2001:db8::', '2002:db8::', np.NaN, None], name='ip_address_v6')
>>> s.ads.feature_type = ['ip_address_v6']
>>> s.ads.feature_domain()
constraints: []
stats:
    count: 6
    missing: 2
    unique: 2
values: IpAddressV6

Returns:: Domain based on the IpAddressV6 feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count).

Examples

>>> s = pd.Series(['2002:db8::', '2001:db8::', '2001:db8::', '2002:db8::', np.NaN, None], name='ip_address')
>>> s.ads.feature_type = ['ip_address_v6']
>>> s.ads.feature_stat()
    Metric  Value
0       count   6
1       unique  2
2       missing 2

Returns:: Summary statistics of the Series provided.
Return type:: Pandas Dataframe

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.ip_address_v6.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pandas.Series) – The data to process.
Returns:: The logical list indicating if the data matches requirements.
Return type:: pandas.Series

ads.feature_engineering.feature_type.lat_long module

The module that represents a LatLong feature type.

Classes:

LatLong: The LatLong feature type.

Functions:

default_handler(data: pd.Series) -> pd.Series: Processes given data and indicates if the data matches requirements.

class ads.feature_engineering.feature_type.lat_long.LatLong

Bases: String

Type representing longitude and latitute.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows the location of given address on map base on longitude and latitute.

Example

>>> from ads.feature_engineering.feature_type.lat_long import LatLong
>>> import pandas as pd
>>> s = pd.Series(["-18.2193965, -93.587285",
                    "-21.0255305, -122.478584",
                    "85.103913, 19.405744",
                    "82.913736, 178.225672",
                    "62.9795085,-66.989705",
                    "54.5604395,95.235090",
                    "24.2811855,-162.380403",
                    "-1.818319,-80.681214",
                    None,
                    "(51.816119, 175.979008)",
                    "(54.3392995,-11.801615)"],
                    name='latlong')
>>> s.ads.feature_type = ['lat_long']
>>> LatLong.validator.is_lat_long(s)
0      True
1      True
2      True
3      True
4      True
5      True
6      True
7      True
8     False
9      True
10     True
Name: latlong, dtype: bool

description = 'Type representing longitude and latitute.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> latlong_series = pd.Series([
    "69.196241,-125.017615",
    "5.2272595,-143.465712",
    "-33.9855425,-153.445155",
    "43.340610,86.460554",
    "24.2811855,-162.380403",
    "2.7849025,-7.328156",
    "45.033805,157.490179",
    "-1.818319,-80.681214",
    "-44.510428,-169.269477",
    "-56.3344375,-166.407038",
    "",
    np.NaN,
    None
    ],
    name='latlong'
)
>>> latlong_series.ads.feature_type = ['lat_long']
>>> latlong_series.ads.feature_domain()
constraints: []
stats:
    count: 13
    missing: 3
    unique: 10
values: LatLong

Returns:: Domain based on the LatLong feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows the location of given address on map base on longitude and latitute.

Examples

>>> latlong_series = pd.Series([
            "69.196241,-125.017615",
            "5.2272595,-143.465712",
            "-33.9855425,-153.445155",
            "43.340610,86.460554",
            "24.2811855,-162.380403",
            "2.7849025,-7.328156",
            "45.033805,157.490179",
            "-1.818319,-80.681214",
            "-44.510428,-169.269477",
            "-56.3344375,-166.407038",
            "",
            np.NaN,
            None
        ],
    name='latlong'
)
>>> latlong_series.ads.feature_type = ['lat_long']
>>> latlong_series.ads.feature_plot()

Returns:: Plot object for the series based on the LatLong feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series) → DataFrame

Generate feature statistics.

Feature statistics include (total)count, unique(count) and missing(count) if there is any.

Examples

>>> latlong_series = pd.Series([
            "69.196241,-125.017615",
            "5.2272595,-143.465712",
            "-33.9855425,-153.445155",
            "43.340610,86.460554",
            "24.2811855,-162.380403",
            "2.7849025,-7.328156",
            "45.033805,157.490179",
            "-1.818319,-80.681214",
            "-44.510428,-169.269477",
            "-56.3344375,-166.407038",
            "",
            np.NaN,
            None
        ],
    name='latlong'
)
>>> latlong_series.ads.feature_type = ['lat_long']
>>> latlong_series.ads.feature_stat()
    Metric  Value
0       count   13
1       unique  10
2       missing 3

Returns:: Summary statistics of the Series or Dataframe provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.lat_long.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pandas.Series) – The data to process.
Returns:: The logical list indicating if the data matches requirements.
Return type:: pandas.Series

ads.feature_engineering.feature_type.object module

The module that represents an Object feature type.

Classes:

Object: The Object feature type.

class ads.feature_engineering.feature_type.object.Object

Bases: FeatureType

Type representing object.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

description = 'Type representing object.'

classmethod feature_domain()

Returns:: Nothing.
Return type:: None

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.ordinal module

The module that represents an Ordinal feature type.

Classes:

Ordinal: The Ordinal feature type.

class ads.feature_engineering.feature_type.ordinal.Ordinal

Bases: FeatureType

Type representing ordered values.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows the counts of observations in each categorical bin using bar chart.

description = 'Type representing ordered values.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> x = pd.Series([1, 2, 3, 4, 5, 6, 7, 8, 9, np.nan], name='ordinal')
>>> x.ads.feature_type = ['ordinal']
>>> x.ads.feature_domain()
constraints:
- expression: $x in [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
    language: python
stats:
    count: 10
    missing: 1
    unique: 9
values: Ordinal

Returns:: Domain based on the Ordinal feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows the counts of observations in each categorical bin using bar chart.

Examples

>>> x = pd.Series([1, 2, 3, 4, 5, 6, 7, 8, 9, np.nan], name='ordinal')
>>> x.ads.feature_type = ['ordinal']
>>> x.ads.feature_plot()

Returns:: The bart chart plot object for the series based on the Continuous feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count), and missing(count) if there is any.

Examples

>>> x = pd.Series([1, 2, 3, 4, 5, 6, 7, 8, 9, np.nan], name='ordinal')
>>> x.ads.feature_type = ['ordinal']
>>> x.ads.feature_stat()
    Metric  Value
0       count   10
1       unique  9
2       missing 1

Returns:: Summary statistics of the Series or Dataframe provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.phone_number module

The module that represents a Phone Number feature type.

Classes:

PhoneNumber: The Phone Number feature type.

Functions:

default_handler(data: pd.Series) -> pd.Series: Processes given data and indicates if the data matches requirements.

class ads.feature_engineering.feature_type.phone_number.PhoneNumber

Bases: String

Type representing phone numbers.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

Examples

>>> from ads.feature_engineering.feature_type.phone_number import PhoneNumber
>>> import pandas as pd
>>> s = pd.Series([None, "1-640-124-5367", "1-573-916-4412"])
>>> PhoneNumber.validator.is_phone_number(s)
0    False
1     True
2     True
dtype: bool

description = 'Type representing phone numbers.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> s = pd.Series(['2068866666', '6508866666', '2068866666', '', np.NaN, np.nan, None], name='phone')
>>> s.ads.feature_type = ['phone_number']
>>> s.ads.feature_domain()
constraints: []
stats:
    count: 7
    missing: 4
    unique: 2
values: PhoneNumber

Returns:: Domain based on the PhoneNumber feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count) if there is any.

Examples

>>> s = pd.Series(['2068866666', '6508866666', '2068866666', '', np.NaN, np.nan, None], name='phone')
>>> s.ads.feature_type = ['phone_number']
>>> s.ads.feature_stat()
    Metric  Value
1       count   7
2       unique  2
3       missing 4

Returns:: Summary statistics of the Series or Dataframe provided.
Return type:: pandas.DataFrame

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.phone_number.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pandas.Series) – The data to process.
Returns:: The logical list indicating if the data matches requirements.
Return type:: pandas.Series

ads.feature_engineering.feature_type.string module

The module that represents a String feature type.

Classes:

String: The feature type that represents string values.

class ads.feature_engineering.feature_type.string.String

Bases: FeatureType

Type representing string values.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows distributions of datasets using wordcloud.

Example

>>> from ads.feature_engineering.feature_type.string import String
>>> import pandas as pd
>>> s = pd.Series(["Hello", "world", None], name='string')
>>> String.validator.is_string(s)
0     True
1     True
2    False
Name: string, dtype: bool

description = 'Type representing string values.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> string = pd.Series(['S', 'C', 'S', 'S', 'S', 'Q', 'S', 'S', 'S', 'C', 'S', 'S', 'S',
        'S', 'S', 'S', 'Q', 'S', 'S', '', np.NaN, None], name='string')
>>> string.ads.feature_type = ['string']
>>> string.ads.feature_domain()
constraints: []
stats:
    count: 22
    missing: 3
    unique: 3
values: String

Returns:: Domain based on the String feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows distributions of datasets using wordcloud.

Examples

>>> string = pd.Series(['S', 'C', 'S', 'S', 'S', 'Q', 'S', 'S', 'S', 'C', 'S', 'S', 'S',
        'S', 'S', 'S', 'Q', 'S', 'S', '', np.NaN, None], name='string')
>>> string.ads.feature_type = ['string']
>>> string.ads.feature_plot()

Returns:: Plot object for the series based on the String feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count) if there is any.

Examples

>>> string = pd.Series(['S', 'C', 'S', 'S', 'S', 'Q', 'S', 'S', 'S', 'C', 'S', 'S', 'S',
        'S', 'S', 'S', 'Q', 'S', 'S', '', np.NaN, None], name='string')
>>> string.ads.feature_type = ['string']
>>> string.ads.feature_stat()
    Metric  Value
0       count   22
1       unique  3
2       missing 3

Returns:: Summary statistics of the Series or Dataframe provided.
Return type:: Pandas Dataframe

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.string.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pd.Series) – The data to process.
Returns:: pd.Series
Return type:: The logical list indicating if the data matches requirements.

ads.feature_engineering.feature_type.text module

The module that represents a Text feature type.

Classes:

Text: The Text feature type.

class ads.feature_engineering.feature_type.text.Text

Bases: String

Type representing text values.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_plot(x: pd.Series) → plt.Axes: Shows distributions of datasets using wordcloud.

description = 'Type representing text values.'

classmethod feature_domain()

Returns:: Nothing.
Return type:: None

static feature_plot(x: Series) → Axes

Shows distributions of datasets using wordcloud.

Examples

>>> text = pd.Series(['S', 'C', 'S', 'S', 'S', 'Q', 'S', 'S', 'S', 'C', 'S', 'S', 'S',
        'S', 'S', 'S', 'Q', 'S', 'S', '', np.NaN, None], name='text')
>>> text.ads.feature_type = ['text']
>>> text.ads.feature_plot()

Returns:: Plot object for the series based on the Text feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.unknown module

The module that represents an Unknown feature type.

Classes:

Text: The Unknown feature type.

class ads.feature_engineering.feature_type.unknown.Unknown

Bases: FeatureType

Type representing third-party dtypes.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

description = 'Type representing unknown type.'

classmethod feature_domain()

Returns:: Nothing.
Return type:: None

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.zip_code module

The module that represents a ZipCode feature type.

Classes:

ZipCode: The ZipCode feature type.

Functions:

default_handler(data: pd.Series) -> pd.Series: Processes given data and indicates if the data matches requirements.

class ads.feature_engineering.feature_type.zip_code.ZipCode

Bases: String

Type representing postal code.

description

The feature type description.

Type:: str

name

The feature type name.

Type:: str

warning

Provides functionality to register warnings and invoke them.

Type:: FeatureWarning

validator: Provides functionality to register validators and invoke them.

feature_stat(x: pd.Series) → pd.DataFrame: Generates feature statistics.

feature_plot(x: pd.Series) → plt.Axes: Shows the geometry distribution base on location of zipcode.

Example

>>> from ads.feature_engineering.feature_type.zip_code import ZipCode
>>> import pandas as pd
>>> import numpy as np
>>> s = pd.Series(["94065", "90210", np.NaN, None], name='zipcode')
>>> ZipCode.validator.is_zip_code(s)
0     True
1     True
2    False
3    False
Name: zipcode, dtype: bool

description = 'Type representing postal code.'

classmethod feature_domain(x: Series) → Domain

Generate the domain of the data of this feature type.

Examples

>>> zipcode = pd.Series([94065, 90210, np.NaN, None], name='zipcode')
>>> zipcode.ads.feature_type = ['zip_code']
>>> zipcode.ads.feature_domain()
constraints: []
stats:
    count: 4
    missing: 2
    unique: 2
values: ZipCode

Returns:: Domain based on the ZipCode feature type.
Return type:: ads.feature_engineering.schema.Domain

static feature_plot(x: Series) → Axes

Shows the geometry distribution base on location of zipcode.

Examples

>>> zipcode = pd.Series([94065, 90210, np.NaN, None], name='zipcode')
>>> zipcode.ads.feature_type = ['zip_code']
>>> zipcode.ads.feature_plot()

Returns:: Plot object for the series based on the ZipCode feature type.
Return type:: matplotlib.axes._subplots.AxesSubplot

static feature_stat(x: Series) → DataFrame

Generates feature statistics.

Feature statistics include (total)count, unique(count) and missing(count).

Examples

>>> zipcode = pd.Series([94065, 90210, np.NaN, None], name='zipcode')
>>> zipcode.ads.feature_type = ['zip_code']
>>> zipcode.ads.feature_stat()
    Metric  Value
0       count   4
1       unique  2
2       missing 2

Returns:: Summary statistics of the Series provided.
Return type:: Pandas Dataframe

validator = <ads.feature_engineering.feature_type.handler.feature_validator.FeatureValidator object>

warning = <ads.feature_engineering.feature_type.handler.feature_warning.FeatureWarning object>

ads.feature_engineering.feature_type.zip_code.default_handler(data: Series, *args, **kwargs) → Series

Processes given data and indicates if the data matches requirements.

Parameters:: data (pd.Series) – The data to process.
Returns:: pd.Series
Return type:: The logical list indicating if the data matches requirements.

Module contents

Address: Type representing address.
Boolean: Type representing binary values True/False.
Category: Type representing discrete unordered values.
Constant: Type representing constant values.
Continuous: Type representing continuous values.
CreditCard: Type representing credit card numbers.
DateTime: Type representing date and/or time.
Document: Type representing document values.
Discrete: Type representing discrete values.
FeatureType: Base class for all feature types.
GIS: Type representing geographic information.
Integer: Type representing integer values.
IpAddress: Type representing IP Address.
IpAddressV4: Type representing IP Address V4.
IpAddressV6: Type representing IP Address V6.
LatLong: Type representing longitude and latitute.
Object: Type representing object.
Ordinal: Type representing ordered values.
PhoneNumber: Type representing phone numbers.
String: Type representing string values.
Tag: Free form tag.
Text: Type representing text values.
ZipCode: Type representing postal code.
Unknown: Type representing third-party dtypes.

Submodules

ads.feature_engineering.exceptions module

exception ads.feature_engineering.exceptions.InvalidFeatureType(tname: str): Bases: TypeError

exception ads.feature_engineering.exceptions.NameAlreadyRegistered(name: str): Bases: NameError

exception ads.feature_engineering.exceptions.TypeAlreadyAdded(tname: str): Bases: TypeError

exception ads.feature_engineering.exceptions.TypeAlreadyRegistered(tname: str): Bases: TypeError

exception ads.feature_engineering.exceptions.TypeNotFound(tname: str): Bases: TypeError

exception ads.feature_engineering.exceptions.WarningAlreadyExists(name: str): Bases: ValueError

exception ads.feature_engineering.exceptions.WarningNotFound(name: str): Bases: ValueError

ads.feature_engineering.feature_type_manager module

The module that helps to manage feature types. Provides functionalities to register, unregister, list feature types.

Classes

FeatureTypeManager
Feature Types Manager class that manages feature types.

Examples

>>> from ads.feature_engineering.feature_type.base import FeatureType
>>> class NewType(FeatureType):
...    description="My personal type."
...    pass
>>> FeatureTypeManager.feature_type_register(NewType)
>>> FeatureTypeManager.feature_type_registered()
            Name        Feature Type                                  Description
---------------------------------------------------------------------------------
0     Continuous          continuous          Type representing continuous values.
1       DateTime           date_time           Type representing date and/or time.
2       Category            category  Type representing discrete unordered values.
3        Ordinal             ordinal             Type representing ordered values.
4        NewType            new_type                             My personal type.

>>> FeatureTypeManager.warning_registered()
    Feature Type             Warning                    Handler
----------------------------------------------------------------------
0     continuous               zeros              zeros_handler
1     continuous    high_cardinality   high_cardinality_handler

>>> FeatureTypeManager.validator_registered()
    Feature Type            Validator                 Condition                     Handler
-------------------------------------------------------------------------------------------
0   phone_number      is_phone_number                        ()             default_handler
1   phone_number      is_phone_number    {'country_code': '+7'}    specific_country_handler
2    credit_card       is_credit_card                        ()             default_handler

>>> FeatureTypeManager.feature_type_unregister(NewType)
>>> FeatureTypeManager.feature_type_reset()
>>> FeatureTypeManager.feature_type_object('continuous')
Continuous

class ads.feature_engineering.feature_type_manager.FeatureTypeManager

Bases: object

Feature Types Manager class that manages feature types.

Provides functionalities to register, unregister, list feature types.

feature_type_object(cls, feature_type: Union[FeatureType, str]) → FeatureType: Gets a feature type by class object or name.

feature_type_register(cls, feature_type_cls: FeatureType) → None: Registers a feature type.

feature_type_unregister(cls, feature_type_cls: Union[FeatureType, str]) → None: Unregisters a feature type.

feature_type_reset(cls) → None: Resets feature types to be default.

feature_type_registered(cls) → pd.DataFrame: Lists all registered feature types as a DataFrame.

warning_registered(cls) → pd.DataFrame: Lists registered warnings for all registered feature types.

validator_registered(cls) → pd.DataFrame: Lists registered validators for all registered feature types.

Examples

>>> from ads.feature_engineering.feature_type.base import FeatureType
>>> class NewType(FeatureType):
...    pass
>>> FeatureTypeManager.register_feature_type(NewType)
>>> FeatureTypeManager.feature_type_registered()
            Name      Feature Type                                  Description
-------------------------------------------------------------------------------
0     Continuous        continuous          Type representing continuous values.
1       DateTime         date_time           Type representing date and/or time.
2       Category          category  Type representing discrete unordered values.
3        Ordinal           ordinal             Type representing ordered values.

>>> FeatureTypeManager.warning_registered()
    Feature Type             Warning                    Handler
----------------------------------------------------------------------
0     continuous               zeros              zeros_handler
1     continuous    high_cardinality   high_cardinality_handler

>>> FeatureTypeManager.validator_registered()
    Feature Type            Validator                 Condition                     Handler
-------------------------------------------------------------------------------------------
0   phone_number      is_phone_number                        ()             default_handler
1   phone_number      is_phone_number    {'country_code': '+7'}    specific_country_handler
2    credit_card       is_credit_card                        ()             default_handler

>>> FeatureTypeManager.feature_type_unregister(NewType)
>>> FeatureTypeManager.feature_type_reset()
>>> FeatureTypeManager.feature_type_object('continuous')
Continuous

classmethod feature_type_object(feature_type: Union[FeatureType, str]) → FeatureType

Gets a feature type by class object or name.

Parameters:

feature_type (Union[FeatureType, str]) – The FeatureType subclass or a str indicating feature type.

Returns:

Found feature type.

Return type:

FeatureType

Raises:

TypeNotFound – If provided feature type not registered.
TypeError – If provided feature type not a subclass of FeatureType.

classmethod feature_type_register(feature_type_cls: FeatureType) → None

Registers new feature type.

Parameters:

feature_type (FeatureType) – Subclass of FeatureType to be registered.

Returns:

Nothing.

Return type:

None

Raises:

TypeError – Type is not a subclass of FeatureType.
TypeError – Type has already been registered.
NameError – Name has already been used.

classmethod feature_type_registered() → DataFrame

Lists all registered feature types as a DataFrame.

Returns:: The list of feature types in a DataFrame format.
Return type:: pd.DataFrame

classmethod feature_type_reset() → None

Resets feature types to be default.

Returns:: Nothing.
Return type:: None

classmethod feature_type_unregister(feature_type: Union[FeatureType, str]) → None

Unregisters a feature type.

Parameters:: feature_type ((FeatureType | str)) – The FeatureType subclass or a str indicating feature type.
Returns:: Nothing.
Return type:: None
Raises:: TypeError – In attempt to unregister a default feature type.

classmethod is_type_registered(feature_type: Union[FeatureType, str]) → bool

Checks if provided feature type registered in the system.

Parameters:: feature_type (Union[FeatureType, str]) – The FeatureType subclass or a str indicating feature type.
Returns:: True if provided feature type registered, False otherwise.
Return type:: bool

classmethod validator_registered() → DataFrame

Lists registered validators for registered feature types.

Returns:: The list of registered validators for registered feature types in a DataFrame format.
Return type:: pd.DataFrame

Examples

>>> FeatureTypeManager.validator_registered()
    Feature Type            Validator                 Condition                     Handler
-------------------------------------------------------------------------------------------
0   phone_number      is_phone_number                        ()             default_handler
1   phone_number      is_phone_number    {'country_code': '+7'}    specific_country_handler
2    credit_card       is_credit_card                        ()             default_handler

classmethod warning_registered() → DataFrame

Lists registered warnings for all registered feature types.

Returns:: The list of registered warnings for registered feature types in a DataFrame format.
Return type:: pd.DataFrame

Examples

>>> FeatureTypeManager.warning_registered()
    Feature Type             Warning                    Handler
----------------------------------------------------------------------
0     continuous               zeros              zeros_handler
1     continuous    high_cardinality   high_cardinality_handler

ads.feature_engineering.schema module

class ads.feature_engineering.schema.Attribute(dtype: str, feature_type: str, name: str, domain: Domain, required: bool, description: str, order: Optional[int] = None)

Attribute describes the column/feature/element. It holds following information - * dtype - Type of data - float, int64, etc. Matches with Pandas dtypes * feature_type - Feature type of data - Integer, String, etc. Matches with ads feature types. * name - Name of the feature * domain - Represented by the Domain class * required - Boolean - True or False * description - Description about the column/feature * order - order of the column/feature in the data

Examples

>>> attr_fruits = Attribute(
...     dtype = "category",
...     feature_type = "category",
...     name = "fruits",
...     domain = Domain(values="Apple, Orange, Grapes", stats={"mode": "Orange"}, constraints=[Expression("in ['Apple', 'Orange', 'Grapes']")]),
...     required = True,
...     description = "Names of fruits",
...     order = 0
... )
>>> attr_fruits
description: Names of fruits
domain:
    constraints:
    - expression: in ['Apple', 'Orange', 'Grapes']
        language: python
    stats:
        mode: Orange
    values: Apple, Orange, Grapes
dtype: category
feature_type: category
name: fruits
order: 0
required: true
>>> attr_fruits.key
'fruits'

description: str

domain: Domain

dtype: str

feature_type: str

property key

name: str

order: Optional[int] = None

required: bool

sort_index: int

to_dict(**kwargs) → dict

Serializes instance of class into a dictionary

kwargs

side_effect: Optional[SideEffect]: side effect to take on the dictionary. The side effect can be either convert the dictionary keys to “lower” (SideEffect.CONVERT_KEYS_TO_LOWER.value) or “upper”(SideEffect.CONVERT_KEYS_TO_UPPER.value) cases.

returns:: A dictionary.
rtype:: Dict

class ads.feature_engineering.schema.BaseSchemaLoader

Bases: ABC

Base Schema Loader which load and validate schema.

load_schema(self): Load and validate schema from a file and return the normalized schema.

load_schema(schema_path): Load and validate schema from a file and return the normalized schema.

exception ads.feature_engineering.schema.DataSizeTooWide(data_col_num: int, max_col_num: int): Bases: ValueError

class ads.feature_engineering.schema.Domain(values: str = '', stats: ~typing.Dict = <factory>, constraints: ~typing.List[~ads.feature_engineering.schema.Expression] = <factory>)

Domain describes the data. It holds following information - * stats - Statistics of the data. * constraints - List of Expression which defines the constraint for the data. * Domain values.

Examples

>>> Domain(values='Rational Numbers', stats={"mean":50, "median":51, "min": 5, "max":100}, constraints=[Expression('$x > 5')])
constraints:
- expression: $x > 5
    language: python
stats:
    max: 100
    mean: 50
    median: 51
    min: 5
values: Rational Numbers

constraints: List[Expression]

stats: Dict

values: str = ''

class ads.feature_engineering.schema.Expression(expression: str, language: str = 'python')

Expression allows specifying string representation of an expression which can be evaluated by the language corresponding to the value provided in langauge attribute

Default value for language is python

Parameters:

exression (Must use string.Template format for specifying the exression) – type: str
language (default value is python. It could be any language. evaluate method expects the expression to be of type python) –

Examples

>>> exp = Expression("($x > 10 and $x <100) or ($x < -1 and $x > -500)")
>>> exp.evaluate(x=500)
False
>>> exp.evaluate(x=20)
True
>>> exp.evaluate(x=9)
False
>>> exp.evaluate(x=-9)
True

evaluate(**kwargs)

expression: str

language: str = 'python'

class ads.feature_engineering.schema.JsonSchemaLoader

Bases: BaseSchemaLoader

Json Schema which load and validate schema from json file.

load_schema(self): Load and validate schema from json file and return the normalized schema.

Examples

>>> schema_loader = JsonSchemaLoader()
>>> schema_dict = schema_loader.load_schema('schema.json')
>>> schema_dict
{'Schema': [{'dtype': 'object',
    'feature_type': 'String',
    'name': 'Attrition',
    'domain': {'values': 'String',
        'stats': {'count': 1470, 'unique': 2},
        'constraints': []},
    'required': True,
    'description': 'Attrition'},
    {'dtype': 'int64',
    'feature_type': 'Integer',
    'name': 'Age',
    'domain': {'values': 'Integer',
        'stats': {'count': 1470.0,
        'mean': 37.923809523809524,
        'std': 9.135373489136732,
        'min': 19.0,
        '25%': 31.0,
        '50%': 37.0,
        '75%': 44.0,
        'max': 61.0},
        'constraints': []},
    'required': True,
    'description': 'Age'}]}

class ads.feature_engineering.schema.Schema(_version: str = '1.1')

Bases: object

Schema describes the structure of the data.

add(self, item: Attribute, replace: bool = False): Adds a new attribute item. Replaces existing one if replace flag is True.

from_dict(self): Constructs an instance of Schema from a dictionary.

from_file(cls, file_path):: Loads the data schema from a file.

to_dict(self): Serializes the data schema into a dictionary.

to_yaml(self): Serializes the data schema into a YAML.

to_json(self): Serializes the data schema into a json string.

to_json_file(self): Saves the data schema into a json file.

to_yaml_file(self): Save to a yaml file.

add(self, item: Attribute, replace=False) → None: Adds a new attribute item. Replaces existing one if replace flag is True.

Examples

>>> attr_fruits = Attribute(
...     dtype = "category",
...     feature_type = "category",
...     name = "fruits",
...     domain = Domain(values="Apple, Orange, Grapes", stats={"mode": "Orange"}, constraints=[Expression("in ['Apple', 'Orange', 'Grapes']")]),
...     required = True,
...     description = "Names of fruits",
...     order = 0,
... )
>>> attr_animals = Attribute(
...     dtype = "category",
...     feature_type = "category",
...     name = "animals",
...     domain = Domain(values="Dog, Cat, Python", stats={"mode": "Dog"}, constraints=[Expression("in ['Dog', 'Cat', 'Python']")]),
...     required = True,
...     description = "Names of animals",
...     order = 1,
... )
>>> schema = Schema()
>>> schema.add(attr_fruits)
>>> schema.add(attr_animals)
>>> schema
schema:
- description: Names of fruits
domain:
    constraints:
    - expression: in ['Apple', 'Orange', 'Grapes']
    language: python
    stats:
    mode: Orange
    values: Apple, Orange, Grapes
dtype: category
feature_type: category
name: fruits
order: 0
required: true
- description: Names of animals
domain:
    constraints:
    - expression: in ['Dog', 'Cat', 'Python']
    language: python
    stats:
    mode: Dog
    values: Dog, Cat, Python
dtype: category
feature_type: category
name: animals
order: 1
required: true
>>> schema.to_dict()
    {'schema': [{'dtype': 'category',
    'feature_type': 'category',
    'name': 'fruits',
    'domain': {'values': 'Apple, Orange, Grapes',
        'stats': {'mode': 'Orange'},
        'constraints': [{'expression': "in ['Apple', 'Orange', 'Grapes']",
        'language': 'python'}]},
    'required': True,
    'description': 'Names of fruits',
    'order': 0},
    {'dtype': 'category',
    'feature_type': 'category',
    'name': 'animals',
    'domain': {'values': 'Dog, Cat, Python',
        'stats': {'mode': 'Dog'},
        'constraints': [{'expression': "in ['Dog', 'Cat', 'Python']",
        'language': 'python'}]},
    'required': True,
    'description': 'Names of animals',
    'order': 1}]}

add(item: Attribute, replace: bool = False)

Adds a new attribute item. Replaces existing one if replace flag is True.

Overrides the existing one if replace flag is True.

Parameters:

item (Attribute) – The attribute instance of a column/feature/element.
replace (bool) – Overrides the existing attribute item if replace flag is True.

Returns:

Nothing.

Return type:

None

Raises:

ValueError – If item is already registered and replace flag is False.
TypeError – If input data has a wrong format.

classmethod from_dict(schema: dict)

Constructs an instance of Schema from a dictionary.

Parameters:: schema (dict) – Data schema in dictionary format.
Returns:: An instance of Schema.
Return type:: Schema

classmethod from_file(file_path: str)

Loads the data schema from a file.

Parameters:: file_path (str) – File Path to load the data schema.
Returns:: An instance of Schema.
Return type:: Schema

classmethod from_json(schema: str)

Constructs an instance of Schema from a Json.

Parameters:: schema (str) – Data schema in Json format.
Returns:: An instance of Schema.
Return type:: Schema

property keys: list

Returns all registered Attribute keys.

Returns:: The list of Attribute keys.
Return type:: Tuple[str]

to_dict()

Serializes data schema into a dictionary.

Returns:: The dictionary representation of data schema.
Return type:: dict

to_json(): Serializes the data schema into a json string. :returns: The json representation of data schema. :rtype: str

to_json_file(file_path)

Saves the data schema into a json file.

Parameters:: file_path (str) – File Path to store the schema in json format.
Returns:: Nothing.
Return type:: None

to_yaml(): Serializes the data schema into a YAML. :returns: The yaml representation of data schema. :rtype: str

to_yaml_file(file_path)

Saves the data schema into a yaml file. :param file_path: File Path to store the schema in yaml format. :type file_path: str

Returns:: Nothing.
Return type:: None

validate_schema(): Validate the schema.

validate_size() → bool

Validates schema size.

Validates the size of schema. Throws an error if the size of the schema exceeds expected value.

Returns:: True if schema does not exceeds the size limit.
Return type:: bool
Raises:: SchemaSizeTooLarge – If the size of the schema exceeds expected value.

class ads.feature_engineering.schema.SchemaFactory

Bases: object

Schema Factory.

register_format(self): Register a new type of schema class.

get_schema(self): Get the YamlSchema or JsonSchema based on the format.

default_schema(cls): Construct a SchemaFactory instance and register yaml and json loader.

Examples

>>> factory = SchemaFactory.default_schema()
>>> schema_loader = factory.get_schema('.json')
>>> schema_dict = schema_loader.load_schema('schema.json')
>>> schema = Schema.from_dict(schema_dict)
>>> schema
Schema:
- description: Attrition
domain:
    constraints: []
    stats:
    count: 1470
    unique: 2
    values: String
dtype: object
feature_type: String
name: Attrition
required: true
- description: Age
domain:
    constraints: []
    stats:
    25%: 31.0
    50%: 37.0
    75%: 44.0
    count: 1470.0
    max: 61.0
    mean: 37.923809523809524
    min: 19.0
    std: 9.135373489136732
    values: Integer
dtype: int64
feature_type: Integer
name: Age
required: true

classmethod default_schema()

get_schema(file_format): Get the YamlSchema or JsonSchema based on the format.

register_format(file_format, creator): Register a new type of schema class.

exception ads.feature_engineering.schema.SchemaSizeTooLarge(size: int): Bases: ValueError

class ads.feature_engineering.schema.YamlSchemaLoader

Bases: BaseSchemaLoader

Yaml Schema which loads and validates schema from a yaml file.

load_schema(self): Loads and validates schema from a yaml file and returns the normalized schema.

Examples

>>> schema_loader = YamlSchemaLoader()
>>> schema_dict = schema_loader.load_schema('schema.yaml')
>>> schema_dict
{'Schema': [{'description': 'Attrition',
    'domain': {'constraints': [],
        'stats': {'count': 1470, 'unique': 2},
        'values': 'String'},
    'dtype': 'object',
    'feature_type': 'String',
    'name': 'Attrition',
    'required': True},
    {'description': 'Age',
    'domain': {'constraints': [],
        'stats': {'25%': 31.0,
        '50%': 37.0,
        '75%': 44.0,
        'count': 1470.0,
        'max': 61.0,
        'mean': 37.923809523809524,
        'min': 19.0,
        'std': 9.135373489136732},
        'values': 'Integer'},
    'dtype': 'int64',
    'feature_type': 'Integer',
    'name': 'Age',
    'required': True}]}

ads.feature_engineering.utils module

The module that represents utility functions.

Functions:

is_boolean(value: Any) -> bool: Checks if value type is boolean.

class ads.feature_engineering.utils.SchemeNeutral

Bases: str

AREA_DARK = '#9E9892'

AREA_LIGHT = '#BCB6B1'

BACKGROUND_DARK = '#E4E1DD'

BACKGROUND_LIGHT = '#F5F4F2'

LINE_DARK = '#47423E'

LINE_LIGHT = '#665F5B'

class ads.feature_engineering.utils.SchemeTeal

Bases: str

AREA_DARK = '#76A2A0'

AREA_LIGHT = '#9ABFBF'

BACKGROUND_DARK = '#D6E5E5'

BACKGROUND_LIGHT = '#F0f6f5'

LINE_DARK = '#2B484B'

LINE_LIGHT = '#3E686C'

ads.feature_engineering.utils.assign_issuer(cardnumber)

ads.feature_engineering.utils.is_boolean(value: Any) → bool

Checks if value type is boolean.

Parameters:: value (Any) – The value to check.
Returns:: bool
Return type:: True if value is boolean, False otherwise.

ads.feature_engineering.utils.random_color_func(z, word=None, font_size=None, position=None, orientation=None, font_path=None, random_state=None): Returns random color function use for color_func in creating WordCloud

Module contents

ads.hpo package

Subpackages

ads.hpo.visualization package

Module contents

Submodules

ads.hpo.ads_search_space module

class ads.hpo.ads_search_space.DecisionTreeClassifierSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.DecisionTreeRegressorSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.ElasticNetSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.ExtraTreesClassifierSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.ExtraTreesRegressorSearchSpace(strategy): Bases: ExtraTreesClassifierSearchSpace

class ads.hpo.ads_search_space.GradientBoostingClassifierSearchSpace(strategy): Bases: GradientBoostingRegressorSearchSpace

class ads.hpo.ads_search_space.GradientBoostingRegressorSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.LGBMClassifierSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.LGBMRegressorSearchSpace(strategy): Bases: LGBMClassifierSearchSpace

class ads.hpo.ads_search_space.LassoSearchSpace(strategy): Bases: RidgeSearchSpace

class ads.hpo.ads_search_space.LinearSVCSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.LinearSVRSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.LogisticRegressionSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.ModelSearchSpace(strategy)

Bases: ABC

Defines an abstract base class for setting the search space and strategy used during hyperparameter optimization

suggest_space(**kwargs)

class ads.hpo.ads_search_space.RandomForestClassifierSearchSpace(strategy): Bases: ExtraTreesClassifierSearchSpace

class ads.hpo.ads_search_space.RandomForestRegressorSearchSpace(strategy): Bases: ExtraTreesClassifierSearchSpace

class ads.hpo.ads_search_space.RidgeClassifierSearchSpace(strategy): Bases: RidgeSearchSpace

class ads.hpo.ads_search_space.RidgeSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.SGDClassifierSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.SGDRegressorSearchSpace(strategy): Bases: SGDClassifierSearchSpace

class ads.hpo.ads_search_space.SVCSearchSpace(strategy)

suggest_space(**kwargs)

class ads.hpo.ads_search_space.SVRSearchSpace(strategy): Bases: SVCSearchSpace

class ads.hpo.ads_search_space.XGBClassifierSearchSpace(strategy)