ads.common package

Subpackages

Submodules

ads.common.analyzer module

ads.common.auth module

class ads.common.auth.APIKey(args: Dict | None = None)[source]

Bases: AuthSignerGenerator

Creates api keys auth instance. This signer is intended to be used when signing requests for a given user - it requires that user’s ID, their private key and certificate fingerprint. It prepares extra arguments necessary for creating clients for variety of OCI services.

Signer created based on args provided. If not provided current values of according arguments will be used from current global state from AuthState class.

Parameters:

args (dict) –

args that are required to create api key config and signer. Contains keys: oci_config, oci_config_location, oci_key_profile, client_kwargs.

  • oci_config is a configuration dict that can be used to create clients

  • oci_config_location - path to config file

  • oci_key_profile - the profile to load from config file

  • client_kwargs - optional parameters for OCI client creation in next steps

create_signer() → Dict[source]

Creates api keys configuration and signer with extra arguments necessary for creating clients. Signer constructed from the oci_config provided. If not ‘oci_config’, configuration will be constructed from ‘oci_config_location’ and ‘oci_key_profile’ in place.

Returns:

Contains keys - config, signer and client_kwargs.

  • config contains the configuration information

  • signer contains the signer object created. It is instantiated from signer_callable, or

signer provided in args used, or instantiated in place - client_kwargs contains the client_kwargs that was passed in as input parameter

Return type:

dict

Examples

>>> signer_args = dict(
>>>     client_kwargs=client_kwargs
>>> )
>>> signer_generator = AuthFactory().signerGenerator(AuthType.API_KEY)
>>> signer_generator(signer_args).create_signer()
class ads.common.auth.AuthContext(**kwargs)[source]

Bases: object

AuthContext used in ‘with’ statement for properly managing global authentication type, signer, config and global configuration parameters.

Examples

>>> from ads import set_auth
>>> from ads.jobs import DataFlowRun
>>> with AuthContext(auth='resource_principal'):
>>>     df_run = DataFlowRun.from_ocid(run_id)

>>> from ads.model.framework.sklearn_model import SklearnModel
>>> model = SklearnModel.from_model_artifact(uri="model_artifact_path", artifact_dir="model_artifact_path")
>>> set_auth(auth='api_key', oci_config_location="~/.oci/config")
>>> with AuthContext(auth='api_key', oci_config_location="~/another_config_location/config"):
>>>     # upload model to Object Storage using config from another_config_location/config
>>>     model.upload_artifact(uri="oci://bucket@namespace/prefix/")
>>> # upload model to Object Storage using config from ~/.oci/config, which was set before 'with AuthContext():'
>>> model.upload_artifact(uri="oci://bucket@namespace/prefix/")

Initialize class AuthContext and saves global state of authentication type, signer, config and global configuration parameters.

Parameters:

**kwargs (optional, list of parameters passed to ads.set_auth() method, which can be:) –

auth: Optional[str], default ‘api_key’

’api_key’, ‘resource_principal’ or ‘instance_principal’. Enable/disable resource principal identity, instance principal or keypair identity

oci_config_location: Optional[str], default oci.config.DEFAULT_LOCATION, which is ‘~/.oci/config’

config file location

profile: Optional[str], default is DEFAULT_PROFILE, which is ‘DEFAULT’

profile name for api keys config file

config: Optional[Dict], default {}

created config dictionary

signer: Optional[Any], default None

created signer, can be resource principals signer, instance principal signer or other

signer_callable: Optional[Callable], default None

a callable object that returns signer

signer_kwargs: Optional[Dict], default None

parameters accepted by the signer

client_kwargs: Optional[Dict], default None

Additional keyword arguments for initializing the OCI client. Example: client_kwargs = {“timeout”: 60}

class ads.common.auth.AuthFactory[source]

Bases: object

AuthFactory class which contains list of registered signers and allows to register new signers. Check documentation for more signers: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html.

Current signers:

  • APIKey

  • ResourcePrincipal

  • InstancePrincipal

  • SecurityToken

classes = {'api_key': <class 'ads.common.auth.APIKey'>, 'instance_principal': <class 'ads.common.auth.InstancePrincipal'>, 'resource_principal': <class 'ads.common.auth.ResourcePrincipal'>, 'security_token': <class 'ads.common.auth.SecurityToken'>}
classmethod register(signer_type: str, signer: Any) → None[source]

Registers a new signer.

Parameters:

  • signer_type (str) – signer type to be registers

  • signer (RecordParser) – A new signer class to be registered.

Returns:

Nothing.

Return type:

None

signerGenerator(iam_type: str | None = 'api_key')[source]

Generates signer classes based of iam_type, which specify one of auth methods: ‘api_key’, ‘resource_principal’ or ‘instance_principal’.

Parameters:

iam_type (str, default 'api_key') – type of auth provided in IAM_TYPE environment variable or set in parameters in ads.set_auth() method.

Returns:

returns one of classes, which implements creation of signer of specified type

Return type:

APIKey or ResourcePrincipal or InstancePrincipal or SecurityToken

Raises:

ValueError – If iam_type is not supported.

class ads.common.auth.AuthSignerGenerator[source]

Bases: object

Abstract class for auth configuration and signer creation.

create_signer()[source]
class ads.common.auth.AuthState(*args, **kwargs)[source]

Bases: object

Class stores state of variables specified for auth method, configuration, configuration file location, profile name, signer or signer_callable, which set by use at any given time and can be provided by this class in any ADS module.

oci_cli_auth: str = None
oci_client_kwargs: Dict = None
oci_config: Dict = None
oci_config_path: str = None
oci_iam_type: str = None
oci_key_profile: str = None
oci_signer: Any = None
oci_signer_callable: Callable = None
oci_signer_kwargs: Dict = None
class ads.common.auth.AuthType[source]

Bases: str

API_KEY = 'api_key'
INSTANCE_PRINCIPAL = 'instance_principal'
RESOURCE_PRINCIPAL = 'resource_principal'
SECURITY_TOKEN = 'security_token'
class ads.common.auth.InstancePrincipal(args: Dict | None = None)[source]

Bases: AuthSignerGenerator

Creates Instance Principal signer - a SecurityTokenSigner which uses a security token for an instance principal. It prepares extra arguments necessary for creating clients for variety of OCI services.

Signer created based on args provided. If not provided current values of according arguments will be used from current global state from AuthState class.

Parameters:

args (dict) –

args that are required to create Instance Principal signer. Contains keys: signer_kwargs, client_kwargs.

  • signer_kwargs - optional parameters required to instantiate instance principal signer

  • client_kwargs - optional parameters for OCI client creation in next steps

create_signer() → Dict[source]

Creates Instance Principal signer with extra arguments necessary for creating clients. Signer instantiated from the signer_callable or if the signer provided is will be return by this method. If signer_callable or signer not provided new signer will be created in place.

Returns:

Contains keys - config, signer and client_kwargs.

  • config contains the configuration information

  • signer contains the signer object created. It is instantiated from signer_callable, or

signer provided in args used, or instantiated in place - client_kwargs contains the client_kwargs that was passed in as input parameter

Return type:

dict

Examples

>>> signer_args = dict(signer_kwargs={"log_requests": True})
>>> signer_generator = AuthFactory().signerGenerator(AuthType.INSTANCE_PRINCIPAL)
>>> signer_generator(signer_args).create_signer()
class ads.common.auth.OCIAuthContext(profile: str = None)[source]

Bases: object

OCIAuthContext used in ‘with’ statement for properly managing global authentication type and global configuration profile parameters.

Examples

>>> from ads.jobs import DataFlowRun
>>> with OCIAuthContext(profile='TEST'):
>>>     df_run = DataFlowRun.from_ocid(run_id)

Initialize class OCIAuthContext and saves global state of authentication type and configuration profile.

Parameters:

profile (str, default is None) – profile name for api keys config file

class ads.common.auth.ResourcePrincipal(args: Dict | None = None)[source]

Bases: AuthSignerGenerator

Creates Resource Principal signer - a security token for a resource principal. It prepares extra arguments necessary for creating clients for variety of OCI services.

Signer created based on args provided. If not provided current values of according arguments will be used from current global state from AuthState class.

Parameters:

args (dict) –

args that are required to create Resource Principal signer. Contains keys: client_kwargs.

  • client_kwargs - optional parameters for OCI client creation in next steps

create_signer() → Dict[source]

Creates Resource Principal signer with extra arguments necessary for creating clients.

Returns:

Contains keys - config, signer and client_kwargs.

  • config contains the configuration information

  • signer contains the signer object created. It is instantiated from signer_callable, or

signer provided in args used, or instantiated in place - client_kwargs contains the client_kwargs that was passed in as input parameter

Return type:

dict

Examples

>>> signer_args = dict(
>>>     signer=oci.auth.signers.get_resource_principals_signer()
>>> )
>>> signer_generator = AuthFactory().signerGenerator(AuthType.RESOURCE_PRINCIPAL)
>>> signer_generator(signer_args).create_signer()
static supported()[source]
class ads.common.auth.SecurityToken(args: Dict | None = None)[source]

Bases: AuthSignerGenerator

Creates security token auth instance. This signer is intended to be used when signing requests for a given user - it requires that user’s private key and security token. It prepares extra arguments necessary for creating clients for variety of OCI services.

Signer created based on args provided. If not provided current values of according arguments will be used from current global state from AuthState class.

Parameters:

args (dict) –

args that are required to create Security Token signer. Contains keys: oci_config, oci_config_location, oci_key_profile, client_kwargs.

  • oci_config is a configuration dict that can be used to create clients

  • oci_config_location - path to config file

  • oci_key_profile - the profile to load from config file

  • client_kwargs - optional parameters for OCI client creation in next steps

SECURITY_TOKEN_BODY_HEADERS = ['content-length', 'content-type', 'x-content-sha256']
SECURITY_TOKEN_GENERIC_HEADERS = ['date', '(request-target)', 'host']
SECURITY_TOKEN_REQUIRED = ['security_token_file', 'key_file', 'region']
create_signer() → Dict[source]

Creates security token configuration and signer with extra arguments necessary for creating clients. Signer constructed from the oci_config provided. If not ‘oci_config’, configuration will be constructed from ‘oci_config_location’ and ‘oci_key_profile’ in place.

Returns:

Contains keys - config, signer and client_kwargs.

  • config contains the configuration information

  • signer contains the signer object created. It is instantiated from signer_callable, or

signer provided in args used, or instantiated in place - client_kwargs contains the client_kwargs that was passed in as input parameter

Return type:

dict

Examples

>>> signer_args = dict(
...     client_kwargs=client_kwargs
... )
>>> signer_generator = AuthFactory().signerGenerator(AuthType.SECURITY_TOKEN)
>>> signer_generator(signer_args).create_signer()
exception ads.common.auth.SecurityTokenError[source]

Bases: Exception

class ads.common.auth.SingletonMeta[source]

Bases: type

ads.common.auth.api_keys(oci_config: str | Dict = '/home/docs/.oci/config', profile: str = 'DEFAULT', client_kwargs: Dict | None = None) → Dict[source]

Prepares authentication and extra arguments necessary for creating clients for different OCI services using API Keys.

Parameters:

  • oci_config (Optional[Union[str, Dict]], default is ~/.oci/config) – OCI authentication config file location or a dictionary with config attributes.

  • profile (Optional[str], is DEFAULT_PROFILE, which is 'DEFAULT') – Profile name to select from the config file.

  • client_kwargs (Optional[Dict], default None) – kwargs that are required to instantiate the Client if we need to override the defaults.

Returns:

Contains keys - config, signer and client_kwargs.

  • The config contains the config loaded from the configuration loaded from oci_config.

  • The signer contains the signer object created from the api keys.

  • client_kwargs contains the client_kwargs that was passed in as input parameter.

Return type:

dict

Examples

>>> from ads.common import oci_client as oc
>>> auth = ads.auth.api_keys(oci_config="/home/datascience/.oci/config", profile="TEST", client_kwargs={"timeout": 6000})
>>> oc.OCIClientFactory(**auth).object_storage # Creates Object storage client with timeout set to 6000 using API Key authentication
ads.common.auth.create_signer(auth_type: str | None = 'api_key', oci_config_location: str | None = '~/.oci/config', profile: str | None = 'DEFAULT', config: Dict | None = {}, signer: Any | None = None, signer_callable: Callable | None = None, signer_kwargs: Dict | None = {}, client_kwargs: Dict | None = None) → Dict[source]

Prepares authentication and extra arguments necessary for creating clients for different OCI services based on provided parameters. If signer or signer`_callable provided, authentication with that signer will be created. If config provided, api_key type of authentication will be created. Accepted values for auth_type: api_key (default), ‘instance_principal’, ‘resource_principal’.

Parameters:

  • auth_type (Optional[str], default 'api_key') –

    ‘api_key’, ‘resource_principal’ or ‘instance_principal’. Enable/disable resource principal identity,

    instance principal or keypair identity in a notebook session

  • oci_config_location (Optional[str], default oci.config.DEFAULT_LOCATION, which is '~/.oci/config') – config file location

  • profile (Optional[str], default is DEFAULT_PROFILE, which is 'DEFAULT') – profile name for api keys config file

  • config (Optional[Dict], default {}) – created config dictionary

  • signer (Optional[Any], default None) – created signer, can be resource principals signer, instance principal signer or other. Check documentation for more signers: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html

  • signer_callable (Optional[Callable], default None) – a callable object that returns signer

  • signer_kwargs (Optional[Dict], default None) – parameters accepted by the signer. Check documentation: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html

  • client_kwargs (dict) – kwargs that are required to instantiate the Client if we need to override the defaults

Examples

>>> import ads
>>> auth = ads.auth.create_signer() # api_key type of authentication dictionary created with default config location and default profile

>>> config = oci.config.from_file("other_config_location", "OTHER_PROFILE")
>>> auth = ads.auth.create_signer(config=config) # api_key type of authentication dictionary created based on provided config

>>> config={
...     user=ocid1.user.oc1..<unique_ID>,
...     fingerprint=<fingerprint>,
...     tenancy=ocid1.tenancy.oc1..<unique_ID>,
...     region=us-ashburn-1,
...     key_content=<private key content>,
... }
>>> auth = ads.auth.create_signer(config=config) # api_key type of authentication dictionary with private key content created based on provided config

>>> config={
...     user=ocid1.user.oc1..<unique_ID>,
...     fingerprint=<fingerprint>,
...     tenancy=ocid1.tenancy.oc1..<unique_ID>,
...     region=us-ashburn-1,
...     key_file=~/.oci/oci_api_key.pem,
... }
>>> auth = ads.auth.create_signer(config=config) # api_key type of authentication dictionary with private key file location created based on provided config

>>> signer = oci.auth.signers.get_resource_principals_signer()
>>> auth = ads.auth.create_signer(config={}, signer=signer) # resource principals authentication dictionary created

>>> auth = ads.auth.create_signer(auth_type='instance_principal') # instance principals authentication dictionary created

>>> signer_callable = oci.auth.signers.InstancePrincipalsSecurityTokenSigner
>>> signer_kwargs = dict(log_requests=True) # will log the request url and response data when retrieving
>>> auth = ads.auth.create_signer(signer_callable=signer_callable, signer_kwargs=signer_kwargs) # instance principals authentication dictionary created based on callable with kwargs parameters
>>> config = dict(
...     region=us-ashburn-1,
...     key_file=~/.oci/sessions/DEFAULT/oci_api_key.pem,
...     security_token_file=~/.oci/sessions/DEFAULT/token
... )
>>> auth = ads.auth.create_signer(auth_type="security_token", config=config) # security token authentication created based on provided config
ads.common.auth.default_signer(client_kwargs: Dict | None = None) → Dict[source]

Prepares authentication and extra arguments necessary for creating clients for different OCI services based on the default authentication setting for the session. Refer ads.set_auth API for further reference.

Parameters:

client_kwargs (dict) – kwargs that are required to instantiate the Client if we need to override the defaults. Example: client_kwargs = {“timeout”: 60}

Returns:

Contains keys - config, signer and client_kwargs.

  • The config contains the config loaded from the configuration loaded from the default location if the default auth mode is API keys, otherwise it is empty dictionary.

  • The signer contains the signer object created from default auth mode.

  • client_kwargs contains the client_kwargs that was passed in as input parameter.

Return type:

dict

Examples

>>> import ads
>>> from ads.common import oci_client as oc

>>> auth = ads.auth.default_signer()
>>> oc.OCIClientFactory(**auth).object_storage # Creates Object storage client

>>> ads.set_auth("resource_principal")
>>> auth = ads.auth.default_signer()
>>> oc.OCIClientFactory(**auth).object_storage # Creates Object storage client using resource principal authentication

>>> signer_callable = oci.auth.signers.InstancePrincipalsSecurityTokenSigner
>>> ads.set_auth(signer_callable=signer_callable) # Set instance principal callable
>>> auth = ads.auth.default_signer() # signer_callable instantiated
>>> oc.OCIClientFactory(**auth).object_storage # Creates Object storage client using instance principal authentication
ads.common.auth.get_signer(oci_config: str | None = None, oci_profile: str | None = None, **client_kwargs) → Dict[source]

Provides config and signer based given parameters. If oci_config (api key config file location) and oci_profile specified new signer will ge generated. Else signer of a type specified in OCI_CLI_AUTH environment variable will be used to generate signer and return. If OCI_CLI_AUTH not set, resource principal signer will be provided. Accepted values for OCI_CLI_AUTH: ‘api_key’, ‘instance_principal’, ‘resource_principal’.

Parameters:

  • oci_config (Optional[str], default None) – Path to the config file

  • oci_profile (Optional[str], default None) – the profile to load from the config file

  • client_kwargs – kwargs that are required to instantiate the Client if we need to override the defaults

ads.common.auth.resource_principal(client_kwargs: Dict | None = None) → Dict[source]

Prepares authentication and extra arguments necessary for creating clients for different OCI services using Resource Principals.

Parameters:

client_kwargs (Dict, default None) – kwargs that are required to instantiate the Client if we need to override the defaults.

Returns:

Contains keys - config, signer and client_kwargs.

  • The config contains and empty dictionary.

  • The signer contains the signer object created from the resource principal.

  • client_kwargs contains the client_kwargs that was passed in as input parameter.

Return type:

dict

Examples

>>> from ads.common import oci_client as oc
>>> auth = ads.auth.resource_principal({"timeout": 6000})
>>> oc.OCIClientFactory(**auth).object_storage # Creates Object Storage client with timeout set to 6000 seconds using resource principal authentication
ads.common.auth.security_token(oci_config: str | Dict = '/home/docs/.oci/config', profile: str = 'DEFAULT', client_kwargs: Dict | None = None) → Dict[source]

Prepares authentication and extra arguments necessary for creating clients for different OCI services using Security Token.

Parameters:

  • oci_config (Optional[Union[str, Dict]], default is ~/.oci/config) – OCI authentication config file location or a dictionary with config attributes.

  • profile (Optional[str], is DEFAULT_PROFILE, which is 'DEFAULT') – Profile name to select from the config file.

  • client_kwargs (Optional[Dict], default None) – kwargs that are required to instantiate the Client if we need to override the defaults.

Returns:

Contains keys - config, signer and client_kwargs.

  • The config contains the config loaded from the configuration loaded from oci_config.

  • The signer contains the signer object created from the security token.

  • client_kwargs contains the client_kwargs that was passed in as input parameter.

Return type:

dict

Examples

>>> from ads.common import oci_client as oc
>>> auth = ads.auth.security_token(oci_config="/home/datascience/.oci/config", profile="TEST", client_kwargs={"timeout": 6000})
>>> oc.OCIClientFactory(**auth).object_storage # Creates Object storage client with timeout set to 6000 using Security Token authentication
ads.common.auth.set_auth(auth: str | None = 'api_key', oci_config_location: str | None = '~/.oci/config', profile: str | None = 'DEFAULT', config: Dict | None = {}, signer: Any | None = None, signer_callable: Callable | None = None, signer_kwargs: Dict | None = {}, client_kwargs: Dict | None = {}) → None[source]

Sets the default authentication type.

Parameters:

  • auth (Optional[str], default 'api_key') –

    ‘api_key’, ‘resource_principal’ or ‘instance_principal’. Enable/disable resource principal identity,

    instance principal or keypair identity in a notebook session

  • oci_config_location (Optional[str], default '~/.oci/config') – config file location

  • profile (Optional[str], default is DEFAULT_PROFILE, which is 'DEFAULT') – profile name for api keys config file

  • config (Optional[Dict], default {}) – created config dictionary

  • signer (Optional[Any], default None) – created signer, can be resource principals signer, instance principal signer or other. Check documentation for more signers: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html

  • signer_callable (Optional[Callable], default None) – a callable object that returns signer

  • signer_kwargs (Optional[Dict], default None) – parameters accepted by the signer. Check documentation: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html

  • client_kwargs (Optional[Dict], default None) – Additional keyword arguments for initializing the OCI client. Example: client_kwargs = {“timeout”: 60}

Examples

>>> ads.set_auth("api_key") # default signer is set to api keys

>>> ads.set_auth("api_key", profile = "TEST") # default signer is set to api keys and to use TEST profile

>>> ads.set_auth("api_key", oci_config_location = "other_config_location") # use non-default oci_config_location

>>> ads.set_auth("api_key", client_kwargs={"timeout": 60}) # default signer with connection and read timeouts set to 60 seconds for the client.

>>> other_config = oci.config.from_file("other_config_location", "OTHER_PROFILE") # Create non-default config
>>> ads.set_auth(config=other_config) # Set api keys type of authentication based on provided config

>>> config={
...     user=ocid1.user.oc1..<unique_ID>,
...     fingerprint=<fingerprint>,
...     tenancy=ocid1.tenancy.oc1..<unique_ID>,
...     region=us-ashburn-1,
...     key_content=<private key content>,
... }
>>> ads.set_auth(config=config) # Set api key authentication using private key content based on provided config

>>> config={
...     user=ocid1.user.oc1..<unique_ID>,
...     fingerprint=<fingerprint>,
...     tenancy=ocid1.tenancy.oc1..<unique_ID>,
...     region=us-ashburn-1,
...     key_file=~/.oci/oci_api_key.pem,
... }
>>> ads.set_auth(config=config) # Set api key authentication using private key file location based on provided config

>>> ads.set_auth("resource_principal")  # Set resource principal authentication

>>> ads.set_auth("instance_principal")  # Set instance principal authentication

>>> ads.set_auth("security_token")  # Set security token authentication

>>> config = dict(
...     region=us-ashburn-1,
...     key_file=~/.oci/sessions/DEFAULT/oci_api_key.pem,
...     security_token_file=~/.oci/sessions/DEFAULT/token
... )
>>> ads.set_auth("security_token", config=config) # Set security token authentication from provided config

>>> signer = oci.signer.Signer(
...     user=ocid1.user.oc1..<unique_ID>,
...     fingerprint=<fingerprint>,
...     tenancy=ocid1.tenancy.oc1..<unique_ID>,
...     region=us-ashburn-1,
...     private_key_content=<private key content>,
... )
>>> ads.set_auth(signer=signer) # Set api keys authentication with private key content based on provided signer

>>> signer = oci.signer.Signer(
...     user=ocid1.user.oc1..<unique_ID>,
...     fingerprint=<fingerprint>,
...     tenancy=ocid1.tenancy.oc1..<unique_ID>,
...     region=us-ashburn-1,
...     private_key_file_location=<private key content>,
... )
>>> ads.set_auth(signer=signer) # Set api keys authentication with private key file location based on provided signer

>>> signer = oci.auth.signers.get_resource_principals_signer()
>>> ads.auth.create_signer(config={}, signer=signer) # resource principals authentication dictionary created

>>> signer_callable = oci.auth.signers.ResourcePrincipalsFederationSigner
>>> ads.set_auth(signer_callable=signer_callable) # Set resource principal federation signer callable

>>> signer_callable = oci.auth.signers.InstancePrincipalsSecurityTokenSigner
>>> signer_kwargs = dict(log_requests=True) # will log the request url and response data when retrieving
>>> # instance principals authentication dictionary created based on callable with kwargs parameters:
>>> ads.set_auth(signer_callable=signer_callable, signer_kwargs=signer_kwargs)

ads.common.base_properties module

ads.common.card_identifier module

credit card patterns refer to https://en.wikipedia.org/wiki/Payment_card_number#Issuer_identification_number_(IIN) Active and frequent card information American Express: 34, 37 Diners Club (US & Canada): 54,55 Discover Card: 6011, 622126 - 622925, 624000 - 626999, 628200 - 628899, 64, 65 Master Card: 2221-2720, 51–55 Visa: 4

class ads.common.card_identifier.card_identify[source]

Bases: object

identify_issue_network(card_number)[source]

Returns the type of credit card based on its digits

Parameters:

card_number (String)

Returns:

String

Return type:

A string corresponding to the kind of credit card.

ads.common.config module

class ads.common.config.Config(uri: str | None = '~/.ads/config', auth: Dict | None = None)[source]

Bases: object

The class representing a config.

Initializes a config instance.

Parameters:

  • uri ((str, optional). Defaults to ~/.ads/config.) – The path to the config file. Can be local or Object Storage file.

  • auth ((Dict, optional). Defaults to None.) – The default authetication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

default() → ConfigSection[source]

Gets default config section.

Returns:

A default config section.

Return type:

ConfigSection

keys() → List[str][source]

Gets the all registered config section keys.

Returns:

The list of the all registered config section keys.

Return type:

List[str]

load(uri: str | None = None, auth: Dict | None = None) → Config[source]

Loads config from a config file.

Parameters:

  • uri ((str, optional). Defaults to ~/.ads/config.) – The path where the config file needs to be saved. Can be local or Object Storage file.

  • auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

A config object.

Return type:

Config

save(uri: str | None = None, auth: Dict | None = None, force_overwrite: bool | None = False) → Config[source]

Saves config to a config file.

Parameters:

  • uri ((str, optional). Defaults to ~/.ads/config.) – The path to the config file. Can be local or Object Storage file.

  • auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

  • force_overwrite ((bool, optional). Defaults to False.) – Overwrites the config if exists.

Returns:

Nothing

Return type:

None

section_exists(key: str) → bool[source]

Checks if a config section exists.

Parameters:

key (str) – A key of a config section.

Returns:

True if a config section exists, Fasle otherwise.

Return type:

bool

section_get(key: str) → ConfigSection[source]

Gets the config section by key.

Returns:

A config section object.

Return type:

ConfigSection

Raises:

KeyError – If a config section not exists.

section_remove(key: str) → Config[source]

Removes config section form config.

Parameters:

key (str) – A key of a config section that needs to be removed.

Returns:

Nothing

Return type:

None

section_set(key: str, info: dict | ConfigSection, replace: bool | None = False) → ConfigSection[source]

Sets a config section to config. The new config section will be added in case if it doesn’t exist. Otherwise the existing config section will be merged with the new fields.

Parameters:

  • key (str) – A key of a config section.

  • info (Union[dict, ConfigSection]) – The config section information in a dictionary or ConfigSection format.

  • replace ((bool, optional). Defaults to False.) – If set as True, overwrites config section with the new information.

Returns:

A config section object.

Return type:

ConfigSection

Raises:

  • ValueError – If section with given key is already exist and replace flag set to False.

  • TypeError – If input info has a wrong format.

to_dict() → Dict[str, Dict[str, Any]][source]

Converts config to a dictionary format.

Returns:

A config in a dictionary format.

Return type:

Dict[str, Dict[str, Any]]

with_dict(info: Dict[str, Dict[str, Any] | ConfigSection], replace: bool | None = False) → Config[source]

Merging dictionary to config.

Parameters:

  • info (Dict[str, Union[Dict[str, Any], ConfigSection]]) – A dictionary that needs to be merged to config.

  • replace ((bool, optional). Defaults to False.) – If set as True, overwrites config section with the new information.

Returns:

A config object.

Return type:

Config

class ads.common.config.ConfigSection[source]

Bases: object

The class representing a config section.

Initializes the config section instance.

clear() → None[source]

Clears the config section values.

Returns:

Nothing

Return type:

None

copy() → ConfigSection[source]

Makes a copy of a config section.

Returns:

The instance of a copied ConfigSection.

Return type:

ConfigSection

get(key: str) → str[source]

Gets the config section value by key.

Returns:

A specific config section value.

Return type:

str

keys() → Tuple[str][source]

Gets the list of the keys of a config section.

Returns:

The list of config section keys.

Return type:

Tuple[str]

remove(key: str) → None[source]

Removes the config section field by key.

Parameters:

key (str) – The config section field key.

Returns:

Nothing

Return type:

None

set(key: str, value: str, replace: bool | None = False) → None[source]

Sets the config section value by key.

Parameters:

  • key (str) – The config section field key.

  • value (str) – The config section field value.

Returns:

Nothing

Return type:

None

to_dict() → Dict[str, Any][source]

Converts config section to a dictionary.

Returns:

The config section in a dictionary format.

Return type:

Dict[str, Any]

with_dict(info: Dict[str, Any], replace: bool | None = False) → ConfigSection[source]

Populates the config section from a dictionary.

Parameters:

  • info (Dict[str, Any]) – The config section information in a dictionary format.

  • replace ((bool, optional). Defaults to False.) – If set as True, overwrites config section with the new information.

class ads.common.config.EventType(value)[source]

Bases: Enum

An enumeration.

CHANGE = 'change'
class ads.common.config.Eventing[source]

Bases: object

The class helper to register event handlers.

on(event_name: str, callback: Callable) → None[source]

Registers a callback for the particular event.

trigger(event: str) → None[source]

Triggers all the registered callbacks for the particular event.

class ads.common.config.ExtendedConfigParser(uri: str | None = '~/.ads/config', auth: Dict | None = None)[source]

Bases: ConfigParser

Class helper to read/write information to the config file.

Initializes a config parser instance.

Parameters:

  • uri ((str, optional). Defaults to ~/.ads/config.) – The path to the config file. Can be local or Object Storage file.

  • auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

read(uri: str | None = None, auth: Dict | None = None) → ExtendedConfigParser[source]

Reads config file.

uri: (str, optional). Defaults to ~/.ads/config.

The path to the config file. Can be local or Object Storage file.

auth: (Dict, optional). Defaults to None.

The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

Config parser object.

Return type:

ExtendedConfigParser

save(uri: str | None = None, auth: Dict | None = None, force_overwrite: bool | None = False) → None[source]

Saves the config to the file.

Parameters:

  • uri ((str, optional). Defaults to ~/.ads/config.) – The path to the config file. Can be local or Object Storage file.

  • auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

  • force_overwrite ((bool, optional). Defaults to False.) – Overwrites the config if exists.

Return type:

None

Raises:

FileExistsError – In case if file exists and force_overwrite is false.

to_dict() → Dict[str, Any][source]

Converts config to a dictionary.

Returns:

Config in a dictionary format.

Return type:

Dict[str, Any]

with_dict(info: Dict[str, Dict[str, Any]]) → ExtendedConfigParser[source]

Populates config with values from a dictionary.

Parameters:

info (Dict[str, Dict[str, Any]]) – Config in a dictionary format.

Returns:

Config parser object.

Return type:

ExtendedConfigParser

class ads.common.config.Mode[source]

Bases: object

READ = 'r'
WRITE = 'w'

ads.common.data module

class ads.common.data.ADSData(X=None, y=None, name='', dataset_type=None)[source]

Bases: object

This class wraps the input dataframe to various models, evaluation, and explanation frameworks. It’s primary purpose is to hold any metadata relevant to these tasks. This can include it’s:

  • X - the independent variables as some dataframe-like structure,

  • y - the dependent variable or target column as some array-like structure,

  • name - a string to name the data for user convenience,

  • dataset_type - the type of the X value.

As part of this initiative, ADSData knows how to turn itself into an onnxruntime compatible data structure with the method .to_onnxrt(), which takes and onnx session as input.

Parameters:

  • X (Union[pandas.DataFrame, dask.DataFrame, numpy.ndarray, scipy.sparse.csr.csr_matrix]) – If str, URI for the dataset. The dataset could be read from local or network file system, hdfs, s3 and gcs Should be none if X_train, y_train, X_test, Y_test are provided

  • y (Union[str, pandas.DataFrame, dask.DataFrame, pandas.Series, dask.Series, numpy.ndarray]) – If str, name of the target in X, otherwise series of labels corresponding to X

  • name (str, optional) – Name to identify this data

  • dataset_type (ADSDataset optional) – When this value is available, would be used to evaluate the ads task type

  • kwargs – Additional keyword arguments that would be passed to the underlying Pandas read API.

static build(X=None, y=None, name='', dataset_type=None, **kwargs)[source]

Returns an ADSData object built from the (source, target) or (X,y)

Parameters:

  • X (Union[pandas.DataFrame, dask.DataFrame, numpy.ndarray, scipy.sparse.csr.csr_matrix]) – If str, URI for the dataset. The dataset could be read from local or network file system, hdfs, s3 and gcs Should be none if X_train, y_train, X_test, Y_test are provided

  • y (Union[str, pandas.DataFrame, dask.DataFrame, pandas.Series, dask.Series, numpy.ndarray]) – If str, name of the target in X, otherwise series of labels corresponding to X

  • name (str, optional) – Name to identify this data

  • dataset_type (ADSDataset, optional) – When this value is available, would be used to evaluate the ads task type

  • kwargs – Additional keyword arguments that would be passed to the underlying Pandas read API.

Returns:

ads_data – A built ADSData object

Return type:

ads.common.data.ADSData

Examples

>>> data = open_csv("my.csv")

>>> data_ads = ADSData(data, 'target').build(data, 'target')
to_onnxrt(sess, idx_range=None, model=None, impute_values={}, **kwargs)[source]

Returns itself formatted as an input for the onnxruntime session inputs passed in.

Parameters:

  • sess (Session) – The session object

  • idx_range (Range) – The range of inputs to convert to onnx

  • model (SupportedModel) – A model that supports being serialized for the onnx runtime.

  • kwargs (additional keyword arguments) –

    • sess_inputs - Pass in the output from onnxruntime.InferenceSession(“model.onnx”).get_inputs()

    • input_dtypes (list) - If sess_inputs cannot be passed in, pass in the numpy dtypes of each input

    • input_shapes (list) - If sess_inputs cannot be passed in, pass in the shape of each input

    • input_names (list) -If sess_inputs cannot be passed in, pass in the name of each input

Returns:

ort – array of inputs formatted for the given session.

Return type:

Array

ads.common.data_serializer module

ads.common.error module

exception ads.common.error.ChangesNotCommitted(path)[source]

Bases: Exception

ads.common.extended_enum module

class ads.common.extended_enum.ExtendedEnum(value)[source]

Bases: Enum

The base class to extend functionality of a generic Enum.

Examples

>>> class TestEnum(ExtendedEnumMeta):
...    KEY1 = "value1"
...    KEY2 = "value2"
>>> print(TestEnum.KEY1.value) # "value1"
classmethod keys()[source]
classmethod values()[source]
class ads.common.extended_enum.ExtendedEnumMeta(name, bases, namespace, **kwargs)[source]

Bases: ABCMeta

The helper metaclass to extend functionality of a generic Enum.

values(cls) → list:[source]

Gets the list of class attributes.

Examples

>>> class TestEnum(str, metaclass=ExtendedEnumMeta):
...    KEY1 = "value1"
...    KEY2 = "value2"
>>> print(TestEnum.KEY1) # "value1"
keys() → list[source]

Gets the list of class attributes names.

Returns:

The list of class attributes names.

Return type:

list

values() → list[source]

Gets the list of class attributes values.

Returns:

The list of class values.

Return type:

list

ads.common.ipython module

ads.common.ipython.configure_plotting()[source]
ads.common.ipython.set_ipython_traceback()[source]

ads.common.model module

class ads.common.model.ADSModel(est, target=None, transformer_pipeline=None, client=None, booster=None, classes=None, name=None)[source]

Bases: object

Construct an ADSModel

Parameters:

  • est (fitted estimator object) – The estimator can be a standard sklearn estimator, a keras, lightgbm, or xgboost estimator, or any other object that implement methods from (BaseEstimator, RegressorMixin) for regression or (BaseEstimator, ClassifierMixin) for classification.

  • target (PandasSeries) – The target column you are using in your dataset, this is assigned as the “y” attribute.

  • transformer_pipeline (TransformerPipeline) – A custom trasnformer pipeline object.

  • client (Str) – Currently unused.

  • booster (Str) – Currently unused.

  • classes (list, optional) – List of target classes. Required for classification problem if the est does not contain classes_ attribute.

  • name (str, optional) – Name of the model.

static convert_dataframe_schema(df, drop=None)[source]
feature_names(X=None)[source]
static from_estimator(est, transformers=None, classes=None, name=None)[source]

Build ADSModel from a fitted estimator

Parameters:

  • est (fitted estimator object) – The estimator can be a standard sklearn estimator or any object that implement methods from (BaseEstimator, RegressorMixin) for regression or (BaseEstimator, ClassifierMixin) for classification.

  • transformers (a scalar or an iterable of objects implementing transform function, optional) – The transform function would be applied on data before calling predict and predict_proba on estimator.

  • classes (list, optional) – List of target classes. Required for classification problem if the est does not contain classes_ attribute.

  • name (str, optional) – Name of the model.

Returns:

model

Return type:

ads.common.model.ADSModel

Examples

>>> model = MyModelClass.train()
>>> model_ads = from_estimator(model)
static get_init_types(df, underlying_model=None)[source]
is_classifier()[source]

Returns True if ADS believes that the model is a classifier

Returns:

Boolean

Return type:

True if the model is a classifier, False otherwise.

predict(X)[source]

Runs the models predict function on some data

Parameters:

X (ADSData) – A ADSData object which holds the examples to be predicted on.

Returns:

Usually a list or PandasSeries of predictions

Return type:

Union[List, pandas.Series], depending on the estimator

predict_proba(X)[source]

Runs the models predict probabilities function on some data

Parameters:

X (ADSData) – A ADSData object which holds the examples to be predicted on.

Returns:

Usually a list or PandasSeries of predictions

Return type:

Union[List, pandas.Series], depending on the estimator

prepare(target_dir=None, data_sample=None, X_sample=None, y_sample=None, include_data_sample=False, force_overwrite=False, fn_artifact_files_included=False, fn_name='model_api', inference_conda_env=None, data_science_env=False, ignore_deployment_error=False, use_case_type=None, inference_python_version=None, imputed_values={}, **kwargs)[source]

Prepare model artifact directory to be published to model catalog

Parameters:

  • target_dir (str, default: model.name[:12]) – Target directory under which the model artifact files need to be added

  • data_sample (ADSData) – Note: This format is preferable to X_sample and y_sample. A sample of the test data that will be provided to predict() API of scoring script Used to generate schema_input.json and schema_output.json which defines the input and output formats

  • X_sample (pandas.DataFrame) – A sample of input data that will be provided to predict() API of scoring script Used to generate schema.json which defines the input formats

  • y_sample (pandas.Series) – A sample of output data that is expected to be returned by predict() API of scoring script, corresponding to X_sample Used to generate schema_output.json which defines the output formats

  • force_overwrite (bool, default: False) – If True, overwrites the target directory if exists already

  • fn_artifact_files_included (bool, default: True) – If True, generates artifacts to export a model as a function without ads dependency

  • fn_name (str, default: 'model_api') – Required parameter if fn_artifact_files_included parameter is setup.

  • inference_conda_env (str, default: None) – Conda environment to use within the model deployment service for inferencing

  • data_science_env (bool, default: False) – If set to True, datascience environment represented by the slug in the training conda environment will be used.

  • ignore_deployment_error (bool, default: False) – If set to True, the prepare will ignore all the errors that may impact model deployment

  • use_case_type (str) – The use case type of the model. Use it through UserCaseType class or string provided in UseCaseType. For example, use_case_type=UseCaseType.BINARY_CLASSIFICATION or use_case_type=”binary_classification”. Check with UseCaseType class to see all supported types.

  • inference_python_version (str, default:None.) – If provided will be added to the generated runtime yaml

  • **kwargs

  • --------

  • max_col_num ((int, optional). Defaults to utils.DATA_SCHEMA_MAX_COL_NUM.) – The maximum column size of the data that allows to auto generate schema.

Returns:

model_artifact

Return type:

an instance of ModelArtifact that can be used to test the generated scoring script

rename(name)[source]

Changes the name of a model

Parameters:

name (str) – A string which is supplied for naming a model.

score(X, y_true, score_fn=None)[source]

Scores a model according to a custom score function

Parameters:

  • X (ADSData) – A ADSData object which holds the examples to be predicted on.

  • y_true (ADSData) – A ADSData object which holds ground truth labels for the examples which are being predicted on.

  • score_fn (Scorer (callable)) – A callable object that returns a score, usually created with sklearn.metrics.make_scorer().

Returns:

Almost always a scalar score (usually a float).

Return type:

float, depending on the estimator

show_in_notebook()[source]

Describe the model by showing it’s properties

summary()[source]

A summary of the ADSModel

transform(X)[source]

Process some ADSData through the selected ADSModel transformers

Parameters:

X (ADSData) – A ADSData object which holds the examples to be transformed.

visualize_transforms()[source]

A graph of the ADSModel transformer pipeline. It is only supported in JupyterLabs Notebooks.

ads.common.model_artifact module

class ads.common.model_artifact.ConflictStrategy[source]

Bases: object

CREATE = 'CREATE'
IGNORE = 'IGNORE'
UPDATE = 'UPDATE'
exception ads.common.model_artifact.InvalidDataType[source]

Bases: Exception

Invalid Data Type.

class ads.common.model_artifact.ModelArtifact(artifact_dir, conflict_strategy='IGNORE', install_libs=False, reload=True, create=False, progress=None, model_file_name='model.onnx', inference_conda_env=None, data_science_env=False, ignore_deployment_error=False, inference_python_version=None)[source]

Bases: Introspectable

A class used to construct model artifacts.

…

artifact_dir

Path to the model artifacts.

Type:

str

conflict_strategy

How to handle version conflicts between the current environment and the requirements of model artifact.

Type:

ConflictStrategy, default: IGNORE

install_libs

Re-install the environment inwhich the model artifact were trained in.

Type:

bool

reload[source]

Reload the model into the environment.

Type:

bool

create

Create the runtime.yaml file.

Type:

bool

progress

Show a progress bar.

model_file_name

Name of the model file.

Type:

str

inference_conda_env

The inference conda environment. If provided, the value will be set in the runtime.yaml. This is expected to be full oci URI format - oci://{bucket}@{namespace}/path/to/condapack.

Type:

str

data_science_env

Is the inference conda environment managed by the Oracle Data Science service?

Type:

bool

ignore_deployment_error

Determine whether to turn off logging for deployment error. If set to True, the .prepare() method will ignore errors that impact model deployment.

Type:

bool

inference_python_version

The version of Python to be used in inference. The value will be set in the runtime.yaml file

Type:

str Optional, default None

reload(self, model_file_name=None)[source]

Reload the files in the model artifact directory.

verify(self, input_data)[source]

Verifies a model artifact directory.

install_requirements(self, conflict_strategy=ConflictStrategy.IGNORE)[source]

Installs missing libraries listed in the model artifact.

populate_metadata(self, model=None, use_case_type=None)[source]

Extracts and populate taxonomy metadata from the model.

save(

self, display_name: str = None, description: str = None, project_id: str = None, compartment_id: str = None, training_script_path: str = None, ignore_pending_changes: bool = False, auth: dict = None, training_id: str = None, timeout: int = None, ignore_introspection=False,

)

Saves this model artifact in model catalog.

populate_schema(

self, data_sample: ADSData = None, X_sample: Union[list, tuple, pd.DataFrame, pd.Series, np.ndarray] = None, y_sample: Union[list, tuple, pd.DataFrame, pd.Series, np.ndarray] = None,

)

Populates input and output schema.

introspect(self) → pd.DataFrame[source]

Runs model introspection.

classmethod from_model_catalog(model_id: str, artifact_dir: str, model_file_name: str | None = 'model.onnx', auth: Dict | None = None, force_overwrite: bool | None = False, install_libs: bool | None = False, conflict_strategy='IGNORE', bucket_uri: str | None = None, remove_existing_artifact: bool | None = True, **kwargs) → ModelArtifact[source]

Download model artifacts from the model catalog to the target artifact directory.

Parameters:

  • model_id (str) – The model OCID.

  • artifact_dir (str) – The artifact directory to store the files needed for deployment. Will be created if not exists.

  • model_file_name ((str, optional). Defaults to "model.onnx".) – The name of the serialized model.

  • auth ((Dict, optional). Defaults to None.) – Default authetication is set using the ads.set_auth() method. Use the ads.common.auth.api_keys() or ads.common.auth.resource_principal() to create appropriate authentication signer and kwargs required to instantiate a IdentityClient object.

  • force_overwrite ((bool, optional). Defaults to False.) – Overwrite existing files.

  • install_libs (bool, default: False) – Install the libraries specified in ds-requirements.txt.

  • conflict_strategy (ConflictStrategy, default: IGNORE) – Determines how to handle version conflicts between the current environment and requirements of model artifact. Valid values: “IGNORE”, “UPDATE” or ConflictStrategy. IGNORE: Use the installed version in case of conflict UPDATE: Force update dependency to the version required by model artifact in case of conflict

  • bucket_uri ((str, optional). Defaults to None.) – The OCI Object Storage URI where model artifacts will be copied to. The bucket_uri is only necessary for downloading large artifacts with size is greater than 2GB. Example: oci://<bucket_name>@<namespace>/prefix/.

  • remove_existing_artifact ((bool, optional). Defaults to True.) – Whether artifacts uploaded to object storage bucket need to be removed or not.

  • kwargs –

    compartment_id: (str, optional)

    Compartment OCID. If not specified, the value will be taken from the environment variables.

    timeout: (int, optional). Defaults to 10 seconds.

    The connection timeout in seconds for the client.

Returns:

An instance of ModelArtifact class.

Return type:

ModelArtifact

install_requirements(conflict_strategy='IGNORE')[source]

Installs missing libraries listed in the model artifacts.

Parameters:

conflict_strategy (ConflictStrategy, default: IGNORE) – Update the conflicting dependency to the version required by the model artifact. Valid values: “IGNORE” or ConflictStrategy.IGNORE, “UPDATE” or ConflictStrategy.UPDATE. IGNORE: Use the installed version in case of a conflict. UPDATE: Force update dependency to the version required by model artifact in case of conflict.

introspect() → DataFrame[source]

Runs model introspection.

Returns:

The introspection result in a dataframe format.

Return type:

pd.DataFrame

populate_metadata(model=None, use_case_type=None)[source]

Extracts and populate taxonomy metadata from given model.

Parameters:

  • model ((Any, optional). Defaults to None.) – The model object.

  • use_case_type ((str, optional). Default to None.) – The use case type of the model.

  • model – This is an optional model object which is only used to extract taxonomy metadata. Supported models: keras, lightgbm, pytorch, sklearn, tensorflow, and xgboost. If the model is not under supported frameworks, then extracting taxonomy metadata will be skipped.

  • use_case_type – The use case type of the model.

Returns:

Nothing.

Return type:

None

populate_schema(data_sample: ADSData | None = None, X_sample: list | tuple | DataFrame | Series | ndarray | None = None, y_sample: list | tuple | DataFrame | Series | ndarray | None = None, max_col_num: int = 2000)[source]

Populate the input and output schema. If the schema exceeds the limit of 32kb, save as json files to the artifact directory.

Parameters:

  • data_sample (ADSData) – A sample of the data that will be used to generate input_schema and output_schema.

  • X_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]) – A sample of input data that will be used to generate input schema.

  • y_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]) – A sample of output data that will be used to generate output schema.

  • max_col_num ((int, optional). Defaults to utils.DATA_SCHEMA_MAX_COL_NUM.) – The maximum column size of the data that allows to auto generate schema.

reload(model_file_name: str | None = None)[source]

Reloads files in model artifact directory.

Parameters:

model_file_name (str) – The model file name.

save(display_name: str = None, description: str = None, project_id: str = None, compartment_id: str = None, training_script_path: str = None, ignore_pending_changes: bool = False, auth: dict = None, training_id: str = None, timeout: int = None, ignore_introspection=True, freeform_tags=None, defined_tags=None, bucket_uri: str | None = None, remove_existing_artifact: bool | None = True, model_version_set: str | ModelVersionSet | None = None, version_label: str | None = None)[source]

Saves the model artifact in the model catalog.

Parameters:

  • display_name (str, optional) – Model display name.

  • description (str, optional) – Description for the model.

  • project_id (str, optional) – Model’s project OCID. If None, the default project OCID config.PROJECT_OCID would be used.

  • compartment_id (str, optional) – Model’s compartment OCID. If None, the default compartment OCID config.NB_SESSION_COMPARTMENT_OCID would be used.

  • training_script_path (str, optional) – The training script path is either relative to the working directory, or an absolute path.

  • ignore_pending_changes (bool, default: False) – If True, ignore uncommitted changes and use the current git HEAD commit for provenance metadata. This argument is used only when the function is called from a script in git managed directory.

  • auth (dict) – Default is None. Default authetication is set using the ads.set_auth() method. Use the ads.common.auth.api_keys() or ads.common.auth.resource_principal() to create appropriate authentication signer and kwargs required to instantiate a DataScienceClient object.

  • training_id (str, optional) – The training OCID for the model.

  • timeout (int, default: 10) – The connection timeout in seconds.

  • ignore_introspection (bool, optional) – Ignore the result of model introspection . If set to True, the .save() will ignore all model introspection errors.

  • freeform_tags (dict(str, str), optional) – Freeform tags for the model.

  • defined_tags (dict(str, dict(str, object)), optional) – Defined tags for the model.

  • bucket_uri ((str, optional). Defaults to None.) – The OCI Object Storage URI where model artifacts will be copied to. The bucket_uri is only necessary for uploading large artifacts which size is greater than 2GB. Example: oci://<bucket_name>@<namespace>/prefix/

  • remove_existing_artifact ((bool, optional). Defaults to True.) – Whether artifacts uploaded to object storage bucket need to be removed or not.

  • model_version_set ((Union[str, ModelVersionSet], optional). Defaults to None.) – The Model version set OCID, or name, or ModelVersionSet instance.

  • version_label ((str, optional). Defaults to None.) – The model version label.

Examples

>>> from ads.common.model_artifact import ModelArtifact
>>> from ads.config import NB_SESSION_OCID

>>> # Getting auth details.
>>> # If you are using API keys
>>> auth=ads.common.auth.api_keys()

>>> # If you are using resource principal
>>> auth=ads.common.auth.resource_principal()

>>> # If you have set the auth type using ads.set_auth()
>>> auth=ads.common.auth.default_signer()

>>> # Preparing model artifacts
>>> model_artifact = prepare_generic_model(
...     "path_to_model_artifacts",
...     force_overwrite=True,
...     data_science_env=True,
...     model=gamma_reg_model,
... )

>>> # Saving model to the model catalog
>>> model_artifact.save(
...     project_id=PROJECT_ID,
...     compartment_id=COMPARTMENT,
...     display_name="RF Classifier 2",
...     description="A sample Random Forest classifier",
...     ignore_pending_changes=True,
...     auth=auth,
...     training_id=NB_SESSION_OCID,
...     timeout=6000,
...     ignore_introspection = True
... )
verify(input_data)[source]

Verifies the contents of the model artifact directory.

Parameters:

input_data (str, dict, BytesIO stream) – Data to be passed into the deployed model. It can be of type json (str), a dict object, or a BytesIO stream. All types get converted into a UTF-8 encoded BytesIO stream and is then sent to the handler. Any data handling past there is done in func.py. By default it looks for data under the keyword “input”, and returns data under teh keyword “prediction”.

Returns:

  • output_data (the resulting prediction, formatted in the same way as input_data)

  • Example – ——– input_dict = {“input”: train.X[:3].to_dict()} model_artifact.verify(input_dict)

    • returns {“prediction”: [30/4, 24.8, 30.7]} *

class ads.common.model_artifact.PACK_TYPE(value)[source]

Bases: Enum

An enumeration.

SERVICE_PACK = 'data_science'
USER_CUSTOM_PACK = 'published'
class ads.common.model_artifact.VersionConflictWarning(version_conflicts)[source]

Bases: object

ads.common.model_artifact.execute(cmd)[source]
ads.common.model_artifact.pip_install(package, options='-U')[source]

ads.common.model_export_util module

ads.common.model_export_util.prepare_generic_model(model_path: str, fn_artifact_files_included: bool = False, fn_name: str = 'model_api', force_overwrite: bool = False, model: Any = None, data_sample: ADSData = None, use_case_type=None, X_sample: list | tuple | Series | ndarray | DataFrame = None, y_sample: list | tuple | Series | ndarray | DataFrame = None, **kwargs) → ModelArtifact[source]

Generates template files to aid model deployment. The model could be accompanied by other artifacts all of which can be dumped at model_path. Following files are generated: * func.yaml * func.py * requirements.txt * score.py

Parameters:

  • model_path (str) – Path where the artifacts must be saved. The serialized model object and any other associated files/objects must be saved in the model_path directory

  • fn_artifact_files_included (bool) – Default is False, if turned off, function artifacts are not generated.

  • fn_name (str) – Opional parameter to specify the function name

  • force_overwrite (bool) – Opional parameter to specify if the model_artifact should overwrite the existing model_path (if it exists)

  • model ((Any, optional). Defaults to None.) – This is an optional model object which is only used to extract taxonomy metadata. Supported models: automl, keras, lightgbm, pytorch, sklearn, tensorflow, and xgboost. If the model is not under supported frameworks, then extracting taxonomy metadata will be skipped. The alternative way is using atifact.populate_metadata(model=model, usecase_type=UseCaseType.REGRESSION).

  • data_sample (ADSData) – A sample of the test data that will be provided to predict() API of scoring script Used to generate schema_input and schema_output

  • use_case_type (str) – The use case type of the model

  • X_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame, dask.dataframe.core.Series, dask.dataframe.core.DataFrame]) – A sample of input data that will be provided to predict() API of scoring script Used to generate input schema.

  • y_sample (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame, dask.dataframe.core.Series, dask.dataframe.core.DataFrame]) – A sample of output data that is expected to be returned by predict() API of scoring script, corresponding to X_sample Used to generate output schema.

  • **kwargs

  • ________

  • data_science_env (bool, default: False) – If set to True, the datascience environment represented by the slug in the training conda environment will be used.

  • inference_conda_env (str, default: None) – Conda environment to use within the model deployment service for inferencing. For example, oci://bucketname@namespace/path/to/conda/env

  • ignore_deployment_error (bool, default: False) – If set to True, the prepare method will ignore all the errors that may impact model deployment.

  • underlying_model (str, default: 'UNKNOWN') – Underlying Model Type, could be “automl”, “sklearn”, “h2o”, “lightgbm”, “xgboost”, “torch”, “mxnet”, “tensorflow”, “keras”, “pyod” and etc.

  • model_libs (dict, default: {}) – Model required libraries where the key is the library names and the value is the library versions. For example, {numpy: 1.21.1}.

  • progress (int, default: None) – max number of progress.

  • inference_python_version (str, default:None.) – If provided will be added to the generated runtime yaml

  • max_col_num ((int, optional). Defaults to utils.DATA_SCHEMA_MAX_COL_NUM.) – The maximum column size of the data that allows to auto generate schema.

Examples

>>> import cloudpickle
>>> import os
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.datasets import make_classification
>>> import ads
>>> from ads.common.model_export_util import prepare_generic_model
>>> import yaml
>>> import oci
>>>
>>> ads.set_auth('api_key', oci_config_location=oci.config.DEFAULT_LOCATION, profile='DEFAULT')
>>> model_artifact_location = os.path.expanduser('~/myusecase/model/')
>>> inference_conda_env="oci://my-bucket@namespace/conda_environments/cpu/Data_Exploration_and_Manipulation_for_CPU_Python_3.7/2.0/dataexpl_p37_cpu_v2"
>>> inference_python_version = "3.7"
>>> if not os.path.exists(model_artifact_location):
...     os.makedirs(model_artifact_location)
>>> X, y = make_classification(n_samples=100, n_features=20, n_classes=2)
>>> lrmodel = LogisticRegression().fit(X, y)
>>> with open(os.path.join(model_artifact_location, 'model.pkl'), "wb") as mfile:
...     cloudpickle.dump(lrmodel, mfile)
>>> modelartifact = prepare_generic_model(
...     model_artifact_location,
...     model = lrmodel,
...     force_overwrite=True,
...     inference_conda_env=inference_conda_env,
...     ignore_deployment_error=True,
...     inference_python_version=inference_python_version
... )
>>> modelartifact.reload() # Call reload to update the ModelArtifact object with the generated score.py
>>> assert len(modelartifact.predict(X[:5])['prediction']) == 5 #Test the generated score.py works. This may require customization.
>>> with open(os.path.join(model_artifact_location, "runtime.yaml")) as rf:
...     content = yaml.load(rf, Loader=yaml.FullLoader)
...     assert content['MODEL_DEPLOYMENT']['INFERENCE_CONDA_ENV']['INFERENCE_ENV_PATH'] == inference_conda_env
...     assert content['MODEL_DEPLOYMENT']['INFERENCE_CONDA_ENV']['INFERENCE_PYTHON_VERSION'] == inference_python_version
>>> # Save Model to model artifact
>>> ocimodel = modelartifact.save(
...     project_id="oci1......", # OCID of the project to which the model to be associated
...     compartment_id="oci1......", # OCID of the compartment where the model will reside
...     display_name="LRModel_01",
...     description="My Logistic Regression Model",
...     ignore_pending_changes=True,
...     timeout=100,
...     ignore_introspection=True,
... )
>>> print(f"The OCID of the model is: {ocimodel.id}")
Returns:

model_artifact – A generic model artifact

Return type:

ads.model_artifact.model_artifact

ads.common.model_export_util.serialize_model(model=None, target_dir=None, X=None, y=None, model_type=None, **kwargs)[source]
Parameters:

  • model (ads.Model) – A model to be serialized

  • target_dir (str, optional) – directory to output the serialized model

  • X (Union[pandas.DataFrame, pandas.Series]) – The X data

  • y (Union[list, pandas.DataFrame, pandas.Series]) – Tbe Y data

  • model_type (str, optional) – A string corresponding to the model type

Returns:

model_kwargs – A dictionary of model kwargs for the serialized model

Return type:

Dict

ads.common.model_introspect module

ads.common.model_metadata module

The module created for the back compatability. The original model_metadata was moved to the ads.model package.

ads.common.model_metadata_mixin module

ads.common.object_storage_details module

exception ads.common.object_storage_details.InvalidObjectStoragePath[source]

Bases: Exception

Invalid Object Storage Path.

class ads.common.object_storage_details.ObjectStorageDetails(bucket: str, namespace: str = '', filepath: str = '', version: str | None = None, auth: Dict | None = None)[source]

Bases: object

Class that represents the Object Storage bucket URI details.

bucket

The Object Storage bucket name.

Type:

str

namespace

The Object Storage namespace. Will be extracted automatically if not provided.

Type:

(str, optional). Defaults to empty string.

filepath

The path to the object.

Type:

(str, optional). Defaults to empty string.

version

The version of the object.

Type:

(str, optional). Defaults to None.

auth

ADS auth dictionary for OCI authentication. This can be generated by calling ads.common.auth.api_keys() or ads.common.auth.resource_principal() If this is None, ads.common.default_signer() will be used.

Type:

(Dict, optional). Defaults to None.

auth: Dict = None
bucket: str
bulk_download_from_object_storage(paths: List[ObjectStorageDetails], target_dir: str, progress_bar: TqdmProgressBar | None = None)[source]

Downloads the files with object versions set in the paths dict parallely.

Parameters:

  • paths – Contains a list of OSS paths along with a value of file version id. If version_id is not available, download the latest version.

  • target_dir – Local directory to save the files

  • progress_bar – an instance of the TqdmProgressBar, can update description in the calling progress bar

Return type:

None

create_os_client()[source]
download_from_object_storage(path: ObjectStorageDetails, target_dir: str, progress_bar: TqdmProgressBar | None = None)[source]

Downloads the files with object versions set in the paths dict.

Parameters:

  • path – OSS path along with a value of file version id. If version_id is not available, download the latest version.

  • target_dir – Local directory to save the files

  • progress_bar – an instance of the TqdmProgressBar, can update description in the calling progress bar

Return type:

None

fetch_metadata_of_object() → Dict[source]

Fetches the manifest metadata from the object storage of a conda pack.

Returns:

The metadata in dictionary format.

Return type:

Dict

filepath: str = ''
classmethod from_path(env_path: str) → ObjectStorageDetails[source]

Construct an ObjectStorageDetails instance from conda pack path.

Parameters:

env_path ((str)) – codna pack object storage path.

Raises:

Exception – OCI conda url path not properly configured.:

Returns:

An ObjectStorageDetails instance.

Return type:

ObjectStorageDetails

is_bucket_versioned() → bool[source]

Check if the given bucket is versioned. :returns: bool :rtype: return True if the bucket is versioned.

static is_oci_path(uri: str | None = None) → bool[source]

Check if the given path is oci object storage uri.

Parameters:

uri (str) – The URI of the target.

Returns:

bool

Return type:

return True if the path is oci object storage uri.

static is_valid_uri(uri: str) → bool[source]

Validates the Object Storage URI.

list_object_versions(**kwargs)[source]

Lists object versions in a given oss path

Parameters:

**kwargs – namespace, bucket, filepath are set by the class. By default, fields gets all values. For other supported parameters, check https://docs.oracle.com/iaas/api/#/en/objectstorage/20160918/Object/ListObjectVersions

Return type:

Object of type oci.object_storage.models.ObjectVersionCollection

list_objects(**kwargs)[source]
Lists objects in a given oss path

Parameters

Return type:

Object of type oci.object_storage.models.ListObjects

namespace: str = ''
property os_client
property path

Full object storage path of this file.

to_tuple()[source]

Returns the values of the fields of ObjectStorageDetails class.

version: str = None

ads.common.oci_client module

class ads.common.oci_client.OCIClientFactory(config=None, signer=None, client_kwargs=None)[source]

Bases: object

A factory class to create OCI client objects. The constructor takes in config, signer and client_kwargs. client_kwargs is passed to the client constructor as key word arguments.

Examples

from ads.common import auth as authutil from ads.common import oci_client as oc

auth = authutil.default_signer() oc.OCIClientFactory(**auth).object_storage # Creates Object storage client

auth = authutil.default_signer({“timeout”: 6000}) oc.OCIClientFactory(**auth).object_storage # Creates Object storage client with timeout set to 6000

auth = authutil.api_keys(config=”/home/datascience/.oci/config”, profile=”TEST”, {“timeout”: 6000}) oc.OCIClientFactory(**auth).object_storage # Creates Object storage client with timeout set to 6000 using API Key authentication

auth = authutil.resource_principal({“timeout”: 6000}) oc.OCIClientFactory(**auth).object_storage # Creates Object storage client with timeout set to 6000 using resource principal authentication

auth = authutil.create_signer(“instance_principal”) oc.OCIClientFactory(**auth).object_storage # Creates Object storage client using instance principal authentication

property ai_language
property artifacts: ArtifactsClient
create_client(client_name)[source]
property data_catalog
property data_labeling_cp
property data_labeling_dp
property data_science
property dataflow
property identity
property limits
property logging_management
property marketplace
property object_storage
property secret
property vault
property virtual_network

ads.common.oci_datascience module

class ads.common.oci_datascience.DSCNotebookSession(config: dict | None = None, signer: Signer | None = None, client_kwargs: dict | None = None, **kwargs)[source]

Bases: OCIDataScienceMixin, NotebookSession

Represents a data science notebook session

To get the information of an existing notebook session: >>> notebook = DSCNotebookSession.from_ocid(NOTEBOOK_OCID) Get the name of the notebook session >>> notebook.display_name Get the subnet ID of the notebook session >>> notebook.notebook_session_configuration_details.subnet_id

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

  • config (dict, optional) – OCI API key config dictionary, by default None.

  • signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.

  • client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

class ads.common.oci_datascience.OCIDataScienceMixin(config: dict | None = None, signer: Signer | None = None, client_kwargs: dict | None = None, **kwargs)[source]

Bases: OCIModelMixin

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

  • config (dict, optional) – OCI API key config dictionary, by default None.

  • signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.

  • client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

property client: DataScienceClient

OCI client

property client_composite: DataScienceClientCompositeOperations
classmethod init_client(**kwargs) → DataScienceClient[source]

Initializes the OCI client specified in the “client” keyword argument Sub-class should override this method and call cls._init_client(client=OCI_CLIENT)

Parameters:

**kwargs – Additional keyword arguments for initializing the OCI client.

Return type:

An instance of OCI client.

ads.common.oci_logging module

class ads.common.oci_logging.ConsolidatedLog(*args)[source]

Bases: object

Represents the Consolidated OCI Log resource.

Usage: Initialize consolidated log instance for oci_log_one and oci_log_two >>> oci_log_one = OCILog( >>> compartment_id=<compartment_id_one>, >>> id=<id_one>, >>> log_group_id=<log_group_id_one>, >>> annotation=<annotation_one> >>> ) >>> oci_log_two = OCILog( >>> compartment_id=<compartment_id_two>, >>> id=<id_two>, >>> log_group_id=<log_group_id_two>, >>> annotation=<annotation_two> >>> ) >>> consolidated_log = ConsolidatedLog(oci_log_one, oci_log_two) Stream, sort and annotate the logs from oci_log_one and oci_log_two >>> consolidated_log.stream() Get the most recent 20 consolidated logs from oci_log_one and oci_log_two >>> consolidated_log.tail(limit=20) Get the most recent 20 raw logs from oci_log_one and oci_log_two >>> consolidated_log.search(limit=20)

Initializes a consolidate log model instance.

Parameters:

args – A list of OCILog instance.

head(source: str | None = None, limit: int = 100, time_start: datetime | None = None) → None[source]

Returns the preceding consolidated log records.

Parameters:

  • source (str, optional) – Expression or OCID to filter the “source” field of the OCI log record. Defaults to None.

  • limit (int, optional.) – Maximum number of records to be returned. If limit is set to None, all logs from time_start to now will be returned. Defaults to 100.

  • time_start (datetime.datetime, optional) – Starting time for the log query. Defaults to None.

search(source: str | None = None, time_start: datetime | None = None, time_end: datetime | None = None, limit: int | None = None, sort_by: str = 'datetime', sort_order: str = 'ASC', log_filter: str | None = None) → List[SearchResult][source]

Searches raw logs.

Parameters:

  • source (str, optional) – Expression or OCID to filter the “source” field of the OCI log record. Defaults to None.

  • time_start (datetime.datetime, optional) – Starting time for the log query. Defaults to None.

  • time_end (datetime.datetime, optional.) – Ending time for the query. Defaults to None.

  • limit (int, optional.) – Maximum number of records to be returned. All logs will be returned. Defaults to None.

  • sort_by (str, optional.) – The field for sorting the logs. Defaults to “datetime”

  • sort_order (str, optional.) – The sort order for the log records. Can be “ASC” or “DESC”. Defaults to “ASC”.

  • log_filter (str, optional) – Expression for filtering the logs. This will be the WHERE clause of the query. Defaults to None.

Returns:

A list of oci.loggingsearch.models.SearchResult objects

Return type:

list

stream(source: str | None = None, interval: int = 3, stop_condition: callable | None = None, time_start: datetime | None = None, log_filter: str | None = None)[source]

Streams consolidated logs to console/terminal until stop_condition() returns true.

Parameters:

  • source (str, optional) – Expression or OCID to filter the “source” field of the OCI log record. Defaults to None.

  • interval (int, optional) – The time interval between sending each request to pull logs from OCI logging service. Defaults to 3.

  • stop_condition (callable, optional) – A function to determine if the streaming should stop. The log streaming will stop if the function returns true. Defaults to None.

  • time_start (datetime.datetime, optional) – Starting time for the log query. Defaults to None.

  • log_filter (str, optional) – Expression for filtering the logs. This will be the WHERE clause of the query. Defaults to None.

tail(source: str | None = None, limit: int = 100, time_start: datetime | None = None, log_filter: str | None = None) → None[source]

Returns the most recent consolidated log records.

Parameters:

  • source (str, optional) – Expression or OCID to filter the “source” field of the OCI log record. Defaults to None.

  • limit (int, optional.) – Maximum number of records to be returned. If limit is set to None, all logs from time_start to now will be returned. Defaults to 100.

  • time_start (datetime.datetime, optional) – Starting time for the log query. Defaults to None.

  • log_filter (str, optional) – Expression for filtering the logs. This will be the WHERE clause of the query. Defaults to None.

class ads.common.oci_logging.OCILog(log_type: str = 'CUSTOM', **kwargs)[source]

Bases: OCILoggingModelMixin, Log

Represents the OCI Log resource.

Usage: (OCI requires display_name to be unique and it cannot contain space) >>> log = OCILog(display_name=”My_Log”, log_group_id=LOG_GROUP_ID).create() Usually it is better to create a log using the create_log() method in OCILogGroup. >>> log.delete() # Delete the resource Get a log object from OCID >>> oci_log = OCILog.from_ocid(“LOG_OCID_HERE”) Stream the logs from an OCI Data Science Job run to stdout >>> oci_log.stream(source=”JOB_RUN_OCID_HERE”) Gets the most recent 10 logs >>> oci_log.tail(10)

Initializes an OCI log model locally. The resource is not created in OCI until the create() or create_async() method is called.

create_async()[source]

Creates a new Log with OCI logging service

delete()[source]

Delete the log

static format_datetime(dt: datetime) → str[source]

Converts datetime object to RFC3339 date time format in string

Parameters:

dt (datetime.datetime) – Datetime object to be formated.

Returns:

A timestamp in RFC3339 format.

Return type:

str

head(source: str | None = None, limit=100, time_start: datetime | None = None) → List[dict][source]

Returns the preceding log records.

Parameters:

  • source ((str, optional). Defaults to None.) – The source field to filter the log records.

  • limit ((int, optional). Defaults to 100.) – Maximum number of records to be returned. If limit is set to None, all logs from time_start to now will be returned.

  • time_start ((datetime.datetime, optional)) – Starting time for the log query. Defaults to None. Logs up to 14 days from now will be returned.

Returns:

A list of log records. Each log record is a dictionary with the following keys: id, time, message.

Return type:

list

search(source: str | None = None, time_start: datetime | None = None, time_end: datetime | None = None, limit: int | None = None, sort_by: str = 'datetime', sort_order: str = 'DESC', log_filter: str | None = None) → List[SearchResult][source]

Search logs

Parameters:

  • source (str, optional) – Expression or OCID to filter the “source” field of the OCI log record. Defaults to None. No filtering will be performed.

  • time_start (datetime.datetime, optional) – Starting UTC time for the query. Defaults to None. Logs from the past 24 hours will be returned.

  • time_end (datetime.datetime, optional) – Ending UTC time for the query. Defaults to None. The current time will be used.

  • limit (int, optional) – Maximum number of records to be returned. Defaults to None. All logs will be returned.

  • sort_by (str, optional) – The field for sorting the logs. Defaults to “datetime”

  • sort_order (str, optional) – Specify how the records should be sorted. Must be “ASC” or “DESC”. Defaults to “DESC”.

  • log_filter (str, optional) – Expression for filtering the logs. This will be the WHERE clause of the query. Defaults to None.

Returns:

A list of SearchResult objects

Return type:

List[oci.loggingsearch.models.SearchResult]

property search_client

OCI logging search client.

stream(source: str | None = None, interval: int = 3, stop_condition: callable | None = None, time_start: datetime | None = None, log_filter: str | None = None)[source]

Streams logs to console/terminal until stop_condition() returns true.

Parameters:

  • source (str)

  • interval (int) – The time interval between sending each request to pull logs from OCI logging service (Default value = 3)

  • stop_condition (callable) – A function to determine if the streaming should stop. (Default value = None) The log streaming will stop if the function returns true.

  • time_start (datetime.datetime) – Starting time for the log query. Defaults to None. Logs up to 14 days from now will be returned.

  • log_filter (str, optional) – Expression for filtering the logs. This will be the WHERE clause of the query. Defaults to None.

Returns:

The number of logs printed.

Return type:

int

sync(**kwargs) → None[source]

Refreshes the properties of the Log model OCI requires both Log OCID and Log group OCID to get the Log model.

This method override the sync() method from OCIMixin to improve performance.

tail(source: str | None = None, limit=100, time_start: datetime | None = None, log_filter: str | None = None) → List[dict][source]

Returns the most recent log records.

Parameters:

  • source ((str, optional). Defaults to None.) – The source field to filter the log records.

  • limit ((int, optional). Defaults to 100.) – Maximum number of records to be returned. If limit is set to None, all logs from time_start to now will be returned.

  • time_start ((datetime.datetime, optional)) – Starting time for the log query. Defaults to None. Logs up to 14 days from now will be returned.

  • log_filter ((str, optional). Defaults to None.) – Expression for filtering the logs. This will be the WHERE clause of the query.

Returns:

A list of log records. Each log record is a dictionary with the following keys: id, time, message.

Return type:

list

class ads.common.oci_logging.OCILogGroup(config: dict | None = None, signer: Signer | None = None, client_kwargs: dict | None = None, **kwargs)[source]

Bases: OCILoggingModelMixin, LogGroup

Represents the OCI Log Group resource.

Using OCILogGroup to create a new log group. OCI requires display_name to be unique and it cannot contain space. >>> log_group = OCILogGroup(display_name=”My_Log_Group”).create() Once created, access the OCID and other properties >>> log_group.id # The OCID is None before the log is created. >>> None Create a log resource within the log group >>> log_group.id # OCID will be available once the log group is created Access the property >>> log_group.display_name Create logs within the log group >>> log = log_group.create_log(“My custom access log”) >>> log_group.create_log(“My custom prediction log”) List the logs in a log group. The following line will return a list of OCILog objects. >>> logs = log_group.list_logs() Delete the resource >>> log_group.delete()

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

  • config (dict, optional) – OCI API key config dictionary, by default None.

  • signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.

  • client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

create_async()[source]

Creates a new LogGroup asynchronously with OCI logging service

create_log(display_name: str, **kwargs)[source]

Create a log (OCI resource) within the log group.

Parameters:

  • display_name (str) – The display name of the log

  • **kwargs – Keyword arguments to be passed into the OCI API for log properties.

Returns:

An instance of OCILog

Return type:

OCILog

delete()[source]

Deletes the log group and the logs in the log group.

list_logs(**kwargs) → list[source]

Lists all logs within the log group.

Parameters:

**kwargs – keyword arguments for filtering the results. They are passed into oci.logging.LoggingManagementClient.list_logs()

Returns:

A list of OCILog

Return type:

list

class ads.common.oci_logging.OCILoggingModelMixin(config: dict | None = None, signer: Signer | None = None, client_kwargs: dict | None = None, **kwargs)[source]

Bases: OCIModelMixin, OCIWorkRequestMixin

Base model for representing OCI logging resources managed through oci.logging.LoggingManagementClient. This class should not be initialized directly. Use a sub-class (OCILogGroup or OCILog) instead.

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

  • config (dict, optional) – OCI API key config dictionary, by default None.

  • signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.

  • client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

property client: LoggingManagementClient

OCI logging management client

create()[source]

Creates a new resource with OCI service synchronously. This method will wait for the resource creation to be succeeded or failed.

Each sub-class should implement the create_async() method with the corresponding method in OCI SDK

to create the resource.

Raises:

  • NotImplementedError – when user called create but the create_async() method is not implemented.

  • oci.exceptions.RequestException – when there is an error creating the resource with OCI request.

create_async()[source]

Creates the OCI logging resource asynchronously. Sub-class should implement this method with OCI Python SDK and return the response from the OCI PythonSDK.

classmethod from_name(display_name: str, **kwargs) → OCILogGroup | OCILog[source]

Obtain an existing OCI logging resource by using its display name. OCI log group or log resource requires display name to be unique.

Parameters:

display_name (str) – Display name of the logging resource (e.g. log group)

Return type:

An instance of logging resource, e.g. OCILogGroup, or OCILog.

classmethod init_client(**kwargs) → LoggingManagementClient[source]

Initialize OCI client

class ads.common.oci_logging.SortOrder[source]

Bases: object

ASC = 'ASC'
DESC = 'DESC'

ads.common.oci_mixin module

Contains Mixins for integrating OCI data models

class ads.common.oci_mixin.MergeStrategy(value)[source]

Bases: Enum

An enumeration.

MERGE = 'merge'
OVERRIDE = 'override'
class ads.common.oci_mixin.OCIClientMixin(config=None, signer=None, client_kwargs=None)[source]

Bases: object

Mixin class for representing OCI resource/service with OCI client.

Most OCI requests requires a client for making the connection. Usually the same client will be used for the requests related the same resource/service type. This Mixin adds a “client” property to simplify accessing the client. The actual client will be initialize lazily so that it is not required for a sub-class To use the client, sub-class should override the init_client() method pass in the “client” keyword argument. For example:

@class_or_instance_method def init_client(cls, **kwargs) -> oci.logging.LoggingManagementClient:

return cls._init_client(client=oci.logging.LoggingManagementClient, **kwargs)

Instance methods in the sub-class can use self.client to access the client. The init_client() method is a class method used to create the client. Any class method using the client should use init_client() to create the client. The call to this method may come from an instance or a class. When the method is called from a class,

the default authentication configured at ADS level with ads.set_auth() will be used.

When the method is called from an instance,

the config, signer and kwargs specified when initializing the instance will be used.

The sub-class’s __init__ method should take config, signer and client_kwargs as argument,

then pass them to the __init__ method of this class.

This allows users to override the authentication and client initialization parameters. For example, log_group = OCILogGroup(config=config, signer=signer, client_kwargs=kwargs)

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

  • config (dict, optional) – OCI API key config dictionary, by default None.

  • signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.

  • client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

property auth: dict

The ADS authentication config used to initialize the client. This auth has the same format as those obtained by calling functions in ads.common.auth. The config is a dict containing the following key-value pairs: config: The config contains the config loaded from the configuration loaded from oci_config. signer: The signer contains the signer object created from the api keys. client_kwargs: client_kwargs contains the client_kwargs that was passed in as input parameter.

property client

OCI client

config = None
classmethod create_instance(*args, **kwargs)[source]

Creates an instance using the same authentication as the class or an existing instance. If this method is called by a class, the default ADS authentication method will be used. If this method is called by an instance, the authentication method set in the instance will be used.

classmethod init_client(**kwargs)[source]

Initializes the OCI client specified in the “client” keyword argument Sub-class should override this method and call cls._init_client(client=OCI_CLIENT)

Parameters:

**kwargs – Additional keyword arguments for initializing the OCI client.

Return type:

An instance of OCI client.

kwargs = None
signer = None
class ads.common.oci_mixin.OCIModelMixin(config: dict | None = None, signer: Signer | None = None, client_kwargs: dict | None = None, **kwargs)[source]

Bases: OCISerializableMixin

Mixin class to operate OCI model. OCI resources are represented by models in the OCI Python SDK.

Unifying OCI models for the same resource

OCI SDK uses different models to represent the same resource for different operations. For example, CreateLogDetails is used when creating a log resource,

while LogSummary is returned when listing the log resources.

However, both CreateLogDetails and LogSummary have the same commonly used attribute like

compartment_id, display_name, log_type, etc.

In general, there is a class with a super set of all properties. For example, the Log class contains all properties of CreateLogDetails and LogSummary,

as well as other classes representing an OCI log resource.

A subclass can be implemented with this Mixin to unify the OCI models,

so that all properties are available to the user.

For example, if we define the Mixin model as class OCILog(OCIModelMixin, oci.logging.models.Log),

users will be able to access properties like OCILog().display_name

Since this sub-class contains all the properties, it can be converted to any related OCI model in an operation. For example, we can create CreateLogDetails from OCILog by extracting a subset of the properties. When listing the resources, properties from LogSummary can be used to update

the corresponding properties of OCILog.

Such convertion can be done be the generic methods provided by this Mixin. Although OCI SDK accepts dictionary (JSON) data instead of objects like CreateLogDetails when creating or

updating the resource, the structure and keys of the dictionary is not easy for a user to remember.

It is also unnecessary for the users to construct the entire dictionary if they only want to update a single value.

This Mixin class should be the first parent as the class from OCI SDK does not call super().__init__()

in its __init__() constructor.

Mixin properties may not be intialized correctly if super().__init__() is not called.

Provide generic methods for CRUDL operations

Since OCI SDK uses different models in CRUDL operations,

this Mixin provides the following method to convert between them.

An OCI model instance can be any OCI model of the resource containing some properties, e.g. LogSummary from_oci_model() static method can be used to initialize a new instance from an OCI model instance. update_from_oci_model() can be used to update the existing properties from an OCI model instance. to_oci_model() can be used to extract properties from the Mixin model to OCI model.

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

param config:

OCI API key config dictionary, by default None.

type config:

dict, optional

param signer:

OCI authentication signer, by default None.

type signer:

oci.signer.Signer, optional

param client_kwargs:

Additional keyword arguments for initializing the OCI client.

type client_kwargs:

dict, optional

CONS_COMPARTMENT_ID = 'compartment_id'
OCI_MODEL_PATTERN = '(oci|feature_store_client).[^.]+\\.models[\\..*]?'
static check_compartment_id(compartment_id: str | None) → str[source]
Checks if a compartment ID has value and

return the value from NB_SESSION_COMPARTMENT_OCID environment variable if it is not specified.

Parameters:

compartment_id (str) – Compartment OCID or None

Returns:

str: Compartment OCID

Return type:

type

Raises:

ValueError – compartment_id is not specified and NB_SESSION_COMPARTMENT_OCID environment variable is not set

delete()[source]

Deletes the resource

classmethod deserialize(data: dict, to_cls: str | None = None)[source]

Deserialize data

Parameters:

  • data (dict) – A dictionary containing the data to be deserialized.

  • to_cls (str) – The name of the OCI model class to be initialized using the data. The OCI model class must be from the same OCI service of the OCI client (self.client). Defaults to None, the parent OCI model class name will be used if current class is inherited from an OCI model. If parent OCI model class is not found or not from the same OCI service, the data will be returned as is.

static flatten(data: dict) → dict[source]

Flattens a nested dictionary.

Parameters:

data (A nested dictionary)

Returns:

The flattened dictionary.

Return type:

dict

classmethod from_dict(data)[source]

Initialize an instance from a dictionary.

Parameters:

data (dict) – A dictionary containing the properties to initialize the class.

classmethod from_oci_model(oci_instance)[source]

Initialize an instance from an instance of OCI model.

Parameters:

oci_instance – An instance of an OCI model.

classmethod from_ocid(ocid: str)[source]

Initializes an object from OCID

Parameters:

ocid (str) – The OCID of the object

classmethod list_resource(compartment_id: str | None = None, limit: int = 0, **kwargs) → list[source]

Generic method to list OCI resources

Parameters:

  • compartment_id (str) – Compartment ID of the OCI resources. Defaults to None. If compartment_id is not specified, the value of NB_SESSION_COMPARTMENT_OCID in environment variable will be used.

  • limit (int) – The maximum number of items to return. Defaults to 0, All items will be returned

  • **kwargs – Additional keyword arguments to filter the resource. The kwargs are passed into OCI API.

Returns:

A list of OCI resources

Return type:

list

Raises:

NotImplementedError – List method is not supported or implemented.

load_properties_from_env()[source]

Loads properties from the environment

property name: str

Gets the name of the object.

property status: str | None

Status of the object.

Returns:

Status of the object.

Return type:

str

sync(merge_strategy: MergeStrategy = MergeStrategy.OVERRIDE)[source]

Refreshes the properties of the object from OCI

to_dict(flatten: bool = False) → dict[source]

Converts the properties to a dictionary

Parameters:

flatten – (Default value = False)

to_oci_model(oci_model)[source]

Converts the object into an instance of OCI data model.

Parameters:

  • oci_model (class or str) – The OCI model to be converted to. This can be a string of the model name.

  • type_mapping (dict) – A dictionary mapping the models. Returns: An instance of the oci_model

to_yaml() → str[source]

Serializes the object into YAML string.

Returns:

YAML stored in a string.

Return type:

str

update_from_oci_model(oci_model_instance, merge_strategy: MergeStrategy = MergeStrategy.OVERRIDE)[source]

Updates the properties from OCI model with the same properties.

Parameters:

oci_model_instance – An instance of OCI model, which should have the same properties of this class.

exception ads.common.oci_mixin.OCIModelNotExists[source]

Bases: Exception

class ads.common.oci_mixin.OCIModelWithNameMixin[source]

Bases: object

Mixin class to operate OCI model which contains name property.

classmethod from_name(name: str, compartment_id: str | None = None)[source]

Initializes an object from name.

Parameters:

  • name (str) – The name of the object.

  • compartment_id ((str, optional). Defaults to None.) – Compartment OCID of the OCI resources. If compartment_id is not specified, the value will be taken from environment variables.

class ads.common.oci_mixin.OCISerializableMixin(config=None, signer=None, client_kwargs=None)[source]

Bases: OCIClientMixin

Mixin class containing OCI serialization/de-serialization methods. These methods are copied and modified from the OCI BaseClient.

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

  • config (dict, optional) – OCI API key config dictionary, by default None.

  • signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.

  • client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

classmethod deserialize(data, to_cls)[source]

De-serialize data from dictionary to an OCI model

serialize()[source]

Serialize the model to a dictionary that is ready to be send to OCI API.

Returns:

A dictionary that is ready to be send to OCI API.

Return type:

dict

type_mappings = None
class ads.common.oci_mixin.OCIWorkRequestMixin[source]

Bases: object

Mixin class containing methods related to OCI work request

get_work_request_response(response: str, wait_for_state: str | tuple, success_state: str, max_wait_seconds: int | None = None, wait_interval_seconds: int | None = None, error_msg: str = '')[source]
wait_for_work_request(work_request_id: str, wait_for_state: str | tuple, max_wait_seconds: int | None = None, wait_interval_seconds: int | None = None)[source]

Wait for a work request to be completed.

Parameters:

  • work_request_id (str) – OCI work request ID

  • wait_for_state (str or tuple) – The state to wait for. Must be a tuple for multiple states.

  • max_wait_seconds (int) – Max wait seconds for the work request. Defaults to None (Default value from OCI SDK will be used).

  • wait_interval_seconds (int) – Interval in seconds between each status check. Defaults to None (Default value from OCI SDK will be used).

Returns:

OCI API Response

Return type:

Response

ads.common.oci_resource module

Contains class wrapping OCI resource search service

class ads.common.oci_resource.OCIResource(config=None, signer=None, client_kwargs=None)[source]

Bases: OCIClientMixin

Contains helper methods for getting information from OCIResourceSearch service.

Usage: Find the compartment ID of an OCI resource: >>> OCIResource.get_compartment_id(“YOUR_OCID”) Search for OCI resource matching free text (Similar to the search box in OCI console): >>> OCIResource.search(“YOUR_FREE_TEXT”) Search for OCI resource matching structured text: >>> OCIResource.search(“STRUCTURED_TEXT”, type=”Structured”)

Initializes a service/resource with OCI client as a property. If config or signer is specified, it will be used to initialize the OCI client. If neither of them is specified, the client will be initialized with ads.common.auth.default_signer. If both of them are specified, both of them will be passed into the OCI client,

and the authentication will be determined by OCI Python SDK.

Parameters:

  • config (dict, optional) – OCI API key config dictionary, by default None.

  • signer (oci.signer.Signer, optional) – OCI authentication signer, by default None.

  • client_kwargs (dict, optional) – Additional keyword arguments for initializing the OCI client.

classmethod get_compartment_id(ocid) → str[source]

Gets the compartment OCID of an OCI resource, given the resource’s OCID.

Parameters:

ocid (str) – OCID of a resource

Returns:

Compartment OCID of the resource

Return type:

str

classmethod init_client(**kwargs) → ResourceSearchClient[source]

Initializes the OCI client specified in the “client” keyword argument Sub-class should override this method and call cls._init_client(client=OCI_CLIENT)

Parameters:

**kwargs – Additional keyword arguments for initializing the OCI client.

Return type:

An instance of OCI client.

classmethod search(query: str, type: str = 'FreeText', config: dict | None = None, tenant_id: str | None = None, limit: int = 500, page: str | None = None, **kwargs) → list[source]

Search OCI resource by free text.

Parameters:

  • query (str) – The content to search for.

  • type (str (optional)) – The type of SearchDetails, whether “FreeText” or “Structured”. Defaults to “FreeText”.

  • config (dict (optional)) – Configuration keys and values as per SDK and Tool Configuration. The from_file() method can be used to load configuration from a file. Alternatively, a dict can be passed. You can validate_config the dict using validate_config(). Defaults to None.

  • tenant_id (str (optional)) – The tenancy ID, which can be used to specify a different tenancy (for cross-tenancy authorization) when searching for resources in a different tenancy. Defaults to None.

  • limit (int (optional)) – The maximum number of items to return. The value must be between 1 and 1000. Defaults to 500.

  • page (str (optional)) – The page at which to start retrieving results.

Returns:

A list of search results

Return type:

list

exception ads.common.oci_resource.ResourceNotFoundError[source]

Bases: Exception

Exception when an OCI resource is not found or user does not have permission to access it. This could mean the resource does not exist, or there is not enough permission to find/access the resource.

class ads.common.oci_resource.SEARCH_TYPE[source]

Bases: str

FREETEXT = 'FreeText'
STRUCTURED = 'Structured'

ads.common.serializer module

This module provides a base class for serializable items, as well as methods for serializing and deserializing objects to and from JSON and YAML formats. It also includes methods for reading and writing serialized objects to and from files.

class ads.common.serializer.DataClassSerializable[source]

Bases: Serializable

Wrapper class that inherit from Serializable class.

to_dict(self) → dict[source]

Serializes the object into a dictionary.

from_dict(cls, obj_dict) → cls[source]

Returns an instance of the class instantiated from the dictionary provided.

classmethod from_dict(obj_dict: dict, side_effect: SideEffect | None = 'lower', ignore_unknown: bool | None = False, **kwargs) → DataClassSerializable[source]

Returns an instance of the class instantiated by the dictionary provided.

Parameters:

  • obj_dict ((dict)) – Dictionary representation of the object

  • side_effect (Optional[SideEffect]) – side effect to take on the dictionary. The side effect can be either convert the dictionary keys to “lower” (SideEffect.CONVERT_KEYS_TO_LOWER.value) or “upper”(SideEffect.CONVERT_KEYS_TO_UPPER.value) cases.

  • ignore_unknown ((bool, optional). Defaults to False.) – Whether to ignore unknown fields or not.

Returns:

A DataClassSerializable instance.

Return type:

DataClassSerializable

to_dict(**kwargs) → Dict[source]

Serializes instance of class into a dictionary

kwargs

side_effect: Optional[SideEffect]

side effect to take on the dictionary. The side effect can be either convert the dictionary keys to “lower” (SideEffect.CONVERT_KEYS_TO_LOWER.value) or “upper”(SideEffect.CONVERT_KEYS_TO_UPPER.value) cases.

returns:

A dictionary.

rtype:

Dict

class ads.common.serializer.Serializable[source]

Bases: ABC

Base class that represents a serializable item.

to_dict(self) → dict[source]

Serializes the object into a dictionary.

from_dict(cls, obj_dict) → cls[source]

Returns an instance of the class instantiated from the dictionary provided.

_write_to_file(s, uri, \*\*kwargs)[source]

Write string s into location specified by uri

_read_from_file(uri, \*\*kwargs)[source]

Returns contents from location specified by URI

to_json(self, uri=None, \*\*kwargs)[source]

Returns object serialized as a JSON string

from_json(cls, json_string=None, uri=None, \*\*kwargs)[source]

Creates an object from JSON string provided or from URI location containing JSON string

to_yaml(self, uri=None, \*\*kwargs)[source]

Returns object serialized as a YAML string

from_yaml(cls, yaml_string=None, uri=None, \*\*kwargs)[source]

Creates an object from YAML string provided or from URI location containing YAML string

from_string(cls, obj_string=None: str, uri=None, \*\*kwargs)[source]

Creates an object from string provided or from URI location containing string

abstract classmethod from_dict(obj_dict: dict, **kwargs) → Serializable[source]

Returns an instance of the class instantiated by the dictionary provided.

Parameters:

obj_dict ((dict)) – Dictionary representation of the object.

Returns:

A Serializable instance.

Return type:

Serializable

classmethod from_json(json_string: str | None = None, uri: str | None = None, decoder: callable = <class 'json.decoder.JSONDecoder'>, **kwargs)[source]

Creates an object from JSON string provided or from URI location containing JSON string

Parameters:

  • json_string ((string, optional)) – JSON string. Defaults to None.

  • uri ((string, optional)) – URI location of file containing JSON string. Defaults to None.

  • decoder ((callable, optional)) – Custom decoder. Defaults to simple JSON decoder.

  • kwargs

  • ------

  • storage (keyword arguments to be passed into fsspec.open(). For OCI object) – For other storage connections consider e.g. host, port, username, password, etc.

  • config="path/to/.oci/config". (this should be) – For other storage connections consider e.g. host, port, username, password, etc.

Raises:

ValueError – Raised if neither string nor uri is provided

Returns:

Returns instance of the class

Return type:

cls

classmethod from_string(obj_string: str | None = None, uri: str | None = None, loader: callable = <class 'yaml.cyaml.CSafeLoader'>, **kwargs) → Serializable[source]

Creates an object from string provided or from URI location containing string

Parameters:

  • obj_string ((str, optional)) – String representing the object

  • uri ((str, optional)) – URI location of file containing string. Defaults to None.

  • loader ((callable, optional)) – Custom YAML loader. Defaults to CLoader/SafeLoader.

  • kwargs ((dict)) – keyword arguments to be passed into fsspec.open(). For OCI object storage, this should be config=”path/to/.oci/config”. For other storage connections consider e.g. host, port, username, password, etc.

Returns:

A Serializable instance

Return type:

Serializable

classmethod from_yaml(yaml_string: str | None = None, uri: str | None = None, loader: callable = <class 'yaml.cyaml.CSafeLoader'>, **kwargs)[source]

Creates an object from YAML string provided or from URI location containing YAML string

Parameters:

  • (string (uri) – YAML string. Defaults to None.

  • optional) – YAML string. Defaults to None.

  • (string – URI location of file containing YAML string. Defaults to None.

  • optional) – URI location of file containing YAML string. Defaults to None.

  • (callable (loader) – Custom YAML loader. Defaults to CLoader/SafeLoader.

  • optional) – Custom YAML loader. Defaults to CLoader/SafeLoader.

  • (dict) (kwargs) – keyword arguments to be passed into fsspec.open(). For OCI object storage, this should be config=”path/to/.oci/config”. For other storage connections consider e.g. host, port, username, password, etc.

Raises:

ValueError – Raised if neither string nor uri is provided

Returns:

Returns instance of the class

Return type:

cls

static serialize(obj)[source]

JSON serializer for objects not serializable by default json code.

abstract to_dict(**kwargs: Dict) → Dict[source]

Serializes an instance of class into a dictionary.

Parameters:

**kwargs (Dict) – Additional arguments.

Returns:

The result dictionary.

Return type:

Dict

to_json(uri: str | None = None, encoder: callable = <class 'json.encoder.JSONEncoder'>, default: callable | None = None, **kwargs) → str[source]

Returns object serialized as a JSON string

Parameters:

  • uri ((string, optional)) – URI location to save the JSON string. Defaults to None.

  • encoder ((callable, optional)) – Encoder for custom data structures. Defaults to JSONEncoder.

  • default ((callable, optional)) – A function that gets called for objects that can’t otherwise be serialized. It should return JSON-serializable version of the object or original object.

  • kwargs

  • ------

  • overwrite ((bool, optional). Defaults to True.) – Whether to overwrite existing file or not.

  • fsspec.open(). (keyword arguments to be passed into)

  • storage (For OCI object)

  • config="path/to/.oci/config". (this could be)

  • host (For other storage connections consider e.g.)

  • port

  • username

  • password

  • etc.

Returns:

Serialized version of object. None in case when uri provided.

Return type:

Union[str, None]

to_yaml(uri: str | None = None, dumper: callable = <class 'yaml.dumper.SafeDumper'>, **kwargs) → str | None[source]

Returns object serialized as a YAML string

Parameters:

  • uri (str, optional) – URI location to save the YAML string, by default None

  • dumper (callable, optional) – Custom YAML Dumper, by default yaml.SafeDumper

  • kwargs (dict) –

    overwrite: (bool, optional). Defaults to True.

    Whether to overwrite existing file or not.

    note: (str, optional)

    The note that needs to be added in the beginning of the YAML. It will be added as is without any formatting.

    side_effect: Optional[SideEffect]

    side effect to take on the dictionary. The side effect can be either convert the dictionary keys to “lower” (SideEffect.CONVERT_KEYS_TO_LOWER.value) or “upper”(SideEffect.CONVERT_KEYS_TO_UPPER.value) cases.

    The other keyword arguments to be passed into fsspec.open(). For OCI object storage, this could be config=”path/to/.oci/config”.

Returns:

Serialized version of object. None in case when uri provided.

Return type:

Union[str, None]

class ads.common.serializer.SideEffect(value)[source]

Bases: Enum

An enumeration.

CONVERT_KEYS_TO_LOWER = 'lower'
CONVERT_KEYS_TO_UPPER = 'upper'

ads.common.utils module

exception ads.common.utils.FileOverwriteError[source]

Bases: Exception

class ads.common.utils.JsonConverter(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]

Bases: JSONEncoder

Constructor for JSONEncoder, with sensible defaults.

If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, float or None. If skipkeys is True, such items are simply skipped.

If ensure_ascii is true, the output is guaranteed to be str objects with all incoming non-ASCII characters escaped. If ensure_ascii is false, the output can contain non-ASCII characters.

If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an RecursionError). Otherwise, no such check takes place.

If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.

If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.

If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation.

If specified, separators should be an (item_separator, key_separator) tuple. The default is (’, ‘, ‘: ‘) if indent is None and (‘,’, ‘: ‘) otherwise. To get the most compact JSON representation, you should specify (‘,’, ‘:’) to eliminate whitespace.

If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a TypeError.

default(obj)[source]

Converts an object to JSON based on its type

Parameters:

obj (Object) – An object which is being converted to Json, supported types are pandas Timestamp, series, dataframe, or categorical or numpy ndarrays.

Returns:

Json

Return type:

A json repersentation of the object.

ads.common.utils.batch_convert_case(spec: dict, to_fmt: str) → Dict[source]

Convert the case of a dictionary of spec from camel to snake or vice versa.

Parameters:

  • spec (dict) – dictionary of spec to convert

  • to_fmt (str) – format to convert to, can be “camel” or “snake”

Returns:

dictionary of converted spec

Return type:

dict

ads.common.utils.camel_to_snake(name: str) → str[source]

Converts the camel case string to the snake representation.

Parameters:

name (str) – The name to convert.

Returns:

str

Return type:

The name converted to the snake representation.

ads.common.utils.copy_file(uri_src: str, uri_dst: str, force_overwrite: bool | None = False, auth: Dict | None = None, chunk_size: int | None = 8192, progressbar_description: str | None = 'Copying `{uri_src}` to `{uri_dst}`', ignore_if_src_not_exists: bool | None = False) → str[source]

Copies file from uri_src to uri_dst. If uri_dst specifies a directory, the file will be copied into uri_dst using the base filename from uri_src. Returns the path to the newly created file.

Parameters:

  • uri_src (str) – The URI of the source file, which can be local path or OCI object storage URI.

  • uri_dst (str) – The URI of the destination file, which can be local path or OCI object storage URI.

  • force_overwrite ((bool, optional). Defaults to False.) – Whether to overwrite existing files or not.

  • auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

  • chunk_size ((int, optional). Defaults to DEFAULT_BUFFER_SIZE) – How much data can be copied in one iteration.

  • progressbar_description ((str, optional). Defaults to “Copying `{uri_src} to {uri_dst}”`.) – Prefix for the progressbar.

Returns:

The path to the newly created file.

Return type:

str

Raises:

FileExistsError – If a destination file exists and force_overwrite set to False.

ads.common.utils.copy_from_uri(uri: str, to_path: str, unpack: bool | None = False, force_overwrite: bool | None = False, auth: Dict | None = None) → None[source]

Copies file(s) to local path. Can be a folder, archived folder or a separate file. The source files can be located in a local folder or in OCI Object Storage.

Parameters:

  • uri (str) – The URI of the source file or directory, which can be local path or OCI object storage URI.

  • to_path (str) – The local destination path. If this is a directory, the source files will be placed under it.

  • unpack ((bool, optional). Defaults to False.) – Indicate if zip or tar.gz file specified by the uri should be unpacked. This option has no effect on other files.

  • force_overwrite ((bool, optional). Defaults to False.) – Whether to overwrite existing files or not.

  • auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

Nothing

Return type:

None

Raises:

ValueError – If destination path is already exist and force_overwrite is set to False.

ads.common.utils.download_from_web(url: str, to_path: str) → None[source]

Downloads a single file from http/https/ftp.

Parameters:

  • url (str) – The URL of the source file.

  • to_path (path-like object) – Local destination path.

Returns:

Nothing

Return type:

None

ads.common.utils.ellipsis_strings(raw, n=24)[source]

takes a sequence (<string>, list(<string>), tuple(<string>), pd.Series(<string>) and Ellipsis’ize them at position n

ads.common.utils.extract_lib_dependencies_from_model(model) → dict[source]

Extract a dictionary of library dependencies for a model

Parameters:

model

Returns:

Dict

Return type:

A dictionary of library dependencies.

ads.common.utils.extract_region(auth: Dict | None = None) → str | None[source]

Extracts region information from the environment variables and signer.

Parameters:

auth (Dict) – The ADS authentication config used to initialize the client. Contains keys - config, signer and client_kwargs.

Returns:

The region identifier. For example: us-ashburn-1. Returns None if region cannot be extracted.

Return type:

Union[str, None]

ads.common.utils.first_not_none(itr)[source]

Returns the first non-none result from an iterable, similar to any() but return value not true/false

ads.common.utils.flatten(d, parent_key='')[source]

Flattens nested dictionaries to a single layer dictionary

Parameters:

  • d (dict) – The dictionary that needs to be flattened

  • parent_key (str) – Keys in the dictionary that are nested

Returns:

a_dict – a single layer dictionary

Return type:

dict

ads.common.utils.folder_size(path: str) → int[source]

Recursively calculating a size of the path folder.

Parameters:

path (str) – Path to the folder.

Returns:

The size fo the folder in bytes.

Return type:

int

ads.common.utils.generate_requirement_file(requirements: dict, file_path: str, file_name: str = 'requirements.txt')[source]

Generate requirements file at file_path.

Parameters:

  • requirements (dict) – Key is the library name and value is the version

  • file_path (str) – Directory to save requirements.txt

  • file_name (str) – Opional parameter to specify the file name

ads.common.utils.get_base_modules(model)[source]

Get the base modules from an ADS model

ads.common.utils.get_bootstrap_styles()[source]

Returns HTML bootstrap style information

ads.common.utils.get_compute_accelerator_ncores()[source]

This method returns the web console link for the given resource. :param resource: identify the type of OCI resource. {model, model-deployments, notebook-sessions, jobs} is supported. :type resource: str :param ocid: OCID of the resource :type ocid: str :param region: The Region Identifier that the client should connect to. :type region: str

Returns:

console_link_url – a valid link to the console for the given resource

Return type:

str

ads.common.utils.get_cpu_count()[source]

Returns the number of CPUs available on this machine

ads.common.utils.get_dataframe_styles(max_width=75)[source]

Styles used for dataframe, example usage:

df.style .set_table_styles(utils.get_dataframe_styles()) .set_table_attributes(‘class=table’) .render())

Returns:

styles – A list of dataframe table styler styles.

Return type:

array

ads.common.utils.get_files(directory: str, auth: Dict | None = None)[source]

List out all the file names under this directory.

Parameters:

  • directory (str) – The directory to list out all the files from.

  • auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

List of the files in the directory.

Return type:

List

This method returns the web console link for the given log ids.

Parameters:

  • log_group_id (str, required) – OCID of the resource

  • log_id (str, optional) – OCID of the resource

  • region (str) – The Region Identifier that the client should connect to.

  • compartment_id (str, optional) – The compartment OCID of the resource.

  • source_id (str, optional) – The OCID of the resource.

Returns:

console_link_url – a valid link to the console for the given resource.

Return type:

str

ads.common.utils.get_oci_config()[source]

Returns the OCI config location, and the OCI config profile.

ads.common.utils.get_progress_bar(max_progress: int, description: str = 'Initializing', verbose: bool = False) → TqdmProgressBar[source]

Returns an instance of the TqdmProgressBar class.

Parameters:

  • max_progress (int) – The number of steps for the progressbar.

  • description ((str, optional). Defaults to "Initializing".) – The first step description.

  • verbose ((bool, optional). Defaults to False) – If the progress should show the debug information.

Returns:

An instance of the TqdmProgressBar.

Return type:

TqdmProgressBar

ads.common.utils.get_random_name_for_resource() → str[source]

Returns randomly generated easy to remember name. It consists from 1 adjective and 1 animal word, tailed by UTC timestamp (joined with ‘-‘). This is an ADS default resource name generated for models, jobs, jobruns, model deployments, pipelines.

Returns:

Randomly generated easy to remember name for oci resources - models, jobs, jobruns, model deployments, pipelines. Example: polite-panther-2022-08-17-21:15.46; strange-spider-2022-08-17-23:55.02

Return type:

str

ads.common.utils.get_sqlalchemy_engine(connection_url, *args, **kwargs)[source]

The SqlAlchemny docs say to use a single engine per connection_url, this class will take care of that.

Parameters:

connection_url (string) – The URL to connect to

Returns:

engine – The engine from which SqlAlchemny commands can be ran on

Return type:

SqlAlchemny engine

ads.common.utils.get_value(obj, attr, default=None)[source]

Gets a copy of the value from a nested dictionary of an object with nested attributes.

Parameters:

  • obj – An object or a dictionary

  • attr – Attributes as a string seprated by dot(.)

  • default – Default value to be returned if attribute is not found.

Returns:

A copy of the attribute value. For dict or list, a deepcopy will be returned.

Return type:

Any

ads.common.utils.highlight_text(text)[source]

Returns text with html highlights. :param text: The text to be highlighted. :type text: String

Returns:

ht – The text with html highlight information.

Return type:

String

ads.common.utils.horizontal_scrollable_div(html)[source]

Wrap html with the necessary html to make horizontal scrolling possible.

Examples

display(HTML(utils.horizontal_scrollable_div(my_html)))

Parameters:

html (str) – Your HTML to wrap.

Returns:

Wrapped HTML.

Return type:

type

ads.common.utils.human_size(num_bytes: int, precision: int | None = 2) → str[source]

Converts bytes size to a string representing its value in B, KB, MB and GB.

Parameters:

  • num_bytes (int) – The size in bytes.

  • precision ((int, optional). Defaults to 2.) – The precision of converting the bytes value.

Returns:

A string representing the size in B, KB, MB and GB.

Return type:

str

ads.common.utils.inject_and_copy_kwargs(kwargs, **args)[source]

Takes in a dictionary and returns a copy with the args injected

Examples

>>> foo(arg1, args, utils.inject_and_copy_kwargs(kwargs, arg3=12, arg4=42))
Parameters:

  • kwargs (dict) – The original kwargs.

  • **args (type) – A series of arguments, foo=42, bar=12 etc

Returns:

d – new dictionary object that you can use in place of kwargs

Return type:

dict

ads.common.utils.is_data_too_wide(data: list | tuple | Series | ndarray | DataFrame, max_col_num: int) → bool[source]

Returns true if the data has too many columns.

Parameters:

  • data (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]) – A sample of data that will be used to generate schema.

  • max_col_num (int.) – The maximum column size of the data that allows to auto generate schema.

ads.common.utils.is_debug_mode()[source]

Returns true if ADS is in debug mode.

ads.common.utils.is_documentation_mode()[source]

Returns true if ADS is in documentation mode.

ads.common.utils.is_notebook()[source]

Returns true if the environment is a jupyter notebook.

ads.common.utils.is_path_exists(uri: str, auth: Dict | None = None) → bool[source]

Check if the given path which can be local path or OCI object storage URI exists.

Parameters:

  • uri (str) – The URI of the target, which can be local path or OCI object storage URI.

  • auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

bool

Return type:

return True if the path exists.

ads.common.utils.is_resource_principal_mode()[source]

Returns true if ADS is in resource principal mode.

ads.common.utils.is_same_class(obj, cls)[source]

checks to see if object is the same class as cls

ads.common.utils.is_test()[source]

Returns true if ADS is in test mode.

class ads.common.utils.ml_task_types(value)

Bases: Enum

An enumeration.

BINARY_CLASSIFICATION = 2
BINARY_TEXT_CLASSIFICATION = 4
MULTI_CLASS_CLASSIFICATION = 3
MULTI_CLASS_TEXT_CLASSIFICATION = 5
REGRESSION = 1
UNSUPPORTED = 6
ads.common.utils.numeric_pandas_dtypes()[source]

Returns a list of the “numeric” pandas data types

ads.common.utils.oci_config_file()[source]

Returns the OCI config file location

ads.common.utils.oci_config_location()[source]

Returns oci configuration file location.

ads.common.utils.oci_config_profile()[source]

Returns the OCI config profile location.

ads.common.utils.oci_key_location()[source]

Returns the OCI key location

ads.common.utils.oci_key_profile()[source]

Returns key profile value specified in oci configuration file.

ads.common.utils.print_user_message(msg, display_type='tip', see_also_links=None, title='Tip')[source]

This method is deprecated and will be removed in future releases. Prints in html formatted block one of tip|info|warn type.

Parameters:

  • msg (str or list) – The actual message to display. display_type is “module’, msg can be a list of [module name, module package name], i.e. [“automl”, “ads[ml]”]

  • display_type (str (default 'tip')) – The type of user message.

  • see_also_links (list of tuples in the form of [('display_name', 'url')])

  • title (str (default 'tip')) – The title of user message.

ads.common.utils.random_valid_ocid(prefix='ocid1.dataflowapplication.oc1.iad')[source]

Generates a random valid ocid.

Parameters:

prefix (str) – A prefix, corresponding to a region location.

Returns:

ocid – a valid ocid with the given prefix.

Return type:

str

ads.common.utils.remove_file(file_path: str, auth: Dict | None = None) → None[source]

Reoves file.

Parameters:

  • file_path (str) – The path of the source file, which can be local path or OCI object storage URI.

  • auth ((Dict, optional). Defaults to None.) – The default authentication is set using ads.set_auth API. If you need to override the default, use the ads.common.auth.api_keys or ads.common.auth.resource_principal to create appropriate authentication signer and kwargs required to instantiate IdentityClient object.

Returns:

Nothing.

Return type:

None

ads.common.utils.replace_spaces(lst)[source]

Replace all spaces with underscores for strings in the list.

Requires that the list contains strings for each element.

lst: list of strings

ads.common.utils.set_oci_config(oci_config_location, oci_config_profile)[source]
Parameters:

  • oci_config_location – location of the config file, for example, ~/.oci/config

  • oci_config_profile – The profile to load from the config file. Defaults to “DEFAULT”

ads.common.utils.snake_to_camel(name: str, capitalized_first_token: bool | None = False) → str[source]

Converts the snake case string to the camel representation.

Parameters:

  • name (str) – The name to convert.

  • capitalized_first_token ((bool, optional). Defaults to False.) – Wether the first token needs to be capitalized or not.

Returns:

str

Return type:

The name converted to the camel representation.

ads.common.utils.split_data(X, y, random_state=42, test_size=0.3)[source]

Splits data using Sklearn based on the input type of the data.

Parameters:

  • X (a Pandas Dataframe) – The data points.

  • y (a Pandas Dataframe) – The labels.

  • random_state (int) – A random state for reproducability.

  • test_size (int) – The number of elements that should be included in the test dataset.

ads.common.utils.to_dataframe(data: list | tuple | Series | ndarray | DataFrame)[source]

Convert to pandas DataFrame.

Parameters:

data (Union[list, tuple, pd.Series, np.ndarray, pd.DataFrame]) – Convert data to pandas DataFrame.

Returns:

pandas DataFrame.

Return type:

pd.DataFrame

ads.common.utils.truncate_series_top_n(series, n=24)[source]

take a series which can be interpreted as a dict, index=key, this function sorts by the values and takes the top-n values, and returns a new series

ads.common.utils.upload_to_os(src_uri: str, dst_uri: str, auth: dict | None = None, parallel_process_count: int = 9, progressbar_description: str = 'Uploading `{src_uri}` to `{dst_uri}`.', force_overwrite: bool = False)[source]

Utilizes oci.object_storage.Uploadmanager to upload file to Object Storage.

Parameters:

  • src_uri (str) – The path to the file to upload. This should be local path.

  • dst_uri (str) – Object Storage path, eg. oci://my-bucket@my-tenancy/prefix`.

  • auth ((Dict, optional) Defaults to None.) – default_signer()

  • parallel_process_count ((int, optional) Defaults to 3.) – The number of worker processes to use in parallel for uploading individual parts of a multipart upload.

  • progressbar_description ((str, optional) Defaults to “Uploading `{src_uri} to {dst_uri}”`.) – Prefix for the progressbar.

  • force_overwrite ((bool, optional). Defaults to False.) – Whether to overwrite existing files or not.

Returns:

Response – The response from multipart commit operation or the put operation.

Return type:

oci.response.Response

Raises:
ads.common.utils.wrap_lines(li, heading='')[source]

Wraps the elements of iterable into multi line string of fixed width

ads.common.word_lists module

Module contents