ads.opctl.distributed package

Subpackages

• ads.opctl.distributed.common package
  • Submodules
  • ads.opctl.distributed.common.abstract_cluster_provider module
    • ClusterProvider
      • ClusterProvider.DEFAULT_CODE_DIR
      • ClusterProvider.SYNC_SCRIPT_PATH
      • ClusterProvider.basic_configuration()
      • ClusterProvider.check_cluster_status()
      • ClusterProvider.configuration()
      • ClusterProvider.create_sync_script()
      • ClusterProvider.execution_failed()
      • ClusterProvider.expected_worker_count()
      • ClusterProvider.export_config_files()
      • ClusterProvider.export_configuration()
      • ClusterProvider.fetch_all_worker_info()
      • ClusterProvider.fetch_code()
      • ClusterProvider.find_self_ip()
      • ClusterProvider.get_oci_auth()
      • ClusterProvider.get_sync_loc()
      • ClusterProvider.get_sync_script()
      • ClusterProvider.profile_cmd()
      • ClusterProvider.reached_endoflife()
      • ClusterProvider.ready()
      • ClusterProvider.run_code()
      • ClusterProvider.setup_configuration()
      • ClusterProvider.setup_extra_configs()
      • ClusterProvider.start()
      • ClusterProvider.start_main()
      • ClusterProvider.start_ps()
      • ClusterProvider.start_worker()
      • ClusterProvider.stop()
      • ClusterProvider.stop_filesystem
      • ClusterProvider.sync()
      • ClusterProvider.tearable()
      • ClusterProvider.when_ready()
  • ads.opctl.distributed.common.abstract_framework_spec_builder module
    • AbstractFrameworkSpecBuilder
      • AbstractFrameworkSpecBuilder.update()
    • update_env_vars()
  • ads.opctl.distributed.common.cluster_config_helper module
    • ClusterConfigToJobSpecConverter
      • ClusterConfigToJobSpecConverter.job_def_info()
      • ClusterConfigToJobSpecConverter.job_run_info()
  • ads.opctl.distributed.common.cluster_provider_factory module
    • ClusterProviderFactory
      • ClusterProviderFactory.get_provider()
      • ClusterProviderFactory.provider
      • ClusterProviderFactory.register()
  • ads.opctl.distributed.common.cluster_runner module
    • ClusterRunner
      • ClusterRunner.run()
  • ads.opctl.distributed.common.framework_factory module
  • Module contents

Submodules

ads.opctl.distributed.certificates module

ads.opctl.distributed.certificates.download_certificates(auth, ca_file_name, cert_file_name, key_file_name)

ads.opctl.distributed.cli module

ads.opctl.distributed.cmds module

ads.opctl.distributed.cmds.cancel_distributed_run(job_id, cluster_file_name, **kwargs)

ads.opctl.distributed.cmds.dask_cmd(code_mount, oci_key_mount, config)

CLI command to run Dask distributed framework on local

Parameters:

• code_mount – source_folder to be mounted

• oci_key_mount – oci keys to be mounted

• config – Job config

• Returns – List: CLI command

• -------

ads.opctl.distributed.cmds.dev_mode_base_uri(cluster_type, version)

Temporary method to mock initialization using bucket.

ads.opctl.distributed.cmds.docker_build_cmd(ini)

Builds the docker image

Parameters:

• ini – config.ini file dictionary

• Returns – None

• -------

ads.opctl.distributed.cmds.get_cmd(ini)

Docker build command

Parameters:

• ini – config.ini file dictionary

• Returns – Docker build command

• -------

ads.opctl.distributed.cmds.horovod_cmd(code_mount, oci_key_mount, config)

CLI command to run horovod distributed framework on local

Parameters:

• code_mount – source_folder to be mounted

• oci_key_mount – oci keys to be mounted

• config – Job config

• Returns – List: CLI command

• -------

ads.opctl.distributed.cmds.increment_tag(ini)

Increments the tag of the image

Parameters:

• ini – config.ini file dictionary

• Returns – updated ini dict

• -------

ads.opctl.distributed.cmds.increment_tag_in_ini(ini)

increments tag of image and update latest in config.ini file

Parameters:

• ini – config.ini file dictionary

• Returns – updated config.ini file

• -------

ads.opctl.distributed.cmds.initialize_workspace(cluster_type, version)

ads.opctl.distributed.cmds.load_ini()

Loads config.ini from local in dict Returns

None

---

ads.opctl.distributed.cmds.local_run(config, ini)

Local run distributed framework on local based on input args

Parameters:

• ini – config.ini file dictionary

• config – Job run config

• Returns – None

• -------

ads.opctl.distributed.cmds.production_mode_base_uri(cluster_type, version)

ads.opctl.distributed.cmds.pytorch_cmd(code_mount, oci_key_mount, config)

CLI command to run Pytorch distributed framework on local

Parameters:

• code_mount – source_folder to be mounted

• oci_key_mount – oci keys to be mounted

• config – Job config

• Returns – List: CLI command

• -------

ads.opctl.distributed.cmds.refresh_ini(ini, ini_file)

write updated config.ini in local

Parameters:

• ini – config.ini file dictionary

• ini_file – Name of the config file

ads.opctl.distributed.cmds.run_cmd(cmd)

Run cli command :param cmd: input cli command :param Returns: None :param ——-:

ads.opctl.distributed.cmds.show_config_info(job_id, work_dir, cluster_file_name, worker_info, **kwargs)

ads.opctl.distributed.cmds.tensorflow_cmd(code_mount, oci_key_mount, config)

CLI command to run Tensorflow distributed framework on local

Parameters:

• code_mount – source_folder to be mounted

• oci_key_mount – oci keys to be mounted

• config – Job config

• Returns – List: CLI command

• -------

ads.opctl.distributed.cmds.update_config_image(config)

updates image name in config

Parameters:

• config – Job config

• Returns – updated config dict

• -------

ads.opctl.distributed.cmds.update_image(config, ini)

Update Image name in job config with latest build image

Parameters:

• config – Job run config

• ini – config.ini file dictionary

• Returns – Updated Job run config

• -------

ads.opctl.distributed.cmds.update_ini(tag, registry, dockerfile, source_folder, config, nobuild)

Stores and updates input args in config.ini in Local

Parameters:

• tag – tag of image

• registry – registry to push to

• dockerfile – relative path to Dockerfile

• source_folder – source scripts folder that will be mounted during a local run

• config – Job config

• nobuild – flag for building the docker

• Returns – dict of config.ini file

• -------

ads.opctl.distributed.cmds.verify_and_publish_image(nopush, config)

verify existence of docker image in OCI registry and publish it :param nopush: Doesn’t push image to OCI registry :param config: Job run config :param Returns: None :param ——-:

ads.opctl.distributed.cmds.verify_image(img_name)

Verify if the input image exists in OCI registry

Parameters:

• img_name – Name of the docker image

• Returns – None

• -------

Module contents