YAML Schema#

ADS Job#

job schema

Key Value
kind string Must be job
apiVersion string
spec dict See job.spec schema.

job.spec schema

Key Value
id string
infrastructure dict See Data Science Job or Data Flow schema.
name string
runtime dict See Job Runtime Schemas.

Following is the YAML schema for validating the YAML using Cerberus:

 1kind:
 2  allowed:
 3    - job
 4  required: true
 5  type: string
 6apiVersion:
 7  nullable: true
 8  required: false
 9  type: string
10spec:
11  type: dict
12  required: true
13  schema:
14    id:
15      type: string
16    infrastructure:
17      type: dict
18      meta: See Data Science Job or Data Flow schema.
19    name:
20      required: false
21      type: string
22    runtime:
23      type: dict
24      meta: See Job Runtime Schemas.

Data Science Job#

infrastructure schema

Key Value
kind string Must be infrastructure
type string Must be dataScienceJob
spec dict See infrastructure.spec schema.

infrastructure.spec schema

Key Value
blockStorageSize integer Minimum: 50
compartmentId string
displayName string
id string
logGroupId string
logId string
projectId string
shapeName string
subnetId string
shapeConfigDetails dict See infrastructure.spec.shapeConfigDetails schema.

infrastructure.spec.shapeConfigDetails schema

Key Value
memoryInGBs string
ocpus integer

Following is the YAML schema for validating the YAML using Cerberus:

 1kind:
 2  required: true
 3  type: string
 4  allowed:
 5    - infrastructure
 6type:
 7  required: true
 8  type: string
 9  allowed:
10    - dataScienceJob
11spec:
12  required: true
13  type: dict
14  schema:
15    blockStorageSize:
16      default: 50
17      min: 50
18      required: false
19      type: integer
20    compartmentId:
21      required: false
22      type: string
23    displayName:
24      required: false
25      type: string
26    id:
27      required: false
28      type: string
29    logGroupId:
30      required: false
31      type: string
32    logId:
33      required: false
34      type: string
35    projectId:
36      required: false
37      type: string
38    shapeName:
39      required: false
40      type: string
41    subnetId:
42      required: false
43      type: string
44    shapeConfigDetails:
45      required: false
46      type: dict
47      schema:
48        memoryInGBs:
49          type: string
50        ocpus:
51          type: integer

Data Flow#

infrastructure schema

Key Value
kind string Must be infrastructure
type string Must be dataFlow
spec dict See infrastructure.spec schema.

infrastructure.spec schema

Key Value
compartmentId string
displayName string
driverShape string
executorShape string
id string
language string
logsBucketUri string
metastoreId string
numExecutors integer
sparkVersion string

Following is the YAML schema for validating the YAML using Cerberus:

 1kind:
 2  allowed:
 3    - infrastructure
 4  required: true
 5  type: string
 6type:
 7  required: true
 8  type: string
 9  allowed:
10    - dataFlow
11spec:
12  type: dict
13  required: true
14  schema:
15    compartmentId:
16      type: string
17    displayName:
18      type: string
19    driverShape:
20      type: string
21    executorShape:
22      type: string
23    id:
24      type: string
25    language:
26      type: string
27    logsBucketUri:
28      type: string
29    metastoreId:
30      type: string
31    numExecutors:
32      type: integer
33    sparkVersion:
34      type: string

Job Runtime Schemas#

The runtime of a job can be one of the following.

Python Runtime#

runtime schema

Key Value
kind string Must be runtime
type string Must be script
spec dict See runtime.spec schema.

runtime.spec schema

Key Value
args list List of string.
conda dict See runtime.spec.conda schema.
env list List of dict. For each element, see runtime.spec.env schema.
scriptPathURI string
entrypoint string
outputDir string
outputUri string
workingDir string
pythonPath list List of string.

runtime.spec.conda schema

Key Value
slug string
type string Must be service

runtime.spec.env schema

Key Value
name string
value number or string

Following is the YAML schema for validating the YAML using Cerberus:

 1kind:
 2  required: true
 3  type: string
 4  allowed:
 5    - runtime
 6type:
 7  required: true
 8  type: string
 9  allowed:
10    - script
11spec:
12  required: true
13  type: dict
14  schema:
15    args:
16      nullable: true
17      required: false
18      type: list
19      schema:
20        type: string
21    conda:
22      nullable: false
23      required: false
24      type: dict
25      schema:
26        slug:
27          required: true
28          type: string
29        type:
30          allowed:
31            - service
32          required: true
33          type: string
34    env:
35      nullable: true
36      required: false
37      type: list
38      schema:
39        type: dict
40        schema:
41          name:
42            type: string
43          value:
44            type:
45              - number
46              - string
47    scriptPathURI:
48      required: true
49      type: string
50    entrypoint:
51      required: false
52      type: string
53    outputDir:
54      required: false
55      type: string
56    outputUri:
57      required: false
58      type: string
59    workingDir:
60      required: false
61      type: string
62    pythonPath:
63      required: false
64      type: list
65      schema:
66        type: string

Notebook Runtime#

runtime schema

Key Value
kind string Must be runtime
type string Must be notebook
spec dict See runtime.spec schema.

runtime.spec schema

Key Value
excludeTags list List of string.
notebookPathURI string
notebookEncoding string
outputUri string
args list List of string.
conda dict See runtime.spec.conda schema.
env list List of dict. For each element, see runtime.spec.env schema.

runtime.spec.conda schema

Key Value
slug string
type string Must be service

runtime.spec.env schema

Key Value
name string
value number or string

Following is the YAML schema for validating the YAML using Cerberus:

 1kind:
 2  required: true
 3  type: string
 4  allowed:
 5    - runtime
 6type:
 7  required: true
 8  type: string
 9  allowed:
10    - notebook
11spec:
12  required: true
13  type: dict
14  schema:
15    excludeTags:
16      required: false
17      type: list
18      schema:
19        type: string
20    notebookPathURI:
21      required: false
22      type: string
23    notebookEncoding:
24      required: false
25      type: string
26    outputUri:
27      required: false
28      type: string
29    args:
30      nullable: true
31      required: false
32      type: list
33      schema:
34        type: string
35    conda:
36      nullable: false
37      required: false
38      type: dict
39      schema:
40        slug:
41          required: true
42          type: string
43        type:
44          required: true
45          type: string
46          allowed:
47            - service
48    env:
49      nullable: true
50      required: false
51      type: list
52      schema:
53        type: dict
54        schema:
55          name:
56            type: string
57          value:
58            type:
59              - number
60              - string

GitPython Runtime#

runtime schema

Key Value
kind string Must be runtime
type string Must be gitPython
spec dict See runtime.spec schema.

runtime.spec schema

Key Value
args list List of string.
branch string
commit string
codeDir string
conda dict See runtime.spec.conda schema.
entryFunction string
entrypoint string or list
env list List of dict. For each element, see runtime.spec.env schema.
outputDir string
outputUri string
pythonPath list List of string.
url string

runtime.spec.conda schema

Key Value
slug string
type string Must be service

runtime.spec.env schema

Key Value
name string
value number or string

Following is the YAML schema for validating the YAML using Cerberus:

 1kind:
 2  required: true
 3  type: string
 4  allowed:
 5    - runtime
 6type:
 7  required: true
 8  type: string
 9  allowed:
10    - gitPython
11spec:
12  required: true
13  type: dict
14  schema:
15    args:
16      type: list
17      nullable: true
18      required: false
19      schema:
20        type: string
21    branch:
22      nullable: true
23      required: false
24      type: string
25    commit:
26      nullable: true
27      required: false
28      type: string
29    codeDir:
30      required: false
31      type: string
32    conda:
33      nullable: false
34      required: false
35      type: dict
36      schema:
37        slug:
38          required: true
39          type: string
40        type:
41          required: true
42          type: string
43          allowed:
44            - service
45    entryFunction:
46      nullable: true
47      required: false
48      type: string
49    entrypoint:
50      required: false
51      type:
52        - string
53        - list
54    env:
55      nullable: true
56      required: false
57      type: list
58      schema:
59        type: dict
60        schema:
61          name:
62            type: string
63          value:
64            type:
65            - number
66            - string
67    outputDir:
68      required: false
69      type: string
70    outputUri:
71      required: false
72      type: string
73    pythonPath:
74      nullable: true
75      required: false
76      type: list
77      schema:
78        type: string
79    url:
80      required: false
81      type: string

Container Runtime#


Following is the YAML schema for validating the YAML using Cerberus:

 1kind:
 2  required: true
 3  type: string
 4  allowed:
 5    - runtime
 6type:
 7  required: true
 8  type: string
 9  allowed:
10    - container
11spec:
12  type: dict
13  required: true
14  schema:
15    image:
16      required: true
17      type: string
18    entrypoint:
19      required: false
20      type:
21      - string
22      - list
23    cmd:
24      required: false
25      type:
26      - string
27      - list
28    env:
29      nullable: true
30      required: false
31      type: list
32      schema:
33        type: dict
34        schema:
35          name:
36            type: string
37          value:
38            type:
39            - number
40            - string

Script Runtime#

runtime schema

Key Value
kind string Must be runtime
type string Must be script
spec dict See runtime.spec schema.

runtime.spec schema

Key Value
args list List of string.
conda dict See runtime.spec.conda schema.
env list List of dict. For each element, see runtime.spec.env schema.
scriptPathURI string
entrypoint string

runtime.spec.conda schema

Key Value
slug string
type string Must be service

runtime.spec.env schema

Key Value
name string
value number or string

Following is the YAML schema for validating the YAML using Cerberus:

 1kind:
 2  required: true
 3  type: string
 4  allowed:
 5    - runtime
 6type:
 7  required: true
 8  type: string
 9  allowed:
10    - script
11spec:
12  required: true
13  type: dict
14  schema:
15    args:
16      nullable: true
17      required: false
18      type: list
19      schema:
20        type: string
21    conda:
22      nullable: false
23      required: false
24      type: dict
25      schema:
26        slug:
27          required: true
28          type: string
29        type:
30          allowed:
31            - service
32          required: true
33          type: string
34    env:
35      nullable: true
36      required: false
37      type: list
38      schema:
39        type: dict
40        schema:
41          name:
42            type: string
43          value:
44            type:
45              - number
46              - string
47    scriptPathURI:
48      required: true
49      type: string
50    entrypoint:
51      required: false
52      type: string

Data Flow Runtime#

runtime schema

Key Value
kind string Must be runtime
type string Must be dataFlow
spec dict See runtime.spec schema.

runtime.spec schema

Key Value
archiveBucket string
archiveUri string
args list List of string.
conda dict See runtime.spec.conda schema.
env list List of dict.
scriptBucket string
scriptPathURI string

runtime.spec.conda schema

Key Value
slug string
type string Must be service

Following is the YAML schema for validating the YAML using Cerberus:

 1kind:
 2  allowed:
 3    - runtime
 4  required: true
 5  type: string
 6type:
 7  allowed:
 8    - dataFlow
 9  required: true
10  type: string
11spec:
12  type: dict
13  required: true
14  schema:
15    archiveBucket:
16      type: string
17    archiveUri:
18      type: string
19    args:
20      schema:
21        type: string
22      type: list
23    conda:
24      type: dict
25      required: false
26      schema:
27        slug:
28          type: string
29        type:
30          allowed:
31            - service
32          required: true
33          type: string
34    env:
35      required: false
36      schema:
37        type: dict
38      type: list
39    scriptBucket:
40      type: string
41    scriptPathURI:
42      type: string