Apache Spark

DataFlow

Oracle Cloud Infrastructure (OCI) Data Flow is a fully managed, serverless, and on-demand Apache Spark Service that performs data processing or model training tasks on extremely large datasets without infrastructure to deploy or manage.

Getting Started with Data Flow

ads integrates with OCI Data Flow allowing users to submit and monitor Spark jobs from Notebook Sessions or from their local machine.

• Quick Start
  • Submit a Dummy Python Script to DataFlow
  • Real Data Flow Example with Conda Environment
• Setup and Installation
  • Notebook Session Development
  • Local Development
  • Logging From DataFlow
• Running your Spark Application on OCI Data Flow
  • Notebook Extension
  • ADS CLI
  • ADS Python SDK
• spark-defaults.conf
  • odsc Command-line
  • Manual
• Data Catalog Metastore
• OCI Data Flow Studio
  • Prerequisite
  • Quick Start
  • Data Flow Spark Magic
  • Basic Spark Usage Examples
• [Legacy]
  • Prerequisite
  • Create a Instance
  • Generate a Script Using a Template
  • Create a Application
  • Example Notebook: Develop Pyspark jobs locally - from local to remote workflows
  • Example Notebook: Using the ADB with PySpark