Data Ingestion and Loading

Retrieving secrets from secret scopes so credentials aren't hardcoded in notebooks.

A Databricks feature for incremental, scalable ingestion of files as they arrive in cloud object storage, built on Structured Streaming.

cloudFiles — used as spark.readStream.format("cloudFiles").

Cloud object storage — Amazon S3, Azure Data Lake Storage (ADLS), and Google Cloud Storage (GCS) — as well as Unity Catalog volumes (/Volumes/...) and Azure Blob Storage.

JSON, CSV, XML, Parquet, Avro, ORC, text, and binaryFile (including pre-compressed files in these formats).

Auto Loader is a streaming/continuous incremental ingest (Structured Streaming), while COPY INTO is an idempotent batch SQL command for loading files.