deployment.md

Deployment

Supported Modes

The repo supports two main execution modes:

• local Spark/Delta for development and tests
• Databricks-style bundle and wheel-task execution

The code is designed so the same platform concepts work in both modes.

Local Development Mode

Install dependencies:

UV_CACHE_DIR=.uv-cache uv sync --extra dev

Run core tests:

.venv/bin/pytest -q tests/unit
.venv/bin/pytest -q tests/integration/test_job_entrypoints.py tests/integration/test_reconciliation_controls.py

Validate docs:

.venv/bin/python scripts/check_docs.py
.venv/bin/mkdocs build --strict

Bundle-Style CLI Execution

The public CLI façade is:

.venv/bin/python -m databricks_ingestion.jobs bronze --config conf/jobs/bronze_job.yml
.venv/bin/python -m databricks_ingestion.jobs silver --config conf/jobs/silver_job.yml
.venv/bin/python -m databricks_ingestion.jobs gold --config conf/jobs/gold_job.yml

These commands mirror the intended wheel-task shape on Databricks.

Repository Deployment Shape

The repo already includes:

• databricks.yml
• resources/jobs/ingestion_jobs.yml
• conf/jobs/*.yml

That means the current deployment story is:

1. build the wheel
2. deploy the bundle
3. point Databricks jobs at the wheel-task entrypoint and config file

Runtime Kinds

Current runtime kinds:

• attached_spark
• local_delta

attached_spark

Use on Databricks or any already-running Spark environment.

Example:

runtime:
  kind: attached_spark
  env: databricks
  catalog: main
  schema: banking
  volume: landing

local_delta

Use for local Delta-backed development when you want more production-like table behavior in tests or local runs.

Example:

runtime:
  kind: local_delta
  warehouse_dir: /tmp/warehouse
  metastore_dir: /tmp/metastore
  catalog: spark_catalog
  schema: banking

Bronze Job Example

runtime:
  kind: attached_spark
  catalog: main
  schema: banking

contract:
  source_id: customer_accounts
  source_system: bank_feed
  format: csv
  landing_path: /Volumes/main/raw/customer_accounts
  expectations_path: /Workspace/Shared/databricks_ingestion/contracts/quality/customer_accounts.yaml

target:
  base_path: /Volumes/main/managed/ingestion
  catalog: main
  schema: banking
  dataset_name: customer_accounts
  stage: bronze

ops:
  tracker_path: /Volumes/main/managed/ops/customer_accounts/tracker
  audit_path: /Volumes/main/managed/ops/customer_accounts/audit

Silver Job Example

contract:
  source_id: customer_accounts_silver
  source_system: bank_feed
  format: table
  landing_path: main.banking.customer_accounts_bronze
  table_options:
    source_table: main.banking.customer_accounts_bronze

target:
  stage: silver
  state_model: scd2

reconciliation_controls:
  - name: bronze_to_silver_current
    left_table: main.banking.customer_accounts_bronze
    right_table: main.banking.customer_accounts_silver
    severity: mandatory

Gold Job Example

target:
  base_path: /Volumes/main/managed/ingestion
  catalog: main
  schema: banking
  dataset_name: customer_accounts_summary
  stage: gold

semantic_spec:
  name: customer_accounts_summary
  source_table: main.banking.customer_accounts_silver AS accounts
  materialization: table
  certification_state: validated

reconciliation_controls:
  - name: silver_current_to_gold_summary
    left_table: main.banking.customer_accounts_silver
    right_table: main.banking.customer_accounts_summary_gold
    severity: mandatory

Operational State

By default, jobs persist:

• audit records
• file tracker state for file-based jobs
• reconciliation evidence when configured

For table promotion:

• file-style tracking is intentionally replaced by a no-op tracker

This is important because table sources do not have stable landed file identities.

Recommended Production Pattern

1. Bronze jobs read landing files
2. Silver jobs promote from Bronze tables
3. Gold jobs publish semantic outputs
4. Mandatory reconciliation controls gate Silver and Gold where required

Recommended Directory and Ownership Pattern

Use consistent storage conventions such as:

/Volumes/main/raw/<domain>/<dataset>
/Volumes/main/managed/ingestion/<stage>/<dataset>
/Volumes/main/managed/ops/<dataset>/audit
/Volumes/main/managed/ops/<dataset>/tracker
/Volumes/main/managed/ops/<dataset>/reconciliation

And keep ownership clear:

• source contract owner
• target dataset owner
• control owner where relevant

Current Gaps

The deployment story is already viable, but not yet complete for a large fleet.

Still missing:

• large-scale config registry
• domain-wide orchestration across many datasets
• release management and approval workflows
• stronger environment governance conventions