Get started

OpenDOPS is the dops CLI for standardizing data engineering project workflows across your data stack. It scaffolds and manages Databricks Declarative Automation Bundles (dops dbx dab), Databricks notebook/workspace projects (dops dbx notebook), and dbt projects (dops dbt) — generating structure, metadata, and GitHub Actions while leaving validation, deployment, and runtime to each domain’s native CLI.

Already have projects? You don’t have to start from scratch — adopt them to bring what you already have under management.

Install

OpenDOPS is a Python package (opendops) that installs the dops command.

# with pip
pip install opendops

# or as an isolated tool with uv
uv tool install opendops

Verify the install:

dops --version
dops --help

Initialize a monorepo

Scaffold a complete Databricks DAB monorepo — bundles, shared libs, conf, helper scripts, metadata, and CI — with one command:

dops dbx dab repo init data-platform
cd data-platform

This creates:

data-platform/
  bundles/            # independently deployable Databricks bundles
  libs/dops_common/   # shared Python library
  conf/               # shared dev / staging / prod config
  scripts/dops/       # sandbox-prefix.sh helper
  .dops/              # OpenDOPS metadata (repo.yaml, bundles/)
  .github/workflows/  # six workflows that call the Databricks CLI directly

If the target directory exists and is not empty, the command fails unless you pass --force.

Add a bundle

Generate a Python-first, independently deployable bundle:

dops dbx dab bundle add customer-events-bronze \
  --owner data-platform \
  --targets sandbox,dev,staging,prod

The generated bundle contains a databricks.yml, resources/ (jobs + variables), a src/ package, and tests — and it registers itself at .dops/bundles/customer-events-bronze.yaml.

Adopt existing projects

Already have a Databricks repo or dbt project? dops adopt brings it under OpenDOPS management by inferring metadata from what’s already on disk and writing the .dops/ registry — without generating, moving, or rewriting any of your files.

cd existing-repo
dops adopt --check          # dry run — show what would be adopted
dops adopt --owner data-platform

It auto-detects the project type at the path:

• a dbt_project.yml → adopted as a dbt project (adapter + schema inferred from profiles.yml)
• any databricks.yml → adopted as a DAB monorepo; every bundle is registered, with its blueprint inferred from the resources it defines (jobs, DLT pipeline, SQL, Unity Catalog, …)

After adopting, dops check --all shows how each project compares to the OpenDOPS standard, and dops list inventories everything. Adoption is idempotent — already-managed projects are skipped. Notebook projects have no single marker file, so they’re adopted explicitly with dops dbx notebook adopt.

Adopted projects record no content lock (OpenDOPS didn’t generate the files), so upgrade treats every file as review until you reconcile it — the honest default.

Working in another domain?

The walkthrough above uses Databricks Asset Bundles, but the workflow is identical in every domain — scaffold (or adopt), check, get native commands, upgrade:

# Databricks notebook / workspace projects (deployed with `databricks sync`)
dops dbx notebook init analytics-notebooks --owner data-platform

# dbt projects (executed by the dbt CLI; branch-isolated sandbox schemas)
dops dbt init analytics --adapter dbt-databricks

From inside any project, the context-aware commands auto-detect the type — dops check, dops commands, dops upgrade, and dops list work the same everywhere. See the domains page for what each one generates.

Inspect your repo

dops dbx dab bundle list      # registered bundles (Rich table)
dops dbx dab bundle check customer-events-bronze   # structural checks
dops dbx dab repo doctor      # is this a healthy OpenDOPS monorepo?
dops dbx dab repo explain     # explains the layout & deployment model

bundle check and repo doctor are structural — they never call the Databricks CLI.

Check the whole repo at once

dops check is context-aware — run it from inside any project (bundle, notebook, or dbt) and it checks that one. Add --all to validate every OpenDOPS project in the tree:

dops check          # the project you're in
dops check --all    # every project — one pass/fail
dops check --all --strict   # warnings fail too
dops check --all --json     # machine-readable, for CI/dashboards
dops list           # inventory of every project (--json supported)

repo init also drops a .pre-commit-config.yaml that runs dops check --all on every commit (pre-commit install).

Enforce org standards with policy

Declare standards once in .dops/policy.yaml and check enforces them across every project — a no-op until you enable a rule:

requireOwner: true
allowedBlueprints: [databricks-python-wheel-job, databricks-dlt-pipeline]
requiredTargets: [sandbox, dev, staging, prod]
namingPattern: "^[a-z][a-z0-9-]+$"

OpenDOPS owns structure and conventions; the native CLIs still own validation and deployment.

Native Databricks commands

OpenDOPS never deploys. It prints the exact Databricks CLI commands for you to run:

dops dbx dab bundle commands customer-events-bronze \
  --target sandbox --branch feature/DOPS-123-add-customer-events

cd bundles/customer-events-bronze

RESOURCE_PREFIX="DEV_feature-dops-123-add-customer-events_"

databricks bundle validate -t sandbox --var="resource_prefix=${RESOURCE_PREFIX}"
databricks bundle deploy   -t sandbox --var="resource_prefix=${RESOURCE_PREFIX}"
databricks bundle run customer-events-bronze-job -t sandbox --var="resource_prefix=${RESOURCE_PREFIX}"

For official targets, the prefix is empty and names stay canonical:

dops dbx dab bundle commands customer-events-bronze --target dev

Detect changed bundles

Map a git diff to exactly the affected bundles — ideal for CI matrices:

dops dbx dab bundle changed --base origin/main --head HEAD
dops dbx dab bundle changed --base origin/main --head HEAD --format github-json

A change under bundles/<name>/ marks that bundle; a change under libs/, .dops/repo.yaml, or scripts/dops/ marks all bundles.

GitHub Actions

Generate (or regenerate) the CI workflows at any time:

dops dbx dab gha generate

This writes six workflows that call the Databricks CLI directly: validate changed bundles, sandbox deploy & cleanup, and dev / staging / prod promotion.

Versioning & deployment tracking

Each bundle’s databricks.yml declares a version per target — the source of truth for what is deployed to each environment. Promote a release by bumping that target’s version (a reviewable git change) and deploying; the Databricks CLI deploys exactly what the file declares.

dops dbx dab bundle version customer-events-bronze   # table of target -> version

Every generated resource is stamped with provenance tags — dops_version, dops_git_sha, dops_env (= the target), dops_bundle, dops_blueprint — so you can open any job/pipeline in the workspace and see exactly which version and commit produced it. The version comes from databricks.yml; CI stamps the commit SHA at deploy (--var git_sha=…), and runs databricks bundle summary -t <target> to confirm what’s live.

Upgrading to newer blueprints

Blueprints improve over time. Every generated project records a .opendops-lock.yaml (a hash of each file the blueprint produced), so upgrade can re-render the blueprint and reconcile it with your project without clobbering your edits:

dops dbx dab bundle upgrade customer-events-bronze --check   # dry run
dops dbx dab bundle upgrade customer-events-bronze           # apply

Each file is classified:

• add / update — applied automatically (you hadn’t edited it).
• conflict — you and the blueprint changed it; your file is kept and the new version is written alongside as <file>.opendops-new for you to merge. Pass --accept-theirs to take the blueprint’s version instead.
• remove — the blueprint dropped a file you never touched; it’s deleted.

The same upgrade command exists for the other project types:

dops dbx notebook upgrade        # from inside a notebook project
dops dbt upgrade                 # from inside a dbt project

Interactive cockpit (optional)

For the workflows a TUI does better than flags, OpenDOPS ships an optional Textual cockpit: a scaffolding wizard (pick a type + blueprint and fill a form) and interactive upgrade (read a live diff per changed file and apply or accept the blueprint’s version for conflicts), plus browsing projects and per-target versions.

pip install 'opendops[ui]'   # or: uv tool install --with textual opendops
dops ui

Web control plane

For a shareable, at-a-glance view of every project, launch the read-only control plane — a local web dashboard with a project inventory, structural health, and a deployment matrix (the declared version of each bundle, per target):

dops web            # → http://127.0.0.1:8787 (opens your browser); local-only, no credentials
dops web --live     # also compare declared vs. actually-deployed (uses your databricks CLI auth)

Each project opens a single pane of glass — Overview (resources + per-target versions), Pipeline (the job task DAG), Code (a read-only source browser), Deploy (the native commands), and Checks. The deployment matrix shows the declared version of every bundle per target; with --live it reads the dops_version tag off the actually-deployed resources and flags each cell synced / out of sync / not deployed — desired-vs-actual drift, the heart of an “ArgoCD for data engineering”.

dops web is local-first and offline by default; --live is opt-in and degrades gracefully to “unknown” if the databricks CLI isn’t authenticated. The dashboard ships in the wheel (no Node toolchain).

AI agents (MCP)

dops mcp runs a Model Context Protocol server that exposes the same grounded context — project inventory, structural checks, the deployment matrix (with live drift), pipeline DAGs, blueprint divergence, and source — to an AI agent. Your agent (Claude Desktop, Claude Code, Cursor, …) can then read your projects and help configure and optimize each resource, with OpenDOPS as the source of truth. It is read-only: the agent advises and proposes the native commands; it never deploys or edits.

pip install 'opendops[mcp]'

Wire it into an MCP client by pointing it at a repo, e.g. in Claude Desktop’s config:

{ "mcpServers": { "opendops": { "command": "dops", "args": ["mcp"] } } }

The server is a fourth adapter over the same UI-agnostic core as the CLI, TUI, and web dashboard — so it adds zero new logic, just structured tools (list_projects, project_overview, check_project, get_file, deployment_matrix, blueprint_diff, …).

Deployment model

Every bundle ships with a stable sandbox target plus stable dev, staging, and prod targets. Branch identity is supplied at runtime via a resource_prefix variable — the databricks.yml is never mutated per branch.

The feature-branch prefix is never promoted — code is promoted by deploying merged code to an official target.

Command reference

For the full reference, see the project README on GitHub.