Releases

Changelog

What's new in OpenDOPS. The dops CLI is in active development — Databricks-first today, with more domains on the way.

0.1.0 — Initial release

The first useful OpenDOPS release. A Python package named opendops that installs a dops command, starting with the Databricks domain.

Added

  • dops adopt — bring an existing dbt or Databricks project under OpenDOPS management. It infers metadata from what’s already on disk (dbt adapter + schema from profiles.yml; each databricks.yml registered as a bundle with its blueprint inferred from the resources it defines) and writes the .dops/ registry — without generating, moving, or rewriting any of your files. Auto-detects the type, is idempotent, and has a --check dry run. Per-domain forms (dops dbx dab repo adopt, dops dbt adopt, dops dbx notebook adopt) give explicit control.
  • dops web — a web control plane (“an ArgoCD for data engineering”): a local browser dashboard where each project is a single pane of glass — inventory, structural health, the per-target deployment matrix, a job-task DAG view, a read-only code browser, native deploy commands, and blueprint-sync diffs. dops web --live reads the dops_version tags off your actually-deployed resources and flags every cell synced / out of sync / not deployed — true desired-vs-actual drift (opt-in, uses your databricks CLI auth, degrades to “unknown” offline). Local-first, ships in the wheel (stdlib HTTP + Jinja + vendored assets, no Node). A third adapter over the same UI-agnostic core as the CLI and TUI.
  • dops mcp — a Model Context Protocol server (a fourth adapter over the same core) exposing your projects’ inventory, checks, deployment drift, pipeline DAGs, blueprint divergence, and source to an AI agent (Claude Desktop / Code, Cursor) so it can help configure and optimize each resource — read-only, with OpenDOPS as the grounded source of truth. pip install 'opendops[mcp]'.
  • dops dbx dab repo init — scaffold a Databricks DAB monorepo (bundles, libs, conf, scripts, metadata, and six GitHub Actions workflows).
  • dops dbx dab repo doctor / repo explain — validate and explain the monorepo.
  • dops dbx dab bundle add — generate an independently deployable, Python-first bundle from the databricks-python-wheel-job blueprint.
  • dops dbx dab bundle list / check — Rich tables and structural diagnostics.
  • dops dbx dab bundle commands — print native Databricks CLI commands with branch-derived sandbox prefixes.
  • dops dbx dab bundle changed — detect changed bundles between two git refs, with --format github-json for CI matrices.
  • dops dbx dab gha generate — generate workflows that call the Databricks CLI directly.
  • dops dbx notebook — a non-DAB Databricks subsystem for notebook/workspace projects deployed via native databricks sync (init / check / commands / gha generate).
  • Six more bundle blueprints — databricks-dlt-pipeline, databricks-multi-task-workflow, databricks-sql-job, databricks-unity-catalog (governance-as-code), and databricks-alerting-job (email/webhook notifications + health rules) join the original databricks-python-wheel-job. bundle check and bundle commands are now blueprint-aware — they adapt to jobs, pipelines, or governance bundles automatically.
  • dops dbt — a second domain. Scaffolds standardized dbt projects (staging/marts models, sources, seeds, tests, env targets, CI) and runs them with the native dbt CLI. Branch-isolated sandboxes build into a DEV_<branch>_<schema> target schema.
  • Reviewable examples — every blueprint’s actual rendered output is committed under examples/.
  • Context-aware CLI — run dops check, dops upgrade, or dops commands from inside any project and OpenDOPS auto-detects the type (bundle / notebook / dbt). dops list shows every OpenDOPS project in a tree.
  • Repo-wide checks — dops check --all validates every project at once (with --strict and --json); repo init ships a .pre-commit-config.yaml that runs it on every commit.
  • Policy enforcement — declare org standards once in .dops/policy.yaml (required owner, allowed blueprints, required targets, naming) and check enforces them everywhere. Opt-in; a no-op until you enable a rule. JSON output added to check, list, and bundle version.
  • dops ui — an interactive Textual cockpit for the things a TUI does better than flags: a scaffolding wizard (pick a type + blueprint and fill a form instead of memorising bundle add --blueprint … --owner …) and interactive upgrade (see each changed file, read a live diff, and apply or accept the blueprint’s version for conflicts). Also browses projects across domains and shows per-target versions. Install with pip install 'opendops[ui]'; built as a thin layer over the same core engine the CLI uses.
  • Deployment versioning — each bundle’s databricks.yml declares a version per target (the source of truth for what’s deployed where; promote by bumping it). Resources are stamped with provenance tags (dops_version, dops_git_sha, dops_env, dops_bundle, dops_blueprint), CI stamps the commit SHA at deploy, and dops dbx dab bundle version <name> shows the per-target versions.
  • upgrade — pull blueprint improvements into existing projects without clobbering local edits. Each project records a .opendops-lock.yaml; upgrade re-renders the blueprint and classifies every file as add / update / conflict / remove (conflicts land as .opendops-new sidecars). Available as dops dbx dab bundle upgrade, dops dbx notebook upgrade, and dops dbt upgrade.
  • Branch-isolated sandbox deployments via a DEV_<sanitized_branch_name>_ prefix — on bundle resource names (DAB) or workspace folders (notebooks).
  • Verified against the real CLIs — the generated output is checked by the actual tools, not just golden comparisons: all six DAB blueprints pass databricks bundle validate (CLI v1.3.0) with zero schema diagnostics, and the dbt project passes the real dbt deps + dbt parse (dbt 1.11 + databricks adapter). make test-integration runs these; they skip cleanly when a CLI is absent. Generated projects now ship a .gitignore so build artifacts (target/, dbt_packages/, .databricks/, Python caches, .opendops-new sidecars) stay out of version control.
  • Deploy-ready generated YAML — every blueprint was audited against the real Databricks and dbt schemas so the rendered output passes databricks bundle validate / dbt parse as-is: Python jobs ship a job cluster (spark_version/node_type_id bundle variables), Unity Catalog provenance lives in the schema comment, dbt profiles tolerate unset credentials, and webhook alerts are opt-in. Notebook projects deploy under /Workspace/Shared. An opt-in integration test (make test-integration) runs the real CLIs against the committed examples in CI.
  • Provenance everywhere — dbt projects carry a query-comment that stamps the project, target, and git SHA into every warehouse query (the dbt-native equivalent of resource tags); notebook deploys write a _dops_provenance.json marker into the workspace folder. Every upgrade records the blueprint version it came from and advanced to, in both the .opendops-lock.yaml and the project metadata.
  • Friendlier failures — running changed-detection outside a git repository now raises a clear, actionable error instead of a raw traceback. A parity test pins the generated sandbox-prefix.sh to the Python prefix engine so the two can never silently drift.
  • Rich-based terminal output and a uv-first developer workflow.

Principles

  • OpenDOPS owns structure; the Databricks CLI owns execution.
  • Stable targets; runtime prefixes — databricks.yml is never mutated per branch.

Roadmap

  • Additional Databricks blueprints (multi-task, DLT, SQL, notebook).
  • New domains: dops dbt, dops airflow, dops snowflake.
  • A future terminal UI built as a thin layer over the same core engine.

See the blueprints page for details, or follow along on GitHub.

← Back to home