Databricks is where a data platform and a machine-learning platform stop being two systems. Delta tables, the features built on them, the models trained on those features and the predictions those models produce all sit under Unity Catalog — one permission model, one lineage graph. That is the reason to choose it, and the part most implementations leave switched off.
Overview
Unity Catalog is the point
Plenty of teams run Databricks as a Spark cluster with notebooks on top, which is a fair way to spend a lot of money on something a warehouse already did. The value arrives when features, models and inference tables are governed alongside the data — because that is what makes a prediction explainable months later, and what makes a data quality problem traceable to the reports and models it touched.
What we deliver
Where ML projects on it actually fail
Almost never on the modelling. They fail on features computed after the event they are meant to predict, on models nobody notices going stale, and on scores that land somewhere the business never opens. Each of those has a specific answer on this platform, and each is a decision made early or not at all.
- Feature Views with point-in-time joins, so training cannot see the future
- MLflow tracking tied to the Delta version behind each run
- Champion/Challenger by alias, promoted on realised outcomes rather than offline metrics
- Lakehouse Monitoring over inference tables, with drift triggering retraining
- Asset Bundles in CI, so the platform is rebuildable from source
- Job compute over all-purpose, and batch over real-time unless latency is genuinely required
Why it matters
When Databricks, and when not
It earns its cost when the same data has to serve analytics and machine learning under one governance model, or when the workload is genuinely large and Spark-shaped. It is the wrong answer for a reporting estate that a warehouse and a BI tool would serve at a fraction of the price. We will say so rather than sell you a lakehouse you do not need.
Workflow
How we work in it.
- 01
Assessment
- Existing estate, workloads and where governance currently stops
- Whether the problem is genuinely Spark-shaped
- Unity Catalog and workspace topology
- Cost model: job versus all-purpose, batch versus serving
- 02
Platform Design
- Catalog per environment, with dev, staging and production separated
- Medallion layering and table design
- Feature Views where features are shared across models
- Access control, lineage and audit expectations
- 03
Build
- Pipelines and tested transformations, versioned with the code
- Training pipelines logged to MLflow with data versions captured
- Deployment by alias, with a quality gate before promotion
- Asset Bundles promoted through environments in CI
- 04
Operate
- Inference profiling and drift metrics with alerting
- Automated retraining, manual promotion
- Cost monitoring against the original forecast
- Handover, documentation and training
Related in Warehouse & lakehouse
