A price forecasting platform that ships itself
End-to-end forecasting on Databricks, from ingestion to a registered, promotable model.
AI Data Engineer: design and implementation
The problem
Forecasts existed, but getting one into production was a manual event. A model lived in a notebook, someone re-ran it by hand, and promoting a new version meant copying artefacts between environments and hoping nothing drifted. There was no registry, no validation gate and no clean way back if a release went wrong.
What I did
- Designed the platform end to end: ingestion, feature engineering, training, registry and serving.
- Built the MLflow experiment and registry layer, including the promotion path from Dev to Prod.
- Wrote the automated validation gate that a model must pass before it can be promoted.
- Implemented the CI/CD workflow that makes a release a pipeline run rather than a manual checklist.
- Owned feature engineering and hyperparameter tuning against the baseline time-series model.
Architecture
- Ingestion lands raw source data into Delta tables under Unity Catalog, so lineage and access control come from the platform rather than convention.
- Feature engineering runs as a versioned PySpark job. Feature definitions live in code, not in a notebook cell, which is what makes a rerun reproducible.
- Training logs every run to MLflow (parameters, metrics and artefacts) so comparing a candidate against the incumbent is a query rather than an argument.
- A candidate is only registered if it clears the validation gate: accuracy against a holdout window, plus schema and range checks that catch the silent failures.
- Promotion moves a registered version through stages. Rollback is selecting the previous version, which is the entire point of doing it this way.
- Serving exposes the promoted model through Databricks Model Serving, consumed by a React front end running as a Databricks App.
What went wrong first
A model that scores well can still be wrong
Aggregate error hid failures on specific segments. I added slice-level checks to the validation gate so a candidate that improves the headline number but degrades a segment does not get promoted. The gate is deliberately stricter than the metric.
Reproducibility is a data problem, not a code problem
Re-running the same notebook gave different results because the feature inputs had moved underneath it. Pinning feature generation to versioned Delta reads made training runs comparable, which in turn made the registry meaningful.
Rollback has to be boring
The first design treated promotion as a deployment. That made rollback a redeployment, which nobody wants to do under pressure. Modelling promotion as a stage transition on a registered version turned recovery into a one-line change.
Results
- production model versions registered
- 3production model versions registered
- reduction in deployment cycle time
- ~40%reduction in deployment cycle time
- MAPE reduction
- 18%MAPE reductionagainst the baseline time-series model
- fewer production rollbacks
- ~30%fewer production rollbacks
The measurable win was cycle time, but the durable one was confidence. When promotion is gated and reversible, a team ships more often because a bad release costs minutes instead of a day.