Skip to content
N Divij
Résumé
← All work

Case study · 2026 · Build with DataHub: The Agent Hackathon

Pareis

Most production ML failures start as silent data failures: a bad load NULLs a column and the models keep serving. Pareis watches DataHub for failing dbt tests over 1.07M real transactions, walks lineage from the broken column to every affected feature and MLflow model, flags them, and blocks retraining in CI until the data is clean again.

Built
2026, for Build with DataHub: The Agent Hackathon
Stack
DataHub · dbt · PostgreSQL · MLflow · scikit-learn · FastAPI · MCP
The hackathon demo. It loads only when you press play.

The problem

The warehouse team sees a failing test; the ML team sees nothing. The link between them, table to feature to model, lives only in the lineage graph, and walking it during an incident is nobody's job. A model retrained while the data was bad also stays poisoned after the data heals.

How it works

  1. 01Ingest real signals. Postgres, dbt test results and MLflow models reach DataHub through official connectors.
  2. 02Scan assertions. Find failing dbt tests in the catalog, with the failed row count.
  3. 03Walk lineage. Re-root to origin columns, then BFS forward into features and models.
  4. 04Gate retraining. Ten checks per model; NO-GO exits 2 and blocks the CI job.
  5. 05Write back. Tags, notes and native DataHub incidents; Slack, Jira or Linear tickets.
  6. 06Heal. When tests go green, flags clear and Pareis writes the postmortem.

Key decisions

Column precision, not table panic

A failing test names its column, so Pareis flags only features that consume a poisoned column. The demo price incident flags 2 of 6 features as MEDIUM instead of all 6 as HIGH.

No LLM between a test and a verdict

The core is deterministic: the same incident always gives the same blast radius, rollback and gate decision, reproducible on a laptop with no API keys. Agents reach it through six read-only MCP tools, and write-back stays with the CLI, CI and dashboard.

Silence is not safety

Unresolvable lineage is never treated as clean, and a model version with no timestamp is UNKNOWN, never CLEAN. I read stored aspects directly instead of search, because index lag turns 'no results' into a false all-clear.

Bound the blind window

Corruption starts before anyone notices. Pareis bounds that window from assertion run history, checks every registry version's training time against it, and names the newest clean version to roll back to.

Results

  • 1.07Mreal UCI Online Retail II transactions in the warehouse
  • 0.78 / 0.82AUC of the two MLflow models: repeat purchaser and return risk
  • 2 of 6features flagged in the demo incident, instead of all 6
  • #18823my fix to DataHub's dbt ingestion, merged upstream

What's not done yet

Pareis reacts to failing tests, so it is blind wherever a column has no test; pareis protect ranks those gaps and generates dbt tests, but someone still has to add them. It gates retrain jobs, not continuous learners, and does not yet trace lineage into serving endpoints.

Next case studyElitOrc-AI →A clinical decision-support agent that grounds each assessment in similar past cases.