Cover: data ontology in Azure Databricks and Microsoft Fabric

Data ontology: the governed context that makes AI reason across Databricks and Fabric

The scene repeats at almost every bank I visit. The lakehouse is ready, Power BI is everywhere, the team bought into the idea of a data copilot — and the POC dies on the first serious question. The reason is never the model. It is that nobody told the LLM that, in that bank, delinquency excludes contracts renegotiated in the last 90 days, that “active customer” follows a different rule in Retail and in Wholesale, and that Account links to Contract through a key that only exists on the mainframe. The model does not fail for lack of data. It fails for lack of governed business context.

Both dominant platforms reached the same conclusion and pushed semantics down, from BI into the data layer. Databricks answered with Unity Catalog Metric Views and Genie Ontology; Microsoft Fabric answered with Fabric IQ Ontology on top of OneLake. Metrics, entities and rules stopped being code inside a report and became catalog artifacts — with lineage, permissions and audit — consumed equally by SQL, BI and agents.

What is a data ontology?

An ontology is the layer that declares what things mean in your business: which entities exist (Customer, Account, Contract, Collateral, Risk Event), how they relate, and which rules define each metric. It is the opposite of a data dictionary in a PDF: it is a versioned, governed, machine-queryable object.

The contrast with the alternative is direct. Without an ontology, meaning lives scattered across three places — in the DAX of the semantic model, in the risk team’s notebook, and in the controller’s spreadsheet. With an ontology, there is a single semantic contract, and every consumer — dashboard, analyst or agent — starts from the same definition.

The problem it solves

The problem is not missing data. It is semantic drift: the same question yields three answers because the metric was reimplemented three times. While that stayed inside BI, the cost was a wrong slide in a meeting. With agents, the cost changes nature:

A collections agent that understands “delinquent” differently from the risk team does not produce a wrong number. It produces a wrong action in production — a dunning sequence fired at a customer who renegotiated yesterday.

Three pains disappear when semantics move up into the catalog:

  1. KPI divergence — every metric gets a named owner and a versioned definition, instead of three competing implementations.
  2. Non-auditable AI — the agent resolves against a governed metric, and the generated SQL is inspectable. That is what makes generative AI acceptable in a regulated environment.
  3. A second copy of the data — when semantics are shared, the physical data does not need to be duplicated to serve both BI and engineering.
Diagram: on the left, semantics declared once — entity ontology, governed metrics, live context and a single physical copy in Delta/Unity Catalog; on the right, the flow of every question — question or event, semantic resolution, governed query and auditable answer — with a governance rail of Unity Catalog, Purview, MLflow 3 and regulatory trail applied at every step.
What you declare once (left) sustains what flows on every question (right) — and governance cuts across both bands.

How it works — step by step

  1. Fix authority over the data. Unity Catalog is the single authority over the physical Delta data; Purview is the corporate compliance plane above it. This decision comes before any tooling discussion.
  2. Conform Silver by entity, not by source system. The Silver layer should reflect Customer, Account, Contract — not the mainframe’s file layout. This is where the ontology stops being a diagram and becomes modeling.
  3. Declare the minimum ontology. Start with 7 entities and ~12 relationships. In retail banking, the core that sustains risk, collections, fraud and regulatory response is: CustomerAccountContractCollateral, plus Relationship (ownership graph), Transaction, Risk Event and Fraud Alert.
  4. Convert critical KPIs into governed metrics. Five are enough to prove the concept. A KPI that depends on pipeline logic is born as a Metric View in Databricks; a presentation metric is born in the Fabric semantic model.
  5. Expose by reference, not by copy. Publish Gold tables into OneLake via shortcut or mirroring. The data stays in Delta under Unity Catalog and shows up in Fabric with no pipeline, no lag window and no second storage bill.
  6. Point AI at the semantics, never at the raw table. Genie, Copilot, Data Agents and Foundry agents must resolve against Metric Views and the ontology. If the agent can read the raw table, you do not have a semantic contract — you have a suggestion.
  7. Instrument the agent as an auditable asset. MLflow 3 records the full trace of every decision — prompt, prompt version, tools called, retrieved context, metric version and response — in a Delta table governed by UC.

Where each concept is born

The most expensive mistake I see is not picking the wrong platform. It is picking one of the two and recreating the definitions in the other. The correct architecture defines where each concept is born and propagates from there.

Concept Azure Databricks Microsoft Fabric Where it should be born
Entities and relationships Genie Ontology — context derived from the catalog and real usage Fabric IQ Ontology — entities and rules bound to OneLake Fabric IQ. This is business vocabulary, maintained by stewards — not by engineering.
Metrics / KPIs Unity Catalog Metric Views Semantic model (certified DAX measures) Databricks when the KPI depends on the pipeline; Fabric when it is a presentation metric.
Physical data Delta Lake governed by Unity Catalog OneLake (Lakehouse, Warehouse, Eventhouse) Databricks, exposed to Fabric via shortcut. One copy only.
AI consumption Genie, Agent Bricks, AI Functions Copilot, Data Agents, Foundry, Copilot Studio Both — with the same definition behind them.
Governance Unity Catalog (RBAC, masking, lineage) Purview, OneLake security, sensitivity labels UC as authority over the data; Purview as corporate compliance.

The practical rule: Databricks is the engineering and ML plane; Fabric is the consumption and business-semantics plane. At almost every sizable FSI customer, the answer is not “one or the other” — it is drawing the boundary by persona and letting OneLake prevent the second copy.

The use case that closes the argument

Take the typical profile: a retail bank with ~12 million customers, ~40 million instant-payment transactions per day, a US$ 11bn credit portfolio, a mainframe core, and a risk team that exports to Excel because it does not trust the number on the dashboard.

Four pains, four architectural answers:

  • 90+ delinquency has three implementations. It becomes a Metric View with a named owner, a definition in Git and a semantic regression test. Divergence ends.
  • The fraud score runs in a 15-minute batch, but the loss happens in 90 seconds. Payments flow through Event Hubs → Eventstream → Eventhouse; the model stays in Databricks; Activator fires the block. Scoring inside the transaction window.
  • A regulator request on economic-group exposure takes 5 days and 4 teams. The Relationship entity in the ontology turns that into a query — the regulatory agent assembles the exposure in minutes.
  • A single dunning sequence for the whole portfolio. A recoverability model in MLflow plus an agent that prioritizes by probability of return, not by days past due.

The side effect nobody anticipates: once there is one official definition per KPI, the risk team stops exporting to Excel. Trust in the dashboard is a consequence of semantic governance, not of user training.

The cost nobody models

Two platforms, two consumption models — and that is where the bill gets away from you.

Databricks charges per DBU/second plus Azure infrastructure. It scales well with irregular load and historical reprocessing. The FinOps risk is well known and always the same: an all-purpose cluster with no auto-terminate. By far the largest source of wasted DBU I find in FSI.

Fabric charges an F capacity (CU) shared across all workloads, with smoothing and bursting. A 1-year reservation cuts roughly 40%. The risk here is the opposite: throttling from undersized capacity and a noisy neighbor on the same F SKU. Remember the licensing step: below F64 you still pay Power BI Pro per reader; from F64 up, readers are free — which changes the math in any scenario with thousands of BI users.

Add inference: around 3,500 input tokens and 900 output tokens per agent query is a reasonable assumption for sizing the curve. Model adoption over 12 months, with foundation and migration in the first months and AI workloads entering from M4 — not as a single step in month 1. And always validate in the Azure Pricing Calculator before any commitment: list price varies by region, exchange rate, taxes and commercial agreement.

Production best practices

  • One physical copy of the data. Fabric accesses it via shortcut or mirroring. Copying Gold with a pipeline pays for storage and compute twice and still creates lag.
  • Every KPI with a named owner, a definition versioned in Git and a semantic regression test in CI/CD. A metric with no owner diverges again within three months.
  • Separate F capacity per critical domain, with consumption alerts before throttling. Do not run heavy ETL inside the F capacity and then blame Power BI.
  • Auto-terminate and autoscale are mandatory on every cluster. Tags per domain, budget per workspace and a monthly idle-DBU review — FinOps from day 1, not from month 9.
  • The agent always resolves against the ontology or a Metric View, with the generated SQL visible to the user. Pointing an agent at raw tables “because it is faster for the POC” does not survive an audit.
  • Sensitivity labels and masking applied in UC/Purview, inherited by BI and by agents. A control applied at the edge is not a control.
  • A business steward on the ontology. An ontology designed by engineering alone becomes just another dead dictionary.

What has to be in place

It is not a question of timing — it is a question of dependency. Each stage only holds if the previous one is closed, which is why an ontology programme that starts with the agent always loops back to the beginning. What follows is declarative: what has to be true, not when.

Foundation — authority over the data. Unity Catalog is the single authority over the physical data, with Purview connected as the corporate compliance plane. The landing zone has Private Link, CMK, Entra and FinOps tags from the very first workspace — not as a retrofit. At least two domains reach Silver conformed by the ontology’s entities, not by the source system’s layout. The minimum ontology is declared, and the most contested KPIs in the house exist as Metric Views with a named owner and a definition in Git. One Gold table already shows up in Fabric through a shortcut, feeding a Direct Lake semantic model.
It is in place when a risk KPI returns the same number in Power BI and in Genie.

Production with context — semantics turning into decisions. The business ontology is published and bound to data that already lives in OneLake, by reference and never by copy. The models that matter — fraud, recoverability, PD/LGD — are in serving, and the critical event crosses streaming into the operational trigger inside the window where the loss actually happens. The first business agent answers in the channel where people already work, resolving against a governed metric, with the generated SQL visible. Every decision it makes leaves a trace, prompt version, tools called and metric version in a Delta table governed by UC.
It is in place when Compliance accepts the traceability without asking for a supporting spreadsheet.

Scale and efficiency — the platform as a product. The ontology has stopped being a single-domain project and covers the rest of the bank, maintained by business stewards rather than engineering. Every critical domain has its own capacity with internal chargeback and consumption alerts before throttling. The semantic contract is versioned in CI/CD, with a regression test that breaks the build when someone changes a KPI definition by accident. Capacity reservations and DBU hygiene — auto-terminate, autoscale, monthly idle review — have stopped being an initiative and become routine. The data marts and legacy extracts that existed only to work around the absence of a single definition have been decommissioned.
It is in place when cost per domain is visible and the dashboard number is no longer argued about in meetings.

Notice that the order is not chronological, it is causal. You do not scale semantics nobody governs, and you do not govern semantics that have no owner. Skipping the foundation to reach the agent faster does not get you there faster — it gets you to the same place twice.

Frequently asked questions (FAQ)

Databricks or Fabric — do I have to choose?

In practice, no. The right question is where to place each workload. Heavy ETL, model training and agent observability stay in Databricks, for per-second elasticity and MLflow. Semantic model, corporate BI and business ontology stay in Fabric, for Direct Lake and reach into the business. OneLake is what prevents the copy between the two.

Isn’t an ontology just a data dictionary with a fancier name?

No. A data dictionary describes; an ontology executes. The practical difference is that the definition of “active customer” in the ontology is the same string the agent queries, the dashboard renders and the auditor reads — with lineage and permissions inherited from the catalog. A PDF dictionary has none of that.

Fabric IQ is in preview. Can I commit architecture to it?

You can commit the pattern, not the timeline. Declare semantics first where they are already GA — Metric Views in Unity Catalog and a certified semantic model — and validate Fabric IQ regional availability before moving corporate vocabulary there.

How do I prove to Compliance that the agent is auditable?

With traces, not slides. MLflow 3 records prompt, prompt version, tools called, retrieved context, metric version and response in a Delta table governed by UC. LLM judges score quality in production and catch regression before the regulator does. That is the difference between “the agent answered this” and “I can reconstruct why it answered this”.

Where do I start if the budget is small?

With one KPI. Pick the most contested metric in the house, give it an owner, convert it into a Metric View and prove that Power BI and Genie return the same number. That costs weeks, not quarters — and it is the argument that unlocks the rest.

Conclusion

The bottleneck of enterprise AI is no longer the model. It is governed business context — and both platforms converged on the same thesis because the alternative does not scale: an agent pointed at raw tables is a prototype, not a system.

The architectural decision left on the table is not “Databricks or Fabric”. It is where each definition is born, who owns it, and how the agent is forced to go through it. Whoever solves that first ships AI to production in a regulated environment. Whoever leaves it for later spends the year explaining why the dashboard number does not add up.

👉 If you are designing the semantic layer of your data platform — especially in financial services, where audit and a single definition are not optional — this is the decision that separates POC from production. Want to talk about ontology, Databricks and Fabric? Reach me on LinkedIn.

Leave a Reply