Delta and Lakeflow: where bad data passes with no error

Declaring a quality rule is not blocking the data. In Delta, NOT NULL and CHECK block; in Lakeflow, an expectation with no action retains the row; keys and unique only declare; MERGE ignores the new column. See what each layer does, how to build a quarantine and how to read the count, with an example run locally.

Continue ReadingDelta and Lakeflow: where bad data passes with no error

Databricks Lakeflow: code moves from dev to prod, state does not

The deploy went green, and a full refresh wiped the bronze table that Kafka could no longer replay. In Databricks Lakeflow, the bundle carries the code from dev to prod, but state stays in each environment: the deployment record, the streaming checkpoints, the tables themselves and the identity that runs the pipeline. This article shows the four states the bundle does not carry, what each command can delete and the guards you declare in the bundle before the first production deploy.

Continue ReadingDatabricks Lakeflow: code moves from dev to prod, state does not

Fabric IQ: the ontology is now a workspace item and refresh has a price

The demo convinces, the workspace disagrees. In Microsoft Fabric the ontology is now a real item: a REST API, a definition in files and a child graph created on its own — and it is that child that issues the bill. This article gathers what the item really is, why every refresh is full and freshness became a cost decision, the four capacity meters, how far binding actually reaches, three points where the official documentation contradicts itself — and what has to be in place before switching this on in production.

Continue ReadingFabric IQ: the ontology is now a workspace item and refresh has a price

Federate, mirror, or ingest: the decision between Databricks and Fabric

Federating, mirroring, and ingesting are not synonyms — and in Fabric "mirroring" already means three different mechanisms. When to use Lakehouse Federation, when to mirror a catalog or a database into Microsoft Fabric, when to ingest with Lakeflow Connect, what is actually free, and why source permissions do not cross the OneLake boundary.

Continue ReadingFederate, mirror, or ingest: the decision between Databricks and Fabric

Data ontology: the governed context that makes AI reason across Databricks and Fabric

The bottleneck of enterprise AI is not the model, it is governed business context. How Unity Catalog Metric Views, Genie Ontology and Fabric IQ form a single semantic layer over one copy of the data — with an FSI use case, cost model and roadmap.

Continue ReadingData ontology: the governed context that makes AI reason across Databricks and Fabric