Delta and Lakeflow: where bad data passes with no error

Declaring a quality rule is not blocking the data. In Delta, NOT NULL and CHECK block; in Lakeflow, an expectation with no action retains the row; keys and unique only declare; MERGE ignores the new column. See what each layer does, how to build a quarantine and how to read the count, with an example run locally.

Continue ReadingDelta and Lakeflow: where bad data passes with no error

Databricks Lakeflow: code moves from dev to prod, state does not

The deploy went green, and a full refresh wiped the bronze table that Kafka could no longer replay. In Databricks Lakeflow, the bundle carries the code from dev to prod, but state stays in each environment: the deployment record, the streaming checkpoints, the tables themselves and the identity that runs the pipeline. This article shows the four states the bundle does not carry, what each command can delete and the guards you declare in the bundle before the first production deploy.

Continue ReadingDatabricks Lakeflow: code moves from dev to prod, state does not

The vector index inherits no permissions: data agents on Databricks, Fabric and Foundry

A vector index is a copy, and a copy inherits no permissions: the Databricks AI Search documentation states that row and column level permissions are not supported. The two routes an agent takes to the data — RAG and text-to-SQL —, where the access rule lives on each one, the Genie Agent's two credentials, and the metric layer that stops the number from changing.

Continue ReadingThe vector index inherits no permissions: data agents on Databricks, Fabric and Foundry

Federate, mirror, or ingest: the decision between Databricks and Fabric

Federating, mirroring, and ingesting are not synonyms — and in Fabric "mirroring" already means three different mechanisms. When to use Lakehouse Federation, when to mirror a catalog or a database into Microsoft Fabric, when to ingest with Lakeflow Connect, what is actually free, and why source permissions do not cross the OneLake boundary.

Continue ReadingFederate, mirror, or ingest: the decision between Databricks and Fabric

Data ontology: the governed context that makes AI reason across Databricks and Fabric

The bottleneck of enterprise AI is not the model, it is governed business context. How Unity Catalog Metric Views, Genie Ontology and Fabric IQ form a single semantic layer over one copy of the data — with an FSI use case, cost model and roadmap.

Continue ReadingData ontology: the governed context that makes AI reason across Databricks and Fabric