Lake-level governance: contracts, lineage, and DLP at rest

Our piece on the secure document pipeline covers data in motion — PII stopped before it reaches a model or a log. The lake has a different problem. By the time data lands there, it’s already at rest, already joined to other tables, already one query away from ending up somewhere it shouldn’t. Governing a lake isn’t about a checkpoint at the door; it’s about controls that hold once the door is behind you.

Data contracts: bad data stops at the source, not in an answer

A data contract is an agreement between the team that produces a dataset and the teams that consume it: schema, freshness, and quality thresholds, checked automatically rather than assumed. Without contracts, a schema change three teams upstream surfaces as a wrong number in a dashboard, or worse, in an AI-generated answer nobody thought to question. With them, the pipeline breaks loudly at the point of the breaking change — which is a better failure than a silent one three hops downstream.

Lineage: every answer has a traceable origin

Retrieval layers built on ungoverned sources have a specific failure mode: an AI system produces a confident, well-written answer, and nobody can say which record it came from, whether that record was current, or whether it should have been visible to the person who asked. Lineage back to the source record is what turns “the model said so” into “here is the record, here is when it was last updated, here is who is allowed to see it.” We build retrieval layers — vector stores and knowledge bases — on top of lineage that already exists at the lake level, not bolted on afterwards as a separate index with no memory of where anything came from.

DLP at rest: the scan doesn’t stop at the front door

Detection and redaction at intake (DLP in the flow, covered in the document-pipeline piece) handles data arriving through a known pipeline. A lake accumulates data from more paths than that over its lifetime — ad hoc loads, one-off exports, a well-meaning analyst’s CSV. DLP at rest means the lake itself is periodically scanned for sensitive fields that shouldn’t be there, independent of how they arrived, with findings routed back to the owning team rather than silently masked and forgotten.

Quality gates: the producer/consumer agreement, enforced

Contracts are only useful if something enforces them. A quality gate sits between a new dataset landing and it becoming queryable by downstream consumers — schema conformance, null-rate thresholds, referential checks against the contract — and it’s the mechanism that makes “bad data stops at the door” a property of the system rather than a hope about the upstream team’s diligence.

Where this fits

None of this is glamorous, and none of it is optional if an organisation wants AI systems that can be trusted with real answers. The document pipeline earns the right to bring data in. Lake-level governance is what keeps that trust intact for every day the data sits there afterwards — which, for most enterprise data, is most of its life.