Loading...

Skip to main content

2. Data Lakehouse — Basic Architecture

The components of a working lakehouse, how a read and a write actually flow through them, and the operational surface the design creates.

The component map​

Three layers do the work, and every one of them is a component you now own. The arrows are the sequence of a single catalog-backed query: the engine asks the catalog first, then goes to storage. Readers pinned to a specific metadata file (Modules 03 and 04) skip the live catalog — which is exactly why they can go stale.

Three further concerns cut across all three layers rather than sitting in any one of them: orchestration (what triggers ingestion, transformation, and maintenance), governance (bucket IAM, catalog ACLs, and engine policies, which do not automatically agree), and observability (snapshot history, commit failures, file and manifest counts, catalog latency). Each gets its own section below.

Object storage​

Where the Parquet data files and the Iceberg metadata files both live. Bucket layout, location alignment with the query engine, encryption, and lifecycle rules are architecture decisions, not defaults.

Table format metadata​

Metadata JSON, manifest lists, and manifests — stored alongside the data, in the customer's bucket. Losing or orphaning these is losing the table, even with every data file intact.

Catalog​

The authoritative pointer store. Iceberg REST catalog, Hive Metastore, or a vendor-managed equivalent. Determines which engines can participate and how credentials are issued. The REST spec now covers credential vending, server-side scan planning, multi-table atomic commits, and views — but each catalog implements a subset. Google's Lakehouse runtime catalog, for example, serves no Iceberg views (checked on 2026-09-25). Check capabilities per catalog, not per spec.

Query and transformation engines​

Readers and writers. Multiple engines may read; the write path needs deliberate design.

Orchestration​

What triggers ingestion, transformation, and maintenance, and what it does when a commit fails.

Governance and access control​

IAM on the bucket, ACLs on the catalog, and whatever fine-grained controls each engine offers. These three do not automatically agree with each other.

Observability and lineage​

Snapshot history, commit failures, file counts, manifest counts, and catalog latency — none of which appear in a warehouse dashboard by default.

How a read works, end to end​

The seven steps​

Steps 1 through 3 are sequential round trips that must complete before any data is touched — they set the latency floor for every query, however small. Steps 4 and 5 (highlighted) are where the scan actually shrinks: everything pruned here is a file never opened in step 6.

Where latency comes from​

Catalog round trip, object-storage round trips per metadata file, and object-open latency per data file. This is why external table access can consume near-native compute yet take twice the wall-clock time.

Why pruning quality is a design output, not a feature​

File size, sort order, partition transform, and column statistics all come from how the writer wrote. A badly written table cannot be pruned well by any engine.

How a write works, end to end​

The commit protocol​

Only step 4 is atomic, and only step 4 makes the write visible. Steps 1 through 3 are ordinary object writes that leave files behind whether or not the commit ever lands.

Failure modes at each step​

A failed commit can leave orphaned data and metadata files in the bucket that are newer than the current table state. Never resolve current metadata by listing GCS and taking the newest file — always ask the catalog.

Idempotency and retry​

What a retried pipeline must guarantee, and why "the job succeeded" and "the commit landed" are different facts.

Single-writer vs multi-writer​

The default recommendation is one authoritative writer per table. Multi-writer is supported by the protocol but requires every writer to honour the same catalog and the same Iceberg feature set.

Table maintenance as a first-class workload​

Compaction​

Rewriting many small files into fewer large ones. The single highest-impact maintenance job, and the one most often unscheduled.

Manifest rewrite​

Planning cost grows with manifest count, not just data volume. Manifests need periodic consolidation.

Snapshot expiry​

Time travel depth is a retention policy with a storage bill attached. Expiry physically deletes files no longer referenced. Iceberg's default is five days and at least one snapshot (history.expire.max-snapshot-age-ms); Trino enforces a seven-day minimum.

Orphan file cleanup​

Removing files left behind by failed commits. Requires care — deleting a file still referenced by a retained snapshot, or one an in-flight commit is still writing, destroys data. Spark's default only removes files older than three days; never lower it below your longest-running write.

Who owns this​

An engine, a schedule, and a named team. In a managed pattern the vendor may own some of it; know exactly which parts.

Governance across a split architecture​

The core problem​

The warehouse enforced access control at one place. A lakehouse has at least three: bucket IAM, catalog ACLs, and engine-level policies — and they only add up to a control if there is no fourth path around them.

Credential vending​

The catalog issues short-lived, scoped storage credentials to the engine, so engines never hold standing bucket access. The mechanism that makes multi-engine governance tractable.

What does not travel between engines​

Row-level policies, column masking, and authorized views are engine features, not table-format features. They must be redesigned per engine, and again per cloud after a migration.

The bypass risk​

Any principal with direct bucket read access bypasses every catalog and engine policy. Analysts must not have both.

Layering data inside a lakehouse​

Medallion or zoned layering on open tables​

Raw, curated, and serving layers as separate namespaces or buckets, with unidirectional flow and different access boundaries.

Where the warehouse still fits​

Serving layers with strict latency requirements often stay in native warehouse storage. A lakehouse does not have to be all-or-nothing.

Failure modes to design for​

Stale metadata​

A consumer pinned to an old metadata file silently serves old data. Detection and alerting must be explicit.

Small-file proliferation​

Streaming or frequent micro-batch writes without compaction. Degrades gradually, then sharply.

Catalog as a single point of failure​

Availability, latency, and quota of the catalog now sit in the query path.

Metadata/data separation in backup and DR​

Backing up the bucket is not backing up the table. Data files, metadata files, and catalog registration must be recoverable as one unit.

What to decide before choosing a GCP pattern​

The seven questions​

Which engines must read? Which must write? Does fine-grained governance have to survive? Is cross-cloud portability committed or hypothetical? Who owns maintenance? What is the freshness requirement? What is the query latency SLO?

How the answers map to Chapter 2​

A preview of how these questions select between BigQuery-managed Iceberg, the Lakehouse runtime catalog, and Dataproc Metastore.