4. GCP Pattern 2 — Lakehouse Runtime Catalog
Data files in customer-owned GCS, table format Iceberg, and a Google-managed Iceberg REST catalog that any compatible engine can use. The genuine multi-engine pattern — and the one with the largest governance gap.
The shape of the pattern
The same frame as Module 03, with one thing moved: the catalog is now a standalone service and BigQuery is a client of it rather than its owner.
Compare with Module 03: every engine's line is now solid in both directions. That is what the pattern buys — and the red box is what it costs.
Component placement
Storage in customer GCS. Iceberg metadata in the same bucket. Catalog is Google's managed Iceberg REST catalog (the Lakehouse runtime catalog, formerly BigLake metastore). Engines are whatever can speak Iceberg REST — Google documents Spark, Flink, Trino, Hive, BigQuery, and AlloyDB. A Hive catalog endpoint (Preview) also exists for Hive-only clients.
The catalog is the product
Unlike Pattern 1, the catalog is an addressable service with a standard API. BigQuery becomes one client of it rather than its owner. Google now recommends this pattern over Pattern 1 "for most modern, open lakehouse workloads" (checked on 2026-09-25).
Naming and the moving target
On 2026-04-20 BigLake became Lakehouse and BigLake metastore became the Lakehouse runtime catalog. APIs, client libraries, CLI commands, and IAM roles did not change — they still say biglake (biglake.googleapis.com, gcloud biglake, roles/biglake.*). Related renames: Dataplex is now Knowledge Catalog, Dataproc is now Managed Service for Apache Spark. Pin the name and date whenever writing this down for a client.
Setting it up
Catalog, namespace, table
gcloud biglake iceberg catalogs create CATALOG \
--catalog-type=biglake \
--default-location=gs://BUCKET \
--credential-mode=vended-credentials
gcloud biglake iceberg namespaces create NAMESPACE --catalog=CATALOG
--catalog-type=biglake is the multiple-bucket catalog (up to 15 buckets in one region group) and the one to use; gcs-bucket is the legacy single-bucket type Google discourages for new projects. Credential mode is a separate choice and can be switched later. Set --primary-location=US|EU if BigQuery will query from a multi-region. Terraform support exists (checked on 2026-09-25).
The catalog service account
A per-catalog service agent that needs roles/storage.objectUser on every bucket the catalog uses. It is eventually consistent, so a grant made immediately after catalog creation can fail — retry. Getting this grant wrong is the most common setup failure.
Reaching it from BigQuery
No dataset to create: BigQuery queries catalog tables directly as project.catalog.namespace.table, with each namespace behaving like a dataset. Reads are GA. DML (INSERT, UPDATE, DELETE, MERGE) and DDL are Preview — on by default for tables BigQuery created, opt-in via gcp.biglake.bigquery-dml.enabled=true for tables created by other engines, and only for v2 copy-on-write tables with simple partitioning, no sort order, and default paths. v3 tables are read-only from BigQuery (checked on 2026-09-25).
Federated catalogs
Besides catalogs it owns, the Lakehouse runtime catalog can hold federated catalogs that expose tables from Unity Catalog, AWS Glue, Snowflake Horizon, and other remote Iceberg REST catalogs (Preview, checked on 2026-09-25). Go deeper in Module 10 (federation) and Module 11 (cross-cloud access).
Connecting Spark, Flink, and Trino
Endpoint https://biglake.googleapis.com/iceberg/v1/restcatalog, warehouse bl://projects/PROJECT/catalogs/CATALOG, Google auth (org.apache.iceberg.gcp.auth.GoogleAuthManager in Spark, iceberg.rest-catalog.security=GOOGLE in Trino), and the X-Iceberg-Access-Delegation: vended-credentials header that makes credential vending work. Open-source reads, writes, and streaming writes are GA; Parquet only, no v1 tables, no rename, no Iceberg views.
Multi-engine access — the payoff
Credential vending
The catalog issues short-lived GCS credentials downscoped to the table's paths and the minimum permission — read-only for a query, write only for principals holding roles/biglake.editor. Engines never hold standing bucket access, which is what makes this pattern governable at all. BigQuery itself uses vended credentials on catalogs that have vending enabled.
Which engines participate
BigQuery natively; Spark, Flink, Trino, and Hive via the catalog endpoints; Snowflake through a catalog integration (GA on Snowflake's side since 2026-06-02). See Chapter 3 for each engine's write semantics and Module 10 for other catalogs reading these tables.
The single-authoritative-writer rule
Multiple engines may read freely. Exactly one catalog must be authoritative for writes, and concurrent writers must share the same Iceberg feature set. Tables with BigQuery DML or table management enabled use strict conflict detection, so concurrent open-source writers will see more commit retries.
The governance gap
What does not carry over
BigQuery policy tags, row access policies, dynamic data masking, and authorized views are BigQuery features. They do not become portable Iceberg controls, and they do not apply to a Spark or Trino reader.
What you get instead
IAM at project, catalog, namespace, and table level (table-level is Preview) plus GCS IAM, with roles/biglake.admin, editor, viewer, and metadataViewer. No row-level security, column-level security, or masking — even from BigQuery, which sees only "table-level security with IAM" on these tables. Knowledge Catalog registers the tables for search, lineage, and profiling, but does not enforce policy tags on them (checked on 2026-09-25).
Governance must be redesigned, not copied
This is the single most under-scoped item in a lakehouse migration. Budget for it explicitly, and again for every additional engine.
The metadata-pinned external table workaround
The workaround restores BigQuery governance over a catalog table, at the price of a window where the two disagree:
The refresh is a step in the writer's post-commit flow, not a separate schedule — and the URI must come from the catalog's metadata-location, never from listing the bucket.
The problem it solves
BigQuery supports no authorized views, row-level security, column-level security, or masking on catalog tables, so fine-grained governance has no home. Wrapping the catalog's Parquet files in an Iceberg external table restores the BigQuery governance surface. Note that Google now labels Iceberg external tables "no longer recommended for most use cases" (checked on 2026-09-25) — the workaround still works, but it runs against Google's direction, and it should be re-tested whenever BigQuery's native governance on catalog tables changes.
The mechanism
The external table pins one exact metadata JSON URI. It does not follow the catalog pointer.
The refresh rule
After every successful Lakehouse commit, resolve the catalog's authoritative metadata-location via the REST API and update the external table's URI.
Never resolve the current metadata file by listing GCS and taking the newest object. Failed commits leave orphaned metadata files that are newer than the current table state. The catalog pointer is the only authority.
The consistency window
Between commit and refresh, the external table serves the previous snapshot. This staleness interval must be measured, bounded, monitored, and alerted on — and disclosed to consumers.
Idempotency and failure handling
No refresh on a failed write, a staleness check before refresh, bounded retry, and an explicit alert on failure. The refresh is a step in the writer's post-commit flow, not a separate schedule.
Why it performs well
The benchmark measured the pinned external table (iceext) at only 1.03x native slot-ms — the most compute-efficient non-native option tested — though at 1.82x native elapsed time, with the gap sitting outside measured compute stages (metadata resolution and object-store I/O).
Performance findings
Direct catalog access was the slowest option tested
4.26x native elapsed time (10,020 ms vs 2,352 ms) at only 1.16x native slot-ms. High wall time, low compute — an I/O and metadata-resolution profile, not a compute profile.
The unresolved byte-accounting anomaly
Direct catalog access reported 3.444 GiB processed where every other variant reported 1.722 GiB — an exact 2x. Cause unknown; could be accounting semantics, duplicate physical reads, or a plan difference.
Do not build an on-demand cost estimate on the direct catalog path until this is explained. Under on-demand billing, a 2x bytes-processed figure is a 2x bill. Inspect the query plan and referenced-table statistics first.
Reading these numbers responsibly
Three repetitions, one workload, one file layout. Directional. The follow-up test plan in _research/benchmark_results.md lists what would make them conclusive.
Operating it
Maintenance ownership
Automatic table management (Preview, opt-in via gcp.biglake.table-management.enabled=true, billed per DCU-hour) handles snapshot expiry and compaction toward 256 MB files for eligible tables — the same v2 copy-on-write constraints as BigQuery DML, plus catalog-level credential vending. It does not remove orphan files, does not expire snapshots on tables with tags or branches, and does not document manifest rewrite. Assign everything it does not cover — at minimum orphan cleanup — to Spark and a schedule at design time (checked on 2026-09-25).
Cost shape
Catalog operations are billed per call: first 5,000 Class A (create, update, list, set IAM) and 50,000 Class B (read, delete) operations per month are free, then roughly 0.90 per million respectively, with metadata files over 1 MB counted per MB. Table management is extra, per DCU-hour. Regional list prices vary — check the pricing page (checked on 2026-09-25).
Quotas
1,500 table modifications per table per day (DML and streaming excluded) and 5 metadata updates per 10 seconds per table. A high-frequency micro-batch writer can hit the second one — check the quotas page before designing one.
Monitoring surface
Catalog latency and availability, commit success rate, snapshot count, manifest count, file-size distribution, and external-table staleness.
Disaster recovery
Data files, metadata files, and catalog registration must be protected and restored as one system. Retained objects without a valid metadata chain are not a backup. The catalog is regional; cross-region metadata replication and failover is Preview, needs a dual-region bucket, and metadata can replicate before the data files it references.
Portability to another cloud
Why this pattern is the cleanest source
The table already expresses schema, partitioning, and semantics in Iceberg, so the target does not have to reverse-engineer them from an export.
What actually moves and what does not
Files and Iceberg metadata move. The Google catalog resource, IAM policies, credentials, views, and engine-specific objects do not.
Current state versus full history
The normal approach rewrites the current snapshot into the target with an Iceberg-aware engine, accepting new snapshot IDs. Preserving history means rewriting storage paths inside metadata and copying every referenced file — Iceberg's rewrite_table_path Spark procedure does the metadata part. Significantly more work, done only when history is a requirement.
The mistake to avoid
Copying the Iceberg directory and registering it unchanged. Its metadata still references the source storage paths.
When to choose Pattern 2
Decision gates
Multiple Iceberg-aware engines must read and write the same table; data and metadata must sit in customer-owned storage; schema, partition, and snapshot semantics must be portable; cross-cloud Iceberg migration is committed, not hypothetical; and the team can own catalog governance, compatibility testing, maintenance, and incident response.
When not to
BigQuery is the only real engine, or nobody owns table maintenance — Pattern 1 is lower risk. If row-level security or authorized views are non-negotiable, neither lakehouse pattern provides them natively: budget for the pinned-table workaround, or keep that layer in native BigQuery or external Parquet.