7. Transformation Engine — Spark
The reference Iceberg engine. Broadest write and maintenance support, at the cost of a runtime to operate.
The role Spark plays
The Iceberg reference implementation
Iceberg's Spark support is the most complete of any engine — procedures, DDL, row-level operations, and maintenance actions land here first.
Why most lakehouses end up with Spark somewhere
Even a BigQuery-first client usually needs Spark for maintenance jobs, complex ingestion, or ML feature engineering.
Deployment options on GCP
Dataproc is now Managed Service for Apache Spark (APIs and gcloud dataproc unchanged): clusters (image 3.0 ships Spark 4.1), serverless batches and interactive sessions, BigQuery stored procedures for Apache Spark, and Spark Connect sessions from BigQuery Studio notebooks. Lightning Engine, Google's accelerated runtime, is available on 2.x images and serverless Premium tier but not yet on 3.0. Cost and operational profiles differ sharply (checked on 2026-09-25).
Catalog compatibility
Read this against the same diagram in the other Chapter 3 modules — the frame is identical, so what changes is which pattern each engine is actually good at. Thick arrow: first-class read and write. Thin: works, with caveats. Dotted: read-only or degraded.
The ranking is the exact inverse of BigQuery's in Module 06. A client who needs both engines writing the same tables has already chosen Pattern 2, whether they know it yet or not.
Lakehouse runtime catalog (Pattern 2)
The primary pairing. On Google's images, one property does the wiring — dataproc:dataproc.lakehouse.catalog.NAME=projects/P/catalogs/C on clusters, or dataproc.lakehouse.catalog.NAME=... on batches. Elsewhere, configure type=rest, the Lakehouse endpoint, a bl:// warehouse, GoogleAuthManager, GCSFileIO, and the vended-credentials header (Module 04). Iceberg is 1.6.1 by default on images 2.2/2.3 (1.10 selectable) and 1.11 on image 3.0. The older BigQueryMetastoreCatalog integration is legacy.
Dataproc Metastore (Pattern 3)
HMS as the Iceberg catalog, or as a plain Hive catalog for legacy tables — image 2.x only, since image 3.0 does not support Dataproc Metastore.
BigQuery-managed Iceberg (Pattern 1)
Reads through exported metadata or BigQuery catalog federation. Writes only through the BigQuery Storage Write API connector — never directly to the Iceberg files, which Google warns "can cause query failure or data loss". The constraint that pushes engagements toward Pattern 2.
The write path
DDL and DML
CREATE TABLE ... USING iceberg, CTAS, MERGE INTO, UPDATE, DELETE, and branch/tag operations.
Copy-on-write versus merge-on-read
Set per table and per operation. The decision that determines whether write cost or read cost dominates — revisit it after measuring, not before.
Commit semantics and retries
Optimistic concurrency, commit retry configuration, and what a Spark job must guarantee for a retried run to be safe.
Streaming ingestion
Structured Streaming into Iceberg, the small-file consequence, and the compaction schedule that has to accompany it.
Maintenance — Spark's most valuable role
The procedures
rewrite_data_files, rewrite_manifests, rewrite_position_delete_files (v2 tables; v3 uses deletion vectors), expire_snapshots, remove_orphan_files, and compute_table_stats for NDV statistics. rewrite_table_path supports relocating a table (Module 04). What each one does, what it costs, and how often to run it. On Pattern 2, automatic table management (Module 04) can take compaction and snapshot expiry for eligible tables, but never orphan cleanup — that stays with Spark.
Scheduling and sizing
Maintenance as a first-class, budgeted, monitored workload rather than an afterthought.
Ordering and safety
Why orphan cleanup must run with a retention buffer — Spark defaults to files older than three days, snapshot expiry to five days — and what breaks if it does not.
Performance and cost shape
Cost drivers
Cluster or batch runtime, shuffle behaviour, file sizes, and partition design. Priced by resource-time, not by bytes scanned — which changes the optimization target relative to BigQuery.
Serverless versus clusters
Autoscaling and per-batch billing against warm clusters and lower per-job latency.
No Aliz benchmark exists for Spark against these catalogs. The comparisons in this knowledge base measure BigQuery paths only. Add a Spark run before making comparative performance claims to a client.
Governance
What Spark can and cannot enforce
Vended credentials scope storage access; catalog ACLs scope table access. Column masking and row-level policies are not available at this layer.
The bypass risk
A Spark job with direct bucket credentials sidesteps the catalog entirely. Credential vending exists to prevent exactly this.
Development experience and orchestration
Languages and testing
PySpark and Scala, local iteration, and how to test Iceberg logic without a live catalog.
Orchestration
Composer/Airflow, workflow templates, and where the post-commit metadata refresh fits when BigQuery consumers sit downstream.
When Spark is the right transformation engine
Decision gates
Complex non-SQL transformation logic, an existing Spark team, table maintenance ownership, streaming ingestion, ML feature pipelines, or a multi-cloud Iceberg estate.
Anti-patterns
Using Spark for plain analytical SQL that BigQuery would run faster and cheaper; running Spark and BigQuery as concurrent writers to the same table without a single-writer design.