Loading...

Skip to main content

11. Cross-Cloud Data Access

Querying Iceberg tables that live in another cloud from BigQuery and Spark on GCP, through a federated catalog, without migrating the files. Module 10 covers how the remote catalog becomes visible; this module covers what it costs to move the bytes.

Preview — checked on 2026-09-25

Cross-cloud data access (marketed as "borderless Lakehouse" since 2026-07-31) is a Preview feature under Google's Pre-GA terms. Source launch dates: Databricks Unity 2026-04-22, AWS Glue 2026-06-05, Snowflake Horizon 2026-07-13, Workday 2026-09-21; SAP BDC was announced as GA on 2026-07-20, but the overview page still calls every source Pre-GA. Details below come from the cross-cloud data access overview and the per-source setup pages as of that date. Re-verify before any client deliverable, and do not design a production dependency on it without confirming its GA status.

What the feature is​

A federated catalog pointed at another cloud​

The Lakehouse runtime catalog links to a remote Iceberg REST catalog and exposes its tables under a four-part name, project.federated_catalog.namespace.table. The data files stay in the other cloud's object storage; GCP engines read them in place.

Supported sources​

Databricks Unity Catalog (AWS or Google Cloud), AWS Glue including S3 Tables, Snowflake Horizon Catalog (AWS or Google Cloud), Workday Data Lake, and SAP Business Data Cloud (requires BigQuery Enterprise Plus). The console may not offer every source in every project — check the regions and capabilities page rather than hard-coding this list.

Not always cross-cloud​

Unity Catalog and Snowflake Horizon can both run on Google Cloud. Federating them is still Module 10, but none of the transport, egress, or residency concerns below apply.

How a cross-cloud query flows​

Metadata comes from the remote catalog; data comes from the cache when it can and from the remote bucket when it cannot. Every cache miss is an egress charge on the other cloud's bill.

Metadata discovery​

The GCP catalog calls the remote Iceberg REST catalog and re-syncs on a refresh interval of at least 300 seconds. Authentication differs per source: an AWS IAM role trusting the catalog's service account via OIDC for Glue (plus Lake Formation grants if used); a Databricks service principal via OIDC (recommended) or a secret, with Unity external locations and EXTERNAL USE SCHEMA; a Snowflake PAT or workload identity federation; OAuth credentials for Workday. Secrets live in Secret Manager in the catalog's region.

Secure transport​

The public internet over TLS is the default and needs no networking work. The private option — Partner or Dedicated Cross-Cloud Interconnect, AWS only today — adds an internal proxy load balancer or direct endpoint routing, Service Directory, and PrivateLink on the AWS side. Pick the GCP region nearest the source (Google publishes an AWS-to-GCP region pairing table), or the same region when the source is on Google Cloud.

Execution and caching​

Data segments are cached in the catalog's GCP region and reused by later queries. Caching is automatic for all cross-cloud queries, has no user switch, and unreferenced blocks expire automatically with no published TTL. Google's pricing page lists no cache line item as of the check date.

Engines​

BigQuery​

Standard SQL against the four-part name, with caching, credential vending, and interconnect routing handled for you. Needs BigQuery Data Viewer, BigQuery Job User, and BigLake Viewer.

Spark​

Managed Service for Apache Spark or standalone Spark with credential vending, through the REST catalog endpoint.

Through the same catalog endpoint. Untested by Aliz.

Read, not write​

Federated catalogs are documented as "read-only views of the remote catalog". Write through the owning platform (Module 10's one-writer rule). The "bidirectional" wording in Google's launch material means publishing GCP-produced tables back to a partner such as SAP BDC, not writing into federated tables.

Transport and cost​

Cross-Cloud Interconnect versus public internet​

Egress is billed by the remote cloud at its standard internet rates. Interconnect lowers that rate and makes latency predictable, at the cost of provisioning and a commitment — Partner Interconnect is sold as a flat-rate subscription. The public internet is quicker to set up and costs more per byte.

The cache is the cost model​

Savings depend on how often the same data is re-read, how fast the remote tables change, and how long the cache retains blocks. A workload that scans fresh partitions once pays full egress every time.

Measuring it​

statistics.query.objectStorageStats in the job metadata reports cacheBytesRead and objectStorageBytesRead per cloudProvider. Measure these on a representative workload before quoting a saving.

Gap

No Aliz benchmark exists for cross-cloud queries. Do not quote latency or egress savings to a client without running one.

Residency, encryption, and governance​

Cached copies live in the GCP region​

If the remote data is in another region or jurisdiction, caching stores data-at-rest copies in the target GCP region. For regulated data this can be a residency violation, not a detail.

CMEK​

Customer-managed encryption keys are not supported for the cache; cached blocks use Google-managed keys. If the organisation enforces the Restrict Non-CMEK Services policy, caching is disabled automatically — and the cost model goes with it.

Access control​

Querying needs roles/bigquery.dataViewer, roles/bigquery.jobUser, and roles/biglake.viewer; managing federated catalogs needs roles/biglake.admin. The remote catalog's own controls apply on the other side. Whether BigQuery row access policies or masking can sit over federated tables is not documented — test it. Two policy surfaces to keep in agreement, as in Module 10.

Known limitations​

Only Iceberg tables sync (Snowflake native tables and Databricks default-storage tables are ignored); Snowflake storage-optimised STRING columns can fail to read without a serialization workaround; Iceberg metrics reporting is unavailable; and SAP data products with / or - in their names break the refresh. Views, Delta, and Iceberg v3 are not documented either way.

Federate, replicate, or migrate​

Federate in place​

The source platform is staying, reads are occasional or cache-friendly, and residency and CMEK constraints allow caching.

Replicate​

High-frequency reads of fast-changing data, where egress on every cache miss outgrows the cost of a copy — or CMEK is mandatory and caching is off.

Migrate​

The source platform is being retired. Federation can serve as the bridge while tables move, then be removed. See the AWS migration analysis in _research/gcp_managed_catalog_iceberg_vs_native_and_aws_migration.md.

When to use cross-cloud data access​

Decision gates​

The remote catalog is one of the supported sources; the workload is read-only on the GCP side; residency rules permit cached copies in the GCP region; and the client accepts a Preview dependency or the feature has reached GA.

Anti-patterns​

Assuming the cache makes egress free; federating regulated data without checking where cached copies land; relying on caching in an organisation that enforces CMEK; and using federation as a permanent substitute for a migration the client has already decided on.