6. Architecture Patterns

Every anonymization engagement starts with the same question: who needs to see what? A telco has data engineers who need full-granularity CDRs for pipeline debugging, analysts who need behavioral metrics without knowing subscriber identities, and partners who receive aggregate mobility reports. An e-commerce platform has data scientists training recommendation models, business analysts running conversion funnels, and brand partners expecting sales breakdowns. Each consumer group has a different privacy threshold — and the architecture must serve all of them from a single platform without copying data into ad-hoc silos.

This module covers the three architectural layers that solve this problem: data zones (where data lives and who can access it), pipelines (how data moves between zones), and query-time masking (how different users see different views within a zone). It then ties all three together in two reference architectures — telco CDR modernization and e-commerce customer data platform. Module 07 covers the cryptographic infrastructure underneath (key management, crypto-shredding). Module 08 covers architecture patterns specific to AI and LLM workloads.

---

Data zones — who sees what

Without a deliberate structure, organizations end up with ad-hoc copies of data for every consumer group — a "clean" export for partners, a "masked" copy for analysts, a "full" copy for engineers — each maintained separately, drifting out of sync, and governed inconsistently. When the DPO asks "who has access to raw subscriber data?", nobody can answer with confidence because the copies have proliferated beyond tracking.

A zoned data model replaces this sprawl with a single, governed progression. Data exists at different privacy levels, each serving different consumers, with access policies that widen as privacy guarantees strengthen. Instead of N copies for N consumer groups, you maintain three (or four) zones with controlled pipelines between them.

The three-zone model

The established industry pattern uses three zones: raw → pseudonymized → anonymized. This progression is described in GCP's de-identification reference architecture, codified in Japan's Act on the Protection of Personal Information (APPI) (personal information → pseudonymously processed → anonymously processed, each with distinct regulatory obligations), and implicit in most GDPR-era data platform designs.

Zone	What's in it	Who can access it	How long it stays
Raw	Unmodified source data with full PII	Ingestion pipelines and break-glass troubleshooting only	Days to weeks — kept only for reprocessing
Pseudonymized	Direct identifiers replaced (format-preserving encryption, deterministic encryption, HMAC keyed hashing); quasi-identifiers at original granularity	Data engineers (full access), analysts (masked via dynamic data masking / DDM), ML pipelines	Months to years, per data classification
Anonymized	Structurally anonymized (k-anonymity) and/or DP aggregate outputs	Broad — business users, partners, dashboards, external sharing	Years or indefinite (outside GDPR scope if truly anonymous)

The zones are logical, not necessarily physical. In BigQuery, they can be separate datasets, separate projects, or even column-level distinctions via DDM:

project: analytics-platform
├── dataset: raw_cdrs              (raw zone)
│   └── Restricted IAM, short retention, audit logging
├── dataset: pseudonymized_cdrs    (pseudonymized zone)
│   └── Policy tags + DDM, role-based column masking
└── dataset: anonymized_cdrs       (anonymized zone)
    └── Broad IAM, k-anonymous tables + DP output tables

Access control maps to zones. Each zone has its own IAM boundary. The raw zone uses project-level IAM with explicit deny for human users in normal operations. The pseudonymized zone uses dataset-level IAM plus policy tag roles (fineGrainedReader for engineers, maskedReader for analysts). The anonymized zone uses broad dataset-level or table-level IAM. VPC Service Controls add a network perimeter around the raw and pseudonymized zones — even a principal with IAM permissions cannot exfiltrate data outside the perimeter.

Data flow is unidirectional: raw → pseudonymized → anonymized, via controlled pipelines. There is no reverse path. IAM prevents write access from downstream zones to upstream zones; VPC-SC prevents data egress from the raw zone except to the pseudonymized zone's project.

When three zones aren't enough — the four-zone extension

The three-zone model treats all anonymized data as a single tier. This works when every anonymized-data consumer has the same threat profile. But when an engagement involves external data sharing, the anonymized tier conflates two fundamentally different guarantees:

• Structural anonymization (k-anonymity, l-diversity — see Module 04) protects against re-identification using the dataset's own quasi-identifier structure. Its guarantees depend on the quasi-identifier set being complete — an attacker with auxiliary data on unmodeled fields can potentially break it. It also doesn't compose: two k-anonymous releases of overlapping data can be cross-referenced.
• Differential privacy makes no assumptions about attacker knowledge. The guarantee holds against arbitrary auxiliary data, up to the epsilon bound, and composes formally.

When these outputs serve different consumers — structurally anonymized records for internal broad access, DP aggregates for external partners — they should have separate access policies:

Zone	Content	Access	Threat model
Structurally anonymized	k-anonymous record-level data	Internal broad — analysts, BI, data products	Bounded adversary — known quasi-identifiers
DP-guaranteed outputs	DP aggregates, synthetic data	Widest — partners, regulators, monetization	Unbounded adversary — arbitrary auxiliary data

Extend to four zones when: data leaves the organization, record-level and aggregate outputs coexist with different consumers, or multiple releases of overlapping data are published. Stay with three when: all anonymized data serves the same consumers, or only aggregate DP outputs are produced.

The four-zone BigQuery implementation adds one dataset:

├── dataset: anonymized_cdrs       (zone 2: structurally anonymized)
│   └── Broad internal IAM
└── dataset: dp_outputs            (zone 3: DP-guaranteed)
    └── Safe for external sharing

When using four zones, DP aggregation sources from zone 1 (pseudonymized), not zone 2 — the DP mechanism needs full-granularity data, not the already-generalized version. The reference architectures later in this module use the four-zone model, as they cover engagements with external sharing. For internal-only architectures, collapse zones 2 and 3 into a single anonymized tier.

Engineer takeaway

Start with three zones — the established baseline. Most internal analytics runs in the pseudonymized zone with DDM. Split the anonymized tier into structurally anonymized + DP-guaranteed only when the engagement involves external sharing or consumers with unknown auxiliary data. Don't over-anonymize for internal use cases where access controls are already in place.

---

Pipelines — moving data between zones

Zones define where data lives — but data doesn't arrive pre-sorted into zones. Raw records flow in from source systems, and pipelines transform them into each successive zone: raw data becomes pseudonymized records, and pseudonymized records become anonymized outputs. These pipelines split into two paths because per-record transforms (pseudonymization) can run in real time, while dataset-level operations (k-anonymity, DP) require the full dataset and are inherently batch. The streaming path produces the pseudonymized zone; scheduled batch jobs produce the anonymized zone.

Anonymize at ingestion

The strongest privacy posture transforms data before it reaches any persistent storage — the pipeline reads from the source, applies per-record de-identification, and writes only transformed output to the data lake. The cryptographic transforms used (FPE, deterministic encryption, HMAC) are covered in Module 07.

This approach reduces the blast radius (a compromised data lake exposes only pseudonymized records), simplifies access control (downstream consumers never need raw data access), and provides regulatory clarity (data residency and retention obligations apply to the source system, not the entire platform).

The trade-off is reprocessing rigidity — if the anonymization logic changes, you cannot re-derive the output from stored data and must re-ingest from the source, which may be impossible for ephemeral streams like Pub/Sub with limited retention. Schema evolution and debugging are also harder without raw data in the lake.

This pattern is the default for greenfield data platforms and for engagements where the client's DPO insists on data minimization at the storage boundary. Avoid it when the engagement requires iterative refinement of anonymization parameters, or when the source cannot be re-queried (one-time data drops, third-party feeds).

Streaming and batch patterns

Per-record transforms work identically in streaming and batch — the choice depends on freshness requirements, not anonymization logic. Module 05 covers the individual tools (Sensitive Data Protection / SDP, BigQuery DDM, ARX, Privacy on Beam); this section shows how they combine.

Streaming (Dataflow + Pub/Sub):

Source system → Pub/Sub topic
  → Dataflow streaming pipeline
    → Parse and validate
    → Per-record de-identification (SDP API, batched)
    → Custom transforms (cell-ID generalization, business logic)
    → BigQuery Storage Write API (exactly-once mode)
  → [Errors → Dead-letter Pub/Sub topic → Retry pipeline]

Key decisions: batch SDP API calls within windows (100–500 records per batch to avoid rate limits), use BigQuery Storage Write API in committed mode for exactly-once semantics, and route failed records to a dead-letter topic rather than dropping them silently.

Batch (Dataflow or Cloud Run jobs):

Source (GCS, BigQuery, JDBC)
  → Dataflow batch pipeline or Cloud Run job
    → Read and partition
    → Per-record de-identification (SDP API, batched)
    → Custom transforms
    → BigQuery load job (WRITE_TRUNCATE or merge)
  → [Failed records → Error table in BigQuery]

Key decisions: partition large datasets by date or logical segment (so failures don't require full re-processing), use idempotent writes (WRITE_TRUNCATE or MERGE with dedup keys), and use Dataflow Flex Templates or Cloud Run for autoscale-to-zero cost optimization.

Structural anonymization and DP are always separate batch jobs. k-anonymity needs the full dataset to compute equivalence classes; DP needs it to calibrate noise. These run on a schedule (weekly, nightly) over the pseudonymized zone and write to the anonymized zone. They cannot run inline in a streaming pipeline.

Deep dive

Late-arriving data

Records that arrive after their logical partition has been structurally anonymized create a consistency challenge — inserting a new record may break the equivalence class structure.

Strategy	Mechanism	Trade-off
Append and regenerate	Late records go to the pseudonymized zone; the anonymized zone is regenerated on the next batch run	Simple, eventually consistent
Micro-batch re-anonymization	Trigger re-anonymization of the affected partition on arrival	Better consistency, higher cost
Cutoff and drop	Records arriving after a threshold (e.g., 7 days) are logged but not processed	Simple but lossy
Separate late-arrival partition	Accumulate in a staging partition, merge on a schedule	Balances consistency with cost

For telco CDR modernization, append-and-regenerate is typical — late CDRs land in the pseudonymized zone immediately, and the weekly batch incorporates them.

---

Query-time masking — different views without separate copies

Zones handle the storage-level question ("where does this data physically exist?"). But within a zone — particularly the pseudonymized zone — different users still need different views: data engineers see raw pseudonymized values, analysts see hashed columns, dashboard service accounts see nullified fields. Creating a separate dataset for every access level is impractical. Query-time masking solves this by applying column-level transforms at query time, based on who is asking, without modifying stored data.

BigQuery DDM and policy tags

BigQuery Dynamic Data Masking provides per-column, per-role masking with three components:

1. Taxonomy — a classification scheme (e.g., PII_Classification) scoped to a project and region.
2. Policy tags — categories within the taxonomy (High_PII, Medium_PII, Low_PII), each with a masking rule (hash, nullify, default value, date-year mask, last-four characters).
3. IAM bindings — fineGrainedReader sees raw data, maskedReader sees masked output, neither role gets a permission error.

Taxonomy design: start sensitivity-based (High/Medium/Low), one taxonomy per data domain. Use role-based taxonomies when different teams need different masking on the same sensitivity level, or regulation-based taxonomies for multi-jurisdictional data. Everything is Terraform-manageable (google_data_catalog_taxonomy, google_data_catalog_policy_tag, google_bigquery_datapolicy_data_policy).

Deep dive

Row-level access and authorized views

Column masking handles "which fields." Row-level access policies handle "which rows" — a regional analyst sees only their region's records:

CREATE ROW ACCESS POLICY region_filter
ON `analytics-platform.pseudonymized_cdrs.cdrs`
GRANT TO ('group:analysts-north@aliz.ai')
FILTER USING (region = 'North');

Column and row policies compose naturally. For complex filtering logic (joining against permissions tables, business rules), use authorized views instead.

When masking is enough — and when it isn't

Masking is sufficient when all consumers query through BigQuery (or tools that use BigQuery — Looker, Connected Sheets, dbt, Dataform) and data does not leave the platform.

Masking is not sufficient when data is exported to files or partner systems (the masking is bypassed), when irreversible transformation is required (masking is access control, not anonymization), or when formal privacy guarantees (k-anonymity, DP) are needed.

BigQuery's DDM extends to Lakehouse for Apache Iceberg tables via compute delegation — when queries are routed through BigQuery's engine (BigQuery Storage API), the same policy tags and masking rules apply. The limitation: this only works for BigQuery-routed queries. If the Iceberg table is read directly by Spark, Trino, or another engine accessing the underlying object store, BigQuery's access control is bypassed entirely — the reader sees raw data. This means DDM guarantees in the pseudonymized zone hold only when all access goes through BigQuery; direct-engine access requires separate authorization controls (Apache Ranger or equivalent).

The common mistake: treating DDM as anonymization. When a client says "we've masked the sensitive columns," ask: does data ever leave BigQuery? If it does, the consumer receives raw values. For data that crosses boundaries, pipeline-integrated transforms are required.

Engineer takeaway

Masking controls who sees what; anonymization transforms what exists. Use masking for internal access tiering within the pseudonymized zone, and pipeline transforms for data that leaves the platform. The pseudonymized zone + DDM handles most internal analytics workloads without ever needing structural anonymization.

---

Putting it together — architecture decision tree

Before diving into the reference architectures, the following decision tree captures the major design choices covered in this module. Each decision maps to a section above and leads to the pattern you'll see in the telco and e-commerce reference architectures below.

---

Reference architecture — telco CDR modernization

This section ties zones, pipelines, and masking together for the most common Aliz anonymization engagement: modernizing a telco's CDR processing pipeline on GCP. Because CDR modernization involves external data sharing (partner feeds, roaming analytics, regulatory reporting), this architecture uses the four-zone model. For telco-specific data types, regulatory overlays, and the full vertical context, see the telco industry deep dive.

End-to-end: CDR ingestion through anonymized analytics

Telco network elements (MSC, GGSN, IMS)
               │
               ▼
┌─────────────────────────────────────┐
│  ZONE 0 — RAW (short retention)     │
│  BigQuery dataset: raw_cdrs         │
│  - Full PII: MSISDN, IMSI, IMEI     │
│  - Access: ingestion SAs only       │
│  - Retention: 7 days                │
│  - VPC-SC perimeter enforced        │
└──────────────┬──────────────────────┘
               │ Dataflow streaming pipeline
               │ (per-record pseudonymization)
               ▼
┌─────────────────────────────────────┐
│  ZONE 1 — PSEUDONYMIZED             │
│  BigQuery dataset: pseudo_cdrs      │
│  - MSISDN → FPE, IMSI → det. enc.,  │
│    IMEI → HMAC                      │
│  - Quasi-identifiers at original    │
│    granularity                      │
│  - DDM: hash masking for analysts,  │
│    raw for engineers                │
└──────────────┬──────────────────────┘
               │ Batch jobs (weekly)
               ▼
┌─────────────────────────────────────┐
│  ZONE 2 — STRUCTURALLY ANONYMIZED   │
│  BigQuery dataset: anon_cdrs        │
│  - k=50, l-diversity l=3 (ARX)      │
│  - Cell tower → cluster, timestamp  │
│    → hour, duration → band          │
│  - Access: internal broad           │
└─────────────────────────────────────┘

       ┌──────────────────────────────────────────┐
       │  ZONE 3 — DP-GUARANTEED OUTPUTS           │
       │  BigQuery dataset: dp_outputs             │
       │  - Subscriber counts, avg duration,       │
       │    mobility matrices (ε=1.0)              │
       │  - Access: partners, external             │
       │  - Sourced from Zone 1 (pseudonymized),   │
       │    NOT Zone 2 — DP needs full-granularity │
       │    data, not already-generalized data      │
       └──────────────────────────────────────────┘

Component	Tool	Stage	What it does
Ingestion	Dataflow (streaming)	Zone 0 → Zone 1	Per-record SDP transforms
Per-record pseudonymization	SDP CryptoReplaceFfxFpeConfig, CryptoDeterministicConfig, CryptoHashConfig	Streaming	FPE for MSISDN, det. enc. for IMSI, HMAC for IMEI
Quasi-identifier generalization	Custom Beam transforms	Streaming (basic) + Batch (structural)	Cell tower → cluster (streaming). Full k-anonymity (batch, ARX)
Dynamic data masking	BigQuery policy tags + DDM	Query time	Role-based column masking in zone 1
Structural anonymization	ARX (Cloud Run job)	Zone 1 → Zone 2 (weekly)	k-anonymity + l-diversity
DP aggregation	BigQuery DIFFERENTIAL_PRIVACY / Privacy on Beam	Zone 1 → Zone 3 (scheduled)	Materialized DP output tables
Key management	Cloud KMS (HSM)	Zones 0–1	See Module 07
Access control	IAM + VPC-SC + DDM	All zones	Zone-based access + column masking

Deep dive

The layered privacy strategy

Each layer addresses threats the others cannot:

Layer 1 — Identifier pseudonymization (streaming): MSISDN → FPE (format compatibility for downstream billing systems), IMSI → deterministic encryption (joinable across datasets), IMEI → HMAC with shared key (one-way, no re-identification needed — destroying the shared key erases all IMEI pseudonyms, which is acceptable since per-device erasure isn't required). See Module 07 for the transform decision framework.

Layer 2 — Quasi-identifier generalization (batch): Cell tower → cluster (5–10 towers), timestamp → hour bucket, duration → band. Target k=50 for internal sharing, k=100 for external. Combined with entropy l-diversity (l=3) on sensitive attributes.

Layer 3 — DP aggregate outputs (batch): Materialized DP tables (ε=1.0, max_groups_contributed=3) for subscriber counts, duration averages, mobility matrices. Post-processing immunity means these are computed once and queried freely.

Threat	Layer 1 alone	+Layer 2	+Layer 3
Direct identifier lookup	Mitigated	Mitigated	Mitigated
Behavioral fingerprinting	Vulnerable	Mitigated	Mitigated
Homogeneity attack	Vulnerable	Mitigated (l-diversity)	Mitigated
Differencing attack on aggregates	Vulnerable	Partially vulnerable	Mitigated
Key compromise	All pseudonyms exposed	QIs already generalized	DP outputs unaffected

Engineer takeaway

This is a template, not a prescription. Adapt zone boundaries, k values, DP parameters, and tool choices per engagement. But the layered structure — identifier pseudonymization in streaming, structural anonymization in batch, DP for aggregates — is the pattern to follow. Each layer addresses threats the others cannot.

---

Reference architecture — e-commerce / retail customer data platform

Where the telco architecture handles a single high-volume event stream (calls), e-commerce handles multiple data domains — transactions, profiles, clickstream, loyalty, delivery — each with different PII densities and update patterns. The four-zone layered pattern is the same, but the data shape and privacy challenges shift. For the full retail vertical context, see the retail industry deep dive.

Data domains

Domain	Key identifiers	Quasi-identifiers	Update pattern
Transactions	Customer ID, card token, email	Timestamp, amount, product category, store	Append-only stream
Customer profiles	Customer ID, name, email, phone, address	Age/DOB, postal code, loyalty tier	Slowly changing dimension
Browsing / clickstream	Customer ID, cookie/device ID, IP	Timestamp, page viewed, session duration, device type	High-volume stream
Loyalty / CRM	Loyalty card number, customer ID	Points balance, tier, preferred store	Mixed
Delivery	Order ID, customer ID, delivery address	Delivery timestamp, carrier, postal code	Event stream

End-to-end data flow

Source systems (POS, e-commerce platform, web analytics, CRM, OMS)
               │
               ▼
┌──────────────────────────────────────────┐
│  ZONE 0 — RAW                            │
│  Per-domain datasets, full PII           │
│  Access: ingestion SAs only              │
│  Retention: 14 days, VPC-SC enforced     │
└──────────────┬───────────────────────────┘
               │ Domain-specific pipelines
               ▼
┌──────────────────────────────────────────┐
│  ZONE 1 — PSEUDONYMIZED                  │
│  Customer ID → det. enc. (shared key     │
│    across all domains for joins)         │
│  Card/email/cookie → HMAC (domain keys)  │
│  Name → redacted, address → postal       │
│    prefix, DOB → age band                │
│  Policy tags + DDM per domain            │
└──────────────┬───────────────────────────┘
               │ Batch jobs (nightly/weekly)
               ▼
┌──────────────────────────────────────────┐
│  ZONE 2 — STRUCTURALLY ANONYMIZED        │
│  Transactions: k=25, l-diversity l=3     │
│  Clickstream: k=50 on session profiles   │
│  Internal broad access                   │
└──────────────────────────────────────────┘

┌──────────────────────────────────────────┐
│  ZONE 3 — DP-GUARANTEED OUTPUTS          │
│  Revenue × category × region × week      │
│  Basket distributions, segment sizes,    │
│  conversion funnels (all DP)             │
│  Partners, external sharing              │
│  Sourced from Zone 1 (pseudonymized),    │
│  NOT Zone 2                              │
└──────────────────────────────────────────┘

Key differences from the telco pattern

1. Cross-domain identity resolution comes first. Telco data anchors to a single identifier (MSISDN). E-commerce spans customer accounts, loyalty cards, cookies, email, and delivery addresses. All must resolve to the same pseudonym for cross-domain analytics. The solution: identity resolution in the raw zone (match cookie → customer ID), then a shared deterministic encryption key for the canonical customer ID across all domain pipelines. Secondary identifiers use domain-specific HMAC keys. See Module 07 for key management patterns.

2. Browsing data has uniquely high re-identification risk. A browsing history of 3–5 product views is often enough to single out a shopper. For many use cases, the pseudonymized zone with DDM is sufficient — recommendation models and conversion optimization run on pseudonymized data within the platform.

3. Transaction sequences are quasi-identifiers. A single purchase of €47.83 isn't identifying, but a sequence of three purchases with timestamps and store locations is a behavioral fingerprint. Structural anonymization must treat the sequence as a quasi-identifier: amount bucketing, session-level aggregation, or per-customer weekly aggregation for partner sharing.

4. Slowly changing dimensions need merge-aware pseudonymization. Customer profiles change over time. Pseudonymize on the canonical customer ID (stable), not on fields that change. Use SCD Type 2 in the pseudonymized zone — the pseudonym is stable across versions; changing fields (address, tier) are treated per their classification.

5. Consent models are more granular. Retail customers interact through multiple touchpoints, each with potentially different consent. The pseudonymized zone may need consent-aware views — row-level access policies that filter records based on consent flags per purpose (loyalty analytics vs. browsing behavior tracking).

Deep dive

Common retail analytics use cases by zone

Use case	Zone	Key consideration
Product recommendations	Zone 1 (pseudonymized)	DDM hides identity; model features are behavioral
Conversion funnel analysis	Zone 1 (internal), Zone 3 (external)	Small-segment funnels risk singling out individuals — use DP for external
Market basket analysis	Zone 2 (structurally anonymized)	Basket composition is the sensitive attribute
Customer segmentation	Zone 1 (pseudonymized)	Segment membership (churn risk, price sensitivity) is itself sensitive
Partner data sharing	Zone 2 + Zone 3	DP for aggregates; structural anonymization only if line-item data is shared
Store location planning	Zone 3 (DP-guaranteed)	DP over transaction locations; suppress small-catchment stores

Engineer takeaway

The e-commerce architecture adds two complexities over the telco template: cross-domain identity resolution (must happen before pseudonymization) and granular consent management (shapes what data enters each domain pipeline). Start the design by mapping the identity graph and consent model — these determine the pipeline topology. The zone model follows from the telco template.

---

Next: Module 07 covers key management and crypto-shredding — the cryptographic infrastructure that powers the pseudonymization transforms in this module's pipelines and reference architectures.

---

Created by Claude Opus with ❤️ for the Aliz Data Engineering team.