Template: Architecture Recommendation
This is a ready-to-adapt example, not a universal standard. It shows a four-zone anonymization architecture for a telco CDR platform on GCP — the most common pattern in Aliz engagements. Every client's architecture will differ based on their vertical, data types, regulatory context, and infrastructure.
How to use this template:
- Pre-populated content shows realistic example values drawn from the knowledge base (telco CDR as the anchor use case). Replace examples with the client's actual data types, systems, and organizational structure.
- Adapt blocks in each section explain what to adjust and why — remove them from the final client-facing document.
See Module 06 — Architecture Patterns for the foundational zone model, Module 05 — Tooling Landscape for GCP service details, and Module 13 — Client Engagement for how this template fits into the engagement workflow.
Four-Zone CDR Anonymization Architecture
Replace with the specific architecture pattern and client context. For non-telco verticals: retail purchase data, financial transactions, energy smart meter readings — the zone model adapts, the services stay largely the same.
Current state
The current CDR analytics platform ingests call detail records from the mediation layer into a BigQuery data warehouse. Subscriber identifiers (MSISDNs, IMSIs) are pseudonymized using a dictionary-based lookup table — a flat mapping file that replaces each identifier with a random token. The dictionary is stored in a Cloud SQL instance accessible to the data engineering team.
Gaps identified during discovery:
- Identifier substitution only. Quasi-identifiers (cell tower IDs, timestamps, call durations, device types) pass through at full granularity. The behavioral structure of the data is intact — a small number of spatio-temporal data points can re-identify the vast majority of individuals regardless of the identifier pseudonymization.
- Single point of failure. The dictionary table is the sole re-identification key for the entire historical dataset. No HSM backing, no access logging beyond Cloud SQL audit logs, no key rotation.
- No structural anonymization. There is no Zone 2 — all downstream analytics and partner feeds consume the pseudonymized dataset directly.
- No re-identification risk assessment. The organization treats the pseudonymized output as "anonymized" without measurement.
Replace CDR-specific details with the client's actual data types and systems. For retail: purchase records, customer IDs, loyalty card numbers. For finance: transaction logs, account numbers, KYC records. The gap patterns are remarkably consistent across verticals.
Proposed architecture
A four-zone architecture that separates data by privacy level, with controlled pipelines between zones and access policies that widen as privacy guarantees strengthen.
| Zone | Content | Access | Retention |
|---|---|---|---|
| Zone 0 — Raw | Unmodified CDRs with MSISDN, IMSI, IMEI, full cell-ID, exact timestamps | Mediation systems and reprocessing pipeline only; VPC-SC enforced | 7 days (reprocessing buffer) |
| Zone 1 — Pseudonymized | MSISDN → HMAC-SHA256; IMSI → HMAC-SHA256; IMEI → HMAC-SHA256 (or suppress if not needed for analytics); cell-ID and timestamps at original granularity | Data engineers (full), analysts (DDM-masked) | Per data classification policy |
| Zone 2 — Structurally anonymized | Cell-ID → cluster (k≥50 subscribers per cluster); timestamp → 1-hour bucket; duration → categorical bucket; k-anonymous with l-diverse sensitive columns | Broad internal — BI, analytics, data products | Years (outside GDPR scope if validated) |
| Zone 3 — DP-guaranteed | Aggregated statistics with calibrated noise (ε ≤ 1.0); synthetic CDR data with DP guarantees | Widest — partners, regulators, data monetization | Indefinite |
Adjust zone retention, k thresholds, and ε values to the client's regulatory context and risk appetite. For external sharing scenarios, consider whether Zone 3 is needed or whether Zone 2 with access controls suffices. For internal-only analytics, Zones 2 and 3 can collapse into a single anonymized tier.
Key components
| Component | GCP service | Role |
|---|---|---|
| Ingestion | Pub/Sub | Receives raw CDR stream from mediation layer |
| PII detection + transformation | Sensitive Data Protection (SDP) | Detects MSISDNs, IMSIs, IMEIs via custom InfoTypes; applies HMAC transforms via CryptoHashConfig with kms_wrapped key |
| Pipeline orchestration | Dataflow (Apache Beam) | Streaming pipeline: SDP transforms → schema validation → BigQuery write (Zone 1). Quasi-identifiers pass through at original granularity. |
| Raw buffer | Cloud Storage | Zone 0: 7-day retention of unmodified CDRs for reprocessing |
| Data warehouse | BigQuery | Hosts Zones 1–3 as separate datasets with policy tags and DDM rules |
| Key management | Cloud KMS (HSM protection level) | Manages the symmetric KEK that wraps the separately generated HMAC secret. Configure KEK rotation explicitly (recommended: 90 days); auto-rotation does not re-wrap existing blobs — explicit re-wrap needed before disabling old KEK versions |
| Access control | BigQuery policy tags + IAM + VPC-SC | Column-level masking, role-based access per zone, network perimeter around Zones 0–1 |
| Classification | SDP discovery service | Continuous profiling of BigQuery datasets; auto-detection of new PII columns |
| Audit | Cloud Audit Logs | Admin Activity + Data Access logs for all anonymization pipeline operations |
| Structural anonymization | ARX (containerized batch job) | k-anonymity + l-diversity on Zone 1 → Zone 2 batch pipeline |
| DP aggregation | Privacy on Beam / BigQuery DIFFERENTIAL_PRIVACY | Calibrated noise on aggregate queries for Zone 3 outputs |
For on-prem or hybrid: replace Pub/Sub with Kafka, Dataflow with Spark Structured Streaming, SDP with Presidio, BigQuery with Hive + Apache Ranger, Cloud KMS with HashiCorp Vault. See Module 05 for the full mapping.
Data flow
Zone 0 (raw buffer). Pub/Sub fans out to both a Cloud Storage sink (Zone 0, 7-day retention for reprocessing) and the Dataflow transformation pipeline. Zone 0 holds unmodified CDRs — accessible only to mediation systems and the reprocessing pipeline, behind VPC-SC.
Zone 0 → Zone 1 (streaming). The Dataflow pipeline applies SDP de-identification templates: HMAC-SHA256 via CryptoHashConfig (with a kms_wrapped HMAC key) for MSISDNs, IMSIs, and IMEIs. Quasi-identifiers (cell-IDs, timestamps, durations) pass through at original granularity — Zone 1 is pseudonymized, not structurally anonymized. Schema validation ensures no unrecognized fields pass through untransformed. Output lands in BigQuery pseudonymized_cdrs dataset with policy tags applied.
Zone 1 → Zone 2 (batch). A scheduled ARX batch job (containerized on Cloud Run) reads from Zone 1 and applies the structural anonymization transforms: cell-ID generalization to clusters, timestamp bucketing to 1-hour windows, duration categorization, and k-anonymity (k≥50) with l-diversity on sensitive columns. This is where quasi-identifiers are coarsened — the step that closes the gap between identifier substitution and genuine anonymization. Output lands in anonymized_cdrs — available for broad internal analytics.
Zone 1 → Zone 3 (batch or query-time). Privacy on Beam aggregates Zone 1 data with calibrated Laplace/Gaussian noise (ε ≤ 1.0) for partner feeds and data monetization outputs. Alternatively, BigQuery's DIFFERENTIAL_PRIVACY clause provides query-time DP for ad hoc analytical queries. Output in dp_outputs is suitable for external sharing, subject to validated DP configuration (contribution bounds, δ, and cumulative budget tracking across releases — see Module 04).
Security and access model
| Zone | IAM boundary | Column-level controls | Network perimeter |
|---|---|---|---|
| Zone 0 | Project-level IAM; explicit deny for human users; pipeline service account is the only reader | N/A — no human query access | VPC-SC perimeter containing Zones 0–1 |
| Zone 1 | Dataset-level IAM | Policy tags with DDM: fineGrainedReader (engineers) sees pseudonyms; maskedReader (analysts) sees nullified/hashed values | VPC-SC perimeter (same as Zone 0) |
| Zone 2 | Dataset-level IAM; broad internal access | No masking needed — data is structurally anonymized | Outside VPC-SC perimeter |
| Zone 3 | Table-level IAM; includes external principals | No masking needed — data has DP guarantees | Outside VPC-SC perimeter |
Key management: The HMAC secret is generated separately and wrapped with a Cloud KMS symmetric KEK at HSM protection level. The wrapped blob is stored in the SDP de-identification template. At transform time, the SDP service agent calls KMS to unwrap the HMAC key — grant it cloudkms.cryptoKeyVersions.useToDecrypt on the KEK. Configure KEK rotation explicitly (recommended: 90 days); auto-rotation does not re-wrap existing blobs — schedule explicit re-wraps before disabling old KEK versions. No human user has direct access to the HMAC key material.
Audit trail: Cloud Audit Logs capture administrative actions (SDP template changes, KMS key operations, IAM policy changes, Dataflow job lifecycle) via Admin Activity logs and data access events (BigQuery queries) via Data Access logs. Logs are exported to a separate audit project with immutable retention.
For clients with stricter key sovereignty requirements (e.g., Swiss banking), use Cloud EKM to hold keys outside Google's infrastructure. For multi-jurisdictional deployments, consider per-region KMS keys with separate DEKs per jurisdiction.
Implementation roadmap
| Phase | Duration | Deliverables | Outcome |
|---|---|---|---|
| 0 — Discovery + quick wins | 2–4 weeks | Gap analysis, classification map, remediation roadmap. Replace unsalted hashing with HMAC. Add policy tags to BigQuery. | Measurable risk reduction; documented plan |
| 1 — Core pipeline (Zones 0–1) | 6–8 weeks | Dataflow streaming pipeline with SDP transforms, Cloud KMS key management, BigQuery zone structure, SDP discovery, Cloud Audit Logs | Functioning pseudonymization pipeline with proper key management and access controls |
| 2 — Structural anonymization (Zone 2) | 6–8 weeks | ARX batch job, generalization hierarchies, automated privacy checks (PII scanning, k-anonymity assertions) | Zone 2 dataset available for broad internal analytics |
| 3 — DP outputs (Zone 3) | 6–8 weeks | Privacy on Beam / BigQuery DP integration, DP budget tracking, synthetic data pipeline (if needed) | Formally private output channel for external consumption |
| 4 — Governance (ongoing) | Parallel with Phases 1–3 | Anonymization policy, RACI matrix, periodic re-assessment cadence, incident response procedures | Sustainable governance framework |
Phase sequencing depends on the driver. Regulatory deadline → compress Phases 0–1. Partner data-sharing deadline → prioritize Phase 3. Compliance audit → start with Phase 4 documentation while building Phases 1–3 in parallel. See Module 13 for alternative sequencing.
Cost and effort estimate
Assumptions: 3-person Aliz engineering team, existing GCP environment with BigQuery and Dataflow, CDR volume ~50M records/day, client data engineering team available for integration work.
| Phase | Aliz effort | Key GCP cost drivers |
|---|---|---|
| Phase 0 | 2–3 person-weeks | Minimal — SDP inspection ($0.03/GB for profiling), ARX risk analysis (open-source) |
| Phase 1 | 8–12 person-weeks | Dataflow (streaming job compute), Cloud KMS (0.15/10K HSM operations), SDP de-identification (per-transform pricing) |
| Phase 2 | 6–8 person-weeks | ARX compute (Cloud Run jobs), BigQuery storage for Zone 2 dataset |
| Phase 3 | 6–8 person-weeks | Privacy on Beam compute, BigQuery storage for DP outputs, synthetic data generation compute (if applicable) |
| Phase 4 | 2–4 person-weeks (initial) | N/A — organizational/documentation work |
Ongoing operational cost: primarily Dataflow streaming compute, BigQuery storage across zones, and Cloud KMS key operations. SDP discovery adds 2,500/unit/month) for organizations with >83 TB to profile.
Estimates assume a single-vertical, single-region deployment. Multi-region, multi-jurisdictional, or on-prem/hybrid deployments add complexity. Client-side effort (governance setup, stakeholder alignment, DPIA preparation) is not included — scope this separately during discovery.