9. Governance and Compliance
Modules 06 through 08 showed how to build an anonymization platform — data zones, pipelines, key infrastructure, AI-specific architecture patterns. But a platform is only as good as the decisions that govern it. Which data is sensitive? Which technique applies? Who decides? What legal basis justifies processing personal data when anonymization isn't feasible? And who is accountable when the answer to "is this data anonymous?" turns out to be wrong?
These are organizational and legal questions, not engineering ones — but they land on the data engineer's desk because the engineer is the one building the pipeline. Without clear answers, engineers make their own calls. Each call is reasonable in isolation. In combination, they're inconsistent: one pipeline uses HMAC (keyed hashing), another uses unsalted SHA-256 because nobody said not to; one dataset is labeled "anonymized" when it's pseudonymized; a partner receives data without a DPIA because the team assumed the data governance lead would handle it.
This module covers the governance layer — the policies, roles, legal frameworks, and organizational structures that make anonymization decisions consistent and defensible. Module 10 covers the technical machinery that enforces these decisions: automated classification, pipeline monitoring, lineage tracking, key rotation, and incident response.
The governance stack described here is calibrated for enterprise-scale data platforms — telcos, banks, large retailers with dedicated teams and formal review processes. Not every client fits that profile. Throughout this module, look for the SME shortcut boxes: they show where smaller organizations can take a lighter-weight approach that still satisfies GDPR requirements. The Scaling down section at the end pulls these together into a complete lightweight alternative.
The anonymization policy
An anonymization pipeline without a policy is a system where everyone improvises. Engineering picks techniques based on what's convenient. Legal applies GDPR conservatively and blocks projects. Security focuses on key management without considering analytical utility. The DPO signs off on "anonymized" datasets without a shared definition of what that means. Each team's decisions are reasonable in isolation and inconsistent in combination.
The policy is the document that makes decisions consistent. It doesn't need to be long — a 5–10 page document that engineering teams can actually reference mid-project is better than a 50-page policy nobody consults. What matters is that it covers seven areas:
| Area | What it defines | Who owns it |
|---|---|---|
| Scope | Which datasets, systems, and data flows are covered. Which are explicitly excluded (e.g., aggregate-only reporting tables). | Data governance lead |
| Classification scheme | Sensitivity tiers (Critical / High / Medium / Low / Public) and the criteria for assigning each tier | Data governance lead + data stewards (DPO advises) |
| Approved techniques | Which anonymization and pseudonymization techniques are sanctioned, per sensitivity tier and use case. References the technique selection framework and architecture patterns. | Data engineering + security |
| Zone mapping | Which data zone each dataset belongs to, and the privacy guarantees each zone provides | Data architecture |
| Retention rules | How long data stays in each zone. Raw zone: days to weeks. Pseudonymized zone: per data classification. Anonymized zone: indefinite if truly anonymous. | Legal + data governance |
| Exception handling | How to request an exception (e.g., retaining raw data longer for a specific investigation), who approves it, and how exceptions are tracked and time-boxed | CDO / CPO (DPO advises) |
| Review cadence | How often the policy is reviewed — annually at minimum, plus triggered reviews after regulatory changes, incidents, or significant architecture changes | DPO + data governance lead |
The templates directory of this knowledge base includes a ready-to-adapt anonymization policy template that can be customized per engagement.
A seven-section policy document is appropriate for organizations with multiple teams consuming anonymized data. For a small team — say, two to five engineers with a single data pipeline — a 2-page internal wiki page is sufficient. Cover three things: (1) which fields are PII and what technique applies to each, (2) who approves exceptions and external sharing, and (3) when to re-review (annually, or after a data breach or regulatory change). The rest can be handled in conversation when the team is small enough to sit in one room.
The policy's critical function is creating a single source of truth for "what technique do we use for this type of data?" Without it, the same decision gets made differently by different engineers on different projects — and inconsistency is how anonymization gaps appear.
Designing the sensitivity taxonomy
The classification scheme in the policy maps data types to sensitivity tiers. These tiers drive both technique selection and access control — they're the bridge between the policy document and the platform enforcement that Module 10 covers:
| Tier | Data types (telco examples) | Anonymization requirement | Access scope |
|---|---|---|---|
| Critical | Raw MSISDN (phone number), IMSI (SIM identifier), IMEI (device identifier), subscriber name, billing address | Must be pseudonymized or redacted before leaving the raw zone | Break-glass only |
| High | IP address, precise location (cell ID), call content, SMS text | Pseudonymized in Zone 1; generalized or suppressed in Zone 2 | Data engineering (masked for analysts) |
| Medium | Call duration, data volume, service type, device model | Bucketed or generalized in Zone 2 | Analysts with dynamic data masking (DDM) |
| Low | Cell cluster ID (post-generalization), time bucket, aggregated counts | No further transformation needed | Broad internal |
| Public | Network coverage maps, aggregate statistics with DP guarantees | Safe for external sharing | Unrestricted |
Five tiers work for large data platforms with diverse consumers. Smaller organizations can collapse to three (High / Medium / Low) without losing clarity. What matters is that every data field the pipeline touches has an assigned tier, and every tier has a defined handling requirement.
Data ownership and stewardship
Once raw data passes through the anonymization pipeline, the ownership question splits. The raw CDR (Call Detail Record) data has a clear owner — the telco's network operations team. But who owns the anonymized output? Who can grant access? Who is accountable if it turns out the "anonymized" data can be re-identified?
The standard data governance model (codified in the DAMA DMBOK framework and widely adopted across industries) distinguishes three stewardship roles. For anonymization, a fourth — the DPO — is essential because GDPR requires an independent advisor on data protection compliance:
| Role | Responsibility | Source |
|---|---|---|
| Data owner (business) | Defines what the data can be used for, approves access grants, accountable for business decisions made on the data — including the classification of outputs as anonymized or pseudonymized | DAMA DMBOK |
| Data steward | Validates that classification is correct, manages retention and deletion schedules, bridges business requirements and technical implementation | DAMA DMBOK |
| Data custodian (engineering) | Operates the pipeline, manages key infrastructure, monitors transform health | DAMA DMBOK |
| DPO | Advises on re-identification risk, monitors compliance with the anonymization policy, provides guidance on DPIAs. Does not own decisions — GDPR Art. 39 defines the DPO as advisory and monitoring; decision accountability stays with the controller (data owner / CDO / CPO) | GDPR Art. 37–39 |
Not every organization uses these exact titles — smaller teams combine roles, and some clients use different terminology. What matters is that the accountability lines are explicit: someone must own the business decision, someone must operate the technical infrastructure, and someone must sign off on the privacy classification. The anonymization policy (specifically its "roles and responsibilities" section) should name the individuals or teams assigned to each role. The DPO role is mandated by GDPR Article 37 for certain organizations (public authorities, large-scale systematic monitoring, large-scale processing of special categories); others may designate it voluntarily. The remaining roles are organizational choices — assigned by senior leadership or the data governance lead, and documented in the policy so that accountability is traceable.
One structural principle cuts across all four roles: separation of duties. No single individual should hold both the pseudonymization key and unsupervised access to the pseudonymized dataset — that combination defeats the purpose of pseudonymization. Similarly, re-identification requests should be approved by a controller-side decision-maker (CDO, CPO, or data owner) outside the engineering team that operates the pipeline, with the DPO consulted and each approval logged for audit. The EDPB Guidelines 01/2025 reinforce this: pseudonymization secrets must be "separately stored and protected" with access controls preventing unauthorized attribution.
The controller (data owner, CDO, or CPO) is accountable for the claim that data is "anonymized" rather than "pseudonymized" — with the DPO advising on the assessment. This is not purely a technical determination — it requires assessing what means of re-identification are "reasonably likely to be used" (GDPR Recital 26), which depends on the recipient's capabilities and available auxiliary data. The CJEU ruling in EDPS v SRB (September 2025) confirmed that the same dataset can be personal data in one organization's hands and anonymous in another's — the assessment is relative to the specific recipient.
RACI for anonymization decisions
A RACI matrix makes the decision-making structure concrete:
| Decision | Responsible | Accountable | Consulted | Informed |
|---|---|---|---|---|
| Data classification policy | Data steward | CDO / CPO | DPO, legal, engineering | Executive leadership |
| Technique selection per field | Data engineering | Data owner (business) | DPO, analytics | Security |
| Pipeline implementation and operations | Data engineering | Data engineering lead | Security, DPO | Data owners |
| Key rotation schedule | Security / platform | CISO | Data engineering, DPO | Data owners |
| Re-identification risk assessment | Security / privacy | CDO / CPO | DPO, data engineering, legal | Analytics consumers |
| External data sharing approval | Data owner | CDO / CPO | DPO, legal | Analytics consumers |
| DPIA execution | Privacy / legal | CDO / CPO | DPO, data engineering, security | Data owners |
| Incident response (re-identification) | Security | CISO / CDO | DPO, legal, data engineering | Executive, affected subjects |
Cross-team coordination
The RACI defines who. Three structures define how:
Privacy engineering guild. A cross-functional working group (biweekly or monthly) with representatives from data engineering, security, legal, and analytics. Reviews: pipeline health metrics, upcoming schema changes, re-identification risk results, DSARs and crypto-shredding status, and new use cases requiring anonymization design decisions. The guild coordinates — it doesn't own execution.
Anonymization design review. Before any new data pipeline or significant modification goes to production, a lightweight review verifies: all PII fields are covered by the Sensitive Data Protection (SDP) template, techniques match the policy for each sensitivity tier, key management follows the established architecture, and monitoring is configured. A 30-minute checkpoint, not a multi-week process.
Shared dashboards. A Looker dashboard (sourced from the BigQuery governance metadata table and SDP profile history) accessible to all stakeholders. Engineering sees pipeline health; legal sees compliance status; the DPO sees re-identification risk trends. A shared view prevents the "everything is fine" problem where each team reports on their own silo without seeing the full picture.
The RACI matrix, privacy engineering guild, and design review process assume an organization large enough to have separate data engineering, security, legal, and analytics teams. For a small team, collapse this to two roles: (1) one person owns the anonymization pipeline, the classification, and the monitoring — they make and document the technical decisions, and (2) a second person (CTO, founder, or another controller-side decision-maker) reviews the privacy classification and approves external data sharing. If the organization has a DPO (or uses an external DPO-as-a-service, available from ~€200/month for SMEs), the DPO advises both roles but does not own the approval decision — that stays with the controller-side reviewer. The formality of the process scales with the team — what doesn't scale is having nobody accountable for the privacy classification.
Training and skills
Anonymization engineering requires a skill mix that most data engineering teams don't have by default:
| Skill area | Who needs it | Typical gap |
|---|---|---|
| Privacy models (k-anonymity, DP) | Data engineers, data scientists | Most engineers know hashing and masking but not formal privacy models |
| GDPR and ePrivacy fundamentals | Data engineers, architects | Engineers often treat privacy as a legal problem, not a design constraint |
| GCP SDP configuration | Data engineers | SDP has a steep configuration surface — de-identification templates, InfoType detectors, discovery actions |
| Key management (KMS, envelope encryption) | Platform engineers | Often handled by a dedicated security team, creating a knowledge silo |
| Re-identification risk assessment | Data engineers, privacy team | Requires statistical background that generalist engineers may lack |
The most effective ramp-up: pair a data engineer with the privacy/legal team on the first anonymization engagement. The engineer learns the regulatory constraints; the privacy team learns the technical trade-offs. After one or two engagements, the engineer can handle future projects with advisory support rather than full pairing.
Build vs buy
| Dimension | Build (custom pipeline + OSS) | Buy (managed platform) |
|---|---|---|
| Technique coverage | Full flexibility — any technique, any combination | Limited to what the platform supports |
| Governance and audit | Must be built — lineage, metadata, dashboards are custom code | Often included — governance UI, audit trails, compliance reports |
| Throughput at scale | Scales with pipeline infrastructure (Dataflow, Spark) | May have API quotas or throughput limits |
| Integration | Fits into existing pipeline architecture | May require architectural changes |
| Regulatory certification | Self-attested compliance | Vendor may provide certifications (SOC 2, ISO 27701) |
| Total cost of ownership | Lower license cost, higher engineering cost | Higher license cost, lower engineering cost |
| Time to first value | Weeks to months | Days to weeks |
For GCP-native shops: the "build on managed components" middle path is usually optimal. Use SDP for detection and transformation, BigQuery policy tags for DDM, Cloud KMS for key management, Knowledge Catalog for governance metadata, and Dataflow for pipeline orchestration. The individual components are managed services; the integration between them is custom code. This avoids both the full-custom burden and the vendor lock-in of an end-to-end anonymization platform.
When a third-party platform makes sense: when the organization lacks privacy engineering expertise and needs governance out of the box, when compliance certifications are a procurement requirement, or when the anonymization scope extends beyond GCP (multi-cloud, on-premises, SaaS data sources).
Lawful processing without anonymization
Everything in this knowledge base up to this point assumes anonymization is the goal. But on real engagements, the path sometimes forks — some processing tasks require data fidelity that anonymization destroys. Module 01 introduced this idea; Module 08 showed concrete examples. The question shifts from "how do we exit GDPR scope?" to "what legal basis justifies this processing, and what safeguards compensate for the absence of anonymization?"
When anonymization is the wrong tool
The common cases:
- LLM workloads that require natural language context (Module 08, Patterns 3–5) — anonymization destroys the linguistic structure the model needs for semantic search, summarization, or entity extraction
- NER model training — the model needs real entity examples to learn recognition patterns; anonymized training data produces a model that can't recognize the entities it was trained to find
- Fraud detection and network security — identifying individual bad actors requires individual-level data that anonymization would obscure
- Longitudinal studies — tracking the same individual over time requires consistent linkability, which anonymization eliminates
- Right-to-erasure requirement — GDPR Article 17 applies only to personal data, so truly anonymized data is exempt. But if the organization needs to process personal data (for the reasons above) and also handle erasure requests, the data must remain pseudonymized rather than anonymized. Erasure can be achieved two ways: physical deletion of the subject's records (feasible at moderate scale, works with any pseudonymization technique including HMAC), or crypto-shredding via per-subject DEK deletion (Module 07, more scalable but requires upfront architecture).
In all these cases, the organization processes personal data — GDPR applies in full.
Lawful basis and compliance obligations
GDPR requires a lawful basis (Article 6) for any processing of personal data. The choice of basis depends on the specific use case — there is no default. Two additional compliance obligations — Data Processing Agreements (Art. 28) and Data Protection Impact Assessments (Art. 35) — are not lawful bases themselves, but separate requirements that may apply alongside whichever basis is chosen.
Legitimate Interest — Article 6(1)(f). A frequently applicable basis for analytics, ML training, and AI workloads where consent is impractical — but not a default. Each use case requires a case-by-case assessment. The EDPB Opinion 28/2024 confirmed legitimate interest as a valid legal basis for AI model training and deployment (not the only one), and the CNIL recommendations (June 2025) called it "the most likely legal basis for AI developers" — while emphasizing that the three-step test below must be passed for each specific use case.
A Legitimate Interest Assessment (LIA) follows a three-step test:
- Identify the legitimate interest — must be lawful, specific, and real (not hypothetical). Examples: improving service quality, fraud detection, network optimization, product development.
- Necessity — is processing personal data necessary to achieve the interest, or could the same goal be achieved with anonymized data or no personal data at all? This is the critical question for data engineers: if anonymization would serve the purpose without unacceptable utility loss, that's the less intrusive path and legitimate interest is harder to justify.
- Balancing test — does the organization's interest outweigh the data subject's rights? Factors that tip the balance favorably: pseudonymization applied, data minimization practiced, opt-out mechanisms available, anti-memorization/regurgitation measures in place for AI models, transparency exceeding Article 13/14 requirements.
Document the LIA. Even when the assessment is favorable, the reasoning must be recorded and available for the DPA on request. A one-page LIA per use case is standard practice.
Data Processing Agreements — Article 28. When using cloud AI services (Vertex AI, Gemini API) with personal data, the cloud provider is a data processor and must operate under a DPA. Google Cloud's Cloud Data Processing Addendum (CDPA) covers this contractually:
- Customer data is not used to train Google's foundation models (paid Gemini API, Vertex AI)
- Data is processed only for the purposes specified in the CDPA (verify service-specific residency and retention terms for each AI service)
- Processing only per CDPA terms — no other purpose
- Zero-data-retention terms available for eligible enterprise customers
For data engineers, the practical implication: when building an AI pipeline that processes personal data through Vertex AI, confirm that the project's CDPA is in place and that the specific AI service is covered. Not all GCP AI services have identical data processing terms.
Data Protection Impact Assessment (DPIA) — Article 35. Required before processing that is likely to result in high risk to individuals. Article 35(3) lists mandatory triggers (systematic profiling with significant effects, large-scale processing of special categories, large-scale public monitoring), and supervisory authorities publish additional lists. "Use of new technologies" (Art. 35(1)) is a risk factor that contributes to the high-risk assessment, not an independent trigger — but in practice, many enterprise AI deployments involving personal data meet at least one mandatory criterion.
A DPIA for an AI system processing personal data must include:
- Systematic description of the processing operations (the pipeline architecture)
- Assessment of necessity and proportionality
- Risk assessment for data subjects (re-identification risk, model memorization risk, unauthorized access risk)
- Technical and organizational measures to address risks — pseudonymization, access controls, monitoring
- Residual risk assessment after measures are applied
DPIAs are not one-time documents. Review triggers include: model updates or retraining, new data sources, use case expansion, regulatory changes, and complaints. Annual reassessment is the recommended baseline.
A DPIA is mandatory only when processing is "likely to result in a high risk" — Article 35 lists specific triggers (systematic profiling, large-scale processing of special categories, large-scale public monitoring). A small e-commerce company running product recommendations on pseudonymized purchase history may not hit any trigger. When in doubt, the Article 29 Working Party criteria provide a 9-factor checklist — if fewer than two factors apply, a DPIA is generally not required. Even when a DPIA is required, proportionality applies — a 2–3 page assessment covering the pipeline architecture, risks, and mitigations is proportionate for an SME processing limited personal data.
The EDPB's Guidelines 01/2025 (post-consultation, pending final adoption) introduce the concept of pseudonymisation domain — the context within which pseudonymisation prevents attribution. The guidelines recommend that controllers define the scope of this domain (organizational unit, external recipients) and ensure that re-identification information is kept outside it. Key takeaway: pseudonymisation "is often not sufficient as a safeguard alone and needs to be combined with other measures." It supports the LIA balancing test but does not replace anonymization for the purpose of exiting GDPR scope.
Technical safeguards
When lawful processing with personal data is the chosen path, the organization deploys a risk mitigation stack that compensates for the absence of anonymization:
| Safeguard | What it mitigates | GCP implementation |
|---|---|---|
| VPC Service Controls | Data exfiltration — even principals with IAM permissions cannot move data outside the perimeter | VPC-SC perimeters around projects containing personal data (Module 08, Pattern 1) |
| Data regionality | Cross-border transfer risk — data stays within the required jurisdiction | BigQuery dataset location, Vertex AI regional endpoints, GCS bucket location |
| No-training guarantees | Model memorization risk — data not used for foundation model training | Google CDPA, Vertex AI data governance settings |
| Purpose limitation | Scope creep — data access restricted to the approved use case | IAM roles, BigQuery authorized views, policy tags with role-based masking |
| Audit logging | Accountability gap — records who accessed what, when, and for what purpose | Cloud Audit Logs (Admin Activity + Data Access), BigQuery Data Access logs (enabled by default) |
| Retention controls | Data accumulation risk — automated deletion when data is no longer needed | BigQuery table expiration, GCS lifecycle rules, Cloud Scheduler + deletion jobs |
| Pseudonymization | Re-identification risk — reduces exposure without full anonymization | SDP transforms (HMAC, deterministic encryption) as in the Module 06 pipeline |
VPC Service Controls add a network-level perimeter on top of IAM — valuable for large organizations where IAM misconfigurations are a realistic risk, but meaningful setup and operational overhead for a small team. For an SME with a single GCP project and a handful of principals, IAM alone (least-privilege roles, no allUsers / allAuthenticatedUsers bindings) combined with BigQuery Data Access audit logs (enabled by default, no setup needed) provides a defensible baseline. Add VPC-SC when the project grows to multiple teams, external service accounts, or cross-project data flows. Data regionality is free and always worth configuring — set your BigQuery dataset location to EU to address data-at-rest residency. Note that regionality alone does not fully resolve GDPR cross-border transfer risk: non-EU support access or subprocessor operations may still constitute a transfer and require contractual safeguards (e.g., Standard Contractual Clauses).
The decision framework
This is the decision that the data owner, data engineering lead, and analytics consumers must make together — with the DPO advising on the privacy implications:
The decision is not "anonymize everything" or "don't anonymize anything." Most production architectures use both: anonymization for outputs that don't need individual-level data (analytics dashboards, partner reports, ML training data), and lawful processing with safeguards for workloads that do (fraud detection, customer support, LLM inference). The data zone architecture from Module 06 supports this naturally — Zone 1 (pseudonymized) serves lawful-processing workloads, Zone 2/3 (anonymized) serves everything else.
Scaling down for smaller organizations
Everything above describes governance calibrated for enterprise-scale data platforms — dedicated DPOs, RACI matrices, formal design reviews, VPC-SC perimeters. For a 50-person e-commerce company with a handful of BigQuery datasets and no DPO on staff, it's overkill that will never get implemented.
GDPR itself acknowledges this tension. Recital 98 encourages small and medium enterprises to use standardized approaches and templates rather than building bespoke governance. The regulation requires appropriate technical and organizational measures — appropriateness is proportional to the risk, the volume of data, and the resources of the controller. A smaller organization processing less data, with fewer consumers and simpler data flows, can achieve equivalent compliance with lighter tooling.
The table below maps each governance area to a pragmatic minimum for smaller teams:
| Area | Full version (this module) | SME shortcut | What you give up |
|---|---|---|---|
| Policy | 7-section policy document with formal review cadence | A 2-page internal wiki page covering: which fields are PII, which technique applies to each, and who approves exceptions. | Formal exception handling, triggered reviews. Acceptable if the team is small enough that decisions are made in conversation. |
| Classification | 5-tier taxonomy with automated SDP discovery | 3 tiers (High / Medium / Low). Manual classification in a spreadsheet. SDP inspection jobs on-demand for validation. | Automated detection of new PII. Mitigate by making classification a checklist item in your pipeline review process. |
| Roles | RACI matrix with 8 decision types across 4+ roles | One person owns the pipeline and classification. A second controller-side person (CTO or founder) reviews privacy classification and approves sharing; DPO (or DPO-as-a-service) advises but does not own the decision. | Formal separation of duties, privacy engineering guild, design reviews. |
| Legal basis | Formal LIA per use case, DPIA where required, DPA review | Same requirements apply — GDPR doesn't exempt SMEs from lawful basis. But the documentation can be lighter: 1-page LIA, 2–3 page DPIA where required. | Depth of analysis. Proportionate to the scale and risk of processing. |
| Safeguards | VPC-SC + IAM + audit logging + retention controls + pseudonymization | IAM (least-privilege) + BigQuery Data Access audit logs (default) + data regionality (EU). Add VPC-SC when complexity grows. | Network-level perimeters. Acceptable when principal count is small and manageable. |
The minimum viable governance for a small team processing personal data on GCP is: (1) a written policy — even two pages — that says what's PII and what technique to use, (2) a one-page LIA per use case, (3) IAM with least-privilege and audit logging enabled, and (4) a ROPA entry that documents the anonymization approach. Everything else in this module is scaling that foundation up — you can add layers as the data volume, team size, and regulatory exposure grow.
Next: Module 10 covers the technical machinery that enforces the governance decisions from this module — automated classification, pipeline monitoring, data lineage, key lifecycle, and what to do when anonymization fails.
Created by Claude Opus with ❤️ for the Aliz Data Engineering team.