Project Guide

note

This is the current official version of the Data Assessment: Project Guide. If you have any feedback or suggestions for improvement, use this working document: Data assessment process | Project Guide

1. Purpose and scope

The document is intended for Tech Leads and other internal stakeholders responsible for overseeing data assessment projects. This guide describes how an Aliz team runs a data migration assessment: a fixed-scope engagement, typically 4–6 weeks at standard depth, that produces a fact-based migration plan before any migration work starts.

Migration types covered

Type	Typical trigger	Main difficulty
On-prem → GCP	Data-centre exit, licence renewal, hardware end of life	Network, hidden on-prem dependencies, legacy ETL and SQL
External cloud → GCP (AWS, Azure, Snowflake, Databricks…)	Consolidation, cost, strategy	Egress costs, IAM mapping, replacing proprietary services
GCP → GCP (carve-out, re-organisation, org merge)	Divestment, M&A, landing-zone redesign	Resource hierarchy, billing, shared networking, cross-project dependencies

Green-field DWH assessments follow the same process, with more weight on business use cases and less on inventory.

What the assessment must answer: what exists (an evidence-based inventory), what depends on what, what should happen to each workload or system group (rehost, replatform, refactor, retire), how and in what order we move it, and what it will cost and what can go wrong.

Assessment depth (agreed in the offer; see the Discovery Guide, section 1.4). Every section below applies at the agreed depth:

• Light (2–3 weeks): main systems and major flows; undocumented processes estimated, not mapped; decisions by system group; estimate as a range.
• Standard (4–6 weeks): tier-1 workloads inventoried and mapped in full, the rest sampled; wave 1 estimate firm.
• Deep (2–3+ months): full inventory and dependency mapping; a decision per workload; an estimate per wave.

2. Project definition (agreed at kick-off)

These must be written down and signed off by the client sponsor in week 1. Most assessment disputes trace back to one of them being vague.

Item	What to agree	Example
Migration objectives	Why the client is doing this, in priority order	Data-centre exit by Q3; 30% lower run cost; enable near real-time reporting
Scope and boundaries	Systems, environments (prod/test/dev), countries or business units, time horizon of historical data	Prod + test only; 5 years of history; BU "Retail HU" only
Out of scope	Explicit exclusions	ML workloads, operational databases, source system changes
Dark data handling	What happens to data nobody owns or uses	Archive to cold storage, 6-month review, then delete
Assessment depth	Light, Standard or Deep, as sold in the offer, and what that means for granularity	Standard: tier-1 in full, the rest sampled
Baseline estimate	The indicative migration estimate given in the offer. The assessment confirms or re-baselines it.	Range of €X–Y and 9–12 months, based on the discovery call
Success criteria for the migration	What "done" means later	SLA parity, cutover window ≤ 1 weekend, 100% row-count and checksum match on tier-1 tables
Success criteria for the assessment	What the client gets from us	Approved target architecture, wave plan, estimate within ±20% for wave 1

3. Process

3.1 Phases

Phase	Duration	Key activities	Exit criteria
0. Prepare	1–2 weeks before start	Scope call, NDA/DPA, send survey workbook, start access requests, book interviews	Access requested, interviews booked, coordinator named
1. Kick-off	Week 1	Kick-off workshop, sign-off of section 2, RACI, comms cadence	Project definition signed off
2. Discover	Weeks 1–3	Interviews (section 4), survey, hands-on checks (section 5)	Inventory complete at the agreed depth; open items listed
3. Analyse	Weeks 2–4	Architecture validation (6), dependency analysis (7), approach per workload (8), risks (9)	Draft handover document
4. Review	Weeks 4–5	Draft walkthrough with client (technical + business), feedback round	Content agreed
5. Handover	Week 5–6	Final readout, handover document, short Q&A window	Client sign-off

A GCP → GCP carve-out with a clean landing zone often takes only 2–4 weeks.

Complexity checkpoint (end of week 2): compare the complexity found so far with the plan. If it is higher, put the trade-off to the sponsor: narrow the scope, sample instead of mapping in full, or extend through a change request.

3.2 Team and roles

Aliz team: a lead data architect (owns the assessment and recommendations, runs the key interviews), data engineer(s) (hands-on checks, inventory, tooling, dependency mapping), a cloud / infra engineer as needed (landing zone, network, IAM, project-move analysis), a project manager (plan, access, logistics, status, risk log) and the account manager.

Rhythm: a weekly 30-minute status call with the client coordinator, an internal sync twice a week, and a shared open-questions log. Answers are expected within 2–3 working days.

4. Questionnaires and interviews

Written input: the Data and AI assessment survey workbook. The key tabs are Foundations, Data survey, and the Datasource, Pipelines, Consumer and BI catalogs. Send it in phase 0 and complete it together during interviews. Don't expect the client to fill it in alone.

Interview plan (60–90 minutes each):

• Business / sponsor: why now, which outcomes matter most, deadlines, what would count as failure.
• Data platform lead: architecture walkthrough, data model, volumes, history, known issues, what is undocumented, how much of the migration can be automated.
• Pipelines / ETL: tools and versions, number of jobs, schedules, failure handling, custom code, ownership, complexity and design patterns.
• Source system owners: interfaces, extract methods, change frequency, planned system changes.
• BI / consumers: tools, report inventory, real usage, freshness needs, exports to third parties, currently unfulfilled needs.
• Infra / network / cloud: hosting, connectivity and bandwidth, landing zone, identity, IaC.
• Security / DPO: PII, GDPR, encryption and keys, residency, access model, audit.
• Operations: SLAs, monitoring, incidents, DR (RPO/RTO), release process.
• Previous attempts (if any): what was tried, how far it got, why it stalled, what must not be repeated.

Interview rule: every important statement should end up either verified by a hands-on check or marked "unverified" in the inventory.

Confidence rule: each inventory item and each finding is marked as verified, sampled or estimated. The handover document summarises this, so the client knows what the depth bought them.

5. Hands-on checks

Hands-on checks are what separate our assessment from a questionnaire exercise. They need read-only access. If the client won't grant access, we provide scripts for them to run.

5.1 Asset inventory

Source type	Automated	Manual / scripted
Legacy DWH (Teradata, Snowflake, Redshift, Oracle…)	BigQuery migration assessment using dwh-migration-dumper metadata and query logs (≥ 2 weeks of logs)	Data dictionary, ERDs, schema extracts
SQL Server	dwh-migration-dumper metadata; T-SQL translation dry run (preview)	Query Store and query-statistics views, SSISDB, SQL Agent history
Databricks	BigQuery migration assessment extraction notebook (preview)	Unity Catalog / metastore listing, job list
On-prem servers and VMs	Migration Center discovery and TCO	CMDB export
GCP (GCP → GCP)	Cloud Asset Inventory export to BigQuery (all resources, IAM policies, org policies), INFORMATION_SCHEMA for BigQuery	Billing export, Terraform state or repos

Minimum inventory content: object, owner, size and growth, environment, last-used date, upstream and downstream dependencies, PII flag, criticality tier, confidence (verified / sampled / estimated).

5.2 Project move analysis (GCP → GCP)

A project move between organisations (Resource Manager project migration, also called Project Mover) is a metadata operation: the project ID, number, resources and direct IAM bindings stay, and workloads keep running. What changes has to be checked project by project:

• Inherited IAM: roles granted at the org or folder level are lost. List what each workload depends on.
• Org policies: source constraints are replaced by destination constraints. Diff the two sets and flag conflicts (e.g. allowed regions, service account key creation, domain-restricted sharing).
• Custom roles, quotas, billing: org-level custom roles must be recreated, and org-level quota overrides don't transfer. The billing linkage stays, but discounts, commitments and the support contract may not.
• Networking: Shared VPC host/service relations, VPC Service Controls perimeters, Interconnect/VPN attachments, private DNS, PSC endpoints.
• Cross-project dependencies: CMEK keys, service accounts used across projects, BigQuery authorised views and datasets, Pub/Sub subscriptions, log sinks, monitoring scopes.
• Security tooling: org-level log sinks, Security Command Center, and audit log retention after the move.

Output: a per-project traffic light. Green means the project can be moved as is. Amber means it can move after listed fixes. Red means rebuild step by step instead (section 8).

5.3 Airflow / orchestration audit

• Environment and Airflow version. Cloud Composer is now Managed Service for Apache Airflow. Gen 1 and the Gen 2 2.0.x versions reached end of life on 15 September 2026, so any client still on them is a priority finding. Airflow 1 runs only on Gen 1, Gen 2 supports Airflow 2, and Gen 3 supports Airflow 2 and 3. Moving from Airflow 1 means rewriting DAGs, not just upgrading.
• Python and PyPI packages: pinned packages that conflict with the target image are a common blocker.
• Custom operators, plugins and hooks: hidden and often undocumented logic.
• DAG inventory: number of DAGs, schedules, run durations, failure rates, SLAs, pools, and dependencies between DAGs (sensors, triggers, datasets).
• Connections, variables and secrets: where credentials live (Airflow DB vs Secret Manager). These are re-created, never copied blindly.

The same checklist applies to other orchestrators (cron, SQL Agent, Control-M, ADF, Talend): jobs, schedules, dependencies, credentials, custom code.

5.4 Data profiling and usage

Measure volume and velocity (total size, daily increment, growth, peak ingestion windows), profile data types and quality hot-spots on a sample of tier-1 tables, and use query logs and BI usage statistics to find hot, cold and dead objects. Dead objects are retire candidates, and often the fastest cost saving we can show.

6. Architecture diagram validation

Collect the documented architecture (diagrams, wiki, old project documents) and draw the as-found architecture from the inventory and dependency data, in the same notation as the target. Everything undocumented, or documented but gone, becomes a finding. Then map current → target component by component (e.g. SSIS → Dataform + Managed Airflow, SFTP drop → GCS + Storage Transfer Service), and validate the result with the client in a 60-minute session before the draft report.

7. Dependency analysis by migration type

For every migration type: upstream producers, downstream consumers (BI, ML, exports, APIs), schedules and SLAs, shared credentials, shared reference or master data, dependencies on parallel client projects (a new ERP, network changes, access provisioning), and the commercial side (licence and contract end dates, cloud commitments, billing split for carve-outs).

	On-prem → GCP	External cloud → GCP	GCP → GCP (carve-out)
Network	Interconnect vs VPN, bandwidth for history + deltas, firewalls	Cross-cloud connectivity, private links	Shared VPC, VPC-SC, peering, PSC
Identity	AD / Entra ID → Cloud Identity, agent service accounts	AWS/Azure IAM → GCP IAM, federation	Org/folder IAM inheritance, cross-org access
Data movement	Bulk + incremental/CDC (Datastream), Transfer Appliance	Storage Transfer Service, BigQuery DTS, egress cost	Dataset copy, cross-org sharing, project move
Services to replace	ETL tools, schedulers, file shares	Redshift, Synapse, Glue, ADF, Databricks jobs…	Usually none; org-level logging, SCC, billing

Output: a dependency matrix (workload × dependency) and a dependency graph used for wave planning.

8. Migration approaches

8.1 Per-workload decision

Made at the agreed depth: per workload (Deep), individually for tier-1 workloads (Standard), or per system group (Light).

• Rehost (lift and shift): time pressure, low complexity, stable workload.
• Replatform: same logic on a managed target, e.g. Teradata → BigQuery with translated SQL, Airflow → Managed Airflow.
• Refactor: the logic is the problem (complex ETL, poor data model), or the business wants new capabilities.
• Retire: no usage in the logs, no owner, or duplicated elsewhere.

8.2 GCP → GCP: project move vs step-by-step rebuild

• Project move (whole projects move to the new org): best when boundaries are clean and section 5.2 is mostly green. Fast, no data copy, no downtime, IDs stay; but tech debt comes along, and inherited IAM and policies need rework. Tooling: Resource Manager API, org policy diff, IAM re-binding scripts.
• Step-by-step rebuild (new projects via IaC, copy data, redeploy): best for mixed ownership, a landing zone redesign or a tech-debt clean-up. Clean target and standards, but more effort, data copy cost and a parallel run. Tooling: Terraform, BigQuery dataset copy (DTS), Storage Transfer Service, Airflow snapshots, Data Validation Tool.

Many carve-outs end up mixed: green projects move, red ones are rebuilt.

8.3 Wave planning and pilot

• Pilot (wave 0): one end-to-end slice (source → pipeline → BigQuery → report) that is valuable but not business-critical. It proves the tooling, the validation approach and the estimate.
• Waves: grouped by the dependency graph, so tightly coupled workloads move together. Order them by business value and risk.
• For each wave: scope, entry and exit criteria, validation checks, parallel-run length, rollback plan, decommissioning step.

9. Risks and special attention

Risk	How we handle it
Client honesty / reality gap ("it's all documented", optimistic volumes)	Verify with hands-on checks and mark statements as verified or unverified. Present gaps neutrally, and never surprise the sponsor in the readout.
Changing requirements ("could you also look at…")	Scope signed off in week 1. Add-ons go to a parked backlog or a change request, with the impact on timeline and estimate recorded.
Legacy black boxes (stored procedures, macros, custom operators)	Name them explicitly and estimate reverse engineering separately. Don't hide it in contingency.
Hidden dependencies (partner exports, shared service accounts)	Log-based lineage, network and egress flows, consumer interviews
Access delays, API limits, transfer constraints	Start access requests in phase 0, with client-run scripts as a fallback. Test transfer throughput during the assessment.
Underestimated complexity (behind plan, effort to reach the agreed granularity keeps growing)	Time-box each workstream and use the week-2 complexity checkpoint (3.1). Prioritise tier-1 workloads, and mark what was sampled or estimated.
Missing program management (parallel client projects nobody coordinates; their results or tasks block ours)	Map parallel projects and their owners in phase 1. Log cross-project dependencies with an owner and due date. Ask the sponsor for a client-side program owner or a regular cross-project sync.
Previous failed attempt	A dedicated interview. Map its root causes to mitigations. Frame it as "we build on what you learned".

Keep a risk register from week 1 and hand it over with the report.

10. Handover document

10.1 Structure

1. Executive summary: objectives, key findings, recommendation, headline estimate (confirmed or re-baselined against the offer)
2. Scope, approach and current state: what was assessed and how, at which depth, a confidence statement (fully assessed / sampled / estimated), as-found architecture, inventory summary
3. Dependency analysis and findings: matrix and graph; risks, tech debt, quick wins, retire candidates
4. Target architecture: diagram, key decisions as ADRs (why we chose each GCP service), governance needs (always assessed; how much is implemented follows the client's priorities)
5. Migration approach: decisions per workload or system group, tooling, wave plan, pilot, validation and cutover strategy (reconciliation rules, parallel run, rollback)
6. Roadmap, estimate and TCO, plus operational readiness (runbooks, monitoring, ownership after migration)
7. Open items, assumptions and appendices: inventories, completed survey workbook, interview log, risk register

10.2 Sample finding

F-07 · Undocumented partner export · Severity: High A nightly SQL Agent job exports sales_daily to a logistics partner's SFTP. It isn't in the architecture diagrams, and there's no owner on the client side. Impact: the export would break at cutover. Recommendation: assign an owner, rebuild it as a scheduled export from BigQuery in wave 1, and notify the partner 4 weeks before cutover.

11. Questions for reviewers

1. Scope and duration: keep green-field in this guide? Do 3–6 weeks (2–4 for GCP → GCP) match reality?
2. Hands-on checks and project move: what is missing from section 5? Which existing scripts should we link? Any carve-out lessons for 5.2?
3. Handover document: does section 10.1 work as our standard template? What would you cut?
4. Reuse and accuracy: which past reports can we anonymise as samples? Is anything wrong or overstated?

---