Telemetry Fixes & Updates

This is a small update regarding the AI Telemetry. The telemetry rollout was a long process due to resourcing issues and the rapid change of the harness APIs. It provided us a good amount of data, but every now and then it was questioned by people whether it showed the real data.

As we do not have that many people on the bench right now, I decided to spend some time on it, check what we have, what we don't have, and what needs to be fixed.

20 PRs, including 70 commits, were merged in after 2.5 days of vibe coding.

Biggest Problems

• No infrastructure as code. The stack was built by hand. 66 existing resources had to be imported into Terraform (#57), and production was still running a configuration from before the merged fixes (#58).
• Calls were overcounted. Cloud Monitoring counted agent, tool and hook spans as model calls (#49). Codex was counted about 4× (22 calls against 5 real ones in the sandbox) (#63).
• Costs were wrong. Codex reasoning tokens were counted twice (#48), cache writes weren't priced separately (#26), rates were out of date (#44), and missing prices showed as $0 (#50). For one user, the September Copilot estimate was $993 against about $400 of actual credit usage (#35).
• Content leaked into telemetry. Copilot subagent calls stored prompts and responses (#28), and Copilot hooks stored their input and output (#47).
• Cache data was missing or expiring. None of the 30,736 Copilot Chat inference logs carried cache fields (#40). The only source was trace spans, which expire after 30 days (#36).

Not Fully Resolved

• Missing Copilot token values (#27, open). Collector CPU throttling is a plausible cause, but it isn't confirmed. The issue stays open until delivery is verified after deployment, and telemetry that already expired or was never sent can't be recovered.
• Copilot Chat cost is still overstated. A call gets the cache discount only when a matching span exists. Calls without one stay at full price (#35).
• Pricing per provider (#62, open). Codex calls are priced with GitHub Copilot rates instead of OpenAI rates. The rate table is still maintained by hand, so new models stay unpriced until someone notices.
• Estimates with known gaps. Long-context calls are still underestimated (#45). codex-auto-review uses the gpt-5.6-luna rate because OpenAI publishes no price for it (#66).
• Limits set by upstream tools. The Copilot desktop app and Copilot CLI both report as copilot-cli, and only GitHub can change that (#20). Codex skill usage shows no rows until a Codex release ships the codex.skill_invocation event (#54).

What changed

Problem	Fix
The GitHub Copilot desktop app (macOS) had no setup guide, and as a GUI app it never sees shell-profile exports.	Added a desktop app section that extends the existing ai.aliz.otel-env LaunchAgent with the same OTel variables. Desktop and CLI calls both report service.name=copilot-cli, which is documented as a known limitation. PR: #20
Copilot calls that carried only a response model had a null model, so model dashboards skipped them and pricing couldn't match them. Issue: #29	Copilot Chat logs and CLI spans now fall back to gen_ai.response.model. The CLI auto placeholder is resolved from the response model. PR: #37
Copilot Chat cache metadata was lost once the source spans expired after the 30-day trace retention window. Issue: #36	An hourly capture job persists cache metadata and log bindings to BigQuery, with an initial backfill and coverage monitoring. PR: #39
Copilot subagent calls logged prompt, response and other content. Issue: #28	Both collectors now apply one content-stripping policy to spans, span events and logs. Client content capture is disabled in the relay setup. PR: #38
Copilot token values were missing; request-based CPU allocation could stall the collector's background batch exports between requests. Issue: #27	Collector deployments now use --no-cpu-throttling. The issue stays open until coverage is verified after deployment. PR: #40
Every coding tool repeated the endpoint and identity in its own telemetry setup. Issue: #34	One JSON source and a renderer now generate the config for Claude Code, Copilot Chat, Copilot CLI and Codex, with shared direct and relay guides. PR: #41
Cache writes were not priced separately, and available Copilot cache-write counts were ignored. Issues: #26, #30	A single accounting helper separates fresh input, cache reads and cache writes before pricing, with an explicit cache_write_multiplier in every pricing seed. Dashboard schemas are unchanged. PR: #43
Copilot Chat logs omit the cache breakdown, so cached input was priced at the full input rate.	Log rows are enriched with cache-read counts from matching chat spans, so matched reads get the cache discount. Unmatched calls stay at full price. PR: #35
Model pricing rates and model-matching patterns were out of date. Issue: #44	Rates were refreshed (verified on 2026-10-01). The Cloud Monitoring cost queries are now generated from the same rate card. PR: #45
Cloud Monitoring cost tiles disagreed with BigQuery on Chat models and cache writes. Issue: #46	The Monitoring generator now applies the same request/response model fallback and per-alias cache-write parsing as the BigQuery views. Re-running it upgrades existing dashboards. PR: #55
Codex reasoning tokens were counted twice in output, and Codex cache writes were discarded. Issue: #48	Codex output now uses output_token_count as-is, since it already includes reasoning. Cache writes are read from cache_write_token_count in both BigQuery and Monitoring. PR: #52
The dashboard missed skill/tool events from several tools, totals left out cache reads, missing prices showed as $0, and viewers couldn't run queries. Issue: #50	The skill/tool view now reads Codex, Copilot CLI and VS Code Copilot events, totals include cache reads, and unpriced calls show as NULL with a new unpriced_tokens column. Provisioning grants viewers BigQuery job access. PR: #54
Copilot hook input and output were stored, and a private-mode deploy could keep a leftover public invoker binding. Issue: #47	Both collectors now strip the hook payloads. In private mode, deploy.sh removes public invoker bindings and fails if one remains. PR: #51
Cloud Monitoring overcounted gen_ai.calls and collector instances wrote conflicting series. Issue: #49	Only real model-call spans are counted now, and collector-owned series carry a per-instance label. Derived metrics are generated only in the Cloud Run collector. PR: #53
The telemetry GCP infrastructure was not managed as code. Issue: #56	Terraform now owns the existing stack: 66 resources were imported with no application changes. Production and sandbox have separate backends, and CI applies are dispatched manually. PR: #57
The production configuration predated the merged capture, accounting and collector fixes. Issues: #56, #36	Those changes are now deployed through Terraform in both environments, in a phased, sandbox-first rollout. PR: #58
An unused collector token secret was still injected into Cloud Run and required by every sandbox plan and CI run. Issue: #59	The env var, the secret, its Terraform inputs and the CI secret were removed in a two-phase rollout. PR: #64
Codex calls were still overcounted in gen_ai.calls (about 4×), with an extra unknown model series. Issue: #63	Only the run_sampling_request span is counted, since it runs exactly once per model call. Other Codex spans are dropped before spanmetrics. PR: #65
Codex codex-auto-review calls had no price, so their estimated cost was NULL. Issue: #61	They are now priced at the gpt-5.6-luna rate, the model Codex uses for the same reviews under API-key auth. PR: #66
The repository had placeholder folders and docs that no longer matched the code. Issue: #60	Placeholders were removed, a new README and AGENTS.md were added, and the docs now describe the Terraform-based deployment. PR: #67

Used Tooling

• Claude Opus 5.5 medium/high. Sometimes Codex 6.1 Sol medium for reviewing.
• Herdr for managing all the agents running in parallel.