We built a reusable Claude Code skill that queries billions of rows of telemetry, catches statistical traps, and produces a structured evaluation report that helps engineers make ship\/no-ship decisions. The team saw significant efficiency gains by automating much of the analysis and reporting process, enabling faster and more consistent evaluations. <\/p>\n\n\n\n
Microsoft Clarity<\/a> runs on millions of websites. Every time we prepare a new version of the Clarity script for release, we face one fundamental question: is this version safe to ship? <\/p>\n\n\n\n That sounds simple. It isn’t. <\/p>\n\n\n\n Clarity captures a rich set of signals: Web Vitals (INP, CLS, LCP), session recording fidelity, script overhead, and more. Before shipping, we compare dozens of metrics across multiple dimensions (device type, recording mode, browser, operating system) spanning hundreds of thousands of websites running different versions simultaneously. A regression in any critical metric could degrade the experience for websites relying on Clarity. <\/p>\n\n\n\n Historically, this was a manual process. An engineer would write ad-hoc queries against our analytics data warehouse, eyeball the results, and make a judgment call. The queries were fragile. Our data warehouse has server-side timeouts, and certain aggregation patterns on billions of rows would regularly fail. There was no systematic protocol for what to check, how to interpret differences between unequally-sampled populations, or how to determine when an apparent regression was actually a statistical artifact. <\/p>\n\n\n\n We decided to encode our entire evaluation methodology into a Claude Code skill, a reusable, invokable AI workflow that generates optimized queries, runs them against our data warehouse, and produces a structured evaluation report. <\/p>\n\n\n\n Claude Code<\/a> is an agentic coding tool that runs in your terminal and can help developers complete software development tasks. A skill is a reusable workflow defined by a SKILL.md file<\/a> in your repository (under .claude\/skills\/). The skill can also include supporting files, such as reference documentation, templates, examples, or scripts that Claude can use as part of the workflow. <\/p>\n\n\n\n When you invoke it, say `\/clarity-evaluation 0.8.59 0.8.60-beta`<\/em>, Claude reads the skill, Claude uses the instructions defined in the skill, generates and executes code, and analyzes the output using the evaluation criteria defined in the skill. All within the context of the repository, tools, and data sources available to the agent. <\/p>\n\n\n\n The key insight is that the skill isn’t just a script. It’s a methodology document that gives an AI agent instructions, context, and guardrails for completing a workflow. <\/p>\n\n\n\n It contains: <\/p>\n\n\n\n In this workflow, the AI agent uses the methodology to generate a TypeScript script tailored to the specific versions and date range, executes the analysis, and applies the defined interpretation rules to produce a report. <\/p>\n\n\n\n Building the skill was iterative. Here’s what we learned. <\/p>\n\n\n\n Our first attempts were plagued by query timeouts. Our data warehouse has a server-side timeout per query, and the underlying table has billions of rows. Naive aggregation queries would simply fail. <\/p>\n\n\n\n The critical fix was discovering the right quantile algorithm. The default `quantile()`<\/em> function in our data warehouse uses reservoir sampling, which is O(n) memory per aggregation state. When you compute 7+ percentiles across multiple GROUP BY groups, memory usage explodes: <\/p>\n\n\n\n “`sql<\/em> <\/p>\n\n\n\n quantile(0.5)(DataSize) as P50,<\/em> <\/p>\n\n\n\n quantile(0.9)(DataSize) as P90,<\/em> <\/p>\n\n\n\n quantile(0.99)(DataSize) as P99,<\/em> <\/p>\n\n\n\n quantile(0.5)(ThreadBlocked) as TB_P50,<\/em> <\/p>\n\n\n\n quantile(0.9)(ThreadBlocked) as TB_P90,<\/em> <\/p>\n\n\n\n quantile(0.75)(INP) as INP_P75,<\/em> <\/p>\n\n\n\n quantile(0.75)(CLS) as CLS_P75<\/em> <\/p>\n\n\n\n “`<\/em> <\/p>\n\n\n\n This looks innocent but kills performance at scale. Seven separate reservoir sampling states, each tracking every row individually. Even with sampling, this exceeded the cluster’s memory budget. <\/p>\n\n\n\n The fix was switching to T-Digest-based quantile functions, which use O(1) memory per state. They also support batched syntax: `quantilesTDigest(0.5, 0.9, 0.99)(DataSize)`<\/em> returns an array `[P50, P90, P99]`<\/em> from a single T-Digest state. Three percentiles for the memory cost of one. <\/p>\n\n\n\n This single change took us from 100% timeout rate to 100% success rate. We encoded it directly into the skill so it’s never forgotten: <\/p>\n\n\n\n > **Always use T-Digest quantiles, never reservoir sampling.** Batch multiple percentiles of the same column into one call. Split heavy queries into parallel sub-queries to reduce per-query memory pressure. <\/p>\n\n\n\n Other optimizations that made it into the skill include:<\/strong> <\/p>\n\n\n\n This was perhaps the most valuable lesson of the entire project, and it applies far beyond our use case. <\/p>\n\n\n\n During one evaluation, our queries showed that a particular client-side diagnostic flag had increased by 10x<\/strong> on the beta version compared to production. This looked like a serious regression. Something in the new code was triggering a warning condition ten times more frequently. <\/p>\n\n\n\n But we were sampling the production population at a much lower rate than the beta population, roughly a 10x difference. When distribution queries return raw counts from two differently-sampled populations, those counts aren’t directly comparable. A metric appearing “10x worse” might simply reflect 10x more data being sampled. <\/p>\n\n\n\n To validate, we drilled down to the individual website level. We identified the top contributing websites and queried them at full resolution (no sampling) on both versions. The rates were identical, within hundredths of a percentage point. The apparent “regression” was a pure sampling artifact. <\/p>\n\n\n\n This experience taught us to add an entire anomaly deep-dive phase to the skill: <\/p>\n\n\n\n > For any metric showing a surprising result (>20% change in either direction), drill down to the individual site level. Compare rates within the same site on both versions. If rates differ at the site level, it’s real. If they’re equal, the aggregate difference is a sampling artifact. <\/p>\n\n\n\n We also made this bidirectional<\/strong>. Sampling artifacts can create false regressions, false improvements, and false stability. An implausibly large improvement deserves the same scrutiny as an apparent regression. <\/p>\n\n\n\n Clarity operates in two modes: analytics-only and session recording. When recording is enabled, the script captures DOM mutations for session replay, which naturally produces more data. Certain metrics are *organically different* between these modes. That’s by design, not a bug. <\/p>\n\n\n\n We learned to always stratify by operating mode and never compare across strata. We also discovered that the recording-enabled cohort is a more reliable signal for evaluation: it represents a larger, more representative population. When the two modes disagree on a metric, we trust the larger cohort. This rule is now encoded in the skill’s interpretation guidelines. <\/p>\n\n\n\n Here’s what a typical evaluation looks like. An engineer types a single command: <\/p>\n\n\n\n “` <\/p>\n\n\n\n \/clarity-evaluation 0.8.59 0.8.60-beta <\/p>\n\n\n\n “` <\/p>\n\n\n\n The skill then:\u00a0<\/strong><\/p>\n\n\n\n For our recent 0.8.60-beta evaluation, the numbers were clear: <\/p>\n\n\n\n | Signal | Result | <\/p>\n\n\n\n |——–|——–| <\/p>\n\n\n\n | Fatal errors | Zero | <\/p>\n\n\n\n | Data transmitted (recording-enabled) | 4-30% reduction | <\/p>\n\n\n\n | Main-thread long tasks (Mobile) | 20-40% reduction | <\/p>\n\n\n\n | Cumulative Layout Shift | 12-30% improvement | <\/p>\n\n\n\n | Largest Contentful Paint | 5-7% improvement | <\/p>\n\n\n\n | One flag with apparent 10x increase | Confirmed sampling artifact via deep-dive | <\/p>\n\n\n\n **Verdict: Ship.** <\/p>\n\n\n\n Without the skill, this analysis would have taken an engineer several days of writing queries, debugging timeouts, and potentially flagging the sampling artifact as a real regression that would have delayed the release. With the skill, we had a grounded, auditable verdict in under 30 minutes. <\/p>\n\n\n\n After iterating through multiple versions of this skill, we’ve identified patterns that make AI skills effective for engineering workflows: <\/p>\n\n\n\n The skill doesn’t say “run this exact SQL.” It explains what each metric means, why we check it, and how to interpret the results. This lets the AI adapt when things go wrong (a column doesn’t exist, a query times out, the data distribution looks weird) while staying true to the evaluation methodology. <\/p>\n\n\n\n Every optimization rule in our skill was learned through failure. `quantilesTDigest`<\/em> replaced `quantile`<\/em> after days of debugging timeouts. The sampling artifact protocol was added after nearly shipping a false alarm. The domain matching rules came from discovering that `GROUP BY domain(Url)`<\/em> is a cluster-killing operation. This institutional knowledge is now codified and reproducible. <\/p>\n\n\n\n The skill explicitly forbids certain comparisons (never compare Playback=0 vs Playback=1), requires validation of surprising findings (the Phase 2.5 deep-dive), and specifies which signals are ship-blockers (FatalLogCount, UnloadPresent drops, Web Vitals regressions). This prevents the AI from drawing incorrect conclusions from noisy data. <\/p>\n\n\n\nWhat Is a Claude Code Skill? <\/h2>\n\n\n\n
\n
\n
\n
\n
The Journey: From Timeouts to a Working Skill <\/h2>\n\n\n\n
Optimizing Queries at a Billion-Row Scale <\/h3>\n\n\n\n
\n
\n
\n
The Sampling Artifact Trap <\/h3>\n\n\n\n
Stratification: Comparing Apples to Apples <\/h2>\n\n\n\n
The Skill in Action <\/h2>\n\n\n\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
What Makes a Good AI Skill? <\/h2>\n\n\n\n
1. Encode Methodology, not Just Procedures <\/h3>\n\n\n\n
\n
2. Capture Hard-Won Operational Knowledge <\/h3>\n\n\n\n
\n
3. Build in Guardrails Against Misinterpretation <\/h3>\n\n\n\n
\n
4. Make it Self-Improving <\/h3>\n\n\n\n