Deploy quality: why two Lighthouse runs disagree (2026)

The hardest problem in scoring deploy quality is not deciding which builder wins. It is that two operators running Lighthouse against the same deployed artifact, on different machines and on different days, repeatedly produce different scores.¹

This is not a flaw in any builder. It is a property of the measurement tool, and the Lighthouse team has documented it for years. But it has hard implications for how a benchmark may report deploy-quality numbers, and those implications were not addressed in the June 2026 axis as originally published. This note addresses them, and the correction notice above records how that failure was resolved.

What this post is and is not

This is a reproducibility note, and after the August 2026 correction it is a method-only document. It scores no builder and it restates no withdrawn number. What it publishes is the protocol a hands-on deploy-quality harness would have to follow, so that a reader can contest the procedure before any builder is scored under it.

The protocol applies symmetrically to every builder assessed. No builder is audited under different conditions from any other, and none is exempt from the run-log requirement in step 8.

The phenomenon

Lighthouse measures a deployed page along five categories: performance, accessibility, best practices, SEO, and progressive-web-app readiness. The BuilderProof deploy-quality axis reports three of those (performance, accessibility, SEO) plus a structured SEO checklist that goes beyond the Lighthouse SEO category.

The Lighthouse team's official variability documentation states the constraint directly: "The median Lighthouse score of 5 runs is twice as stable as 1 run."² The implication is the inverse: a single Lighthouse run is, by Google's own measurement of its own tool, the least stable reporting unit available.

The public-facing magnitude of the drift is reported by independent practitioners in roughly the same range. A 2026 community write-up summarising the published guidance puts it at "a score fluctuation of 5 to 10 points between consecutive Lighthouse runs on the same page is completely normal. A fluctuation of 15 points or more" usually indicates a real change.³ The withdrawn June 2026 deploy-quality table reported single-run numbers. That reporting choice is exactly why a reader reproducing it could land 5 to 10 points away without anything having changed, and it is part of why the figures were withdrawn rather than annotated.

Seven sources of variability

A reproducibility protocol is only useful if it names what it is trying to control. Google's variability documentation lists seven primary sources of Lighthouse score variability. Translated to the AI-builder benchmark context, with notes on which sources the protocol can address and which it cannot:

Four of the seven sources are addressable by the operator running the audit. Three (web-server variability, cross-geo latency residual, and browser nondeterminism) are not, and the median-of-five protocol is the residual mitigation for those.

Withdrawn: how the v1 table was produced

This section previously described the conditions under which the June 2026 deploy-quality table was produced: a single operator machine, a named browser version, one Lighthouse run per dimension per builder, an axe-core accessibility pass, and a single afternoon of sequential audits.

That account has been withdrawn as of August 21, 2026. The table it described has itself been withdrawn, and BuilderProof does not hold the per-run output, the machine record or the audit log that would let a reader verify any part of the description. A procedural account of a run that cannot be evidenced is not a methodology disclosure, it is a claim, and this lab does not publish claims it cannot support. Nothing is substituted in its place.

The consequence is worth stating plainly rather than burying: the protocol below has never been executed. It is a pre-registration, not a report. Publishing it still has value, because a protocol published before any score exists can be argued with on its merits instead of being reverse-engineered from a result someone already dislikes. But a reader should not infer from its specificity that a run happened.

The protocol the next iteration will adopt

The next iteration of the deploy-quality axis will follow this protocol on each of the seven builders in the cohort. The protocol is published here so that an external party reproducing or contesting the next deploy-quality table has a documented procedure to compare against.

The protocol does not eliminate variance. It controls four of the seven sources of variance and reports the residual three honestly through the median-and-IQR pair. A counter-result that disagrees with a published median is now a falsifiable claim that has to be argued against the protocol, not against a single number.

Worked example: when median changes the recommendation

A worked example shows why the median-and-IQR pair matters more than the single number.

Suppose two hypothetical builders (call them X and Y) are audited on the deploy-quality performance dimension. Each gets five Lighthouse runs under the protocol above.

Single-run reporting (the v1 approach). If only run 1 is reported, X scores 91 and Y scores 94, and Y looks better by 3 points. If only run 4 is reported, X scores 87 and Y scores 81, and X looks better by 6 points. The single-run reporting is unstable enough that, on this synthetic data, the direction of the comparison flips depending on which run the operator picks.

Median-of-five (the v2 protocol). X's runs sorted are 87, 89, 90, 91, 92; median 90, Q1 89, Q3 91, IQR 2. Y's runs sorted are 81, 86, 90, 92, 94; median 90, Q1 86, Q3 92, IQR 6. The medians are tied at 90. The IQR is informative: X is much more stable (IQR 2) than Y (IQR 6) on this dimension. A reader optimising for predictable performance picks X; a reader optimising for peak performance and tolerant of variance picks Y. Both readings are defensible from the same five runs, and both are obscured by single-number reporting.

The published table cell under the v2 protocol would read something like "90 (IQR 2)" for X and "90 (IQR 6)" for Y. The interesting comparison moves from the median to the IQR, which is what the data actually supports.

Where the protocol runs out of road

Reproducibility protocols address measurement variance. They do not address artifact variance, which is the larger problem when scoring an AI builder.

The published v2 first-build stability proposal introduced a six-category failure taxonomy (F1 through F6) for first-build outcomes. Two of those categories are directly relevant to deploy-quality reproducibility:

• F4: deploy failure. The builder produces an artifact that does not deploy successfully on the first attempt; a manually patched deploy is then audited. The deploy-quality score of the patched build is not the same data point as the deploy-quality score of an unpatched build, but a single number does not surface that distinction. The next iteration of the axis will tag any cell where the audited build required a manual patch.
• F6: vendor outage during the audit window. If the builder's hosting layer or CDN is degraded during the audit, the Lighthouse score reflects the outage rather than the deployed artifact. The protocol's audit-location and cooldown rules cannot help here. The next iteration will flag any run that overlaps a vendor incident (cross-referenced against the vendor's public status page) and exclude it from the median computation, with the exclusion noted in the run-log.

Both of these are outside the protocol's measurement-variance scope and are honest acknowledgements that even a well-designed reproducibility protocol has a floor below which the data is contaminated by the builder's own operational state.

What this means for a reader comparing deploy quality today

With the June 2026 table withdrawn, there is no BuilderProof deploy-quality result to interpret. A reader comparing builders on deploy quality in the meantime is better served by the constraint than by any number:

1. Treat any single Lighthouse figure you see quoted anywhere, by any publication, as a point estimate carrying roughly plus-or-minus 5 to 10 points of noise, consistent with Google's own variability guidance and the community-reported drift range. That includes figures published by vendors about themselves.
2. Prefer categorical signals over score signals. A recurring failure mode ("ships a client-only bundle", "no server-rendered HTML for crawlers") reproduces across runs; a performance score does not.
3. Prefer axe-core style categorical accessibility findings over the Lighthouse accessibility score, for the same reason: an element either has an accessible name or it does not.
4. If a comparison hinges on a gap smaller than 15 points, the gap is inside the noise floor and no single-run source, this one included, can settle it. Run the median-of-five protocol below on the two candidates yourself.

The documentation-derived deploy-quality assessment remains published and is explicit about being read off vendor documentation rather than measured. It is the honest substitute for a withdrawn measurement, and it is not the same thing as one.

Open questions

1. Five runs versus eleven. Google's own documentation recommends a minimum of five and notes diminishing returns above that. Some practitioners argue eleven runs is the floor for reliable IQR estimation. The protocol fixes five for the next pass; a contributor argument for eleven is welcome.
2. Throttling profile. Simulated throttling is more reproducible than DevTools throttling but less faithful to a real client. The protocol picks reproducibility; a counter-argument for fidelity (with a worked example showing where the recommendation would change) would shift the choice.
3. Per-dimension medians versus a composite. The current table reports three dimensions independently. A composite "deploy quality" number is computable but obscures the dimension that drove the median. The protocol keeps the dimensions independent. A reader who prefers a composite view should construct it themselves from the run-log.
4. F6 vendor-outage cross-reference. Several builders in the cohort do not publish a status page granular enough to cross-reference a five-minute Lighthouse run against. The protocol falls back to operator-recorded incident notes in those cases. A more robust mechanism is open.

Responses to these open questions are best raised against this lab note rather than against the June 2026 deploy-quality table, which this note does not modify.

FAQ

Q1. Is there a published BuilderProof deploy-quality score?
No, not as of August 21, 2026. The June 2026 table was withdrawn at source. This note documents the reporting protocol a future hands-on harness would have to meet before any score is published again.

Q2. Why is a 5-to-10-point Lighthouse difference normal?
Because Lighthouse measures a real deployed page across a noisy chain: network, server, client hardware, browser execution order. Google's own variability documentation lists seven sources of variability and reports that five-run medians are roughly twice as stable as single runs.

Q3. What would a run under this protocol change?
There is no prior ranking for it to change, because the v1 results were withdrawn. What the protocol buys is that any future ranking arrives with a median, an interquartile range and a published run-log, so a gap smaller than the noise floor is visible as such instead of being read as a difference.

Q4. Why publish the protocol before the run?
So that a reader who disagrees with the protocol can contest it before any builder has been scored under it. Publishing the rubric before producing scores is the same discipline applied in the v1 methodology and the v2 first-build stability proposal.

Q5. What about Lighthouse versus PageSpeed Insights?
PageSpeed Insights runs Lighthouse on Google infrastructure, which adds tier-1 network variability the protocol cannot control. The protocol pins audits to Lighthouse running locally on the fixed audit machine.

Q6. Does the protocol apply to axe-core accessibility checks?
Less directly. axe-core failures are categorical (an element either has a missing label or it does not). The single-run-per-builder model stays for the accessibility axe-core pass; only the Lighthouse-score dimensions move to median-of-five.

Q7. When will a deploy-quality table be published?
No date, and stating one would be dishonest. The blocking dependency is not the protocol, it is hands-on access: executing it requires running each builder in the cohort through a controlled build on a real account. Until that exists, this axis publishes method and a documentation-derived assessment, and no measured scores.

BuilderProof editorial team. "Deploy quality: why two Lighthouse runs disagree (2026)". BuilderProof, June 2026. https://www.builderproof.org/benchmarks/deploy-quality-reproducibility-why-two-lighthouse-runs-disagree-june-2026.

@misc{builderproof-deploy-quality-reproducibility-why-two-lighthouse-runs-disagree-june-2026,
  title  = {{Deploy quality: why two Lighthouse runs disagree (2026)}},
  author = {{BuilderProof editorial team}},
  year   = {2026},
  month  = {jun},
  howpublished = {\url{https://www.builderproof.org/benchmarks/deploy-quality-reproducibility-why-two-lighthouse-runs-disagree-june-2026}},
  note   = {BuilderProof, builderproof.org}
}