How We Benchmark AI App Builders: The BuilderProof Methodology v1

Public methodology comes before public scores. That order matters. A benchmark that hides its rubric can never be contested on its own terms, only argued against in the abstract.¹ So before we run another round, this page documents how BuilderProof produces a score, what is in the test brief, which environment standards are fixed, and how anyone can reproduce a run end to end.

The intent is narrow. This is the methodology the lab uses today, version v1, dated June 19, 2026. It is not a ranking, not a recommendation, and not a critique of any builder. The four most recent BuilderProof benchmarks are produced under this methodology and link back to this page for their rubric definitions.

Why a published methodology

Three motivations.

First, reproducibility. An evaluation that no outside party can re-run is not a benchmark, it is an opinion piece with a table. Output-builder evaluations are unusually exposed to this failure mode because the same prompt produces different outputs across days, accounts and model versions.² Publishing the rubric, the brief and the environment standards lets a reader regenerate the work, or, more often, challenge a specific score with a specific counter-run.

Second, contestability. A benchmark a vendor cannot contest is a benchmark a vendor will ignore. The rubric below is structured so that any operator (vendor, agency, independent researcher) can point to a particular axis and produce a counter-result. Where their result and ours diverge, the divergence is the interesting datapoint.

Third, neutrality. Every builder in the cohort is scored by the same rubric, the same brief and the same environment, with no builder-specific tuning. The defence against bias is not editorial intent, it is the published rubric: every score, for every builder, is produced the same way and can be reproduced against the same rubric.³

The four axes (v1)

Methodology v1 scores four capability axes. Each axis has its own dedicated benchmark post; this page documents the rubric definitions that all four share.

A fifth axis, first-build stability, is on the v2 candidate list (see Open questions below). It is not part of v1 and not weighted in any v1 result.

The test brief

Every builder is scored against the same test brief. The brief defines what the system under test is asked to build, with enough specificity to make outputs comparable and enough latitude to let each builder use its own idioms.

The brief intentionally mirrors the public 2026 ai-agents-benchmark.com brief at the spec level so external readers can compare BuilderProof's scoring against a second, unrelated corpus. That benchmark has no named author and no published governance, and its own results place Totalum first, so we mirror its spec and not its conclusions.⁴ Where the briefs diverge is in our explicit deploy-quality and agency-suitability axes, which the ai-agents-benchmark brief does not score.

The brief is version-stamped. OQ-7 is the brief used in all four June 2026 benchmarks linked from this page. A change to the brief mints a new version (OQ-8) and triggers re-scoring; old scores are retained in the change log under their original brief.

Environment standardization

Builder output is sensitive to environment. v1 fixes the variables below as the standard a hands-on run must meet. These are requirements for a future run, not a description of runs already performed: as of August 21, 2026 no BuilderProof result set has been produced under them.

The two most-disputed variables are model selection and account state. Defaulting to the builder's recommended model removes a parameter the team would otherwise have to defend per result. Forcing a fresh account on every run isolates the score from any cached project state or hidden personalization on the platform.

Scoring and weights

Each axis returns a 0 to 100 score per builder, produced by axis-specific rubrics documented in the corresponding benchmark post. A weighted roll-up is used only when an article is comparing builders on a multi-axis question; standalone axis posts report unweighted axis scores.

The roll-up weights default to:

These weights reflect the editorial team's read of the audience BuilderProof writes for, which skews toward operators who care about shipping rather than prototyping. The weights are published so that a reader who weights differently (for example, an audience that values speed over agency surface) can recompute the roll-up from the per-axis scores without re-running the benchmark.

A roll-up never replaces a per-axis score in our writing. The per-axis tables are always present alongside any weighted result.

Reproducibility

Reproducing a BuilderProof run end to end requires four artifacts, all of which are public on this site. They are published so that a reader, a vendor or an independent researcher can run the protocol themselves. Following them will not reproduce a BuilderProof number, because as of August 21, 2026 there is no published v1 number to reproduce; the four axis pages linked below currently document method only.

1. The v1 axis definitions on this page.
2. The OQ-7 test brief above.
3. The environment standards table above.
4. The per-axis rubric in the relevant benchmark post:
   • Output quality (June 2026)
   • Speed (June 2026)
   • Deploy quality (June 2026)
   • Agency suitability (June 2026)

A reader who reproduces a run and gets a materially different score is encouraged to publish the counter-result. Where the counter-result reflects a real environment difference (different region, different plan, different model version), v1 will be updated to record the parameter as variable rather than fixed.

Versioning and change log

Methodology v1 is dated June 19, 2026. Future versions are minted when any of the following change: the axis set, the rubric for an axis, the test brief, or the environment standards. Each version mint is logged below with the reason for the change. Older versions remain accessible for citation.

Neutrality safeguards

A benchmark is only as trustworthy as the constraints that keep it honest. The relevant safeguards for a reader evaluating this methodology:

1. Every builder in the cohort is scored by the same rubric and the same brief. No builder receives a different test or a different scale.
2. The rubric, the test brief, the environment standards and the weights are published in full before any builder is scored, so the rubric cannot be tuned to favour a result after the fact.
3. The methodology is the binding constraint. A score that contradicts what we might have expected or preferred is still published.
4. Counter-results from any source, including any vendor, are reviewed against the same rubric. Where a counter-result reproduces the published rubric and produces a different number, the published score is corrected.
5. A published number the lab cannot substantiate is withdrawn, not softened. The August 21, 2026 withdrawal of all four v1 result sets is the first application of this rule and is recorded in the change log below.

Published constraints do not eliminate bias risk on their own; they surface the method so a reader can weigh it. The structural defence is the rubric.

Builders scored under v1

No builder is currently scored under v1. The four June 2026 result sets were withdrawn on August 21, 2026 (see the v1.1 update at the top of this page), so there is no v1 cohort standing behind a published number.

An earlier version of this section named a seven-builder June 2026 cohort. That listing has been removed because it described the membership of a result set that no longer exists, and because it did not match the builder set used in BuilderProof's later documentation-derived axis work. The builders the lab currently tracks are listed on the builders page; tracking a builder is not the same as scoring one, and this page will not imply otherwise again.

When a hands-on harness runs, its cohort will be declared before the run rather than reconstructed after it.

Open questions

Five items are on the v2 candidate list. They are open questions, not commitments.

1. First-build stability. Whether to score the fraction of OQ-7 prompts that complete without manual intervention as a fifth axis, or whether it belongs inside output quality. Arguments either way.
2. Multi-run averaging. v1 reports one run per builder per axis. Multi-run averaging would lower variance but multiplies cost. The threshold above which a result deserves a multi-run is not yet defined.
3. Model-pinning. Whether to pin a specific model version per builder, accepting that the builder's default may drift, or accept default-drift as part of what the benchmark measures.
4. Localization. Whether to mint a Spanish-language OQ-7-ES variant for non-English builder evaluations.
5. External contributor flow. Whether to accept counter-results as full peer-reviewed contributions rather than the current corrections process.

Comments on any of these are welcome via the editorial contact form. Contributors are credited by real, verifiable work, not by attribution alone.

BuilderProof editorial team. "How We Benchmark AI App Builders: The BuilderProof Methodology v1". BuilderProof, June 2026. https://www.builderproof.org/benchmarks/how-we-benchmark-ai-app-builders-methodology-v1.

@misc{builderproof-how-we-benchmark-ai-app-builders-methodology-v1,
  title  = {{How We Benchmark AI App Builders: The BuilderProof Methodology v1}},
  author = {{BuilderProof editorial team}},
  year   = {2026},
  month  = {jun},
  howpublished = {\url{https://www.builderproof.org/benchmarks/how-we-benchmark-ai-app-builders-methodology-v1}},
  note   = {BuilderProof, builderproof.org}
}