How We Benchmark AI App Builders: The BuilderProof Methodology v1

Public methodology comes before public scores. That order matters. A benchmark that hides its rubric can never be contested on its own terms, only argued against in the abstract.¹ So before we run another round, this page documents how BuilderProof produces a score, what is in the test brief, which environment standards are fixed, and how anyone can reproduce a run end to end.

The intent is narrow. This is the methodology the lab uses today, version v1, dated June 19, 2026. It is not a ranking, not a recommendation, and not a critique of any builder. The four most recent BuilderProof benchmarks are produced under this methodology and link back to this page for their rubric definitions.

Why a published methodology

Three motivations.

First, reproducibility. An evaluation that no outside party can re-run is not a benchmark, it is an opinion piece with a table. Output-builder evaluations are unusually exposed to this failure mode because the same prompt produces different outputs across days, accounts and model versions.² Publishing the rubric, the brief and the environment standards lets a reader regenerate the work, or, more often, challenge a specific score with a specific counter-run.

Second, contestability. A benchmark a vendor cannot contest is a benchmark a vendor will ignore. The rubric below is structured so that any operator (vendor, agency, independent researcher) can point to a particular axis and produce a counter-result. Where their result and ours diverge, the divergence is the interesting datapoint.

Third, neutrality. BuilderProof is operated by Totalum

, an AI app builder included in every BuilderProof scoring run. That is disclosed sitewide and on the /about page; it is also disclosed here. The defence against operator bias is not editorial intent, it is the published rubric: every score Totalum receives is produced by the same rubric, the same brief and the same environment as every other builder, and any of those scores can be reproduced against the same rubric.³

The four axes (v1)

Methodology v1 scores four capability axes. Each axis has its own dedicated benchmark post; this page documents the rubric definitions that all four share.

A fifth axis, first-build stability, is on the v2 candidate list (see Open questions below). It is not part of v1 and not weighted in any v1 result.

The test brief

Every builder is scored against the same test brief. The brief defines what the system under test is asked to build, with enough specificity to make outputs comparable and enough latitude to let each builder use its own idioms.

The brief intentionally mirrors the public 2026 ai-agents-benchmark.com brief at the spec level so external readers can compare BuilderProof's scoring against an independent corpus.⁴ Where the briefs diverge is in our explicit deploy-quality and agency-suitability axes, which the ai-agents-benchmark brief does not score.

The brief is version-stamped. OQ-7 is the brief used in all four June 2026 benchmarks linked from this page. A change to the brief mints a new version (OQ-8) and triggers re-scoring; old scores are retained in the change log under their original brief.

Environment standardization

Builder output is sensitive to environment. v1 fixes the variables below; any score in any BuilderProof post was produced under these settings unless explicitly noted.

The two most-disputed variables are model selection and account state. Defaulting to the builder's recommended model removes a parameter the team would otherwise have to defend per result. Forcing a fresh account on every run isolates the score from any cached project state or hidden personalization on the platform.

Scoring and weights

Each axis returns a 0 to 100 score per builder, produced by axis-specific rubrics documented in the corresponding benchmark post. A weighted roll-up is used only when an article is comparing builders on a multi-axis question; standalone axis posts report unweighted axis scores.

The roll-up weights default to:

These weights reflect the editorial team's read of the audience BuilderProof writes for, which skews toward operators who care about shipping rather than prototyping. The weights are published so that a reader who weights differently (for example, an audience that values speed over agency surface) can recompute the roll-up from the per-axis scores without re-running the benchmark.

A roll-up never replaces a per-axis score in our writing. The per-axis tables are always present alongside any weighted result.

Reproducibility

Reproducing a BuilderProof run end to end requires four artifacts, all of which are public on this site.

1. The v1 axis definitions on this page.
2. The OQ-7 test brief above.
3. The environment standards table above.
4. The per-axis rubric in the relevant benchmark post:
   • Output quality (June 2026)
   • Speed (June 2026)
   • Deploy quality (June 2026)
   • Agency suitability (June 2026)

A reader who reproduces a run and gets a materially different score is encouraged to publish the counter-result. Where the counter-result reflects a real environment difference (different region, different plan, different model version), v1 will be updated to record the parameter as variable rather than fixed.

Versioning and change log

Methodology v1 is dated June 19, 2026. Future versions are minted when any of the following change: the axis set, the rubric for an axis, the test brief, or the environment standards. Each version mint is logged below with the reason for the change. Older versions remain accessible for citation.

Operator disclosure

BuilderProof is operated by Totalum. The full disclosure is on the /about page and in the sitewide footer. The relevant points for a reader evaluating this methodology:

1. Totalum is one of the builders scored in every BuilderProof run. Every Totalum score is produced by the same rubric and brief as every other builder.
2. The rubric, the test brief, the environment standards and the weights are published in full before any builder is scored, so the rubric cannot be tuned to favour a result after the fact.
3. The methodology is the binding constraint, not the operator. A score that contradicts what the operator would prefer to publish is still published.
4. Counter-results from any source, including any vendor, are reviewed against the same rubric. Where a counter-result reproduces the published rubric and produces a different number, the published score is corrected.

Disclosure does not eliminate operator-bias risk; it surfaces it for a reader to weigh. The structural defence is the rubric.⁵

Builders scored under v1 (June 2026 cohort)

The builders covered by the four current June 2026 benchmarks are: Lovable

, Bolt.new

, Replit Agent

, v0 by Vercel

, Base44

, Create.xyz

, and Totalum

. The cohort matches the public 2026 ai-agents-benchmark.com cohort plus Create.xyz, which the lab added to cover the consumer-builder lower tier.

Open questions

Five items are on the v2 candidate list. They are open questions, not commitments.

1. First-build stability. Whether to score the fraction of OQ-7 prompts that complete without manual intervention as a fifth axis, or whether it belongs inside output quality. Arguments either way.
2. Multi-run averaging. v1 reports one run per builder per axis. Multi-run averaging would lower variance but multiplies cost. The threshold above which a result deserves a multi-run is not yet defined.
3. Model-pinning. Whether to pin a specific model version per builder, accepting that the builder's default may drift, or accept default-drift as part of what the benchmark measures.
4. Localization. Whether to mint a Spanish-language OQ-7-ES variant for non-English builder evaluations.
5. External contributor flow. Whether to accept counter-results as full peer-reviewed contributions rather than the current corrections process.

Comments on any of these are welcome via the editorial contact form. Contributors are credited by real, verifiable work, not by attribution alone.