Version Drift: Why an AI App Builder Benchmark Only Holds for the Build It Tested (2026)

Abstract. A benchmark number is a measurement, and every measurement has to record when it was taken and on what. AI app builders ship changes to their default models, agents, and scaffolding on a near-weekly cadence, which means a score collected one week can quietly stop describing the product a reader tries the next. This lab note documents the release velocity we observed across v0, Bolt, Lovable, Replit, and Base44 in the first half of 2026, read directly from each vendor's own changelog, explains why undated scores decay, and sets out the versioning protocol BuilderProof uses to keep its scores reproducible. It applies to every builder in the cohort equally. None is singled out.

Quick answer (July 2026)

A benchmark score for an AI app builder is only valid for the exact build and default model it was collected on. In the first half of 2026 the major builders each shipped changes that move the very attributes benchmarks measure. Bolt replaced its named model picker with two agents, Standard and Max, and added the Opus 4.6 model. Lovable made server-side rendering the default for new apps as of May 13, 2026. v0 rebuilt around Git, an in-app code editor, and Mini/Pro/Max model tiers. Replit added an install-time package firewall. Base44 added Fable 5 as a model option. Any responsible benchmark therefore has to do two things: stamp the version and date it tested, and re-run when a builder ships a material change. BuilderProof pins the build and re-runs on a monthly clock. A score without a version stamp is noise, and that includes ours.

A measurement without a timestamp is noise

Reproducibility is the whole reason a benchmark is worth more than a screenshot. If a reader cannot tell which build produced a number, they cannot check it, cannot compare it to a later run, and cannot know whether it still holds. This is the principle behind our published rubric (see How We Benchmark AI App Builders: the BuilderProof methodology v1), and it is why even a single axis can produce different numbers on two consecutive runs when the tooling underneath shifts (we walked through one such case in why two Lighthouse runs disagree).

For static software, versioning is a solved habit: you cite the commit, the release tag, the date. AI app builders make this harder in two ways. The default model behind the "build" button changes without a version bump the user sees, and the scaffolding the builder emits changes with it. Both are invisible unless you go read the changelog. So we go read the changelog.

What actually shipped in the first half of 2026

Every row below is drawn from the vendor's own public release notes, dated to 2026. The point is not to rank these changes. It is to show that the surface a benchmark measures moved under all five builders inside a single six-month window.

A reader who benchmarked Bolt in March 2026 on a named model was not testing the same product a reader who opened Bolt in June found waiting behind the Standard agent. A Lovable deploy-quality score from April, taken before server-side rendering became the default, describes a scaffolding the builder no longer emits for new apps. Neither original number was wrong. Both simply expired.

Why release velocity breaks naive benchmarks

Map the changes above onto what benchmarks actually score and the failure mode is obvious:

• Default-model swaps move output quality and iteration fidelity. When the model behind the build button changes, the code it writes and how well it revises on the second prompt change with it. Bolt's move to Standard/Max agents and Base44's Fable 5 option both sit on this axis.
• Scaffolding changes move deploy quality and portability. Lovable defaulting to server-side rendering changes the shape of the shipped app, its SEO surface, and its Lighthouse profile. v0's rebuild around Git and an editor changes how portable the output is.
• Guardrail changes move security posture. Replit's install-time package firewall changes the security story of what gets shipped, independent of the model.

An undated leaderboard flattens all of this into a single "current" claim it cannot actually support. The honest unit is not "Builder X scores N." It is "Builder X scored N on [default model], tested [date]."

How BuilderProof pins the build

Our protocol is deliberately boring, because boring is what reproducible looks like:

1. Record the build coordinates. For every scored run we log the test date, the default model or agent in effect, and the plan tier used. These travel with the score.
2. Version the scorecard, not just the post. A re-run produces a new dated row, and the prior row is kept, not overwritten, so the delta is visible.
3. Re-run on a monthly clock, and on material releases. A default-model swap, a scaffolding change, or a new guardrail (exactly the five events above) triggers a re-benchmark rather than a silent edit to an old number.
4. Publish the caveat, not just the number. Where a score predates a change we have logged, we say so and mark it for re-run instead of presenting it as current.

This is also why our cohort work leans on documentation-sourced axes wherever possible: a claim traced to a dated changelog line is re-checkable by anyone, in a way a one-off hands-on impression is not. It is the same reason we treat independent academic efforts as complementary rather than competing, which we mapped in Academic AI App Builder Benchmarks, Mapped (2026).

Where this leaves our own scores (the honest part)

Two of the changes documented above land on cohort members we scored earlier in 2026, which means some of our own published numbers are now pre-change snapshots awaiting a re-run. We do not get to exempt ourselves from the rule in this note. Every BuilderProof score carries its test date for exactly this reason, and the ones the 2026 changelogs have overtaken are queued for the next monthly cycle. We apply the same clock to every builder in the cohort. No vendor gets a frozen favorable snapshot, and no vendor gets held to a stale bad one. BuilderProof is the youngest of the reproducible benchmarks in this space, and version drift is one of the places that youth shows: the discipline is in admitting which numbers have expired, not in pretending they never do.