The 2026 Research Wave on AI App Builders: What Six New Papers Actually Measure (August 2026)

BuilderProof withdrew its own numeric results on August 21, 2026. That makes this a strange week to publish a piece about measurement, and a good one. Between June 17 and August 17, 2026, six empirical papers landed on arXiv that touch, directly or obliquely, on the question this lab exists to answer: what actually comes out of an AI app builder. This post maps them. It reports what each one measures, what it does not, and where its own authors say it is thin. None of these results are ours. That is the point of publishing them.

Quick Answer (August 2026)

Six empirical papers on vibe coding and AI app builders were published between June 17 and August 17, 2026. Only one of the six, arXiv:2608.16302 by da Mota and Gama at UFPE, measures the commercial products people actually buy: it runs Lovable, v0 and Replit through one identical prompt three times each, nine apps total, and analyses the output with SonarQube. Its headline is that the three tools have different shapes of problem rather than a clean ranking. The other five measure coding agents, evaluation harnesses, deployed applications, or the research literature itself. The single largest measured finding across the set comes from the security study arXiv:2606.23130, which audited 200 real vibe-coded repositories and found that 180 of them, 90 percent, contained at least one vulnerability, with broken access control present in 75.5 percent of audited repositories. The honest summary for a buyer is that the 2026 literature is much stronger on how AI-generated software fails than on which product to choose.

The six papers, at a glance

This is a snapshot of one window, not a field survey. A broader survey already exists: A Survey of Vibe Coding with Large Language Models (2510.12399, October 2025, revised December 2025) works from a systematic analysis of over 1,000 papers. What follows is narrower and deliberately so. It asks what the newest empirical work measures against the question a person choosing a builder is actually asking.

Finding 1: five of the six do not measure the products people buy

This is the most useful thing to know before reading any of them, and it is easy to miss because the vocabulary overlaps almost perfectly.

2608.16302 evaluates Lovable, v0 and Replit as products. StaminaBench evaluates six agent harnesses paired with seven open-source language models. ICAE-Bench evaluates coding agents. The multivocal literature review evaluates the literature. The security study evaluates deployed applications, using Lovable and Claude Code fingerprints as a discovery mechanism rather than as a comparison cohort.

VideoVIBE is the one most likely to be mis-cited, because its subject is the evaluator rather than the builder. It converts 6,338 verified failures in generated webpages into roughly 1,700 diagnostic video question-answering instances, then scores thirteen video multimodal models on how well they can identify what went wrong. Its leaderboard number, Gemini-2.5-Flash at 64.54, with the authors' V2Lens system reaching 71.72, is a score for a diagnostician, not for a website generator. Anyone reading a headline about a "one-shot website generation benchmark" and inferring a builder ranking has mis-read it.

The practical consequence: if you are choosing between builders in August 2026, there is exactly one recent peer-reviewable comparison, it covers three products, and the next section is about how far it can carry you.

Finding 2: the one comparison that exists is nine applications, and the authors say so first

2608.16302 is an Innovative Ideas and Emerging Results submission to the 40th Brazilian Symposium on Software Engineering. It describes its own results as preliminary in the abstract. Its design is clean and worth stating precisely, because the precision is what makes it useful.

One zero-shot prompt, identical for every generation, specifying a "Nature Park Wildlife App" with wildlife sighting registration, safety issue reporting, notification subscriptions and a geolocated map. Three independent generations per tool. Nine applications. Generation window May 6 to May 21, 2026, on free tiers. Because these are continuously updated platforms with no public version numbers, the authors record build identifiers instead: Lovable on the tanstack_start_ts template via the gpt-engineer-app agent, v0 producing Next.js 16.2.4, Replit on Nix channel stable-25_05 with Node.js 24. Static analysis with SonarQube.

Their consolidated table, reproduced here with the source cited, is the only cross-builder measured comparison published in this window:

Source: da Mota and Gama, arXiv:2608.16302v1, Table 6, August 2026.

The authors conclude that v0 was the most balanced of the three. They also report the qualitative shape, which matters more than the ordering: Lovable's issues are about 86 percent MINOR severity but concentrate the highest remediation effort, with rule S1874 on deprecated APIs accounting for 204 occurrences and more than half of that effort. Replit produced no CRITICAL issues at all but the highest share of MAJOR ones, 213 occurrences at 46 percent of its total.

Their stated limitations are unusually candid and should travel with any citation of the numbers. Three projects per tool is a small sample, and the paper quantifies the resulting instability itself: the standard deviation of the code-smells metric was 62.1 for Lovable against 7.0 for v0 and 20.3 for Replit, driven largely by one Lovable project that recorded 58 code smells where its siblings recorded 164 and 167. They cite Marcilio and colleagues, who studied 421,976 issues across 246 projects and found that on average only about 13 percent of SonarQube-reported issues are ever fixed, as a caution against reading the technical-debt index as ground truth. And they flag, explicitly labelling it speculation rather than a claim, that free-tier resource allocation is opaque and lower-capacity models may be substituted at peak times.

Finding 3: per-KLOC normalisation is not neutral to duplication, and here it changes the order

This is our own reading of the published table rather than a claim the paper makes, and it is checkable by anyone, because the authors released their artifacts.

Rank the three tools by raw issue count and you get v0 with 254, then Lovable with 390, then Replit with 463. Rank them by issues per thousand lines and you get v0 at 9.21, then Replit at 9.56, then Lovable at 22.83. Lovable and Replit swap places. v0 wins either way, so the paper's headline conclusion is not disturbed, but the second and third positions depend entirely on which normalisation you read.

The reason the normalisation is doing so much work is in the same table. Replit duplicated roughly 60 percent of its lines. v0 duplicated at most 7 percent. Lovable duplicated none. Lines of code is the denominator, and one member of the cohort has a denominator that is majority duplicate. A tool that solves the same pinned functional requirement in 48,411 lines with 60 percent duplication and a tool that solves it in 17,081 lines with no duplication are not being divided by the same kind of quantity.

The paper anticipates part of this. Its own text argues that raw figures remain essential because "there is no merit in fulfilling the same functional requirements with more lines of code, especially when there is massive duplication, as in the case of Replit." What it does not do is close the loop and note that the per-KLOC column it publishes alongside inherits exactly that distortion.

We want to be careful about the direction of the bias, because it is not determinable from the published table. Duplicated code inflates the denominator, which pushes density down. But duplicated code that contains a violation also inflates the numerator, because most static-analysis rules raise one issue per occurrence, which pushes density back up. Which effect dominates depends on whether Replit's issues are concentrated inside or outside its duplicated blocks. That question is answerable: the full SonarQube reports and all nine projects are published as research artifacts. It is a good candidate for exactly the kind of re-analysis an open benchmark should do rather than assert.

The generalisable rule, and the one we are adopting for our own rubrics: when a functional requirement is pinned by an identical prompt, the delivered functionality is the constant and lines of code is a tool-dependent variable. Normalising by the variable can quietly reward verbosity.

Finding 4: three teams independently took the language model out of the measuring instrument

Read the four benchmark papers side by side and a design consensus appears that none of them announces, because each team seems to have arrived at it separately.

StaminaBench generates its tests "fully programmatically without LLM involvement, ensuring reproducibility and reliability," and runs the agent and the server in an isolated environment communicating over HTTP so that testing is fully black-box and language-agnostic. ICAE-Bench uses standardised black-box tests, and grounds its simulated user in what it calls User Agent Data specifically so the simulator can reveal hidden constraints "without inventing new requirements or leaking implementation artifacts." 2608.16302 uses SonarQube, a deterministic static analyser, and justifies the choice partly on comparability and reproducibility.

VideoVIBE is the instructive exception. It deliberately puts models in the instrument, and then reports how well that works: the strongest of thirteen video models scores 64.54, and the authors' own multi-agent verification system, which re-checks initial diagnoses against visual and source-code evidence, reaches 71.72. Both numbers are a long way from a reliable judge. The paper is honest about this and its whole contribution is the machinery for making the judgment more evidence-grounded.

Taken together, that is a fairly strong signal about where the field has landed on evaluation hygiene in 2026, and it is a signal we would rather report from other people's designs than assert from our own.

Finding 5: StaminaBench measures a variable nobody scores in commercial builders

StaminaBench asks a question the single-shot benchmarks structurally cannot: how many consecutive change requests can an agent absorb before it breaks. It has agents implement a REST API server and then modify it across 100 procedurally generated follow-up change requests, producing codebases of up to 6,000 lines, across 20 scenarios, with six agent harnesses and seven open-source models.

Three findings, as reported in the paper's abstract:

1. Every tested model failed within five to six turns.
2. Feeding test results back to the agent and letting it retry improved the passed turn count by up to twelve times.
3. A stronger model showed up to a six times gap between its best and worst harness, while weaker models failed under any harness.

Points two and three are the transferable ones, and they cut directly at how builder comparisons are usually conducted. A commercial AI app builder is a harness. It owns the retry loop, decides whether test or runtime feedback returns to the model, chooses the error surface, and manages context across turns. If harness choice can move the same model by a factor of six, then a benchmark that scores one prompt and one generation is measuring the harness and the model together while reporting a single number, and it is measuring the part of the harness that matters least, because turn one is where the retry loop has not yet done anything.

We are not aware of any published benchmark that scores commercial builders across a long change-request sequence. That is a gap, and StaminaBench shows it is tractable, since its testing is black-box over HTTP and language-agnostic by construction. It is not tractable for us today, because it needs paid accounts on five platforms and a hands-on harness we have not built.

Finding 6: the security study is the largest measured result in the window

2606.23130, by Deng, Fan and Meng, is the most quantitatively substantial of the six. Its pipeline is worth understanding because its selection criteria shape what the numbers mean.

Candidate applications were discovered on GitHub by agent fingerprints: a .claude/ configuration directory for Claude Code, and an author metadata tag naming Lovable for Lovable. That produced 74,800 candidates, narrowed to 37,962 by documentation and size filters, then narrowed again by requiring the first commit to be AI-authored and both AI-authored commits and AI-authored lines to exceed 85 percent. From that pool, 200 repositories were audited in depth, yielding 1,471 validated vulnerabilities.

The prevalence result: 180 of 200 audited repositories, 90 percent, contained at least one vulnerability. Among vulnerable repositories the median count was 7 and the mean 8.1, with an interquartile range of 3 to 11. The distribution is broad rather than outlier-driven.

The counter-intuitive result: vulnerability density falls as repositories grow, from 1.62 per thousand lines in repositories under 2,000 lines to 0.06 in repositories over 100,000 lines, while absolute counts rise. The authors' conclusion is the one to carry: small vibe-coded applications should not be assumed safer because they are small.

The category breakdown, with the paper's own comparison against OWASP's reported application-level incidence rates:

Source: Deng, Fan and Meng, arXiv:2606.23130v2, Table II, June 2026. OWASP baseline figures are as cited in that table.

Two cautions belong with this table. The last two columns are not measured the same way: one is the share of 200 deliberately-selected repositories found to contain a category during a targeted audit, the other is OWASP's application-level incidence rate across a very different population and methodology. The gap is large enough to be interesting and the comparison is the paper's own, but it is not a like-for-like ratio and should not be quoted as one. Second, the corpus is public GitHub repositories that are predominantly AI-authored. Applications built on a hosted platform and never pushed to a public repository are invisible to this method, and there is no reason to assume they resemble the ones that were.

The qualitative finding is the one that connects to rubric design. The recurring patterns the authors identify, placeholder logic, unfiltered input and secret exposure, are described as differing from those common in conventional development, and are attributed to systematic limitations of the agents themselves: memory loss across a long session, locally optimised objectives, and insufficient security knowledge. Better models and better prompting reduce the incidence without eliminating it.

That is independent measured support for the shape of several axes this lab proposed from documentation alone, including auth and access-control posture. It supports the choice of what to measure. It does not validate any score, because we do not currently publish any.

What none of the six measures

Stated plainly, because gaps are the useful output of a mapping exercise:

• Commercial builders over a long change-request sequence. StaminaBench has the method and evaluates open-source models under research harnesses. Nobody has pointed it at the products.
• The premium tiers. 2608.16302 used free tiers and lists premium replication as future work. Every published cross-builder number in this window is a free-tier number.
• More than three builders. Bolt.new and Base44 do not appear in any of the six as measured subjects.
• Anything after May 2026 in the one comparison that exists. The generation window closed May 21, 2026. These platforms ship continuously and expose no version numbers, which is precisely why the authors recorded build identifiers instead.
• Runtime behaviour of the emitted application. SonarQube reads structure. VideoVIBE watches recordings of behaviour but scores the watcher. Neither yields a behavioural score for a builder.
• Whether static-analysis findings predict real defects here. The paper's own cited caution, that roughly 13 percent of SonarQube issues get fixed in practice, applies to its own numbers too.

How we are treating this at BuilderProof

We should be direct about our position, because it changes how this post should be read.

On August 21, 2026 this lab withdrew its June output-quality result set and the composite ranking derived from it, after concluding we could not produce the artifacts to support them. We currently publish a documented benchmark methodology and a series of proposed scoring axes, and no results of our own. A lab in that position has no business grading other people's measurements. What it can usefully do is read them carefully, report what they contain, and be explicit about which conclusions the evidence will and will not carry.

Three things we are changing as a result. First, the normalisation point in Finding 3 goes into our rubric guidance: where a prompt pins the functional requirement, prefer the pinned constant over a tool-dependent denominator, and if you publish a normalised column, publish the distortion alongside it. Second, Finding 5 becomes an open question on our side rather than a proposed axis, because we cannot execute it. Third, we are treating the artifact-availability standard set by 2608.16302, full prompt, all nine projects, complete analyser reports, as the bar any result of ours has to clear before publication.

BuilderProof Editorial Team. "The 2026 Research Wave on AI App Builders: What Six New Papers Actually Measure (August 2026)". BuilderProof, August 2026. https://www.builderproof.org/benchmarks/ai-app-builder-research-wave-2026-what-six-new-papers-measure.

@misc{builderproof-ai-app-builder-research-wave-2026-what-six-new-papers-measure,
  title  = {{The 2026 Research Wave on AI App Builders: What Six New Papers Actually Measure (August 2026)}},
  author = {{BuilderProof editorial team}},
  year   = {2026},
  month  = {aug},
  howpublished = {\url{https://www.builderproof.org/benchmarks/ai-app-builder-research-wave-2026-what-six-new-papers-measure}},
  note   = {BuilderProof, builderproof.org}
}