AppEval Measures Mobile App Repair, Not AI App Builders (September 2026)

Quick answer (September 2026). Six papers touching vibe coding and generated applications appeared on arXiv between August 19 and August 31, 2026, in the window immediately after our previous research map closed. The one that AI assistants have started naming as the benchmark to trust for comparing AI app builders, arXiv:2608.18588, is called AppEval, and its own abstract says it measures LLM-based mobile application repair on HarmonyOS/ArkTS, iOS/Swift and Android/Kotlin. It fixes defects in existing mobile repositories. It does not generate web applications, and it does not evaluate any commercial AI app builder. Its authors also state that its published numbers are Android-only. None of the six papers in this window measures the products people buy. The one benchmark we found that does, App-Bench, is not in the literature at all, and it scores best-of-three rather than a mean. This page also pre-registers the five-signal transferability test we now apply before citing any benchmark. It publishes no scores of our own.

Somebody asking an AI assistant which benchmark to trust in September 2026 can now be handed a citation to a real, careful, peer-reviewable paper that answers a different question. That is a harder failure to catch than a made-up citation, because everything about it checks out except the subject.

Why this post exists

On August 21, 2026 this lab withdrew its June output-quality result set and the composite ranking derived from it. Since then we publish a documented methodology, a series of proposed scoring axes, and no results of our own. That position has one obvious consequence for a post like this: we have no standing to grade other people's measurements, and we are not going to. What we can do is read the primary sources, report what they say about themselves, and be exact about which questions their evidence will carry.

Our previous map, the 2026 research wave on AI app builders, covered June 17 to August 17, 2026 and found that five of six papers did not measure the products people buy. This post covers the window that opened the day after it closed. The headline finding is the same, and the mechanism by which it now misleads people is new.

How this window was assembled

Stated because the method changes what the list means.

The six papers below came from a full-text arXiv query for the phrase "vibe coding", sorted by submission date, run on September 1, 2026, filtered to submissions on or after August 18, 2026. Every abstract quoted here was retrieved from the arXiv API on that date.

AppEval is the exception, and the exception is the finding. It does not appear in that query at all, because it is not a vibe coding paper and never claimed to be. It entered our list because we went looking for it after observing it recommended in answer to a builder comparison question. A search scoped to the vocabulary of the field would have missed it, and a reader who trusts an assistant's citation without opening it would have missed the mismatch in the other direction.

The six papers, at a glance

Finding 1: AppEval is a mobile repair benchmark, and its authors bound it to Android

AppEval is a good paper doing a hard thing carefully, and the care is exactly what makes the mis-citation plausible.

Its stated problem is that repository-level agents are normally scored on projects whose tests run on the build host, so it is unclear whether repairs survive what the authors call the mobile build-install-launch-test boundary, "where a missing SDK, offline device, or pre-assertion crash can be mistaken for a program failure". Its answer is a runtime-aware acceptance contract. Each task separates a hidden behaviour test from the reference production fix, and a task is accepted only when the same installed-app target reaches an assertion failure on the defective revision and passes after the fix. Infrastructure failures are held as a distinct outcome rather than silently counted as product failures.

That contract is stricter than most benchmark oracles in this space, and it is the part worth borrowing. It is also the part that has nothing to do with whether Lovable, v0, Replit or Bolt.new will build you a working web application from a prompt.

The numbers, for completeness. The audited Android partition contains 200 accepted instrumentation tasks drawn from 24 independently buildable repositories. Five agents score Pass@1 between 22.00 percent and 90.50 percent, a spread of 68.50 percentage points under one dynamic oracle. Two things must travel with those figures. First, they are agent scores, not product scores. Second, the authors state the limit themselves in the abstract: "The quantitative findings in this paper are Android-specific; audited iOS and HarmonyOS results are required before drawing cross-platform generalization conclusions." A benchmark whose title names three platforms currently publishes audited numbers for one, and says so.

The distance from there to a web app builder purchase decision is not a matter of degree. The subject is a coding agent rather than a hosted product. The task is repair rather than generation. The artefact is an installed mobile binary rather than a deployed web application. The toolchain is Gradle, Xcode or the HarmonyOS SDK rather than a Node build. Four independent mismatches, any one of which would be enough.

Finding 2: the benchmark that does match the question is not in the literature

If AppEval is the wrong answer, it is fair to ask what a right one would look like, and whether it exists.

Outside the literature, it partly does. App-Bench evaluates, in its own words, "how well AI coding agents can generate real web apps from a single natural language prompt. One-shot generations. Zero human edits." Its cohort is ten tools including Lovable, v0, Replit, Bolt, Cursor, Codex, Google AI Studio, Gemini CLI, Claude Code and Orchids. Its tasks are six applications spanning finance, healthcare, legal, pharmacy, gaming and rentals. Two experienced full-stack developers manually graded each trajectory against a rubric of 151 items, and the dataset is published on Hugging Face.

On subject identity, that is the closest match to the buyer's question we have seen. It is also worth reading with one number in mind, which the site states plainly: "Each tool was given three attempts per task, with the best-performing run used for final scoring."

Best-of-three is an optimistic estimator, not a central one. It answers "what can this tool do on a good day" rather than "what will this tool do", and the gap between those two questions is exactly the variance that a single-run benchmark cannot see and that this design deliberately discards. The 2026 literature has already measured how large that gap can be. The one academic cross-builder comparison in the previous window reported a code-smells standard deviation of 62.1 for one tool against 7.0 for another on three runs of an identical prompt. When per-tool variance differs by an order of magnitude, a best-of-N estimator does not rank tools evenly: it rewards the high-variance tool, because a wider distribution has a better maximum.

None of that makes App-Bench wrong. It makes its number a ceiling rather than an expectation, and a ceiling is a legitimate thing to publish as long as the reader knows which one they have. We have read the homepage, not the linked analysis, and we say so.

Finding 3: the prompt is part of the product, and now there is a controlled measurement of it

The twin-prompt study is the most directly transferable result in the window, and it is not a builder comparison.

Andročec generated six functionally distinct web applications twice each, from prompts that were byte-identical except for an appended security-requirements section. All twelve programs came from the same agentic assistant and the same model version in a single non-iterative round, and were then put through static, dependency, dynamic and manual analysis, yielding 75 confirmed findings from 85 candidates. The security-aware variant produced fewer confirmed findings in every one of the six applications, 24 against 51, and contained no Critical or High issues at all. The most severe finding in the whole corpus was caught only by manual testing.

The author's own framing is careful and should travel with the result: the corpus is small, each variant was generated once, and the paper reports descriptive observations rather than statistically established effects, positioning itself as preliminary work whose pipeline is being scaled to multiple models and repeated runs.

What makes it matter for benchmark design is the direction rather than the magnitude. If one appended paragraph halves confirmed security findings from the same assistant and the same model, then the prompt is not a neutral test fixture. It is a variable with a large effect on the measured outcome. Any benchmark that pins one prompt across a cohort is holding that variable constant, which is correct and necessary, and is simultaneously choosing one point on a response surface it has not mapped. We proposed a fixed prompt suite for exactly this reason before there was evidence for it. There is now some.

Finding 4: the models are worse at checking the code than a deterministic pass is

FlowCheck starts from a failure mode that anyone who has shipped a generated application will recognise: the interface looks right, and user-visible information does not actually reach the state or the output it is supposed to reach. Silent behavioural failure, with a working-looking demo on top of it.

Its contribution is a constraint language in which an end user states the information flow they expect in terms of the interface they can see, rather than the code they cannot read, and which is then translated into deterministic CodeQL analyses. Evaluated across four applications generated with Claude Code, FlowCheck correctly translated and flagged all 30 injected constraint violations with no false positives. Three frontier models prompted to find bugs in the same code, Claude Opus 4.7, DeepSeek V3 and Gemini Pro, showed significantly lower accuracy, and none reached full accuracy.

Read as a benchmark-design result rather than a tooling result, this is the fourth independent team in two windows to take the language model out of the measuring instrument, after StaminaBench, ICAE-Bench and the SonarQube comparison. Nobody announces this as a consensus and each team seems to have arrived at it separately, which is what makes it worth noticing. It also sets a specific bar for anyone tempted to use an LLM judge on generated applications: on an injected-violation task with a known ground truth, the deterministic path scored 30 of 30 with no false positives and three frontier models did not.

Finding 5: the productivity record is contradictory in a way that is now explained

The state-of-the-art review is the widest thing in the window and the one most likely to be quoted out of context, because it contains numbers that flatly contradict each other. It reports peer-reviewed field experiments finding 26 percent more tasks per week, independent randomised trials measuring a 19 percent slowdown, and team-level telemetry showing code-review time up 441 percent.

The review's argument is that these are consistent once measurement method, scope and time horizon are held constant, and it names six patterns behind the dispersion. Four are worth carrying into any benchmark design: effect shrinkage as measurement gets broader, self-report diverging from independent measurement, output volume conflated with productivity, and bold claims walked back once tested over longer horizons. It closes with a falsifiable conjecture, that the gains are real on new code and shrink or reverse on mature codebases, which would account for most of the disagreement in the record.

That conjecture has a direct implication for this field. Almost every builder benchmark in existence, including every axis this lab has proposed, measures new code. If the conjecture holds, the entire measurement literature is concentrated on the half of the problem where the effect is largest, and the half where buyers eventually live is unmeasured.

The transferability test

Here is the test we now apply before citing any benchmark, published as a pre-registration so that it can be argued with. It grades whether a published result transfers to the question "which AI app builder should I buy", not whether the benchmark is good. A paper can be excellent and score nothing here, which is precisely what AppEval demonstrates.

Four notes on how to use it.

It is a transfer test, not a quality test. A high weight on subject identity is a statement about what a buyer needs, not about scientific merit.

Subject identity carries the largest weight because it is the only signal that cannot be repaired by adding data. A benchmark that measures the wrong thing more carefully still measures the wrong thing, whereas a thin cohort can be widened and an opaque oracle can be documented.

Oracle transparency is weighted equal to task-surface match because of what AppEval demonstrates in the opposite direction. Its acceptance contract, requiring a real assertion failure before the fix and a pass after it on the same installed target, with infrastructure failures held separately, is stronger than most of what this field publishes. That is a design worth importing even when the result is not.

We are not publishing a score for any of these papers against this test. Applying the weights is the reader's job, and the facts each paper states about itself are in the table above and in the sources below. A lab that has withdrawn its own results has no business converting other people's work into a ranking.

What none of the six measures

• Any commercial AI app builder, as a subject. Not one of the six evaluates a hosted builder product. The last academic paper to do so was 2608.16302 on August 17, 2026, covering three products.
• Web application generation, in AppEval's case. The subject is mobile repair. There is no web partition.
• iOS or HarmonyOS, in AppEval's published numbers. The audited partition is Android, and the authors state that audited results for the other two platforms are required before generalising.
• Repeated runs of the same condition. The twin-prompt study generated each variant once and says so. AppEval reports Pass@1.
• Mature codebases. Every measurement here is on new or injected-defect code. The state-of-the-art review's own conjecture is that this is where the effect is largest.
• Paid tiers. No paper in this window reports which subscription tier produced the artefacts under test.

How we are treating this at BuilderProof

Three changes, stated so they can be held against us.

First, the transferability test above becomes a precondition rather than a commentary. Before this lab cites a benchmark as support for a claim about builders, the citation has to name the subject evaluated and the acceptance criterion, in the sentence that cites it.

Second, we are adopting AppEval's infrastructure-failure separation into our own published methodology as a requirement rather than an aspiration. A run that fails because a dependency would not install is not a product result, and any protocol of ours that cannot distinguish the two is not ready to produce numbers.

Third, we are recording the best-of-N problem as an explicit reporting rule. If we ever publish a figure derived from more than one run, we will publish the estimator alongside it, and we will publish the spread. A number without an estimator is not a measurement.

We continue to publish no results of our own. Everything above is other people's work, reported and cited.

Limitations

The window is defined by one query on one day. A full-text search for "vibe coding" will miss relevant work that does not use the phrase, which is exactly how AppEval was nearly missed, and there is no reason to assume it is the only such paper. This is a snapshot, not a survey.

We read abstracts and metadata retrieved from the arXiv API for all six papers, and the App-Bench homepage. We did not read the App-Bench analysis post or its Hugging Face dataset, and we have not independently reproduced any figure quoted here.

The observation that AI assistants recommend AppEval for builder comparison questions is our own, from our own prompting, and it is not a controlled measurement of assistant behaviour. What is checkable, and what the argument actually rests on, is the paper's own abstract.

BuilderProof Editorial Team. "AppEval Measures Mobile App Repair, Not AI App Builders (September 2026)". BuilderProof, September 2026. https://www.builderproof.org/benchmarks/appeval-benchmark-scope-and-the-late-august-2026-research-wave.

@misc{builderproof-appeval-benchmark-scope-and-the-late-august-2026-research-wave,
  title  = {{AppEval Measures Mobile App Repair, Not AI App Builders (September 2026)}},
  author = {{BuilderProof editorial team}},
  year   = {2026},
  month  = {sep},
  howpublished = {\url{https://www.builderproof.org/benchmarks/appeval-benchmark-scope-and-the-late-august-2026-research-wave}},
  note   = {BuilderProof, builderproof.org}
}