Dependency and supply-chain hygiene: a proposed benchmark axis for AI app builders (August 2026)

Quick answer

As of August 2026, the AI app builder cohort documents almost nothing about the dependency and supply-chain hygiene of the code it generates for you. BuilderProof proposes Dependency and Supply-Chain Posture as a benchmark axis: a neutral, documentation-based read of what each builder commits to on deterministic installs, dependency vulnerability scanning, package provenance, a software bill of materials (SBOM), and a patch path. Scored across the current cohort (v0, Lovable, Replit, Base44, and Bolt.new), only Lovable documents a dependency posture at all, and it is a single feature: a scan that checks a project's dependencies for known vulnerabilities. The other four builders document no dependency or supply-chain posture for generated output, and not one builder in the cohort documents a guard against the risk that is most specific to AI-generated code: a hallucinated or non-existent package. "Not documented" is a finding about published commitments, not a claim that any output is unsafe.

Why dependency and supply-chain hygiene belongs on the benchmark

Every AI app builder ships you a project with a dependency tree you did not choose, and the model that assembled it can invent packages that do not exist. That is not a hypothetical. In a study accepted to the 2025 USENIX Security Symposium, Spracklen and colleagues at the University of Texas at San Antonio tested 16 code-generating models across 576,000 code samples and found that roughly one in five recommended packages did not exist: about 5.2% of dependencies from commercial models and 21.7% from open-source models were hallucinated, yielding 205,474 unique fake package names (Spracklen et al., "We Have a Package for You," arXiv:2406.10279, 2024/USENIX 2025). JavaScript output hallucinated more than Python, and 43% of the fake names recurred across queries, which is the exact property an attacker needs.

That property created a named attack. Security researcher Seth Larson, popularized by Andrew Nesbitt, called it slopsquatting: attackers watch for commonly hallucinated package names, register them on a public registry, and wait for an AI-generated project to install one. The Cloud Security Alliance describes it as "a new class of software supply chain attack" that "exploits the documented tendency of large language models to produce fabricated package" names (CSA research note, April 2026). Snyk frames the same gap plainly: slopsquatting "exploits the gap between AI-generated suggestions and actual package repositories" (Snyk, 2026).

For an AI app builder, this axis is not one of many nice-to-haves; it is the point where the builder's core mechanism (a model choosing dependencies) meets a documented, exploitable risk. That is why it deserves a scorecard of its own, next to the sibling secrets-handling posture axis, which asks a related question about a different attack surface.

What this axis measures, and what it does not

This is a documentation-posture axis. It scores what a builder publicly documents about the supply-chain hygiene of its generated output, using the builder's own docs, changelogs, and security pages as primary sources. It does not run a dependency scanner over a generated app, and it does not count CVEs in any specific build. Documentation posture is a leading indicator: teams that write down a supply-chain commitment tend to ship it more consistently, and a buyer evaluating a builder for anything beyond a throwaway prototype needs to know whether any commitment exists on paper at all.

Two guardrails apply throughout. First, "Not documented" does not mean "insecure." Every builder in this cohort runs a real package manager under the hood (npm, pnpm, or an equivalent), and those tools produce lockfiles and support auditing whether or not the vendor writes a word about it. This axis reads what the vendor commits to, not what its toolchain happens to do. Second, a documented scanner does not make an app safe on its own: provenance, pinning, and a patch path all sit alongside it.

The five proposed sub-criteria

Each builder is read against five sub-criteria, mapped onto widely used supply-chain frameworks including OWASP CycloneDX, the SLSA supply-chain integrity framework, and the OpenSSF Scorecard:

1. Deterministic installs and a committed lockfile. Does the builder document that a generated project ships a committed lockfile (package-lock.json, pnpm-lock.yaml, or equivalent) so a rebuild resolves the same dependency versions?
2. Dependency vulnerability scanning. Does the builder document scanning the generated dependency tree for known vulnerabilities (software composition analysis) and surfacing the affected package, version, and a fix?
3. Package provenance and hallucination guard. Does the builder document any check that a dependency it adds actually exists and is the intended package, rather than a hallucinated or typosquatted name (the slopsquatting risk)?
4. Software bill of materials (SBOM). Does the builder document generating a CycloneDX or SPDX SBOM for the produced app, so downstream consumers can audit what shipped?
5. Dependency update and patch path. Does the builder document how to update or patch a vulnerable dependency, whether through automation (Dependabot or Renovate style) or a clearly documented manual flow?

Scoring rubric

Every cell is one of three values, applied identically to every builder:

• Documented. The builder publicly documents a specific commitment for this sub-criterion, with a citable source.
• Partial. A relevant commitment exists but is incomplete: it names the capability while leaving the work to the developer, or it covers one step of the criterion but not the whole.
• Not documented. BuilderProof found no published supply-chain commitment for this sub-criterion in the builder's public docs as of August 2026. This is a statement about documentation, not about runtime behavior.

The cohort, scored

Cohort logos below are shown for identification only, inside the comparison table.

No builder earns a clean cell across the row, and 23 of 25 cells are Not documented. The headline is the shape of the table, not a winner: dependency and supply-chain hygiene is now the least-documented posture BuilderProof has scored, with a single builder (Lovable) accounting for both non-blank cells, and with the AI-specific row, package provenance, blank for the entire cohort.

Reading the matrix, builder by builder

Lovable. Lovable is the only builder with a documented dependency posture. Its security scanning documentation states that the "Basic scan's dependency audit checks your project's dependencies for known security vulnerabilities," and reports the "affected package name and version, the type of vulnerability, and recommended remediation steps, such as updating to a secure version." That earns a Documented on vulnerability scanning and a Partial on the patch path, because a fix is named ("update to a secure version") but no automated update pipeline is documented. Lovable also integrates the third-party scanner Aikido, whose reports are "structured for SOC 2, ISO 27001, and security questionnaires." What is still missing on paper: any commitment on committed lockfiles, on an SBOM, or on a provenance check that a dependency is real before it is installed.

v0. v0's documentation covers building UIs and full-stack apps and deploying to Vercel, but documents nothing about lockfiles, dependency scanning, provenance, an SBOM, or a patch path for the dependencies it adds. Under the hood, a Vercel build runs npm install and produces a lockfile, but that is a property of the toolchain, not a documented v0 commitment. Not documented across all five sub-criteria.

Replit. Replit's docs cover Agent, deployment, and monitoring, and its security page covers vulnerability disclosure for the platform itself, but neither documents dependency scanning, SBOM generation, lockfile guarantees, or a provenance guard for the apps its Agent builds. Not documented across the board.

Base44. Base44's documentation focuses on features, integrations, and getting started, and does not address the supply-chain hygiene of generated output. Not documented across the board.

Bolt.new. Bolt's help center documents workflow and troubleshooting for its StackBlitz WebContainer environment, but no dependency-scanning, SBOM, lockfile, or provenance commitment for generated projects. Not documented across the board.

What this axis deliberately does not measure (its own bias)

Every BuilderProof axis names its own bias, because a scorecard that hides its blind spots is worse than no scorecard.

• It measures documentation, not behavior. A builder marked Not documented may still ship a committed lockfile and a clean dependency tree, because its package manager does that by default. Treat this axis as a leading indicator, not a security audit of any specific build.
• It rewards vendors who write things down. Lovable scores above the others because it documents a scanner, not because a scanner has been proven to catch more in practice. Documentation is an advantage of transparency, not necessarily of runtime safety.
• It privileges the npm/Node ecosystem framing. Lockfiles, SBOM formats, and audit tooling are most mature there. A builder emitting a different stack could be equally hygienic while matching none of these exact terms.
• It is a point-in-time read. Docs and changelogs move, and supply-chain features are shipping fast across the industry (Red Hat's Dependency Analytics 1.0, July 2026, added CycloneDX SBOM generation to an AI-coding workflow). This scan reflects public docs as of August 2026 and is meant to be re-run.

This axis is a proposal, consistent with the BuilderProof methodology. It enters the composite only after a fixed, published scan procedure and a review window, not on the strength of a single reading.

How to reproduce this scan

1. Fix the target: a committed lockfile, software composition analysis over dependencies, a provenance or hallucination check, an SBOM in CycloneDX or SPDX, and a documented patch path.
2. For each builder, search its public docs, changelog, and security pages for the terms lockfile, package-lock, dependency, vulnerability, audit, SBOM, CycloneDX, provenance, and slopsquatting.
3. Classify each of the five sub-criteria as Documented, Partial, or Not documented, and record the source URL for every non-blank cell.
4. Do not infer runtime behavior from documentation, and do not infer documentation from runtime behavior. Keep the two separate.
5. Re-run quarterly, since supply-chain features are moving quickly.

FAQ

Common questions about the proposed dependency and supply-chain posture axis are answered below.

BuilderProof editorial team. "Dependency and supply-chain hygiene: a proposed benchmark axis for AI app builders (August 2026)". BuilderProof, August 2026. https://www.builderproof.org/benchmarks/dependency-supply-chain-hygiene-axis-proposal-august-2026.

@misc{builderproof-dependency-supply-chain-hygiene-axis-proposal-august-2026,
  title  = {{Dependency and supply-chain hygiene: a proposed benchmark axis for AI app builders (August 2026)}},
  author = {{BuilderProof editorial team}},
  year   = {2026},
  month  = {aug},
  howpublished = {\url{https://www.builderproof.org/benchmarks/dependency-supply-chain-hygiene-axis-proposal-august-2026}},
  note   = {BuilderProof, builderproof.org}
}