Did It Actually Go Through? A Proposed Axis for Outbound-Call Failure (September 2026)

Quick answer. Nothing in an AI app builder's normal output stops a generated application from calling a third-party API with no deadline, retrying that call on any error at all, and repeating a side effect the provider already performed. This page proposes an axis that scores what the emitted code does when an outbound call does not return a usable answer, and it is a pre-registration rather than a result. We have not run it against any builder, and no tool is placed or scored below.

A generated application is almost never self-contained. It charges a card, sends a message, resizes an image, geocodes an address, asks a model for a paragraph. Each of those is an HTTP request to somebody else's server, made from inside a request the user is waiting on. The interesting question is not whether the builder wires the call up correctly. They almost all do. The interesting question is what the emitted code does on the day the other server is slow, over quota, or silent.

There is a specific version of that day which is worse than the rest, and it has a name in Stripe's own documentation. The request went out. No answer came back. The application does not know whether the work happened.

What this axis measures, and what it does not

Three published axes touch this territory and the boundaries are worth stating in both directions, because a reader who cannot see the boundary is entitled to assume there is not one.

Our concurrent-write safety axis puts 15 points on the idempotency of retried mutations. Read its wording closely and every remedy it names is local: an idempotency key held in our own tables, a natural dedupe key, an upsert against a unique constraint. It is scoring whether our own state survives a duplicate, and our own state is something we can inspect, count and repair. This axis scores whether somebody else's state survives a duplicate, and somebody else's state is something we can only ask about. The two move in opposite directions, both ways. An application can hold a flawless unique constraint on its payments table and still charge a customer twice at the provider, because the outbound call carried no key and the local row was written once from the second response. An application can send a perfectly keyed request to the provider and still write two rows locally, because the retry path inserts before it reads. Neither result predicts the other.

That axis also concedes the gap in its own limitations section. Of its 8 points on retry discipline it writes that the signal "may be too generous for a cohort whose emitted code rarely raises the isolation level in the first place, in which case the signal is measuring the absence of a problem". Its retry signal is about a PostgreSQL serialization failure, which only arises under isolation levels the cohort mostly does not select. The retry that actually happens in generated code is the one around fetch.

Our transactional email deliverability axis is the closest neighbour, and it is closer than it looks. Its fourth signal carries 15 points and its failing case reads "Send is inline in the request, no retry, resend duplicates". That is an outbound call with a robustness problem. The axis is nonetheless scoped tightly, and it says so: it asks whether a message is "physically capable of arriving", which it calls a question about transport and reputation. The distinction that matters here is what the calling code needs back. An email send is fire-and-forget by construction, the provider owns the delivery retry after it accepts the message, and a duplicate is an annoyance. A charge, a model completion or a geocode is request-and-response, the answer is the thing the user is waiting for, and a duplicate is a second irreversible act. Both ways again: an application can queue every email off the request path and still block a sign-up on an uncapped synchronous call to a model provider, and an application can handle every model call impeccably while sending its password resets inline through a default test sender.

Our abuse and rate-limiting axis asks whether the generated app returns 429 with Retry-After when somebody calls it too often. This axis asks what the generated app does when somebody sends it a 429. Same header, opposite direction, and the two are genuinely independent: an app can throttle its own callers correctly and hammer its provider through a spend cap, or read every retry-after it receives while leaving its own login endpoint unlimited.

Finally, this axis is not a penetration test and it does not score the provider. A provider that returns unhelpful errors is a fact about the provider. What is scored is the emitted code's response to it.

The finding: retrying is a layered property and nobody documents the stack

Three independent layers in a generated application can each decide to retry an outbound call, and each is documented in isolation by a party that cannot see the others.

The standard layer. RFC 9110, section 9.2.2, is explicit that automatic retry is a privilege of idempotent methods, and that POST is not one of them. Verbatim: "A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied." It is unusually blunt about what happens when people guess: "Some clients take a riskier approach and attempt to guess when an automatic retry is possible." And it closes the section with a sentence that is the whole point of this axis: "A client SHOULD NOT automatically retry a failed automatic retry."

The SDK layer. Stripe's advanced error handling page opens by telling you the library already does this: "The official Stripe SDKs can handle most details involving HTTP and retries." Anthropic's rate-limit documentation refers in passing to "the SDKs' automatic retries". Both statements are true and neither is wrong. Neither can know what the application has wrapped around it.

The platform layer. Vercel documents that "If a Vercel Function doesn't complete within the duration, a 504 error code (FUNCTION_INVOCATION_TIMEOUT) is returned", with a default duration of 300 seconds on its Hobby tier. That is a deadline, it applies to the whole invocation rather than to any one call, and it was set by somebody who does not know what your outbound request does.

Compose the three and you get the configuration RFC 9110 names as wrong, assembled out of parts that are each individually correct: an application try/catch that retries, wrapped around an SDK that already retried, inside a platform that will eventually kill the invocation and return a 504 to a browser that stopped waiting minutes ago. We have not found a document that describes the composition, because each document is scoped to one layer. That is the gap this axis exists to measure.

The default is not a choice

There is no request timeout in fetch. To get one you must construct a signal and pass it, and MDN documents the mechanism as a deliberate, separate act: AbortSignal.timeout() "returns an AbortSignal that will automatically abort after a specified time", aborting with a TimeoutError. It also records a property that matters more in production than in a preview. Verbatim: "The timeout is based on active rather than elapsed time, and will effectively be paused if the code is running in a suspended worker, or while the document is in a back-forward cache." A deadline that pauses is not a wall-clock deadline.

So an await fetch(url, options) written without a signal has no deadline of its own. Its effective deadline is whichever platform ceiling fires first, which on the default Vercel configuration is five minutes. This is the third pattern of its kind this benchmark has recorded, after byte equality being the default comparison rule and an unbounded session being the default session, and the rubric below is shaped accordingly: it scores whether a decision was made, not which decision was made.

Two documented tensions, in the vendors' own sentences

The key you must reuse is the key you must not reuse. Stripe is unambiguous that the in-doubt case requires the same key. Verbatim: "When intermittent problems occur, clients are usually left in a state where they don't know whether or not the server received the request. To get a definitive answer, they should retry such requests with the same idempotency keys and the same parameters until they're able to receive a result from the server." The same page is equally unambiguous that for a client error the same key is a trap, because the API caches the outcome whatever it was: "A request that returns a 400 sends back the same 400 if followed by a new request with the same idempotency key. Generate a fresh idempotency key when modifying the original request to get a successful result." And it names the exception to its own exception: "a request that's rate limited with a 429 can produce a different result with the same idempotency key because rate limiters run before the API's idempotency layer." Its conclusion is a rule of thumb rather than a rule: "the safest strategy where 4xx errors are concerned is to always generate a new idempotency key."

No option is simply correct. Whether to reuse the key is decided by the failure class, and the failure class is exactly what a blanket catch (e) { retry() } cannot see. The resolution is classification, not choice, which is why signal 2 below scores the classification rather than the retry.

The remedy makes one case worse. Anthropic's rate limits documentation describes an ordinary rate limit as retryable and self-describing: exceed requests or tokens per minute and you receive "a 429 error describing which rate limit was exceeded, along with a retry-after header indicating how long to wait", where the header means "The number of seconds to wait until you can retry the request. Earlier retries will fail."

Then it describes a second condition that is not retryable at all and wears the same clothes. On reaching an organisation spend cap, usage pauses "until 00:00 UTC on the first day of the next month" and requests return HTTP 429. Verbatim: "The error type is rate_limit_error, the same as for a rate limit, but the response has no retry-after header. Retrying, including the SDKs' automatic retries, fails until access resumes." The documented way to tell them apart is a nested field: "On the Messages API, error.details.error_code is enforced_spend_limit_reached. Use it to tell this response apart from a rate limit."

Two things follow, and both are uncomfortable. The automatic retry that is the correct behaviour for the first case is the wrong behaviour for the second, and the status code does not distinguish them. And the recovery window can be weeks, which no backoff schedule reaches. A third variant makes the point sharper still: a spend limit you set yourself returns HTTP 400 with invalid_request_error, so one underlying condition, having run out of money, surfaces as 429 under one configuration and 400 under another.

The same page also documents a limit that is triggered by growth rather than by volume, which is worth quoting because it is the failure mode a successful launch produces: you may hit acceleration limits "if your organization has a sharp increase in usage", and the advice is to "ramp up your traffic gradually and maintain consistent usage patterns."

The seven signals

We propose scoring this axis from seven weighted signals summing to 100. Every one is observable by reading the exported application and by exercising it against a provider that is made to fail, which keeps the axis reproducible without privileged access to any builder.

1. A deadline on every outbound call (22). This carries the heaviest weight because its failure scales with somebody else's bad day rather than with anything in the code, and because every other signal is moot without it. A call with no deadline does not fail, it waits, and while it waits it holds everything behind it. That includes a database connection, which is why this axis and our database connection and query cost axis compose rather than overlap: an uncapped outbound call made while a connection is checked out converts a slow third party into connection exhaustion, and the symptom appears in a part of the system that is working correctly. Full marks require a deadline chosen for the call, not inherited from the platform.

2. The retry decision is keyed to the failure class (18). The two vendor documents above agree on the shape of the problem and disagree about nothing: a network error and a 400 demand opposite handling, and a 429 can mean wait thirty seconds or wait three weeks. A code path that cannot name which case it is in cannot be correct in more than one of them. We score whether the emitted code distinguishes at least three classes: no response received, a response saying the request was rejected, and a response saying to come back later.

3. The provider's wait instruction is read and obeyed (15). RFC 9110 section 10.2.3 defines Retry-After as either an HTTP date or a delay in seconds, and its language is advisory: servers send it to indicate how long the client "ought to wait". Providers are less relaxed. Anthropic states plainly that "Earlier retries will fail." We score whether the header is parsed at all, whether both permitted formats are handled, and whether the absence of the header is treated as information rather than as a licence to retry immediately.

4. Side-effecting retries carry a stable idempotency key (15). This is the signal that separates an app which is merely resilient from one which is safe. Stripe's mechanism is precisely the "means to know that the request semantics are actually idempotent" that RFC 9110 requires before a POST may be retried at all, which is why the standard and the vendor fit together here rather than conflicting. We score three things: whether a key is sent on side-effecting calls, whether it is stable across the retries of one logical attempt rather than regenerated per attempt, and whether the code accounts for the documented lifetime. Stripe notes that keys may be removed "after they're at least 24 hours old" and that "We generate a new request if a key is reused after the original is pruned", so a retry queue that backs off past a day has quietly lost its protection.

5. The call is off the critical path when its answer is not the reply (12). Not every outbound call deserves this. If the user is waiting for the model's paragraph, the paragraph has to be fetched before the response is sent. The signal is whether the emitted code can tell the difference, and whether the calls that are genuinely incidental to the reply are deferred rather than awaited.

6. Failure resolves to a distinguishable state (10). After an in-doubt failure, the application should be in a state that can be reconciled: a persisted attempt with the key that was used, a status that is not "succeeded" and not "failed", and something a later job or a human can take to the provider and ask about. An app that returns a generic error and writes nothing has thrown away the only evidence that the attempt happened.

7. Outbound failures are observable (8). The lowest weight, and still not zero, because everything above is unverifiable without it. We score whether a failed outbound call leaves a record naming the provider, the status or error class, the request identifier where the provider supplies one, and the elapsed time.

Four postures

Level 0, Optimistic. The call is awaited with no deadline, there is no retry, and there is no key. This is not negligence so much as the default: it is what the code looks like when nobody decided anything. It works until the provider has a bad minute.

Level 1, Blind retry. A retry exists and is applied uniformly. This is better than Level 0 against a dropped packet and worse than Level 0 against a spend cap, a bad API key and a validation error, because it turns a fast clear failure into a slow unclear one. Level 1 is also the first level capable of double-charging a customer, which is why we do not treat the boundary from 0 to 1 as unambiguous progress.

Level 2, Classified. Failure classes are distinguished, the provider's wait instruction is read, and every outbound call has a deadline chosen for it. The app now fails quickly, correctly and legibly. Side effects may still duplicate, because classification tells you whether to retry and not whether the first attempt landed.

Level 3, In-doubt-safe. Level 2 plus stable idempotency keys on side-effecting calls, plus a resting state that can be reconciled against the provider afterwards. The boundary from 2 to 3 is the one that cannot be added later by configuration, because it requires the calling code to have generated and persisted a key before the request it is now unsure about.

The trap: the healthy-provider illusion

Every axis in this series names the illusion that hides its defect. This one is the healthy-provider illusion.

Every observation available while building an application is made against a third party that is up, fast, well under quota, and answering exactly one caller: you. Your API key is new, your spend is near zero, your request rate is a handful per hour, and the provider's status page is green. Under those conditions a Level 0 application and a Level 3 application are byte-for-byte identical in every trace you can collect. The charge succeeds. The model answers. The geocode returns. There is nothing to see, because the code paths that separate the four postures are precisely the ones that never execute.

It joins a family this benchmark keeps finding. Fire-and-forget, honest-caller, one-keyboard, fresh-session, still-signed-in and empty-waiting-room all share the same structure: every observation available at test time is produced by a single actor, so the defect is invisible rather than subtle. This one has an additional property that only two of the others share. It gets worse as the project succeeds. A rate limit is reached by having users, a spend cap is reached by being used, and Anthropic's acceleration limit is documented as responding to "a sharp increase in usage", which is the definition of a good week.

A ten-step measurement protocol

1. Pin the target. Record the builder, the deployment target and the date, because the platform's invocation ceiling is part of the answer and differs between hosts.
2. Generate from the fixed brief, which must include at least one side-effecting outbound call and at least one read-only one, and export the application untouched.
3. Establish the healthy-provider control first. Exercise both calls against the real provider and confirm they succeed. This is the baseline that makes every later failure attributable, and skipping it is how a broken fixture gets recorded as a builder defect.
4. Read the source and record what is there before touching anything: every outbound call site, whether a timeout or signal is passed, whether an SDK or a raw client is used, whether any key is generated, and where the retry lives if one exists.
5. Inject no response. Point the call at an endpoint that accepts the connection and never replies. Record the wall-clock time until the user-visible response, the verbatim text the user sees, and the state of the database afterwards.
6. Inject a 429 with a retry-after of 30. Record whether the app waits, how long it waits, and how many requests it sends.
7. Inject a 429 with no retry-after. This is the spend-cap shape. Record whether the app distinguishes it from step 6, and report the number of requests it sends rather than a pass or fail, because the count is the finding.
8. Inject a 400 with a validation body. Record whether the app retries it, and if it does, whether it reuses the same idempotency key.
9. Inject a reset after the request is accepted but before the response is sent. This is the in-doubt case and the only one that can produce a duplicate side effect. Record what the provider's own dashboard shows afterwards, and what the app's database shows, and whether the two can be reconciled from the app's records alone.
10. Run the documentation control. Separately from all of the above, record what the builder's own documentation claims about reliability, retries or resilience. Score steps 4 through 9 only, then report the documentation claim beside the measurement. An axis that scores the documentation is scoring a marketing decision.

Steps 5 through 9 are five different defects that can all produce the same user-visible symptom, which is a spinner and then an error. Reporting them as one number would discard the entire finding.

Limitations and open questions

We cannot separate the SDK from the application. If a builder emits a Stripe SDK call with sensible defaults, the application scores well on signals 2 and 3 without anyone having decided anything. That is a real outcome for a real user and we are inclined to credit it, but it measures a dependency choice rather than emitted logic, and a reader who thinks those should be scored separately has an argument we would like to hear.

Signal 5 needs a judgement call we have not removed. Deciding that a welcome email is incidental while a model completion is not introduces a subjective step into an otherwise mechanical rubric. Fixing the brief so that the classification is unambiguous is one option, and we lean that way without being sure.

Step 9 is the hardest to standardise and carries the most weight. Producing a genuine mid-response reset reliably, against a provider whose sandbox you do not control, is fiddly enough that two careful people could produce different numbers. Until that is pinned down, the signal with the clearest safety consequence is the least reproducible thing in the rubric, which is uncomfortable.

The weights are a first draft, and signal 1 may be too heavy. Twenty-two points assumes the platform ceiling is a poor substitute for a real deadline. On a host with a short, aggressive invocation limit the practical gap between having a deadline and inheriting one narrows considerably, and the score would then be measuring the host.

We do not know how much of this a cohort will fail. It is entirely possible that modern builders emit SDK calls almost everywhere and land at Level 2 by default, in which case the useful distribution is between Level 2 and Level 3 and the lower levels are dead weight. We would rather say that in advance than discover it and re-weight afterwards.

How to contribute

BuilderProof axes are community-editable and this one is deliberately unrun. The most useful contributions, in order: a disagreement with the weights that comes with a reason; a reproducible recipe for step 9 that does not depend on a particular provider's sandbox; and a documented case of a fourth retrying layer we have not accounted for, such as a proxy or a gateway sitting between the application and the provider. Corrections to any quotation on this page are welcome and will be made with a dated note rather than silently.

BuilderProof editorial team. "Did It Actually Go Through? A Proposed Axis for Outbound-Call Failure (September 2026)". BuilderProof, September 2026. https://www.builderproof.org/benchmarks/did-it-actually-go-through-outbound-call-failure-axis-september-2026.

@misc{builderproof-did-it-actually-go-through-outbound-call-failure-axis-september-2026,
  title  = {{Did It Actually Go Through? A Proposed Axis for Outbound-Call Failure (September 2026)}},
  author = {{BuilderProof editorial team}},
  year   = {2026},
  month  = {sep},
  howpublished = {\url{https://www.builderproof.org/benchmarks/did-it-actually-go-through-outbound-call-failure-axis-september-2026}},
  note   = {BuilderProof, builderproof.org}
}