Contents
Research · October 1, 2026

The Token Slot Machine: Unpredictability in Frontier Coding Agents

Frontier coding agents are slot machines: you pay per pull and hope. The Avastha research team pulled the lever nearly 7,000 times across three frontier models. The results were surprising: success is often down to luck, costs vary widely across tasks, and failures are indistinguishable from successes. Then we looked for ways to beat the house.

Share of successful outcomes that is due to luck in an agentic coding benchmark
Medium effort. Luck share calculation is described below.

Spending on agentic coding is growing at a rapid pace [1], but users are struggling to translate that spending into reliable outcomes. Part of the difficulty lies in a mismatch between how these systems are billed and how they work: token-based billing charges for consumption, while the value of an agent's work depends on what it achieves. This makes seemingly straightforward questions difficult to answer: for example, how much spending is necessary to get a desired outcome?

In addition, due to the inherent randomness of LLMs, repeating the same task can produce substantially different costs and outcomes — and do so in unintuitive, model-specific ways. The result is a highly complex, constantly-changing decision problem where relying on heuristics or user intuition is suboptimal. In this research note, we examine just how random agentic coding with frontier models is — and offer strategies to harness that randomness in the user's favor.

Key Findings

  • Randomness plays a significant part in whether a task succeeds.This is especially true for the Claude models: up to 51% of Fable 5 tasks and 54% of Opus 5 tasks show at least one success and one failure across four identical runs.
  • Randomness also plays a significant role in task cost (tokens spent).In up to 9% of tasks, the most expensive trial costs twice as much as the cheapest trial, under identical settings. The variance across tasks is even greater: even within this fairly homogeneous benchmark, the most expensive task costs up to 33× the cheapest one.
  • Strikingly, failed attempts cost no less on average than successful ones.Failed attempts are largely indistinguishable from successful attempts in tokens spent, agent steps, and the agent's own belief and reporting.
  • Cost and success rates both vary with the effort level chosen, but in different — and sometimes highly unintuitive — ways.Success in particular often plateaus: on 21% of tasks, Opus 5 performs no better on higher settings than on medium. And Astra is cheaper at xhigh than at high effort on 16% of tasks.
  • Careful orchestration that combines models and runs can elicit better outcomes at lower costs.For example, with Opus 5 at high effort, a second parallel run costs $63 per additional solved task; moving from high to xhigh instead costs $1,327.

Introduction

LLMs are stochastic models: they exhibit intrinsic randomness. Partly this is due to hardware- and kernel-level nondeterminism: even at a sampling temperature of zero, the same request can return different outputs [2][3]. However, recent closed-source models from the leading labs no longer permit controlling parameters such as temperature at all [4][5], rendering output even more variable. To date, there has been little systematic study of the impact of this variability on agentic coding outcomes.

To address this gap, we analyzed repeated rollouts of three frontier models on DeepSWE [6], a widely tracked and demanding benchmark of 113 original, long-horizon coding tasks — features, bug fixes and refactors across 91 real repositories in TypeScript, Go, Python and other languages. The models tested are:

  • GPT-6 Astra
  • Claude Fable 5
  • Claude Opus 5

Models have a further user-selected parameter: the effort level. In theory, higher effort levels map to more tokens spent by the model, with an increased chance of task success. As we discuss below, this straightforward intuition is often at odds with reality. For this report, each task was attempted four times by each model at each of the effort levels (low, medium, high, xhigh and max), resulting in 20 trials per model per task. The success or failure of a task is measured by a suite of hidden tests. Further methodological details are given in Detailed Methodology.

Note: this report focuses on a relatively homogeneous setting, a single closed-form coding benchmark. In coding tasks in the wild, we expect even greater variability in both cost and success.

1. Success is highly random — and moreso for some models than others

Figure 1: Tasks that both passed and failed
Share of tasks with at least one pass and one failure across four identical runs of the same model and effort setting.

Our first finding is the extent to which success on a task is a matter of luck. Running the same model on the same task four times — under identical settings — often results in different outcomes. At low effort, 54% of tasks for Opus 5 and 51% for Fable 5 had at least one success and one failure in the four trials. For Astra, this number was 29%.

The variability of outcome decreases as effort increases, but remains significant at all levels: even at max effort, about a third of the Claude models' tasks show mixed outcomes (Figure 1).

Luck or capability?

Figure 2: How much of success is luck
Shaded band corresponds to 95% interval.

Is a run failing an indication of bad luck, or is it due to an incapability of the model? Although four trials per task are not enough to estimate each task's success probability reliably, an elegant mathematical result (see details below) lets us answer this question in aggregate. The results are shown in Figure 2 above. Across all models and effort levels, luck accounts for at least 20% of the variation in outcomes, and the Claude models both have much higher luck shares than Astra. At medium effort, about 50% of the variation in whether a Claude run succeeds is luck, against 32% for Astra.

The math: how the luck share is calculated

Suppose task i has a true success probability pi, and you choose a task at random and run it once. Let Y ∈ {0, 1} be the outcome. Then the law of total variance [7] states that:

Var(Y)=Var(pi)+E[ pi(1 − pi) ] 'capability''luck'

The first term of the right-hand side of the equation describes how much tasks differ in difficulty. The second term is the task-specific randomness averaged over all tasks — i.e., the average 'luck' component. The left-hand side of the equation is the overall variance, which we can calculate simply from the average success rate p. Applying this decomposition, we obtain the luck share for each model and effort level (Figure 2).

2. Cost is particularly variable across tasks

Figure 3: Cost per task
At high effort. Each stroke corresponds to the four trials of a particular task; tasks are sorted in increasing order of median cost. Filled dots succeeded, hollow dots failed. Each panel has its own cost axis.

Cost behaves differently from success. Across the 15 configurations, 85% of the variance in run cost lies between tasks. And tasks differ enormously, even within this one benchmark: at identical settings, the most expensive task costs up to 33× the cheapest one (Figure 3).

Figure 4: Most expensive vs. least expensive trial cost
Share of task and setting combinations where the priciest of four identical runs cost at least this multiple of the cheapest, pooled over five effort levels.

However, rerunning a task also introduces cost variability. For Astra, for example, 41% of tasks have their most expensive trial at least 50% more expensive than the cheapest, under identical settings; in 9% of cases it is twice as expensive (Figure 4).

The tail is what makes budgeting challenging in particular. On one task, Fable 5 at low effort executed three trials costing from $3.74 to $5.12 each. The fourth identical run cost $35.89 (and failed). Such outliers are not limited to failures, and they occur for Opus and Astra as well. The same request, at the same settings, can cost several times as much as the previous run, with no prior warning.

3. Failures are largely indistinguishable from successes in both cost and agent behavior

Figure 5: Comparison of failing and passing runs
Ratio of failed trials to successful trials, averaged across tasks and effort levels, for the Claude models. Dot is median, whiskers show the interquartile range. 1× means no difference.

To all appearances, failing runs are nearly identical to successful runs. They have the same total cost (tokens spent), number of agent steps, peak context size, number of read and edit commands, and number of lines changed (Figure 5). They also end the same way: 89% of failing runs finished with their own last test run passing, against 90% of passing runs. The agent's own testing surfaced the problem in just 5 of 173 reviewed failures. In other words, a customer paying per token doesn't know which of the runs are failures, yet they pay the same for it as they would for a successful run.

4. Increasing effort level often leads to no improvement in outcomes

Figure 6: Success by effort level

An important parameter of control for the models is the effort level. In theory, raising the effort level induces higher costs but provides better success rates. In practice, we find the picture is much more complex.

First, we show success rates against effort level for each of the models (Figure 6). For Astra and Opus, the average success rate barely improves beyond medium effort; Fable, however, shows a more reliable progression.

Figure 7: Cost per effort level
Line is median run; shaded area corresponds to interquartile range. Log scales except steps.

This is despite the average cost of each model increasing significantly as effort levels are increased (Figure 7). The mechanism also differs between Astra and the Claude models. The Claude agents take many more steps as effort is increased, while Astra keeps its step count nearly flat and increases its thinking per step instead.

Figure 8: Per-task cost ratio going from one effort level to the next
Each dot is a task, showing its mean cost at the higher setting divided by its mean cost at the lower. Orange dots, below the line, got cheaper with more effort; the figure under each column is the share of tasks that did. Log scale.

Despite the seeming predictability of cost increase with effort level, there are still subtleties and unintuitive behaviors at the level of individual tasks. For example, Astra is cheaper at xhigh than at high effort on 16% of tasks (Figure 8).

5. Leveraging randomness in our favor

Figure 9: Success and cost for 1–4 runs of every configuration
Success rate against expected cost per task, with runs in parallel. Each dot is one to four runs of one model at one effort level. The highlighted line is the Pareto frontier. Log scale.

If more effort buys little reliable success, what can we do instead?

One idea is to leverage LLM randomness to our benefit, rather than view it as a drawback. Given a reliable verification method, we can run multiple trials for a particular task at a lower effort, boosting the chances that one of them is successful — while still spending less than a single run at a higher effort level. These multiple trials can be run in parallel, or sequentially — continuing to the next run only if the current run fails.

The Pareto frontier of this strategy is shown in Figure 9. In general, it is far more effective to run multiple trials at lower effort. For example, with Opus 5 at low effort, up to 4 sequential trials solve 85.7% of tasks, while a single Opus 5 run at max effort solves 73.3%; and the low-effort trials do so at a 3.7× lower cost. The exact optimal strategy depends on the model and effort level in question.

Mixing models

Figure 10: Every task, every configuration
113 tasks, sorted from most to least often solved across all 15 configurations. White: always solved. Orange: solved on some runs but not others, brighter for more passes. Hollow: never solved.

Models exhibit diverse, and often complementary, strengths and weaknesses. Strikingly, no task was beyond all three models tested. All of the 113 tasks were solved by at least one configuration, and each of the 22 tasks GPT-6 Astra never solved at xhigh was solved by a Claude model at some setting (Figure 10).

Figure 11: Don't repeat one model; mix them
At medium effort. Success with one, two or three parallel runs, for repeats of one model and for mixes across models, with cost per task.

This observation allows for a significantly stronger strategy than running a single model multiple times. Because the models fail on different tasks, mixing them beats repeating one (Figure 11). For example, at medium effort, one Astra run plus one Opus 5 run solves 86.7% of tasks for $6.41, against 77.5% for two Astra runs ($6.14) and 80.9% for two Opus 5 runs ($6.68). One run of each of the three models reaches 89%–91% at every effort level. The best single run of any configuration only reaches 74.1%.

Removing the risk from AI spend

Every finding above focuses on the same fundamental questions, asked task by task: what will this cost, and how likely is it to work?

To predictWhat makes it hard
What a task will costTask prices vary significantly even at identical settings, reruns can double the bill, and each configuration and model family interacts differently
Whether it will succeedA large portion of task success is due to run-level luck, and more effort does not reliably help with this
Which model and setting to usePass rates correlate only moderately between models, and the best choice changes from task to task
Whether a run workedFailing runs look identical to passing ones, and the agents claim success just as often

Today, whether it is an individual developer trying to answer the above questions per task, or a CIO estimating the budget needed and crafting an AI strategy to hit KPIs, these questions are faced largely without guidance — no estimate of what tasks will cost or whether they will succeed, and no alternative to paying per token and hoping for success. Moreover, AI model release velocity is faster than ever, so any heuristics that are learned via usage are quickly rendered obsolete.

About Avastha

At our core, we build technology to answer the above questions.

We also partner with organizations to build their own in-house version of our technology. Our team consists of experts in machine learning and finance — including ex-DeepMinders, faculty at leading universities, and Olympiad winners. Together, we have 30,000 citations and have published in leading journals and conferences across our fields.

We are more than happy to chat about anything in this report or the services we provide. If you'd like to learn more, contact us at founders@avastha.dev, or join the waitlist for our upcoming closed beta.

Join the waitlist

References

  1. [1]Gartner. “Gartner Predicts AI Coding Costs Will Surpass Average Developer’s Salary by 2028 as Token Consumption Surges.” Press release, June 24, 2026. https://www.gartner.com/en/newsroom/press-releases/2026-06-24-gartner-predicts-ai-coding-costs-will-surpass-average-developer-salary-by-2028-as-token-consumption-surges
  2. [2]H. He and Thinking Machines Lab. “Defeating Nondeterminism in LLM Inference.” Thinking Machines Lab: Connectionism, September 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
  3. [3]J. Yuan, H. Li, X. Ding, W. Xie, Y.-J. Li, W. Zhao, K. Wan, J. Shi, X. Hu and Z. Liu. “Give Me FP32 or Give Me Death? Challenges and Solutions for Reproducible Reasoning.” arXiv:2506.09501, 2025. https://arxiv.org/abs/2506.09501
  4. [4]OpenAI. “Using GPT-6.” OpenAI API documentation, 2026. https://developers.openai.com/api/docs/guides/latest-model
  5. [5]Anthropic. “Claude Platform release notes” (Claude Opus 4.7, April 16, 2026, and later models). Claude Platform documentation, 2026. https://platform.claude.com/docs/en/release-notes/overview
  6. [6]W. Huang, C. Lee, L. Tng and S. Ge. “DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks.” arXiv:2607.07946, 2026. https://arxiv.org/abs/2607.07946
  7. [7]“Law of total variance.” Wikipedia. https://en.wikipedia.org/wiki/Law_of_total_variance
  8. [8]Datacurve. “DeepSWE v1.1.” 2026. https://deepswe.datacurve.ai/blog/deepswe-v1-1
Detailed Methodology

The analysis uses the per-attempt records, transcripts, final patches and verifier reports from DeepSWE v1.1 by Datacurve [8] (Apache-2.0); runs took place between June 12 and September 1, 2026. Of 6,776 attempts, 93 were excluded as infrastructure errors: 73 Claude Fable 5 attempts that could not run after access to the model was suspended partway through the sweep, 17 Opus 5 verifier timeouts and 3 others. Agent timeouts count as failures, and Fable 5 declined one task on all 20 attempts, which also count as failures. Every model ran in the same agent harness, mini-swe-agent; the effects of other harnesses such as Claude Code or Codex were not tested. Costs are calculated from token counts priced at list rates. Mixed-outcome shares, success rates and luck shares use tasks with four scored runs at that setting, and luck-share intervals resample tasks. Shares of cost variance set by the task are intraclass correlations of log cost. Task costs in the cost charts are the median of each task's four runs, while per-task comparisons between effort settings use each task's mean cost. Retry figures, such as the cost per additional solved task, assume the passing attempt can be accurately recognized by DeepSWE's hidden tests. Per-task transcripts and rollouts were reviewed primarily by LLMs.