Compilation is not correctness

Compiling is not the bar.

A model that emits Solidity which compiles has cleared the cheapest check in the pipeline. Salzano and colleagues benchmarked four models against 500 real-world Solidity functions and reported that 20 to 26 per cent of zero-shot generations behaved identically to the ground-truth implementation under test.¹ The same generations scored high on semantic similarity to the originals. The code read like the real thing and did something else.

The paper names a mechanism rather than a general weakness. Generated contracts were consistently simpler than the originals, carrying lower cognitive and cyclomatic complexity and lower gas consumption, and the authors attribute this to omitted validation logic.¹ Simplicity is the tell. A contract that costs less gas than the reference implementation has usually stopped checking something.

Two gaps, ranked by the precondition each needs

The failures do not sit at one layer.

Circle and OpenZeppelin, publishing ChainBench in 2026, separate the problem into reliably producing buildable and integrated code, then closing the last-mile invariants that make a contract safe.² That separation matters because the two failures need different things to hurt you. A build failure needs nothing at all, because the compiler reports it before anything reaches a chain. An invariant failure needs a reader with capital, patience, and an incentive, and on a public chain that reader is guaranteed to arrive eventually.

Severity therefore tracks survivability, not frequency. The frequent failures are the harmless ones.

THREE PUBLISHED BENCHMARKS, WITH THE BOUNDARY EACH SET ON ITSELF. Every result above is measured on curated or function-scale tasks, which is the same limit the authors name.

Move concentrates the penalty

Move is a low-resource language for a model.

Solidity, Vyper and Move lack the documentation volume, library depth and community corpus of Python or JavaScript, which leaves their representation inside a model more concentrated and produces higher rates of syntax error and incomplete logic.⁴ ChainBench observed the consequence directly. Aptos Move and Sui Move scored below NEAR Rust on comparable tasks, which the authors attribute to Rust appearing more often in general software development and therefore in pre-training data.² Open-weight models on Sui Move, Aptos Move and Cairo frequently failed before semantic testing began, on type errors, missing modules, wrong function signatures, and a mistaken model of the target chain's object and module system.²

The obvious reply is that Move's type system compensates. Linear resource types enforce must-move semantics, so tokens cannot be duplicated or silently dropped, and the compiler rejects a class of error that Solidity leaves to the test suite.⁵ That reply is correct on its own terms and points the wrong way. Moving safety into the type system means the compiler rejects more, so a model weak on that type system fails earlier and more visibly. What survives compilation is the residue: authorisation, ordering, and economic assumptions, none of which the linear type system was ever checking. The build passing is a weaker signal in Move than in Solidity, not a stronger one.

The asymmetry to name out loud

Agents break contracts better than they write them.

EVMbench scored GPT-5.3-Codex at 72.2 per cent in exploit mode against 31.9 per cent for GPT-5, and reported weaker performance on detect and patch, with agents in detect mode sometimes halting after a single finding rather than auditing exhaustively, and patch mode struggling to remove a vulnerability while preserving intended behaviour.³ Read in order, the capability ranking runs exploit, then detect, then patch, and generation sits below all three because no benchmark grades it against an adversary at all.

The awkward part of that figure belongs here rather than in a footnote. OpenZeppelin reviewed the EVMbench dataset and reported methodological flaws and invalid vulnerability classifications, including at least four issues labelled high severity that are not exploitable in practice.⁶ The 72.2 per cent is therefore an upper bound on a dataset with known labelling errors, published 12 days after the benchmark itself. Anyone quoting the number without that qualification is quoting a headline.

What a decisive experiment requires

No published benchmark settles the question for Move.

The existing work measures function-level generation, curated vulnerability sets, or translation tasks, and each paper says so.¹²³ A test that would actually settle whether a model can write a Move module fit to hold value needs five properties, and current benchmarks hold two or three of them at a time.

FIVE PROPERTIES A CONCLUSIVE MOVE BENCHMARK NEEDS. The third row is the one that voids most published results, because a model that saw the reference implementation during training is being tested on memory.

The cost, and who pays it

The saving is real and it moves rather than disappears.

Drafting with a model compresses the hours spent producing a first module. Those hours reappear in review, and they reappear at a higher rate, because reviewing unfamiliar code that looks correct is slower per line than writing it. Retrieval-augmented generation improved functional correctness by up to 45 per cent in Salzano's setup, which changes the size of the review burden and not its existence.¹ The position that follows is narrow. Use the model to draft, scaffold and translate, and treat every line as unreviewed until an independent test says otherwise.

The trigger that would reverse that position is a benchmark holding all five rows of the table above, on Move, with released code. As of 29 Aug 2026 there is none to point at.

What this does not cover

Sources

Footnotes

1. SALZANO, SCALABRINO, OLIVETO AND PARESCHI, BEYOND CODE SIMILARITY. arxiv.org

2. CIRCLE AND OPENZEPPELIN, CHAINBENCH. circle.com

3. OPENAI AND PARADIGM, INTRODUCING EVMBENCH. openai.com

4. GUIDING LLM-BASED SMART CONTRACT GENERATION WITH FINITE STATE MACHINE. arxiv.org

5. SMART CONTRACT LANGUAGES, A COMPARATIVE ANALYSIS. arxiv.org

6. OPENZEPPELIN, WE AUDITED OPENAI'S EVMBENCH. openzeppelin.com