PL·BENCH — agent language rig
● RUN COMPLETEGPT-5.6 SOL1,878 ZSTD RUNS LOGGED
Rig 01 · 25 languages · 2 benchmarks · 40 runs each
Dan Luu pointed a coding agent at the same two problems in twenty-five languages and logged every run. The static-versus-dynamic argument that has burned a decade of threads shows up in the telemetry as this:
Static camp mean minus dynamic camp mean
—cases
of 34
Three quarters of one test case, on a rig where a single language swings 14.5 cases between its own best and worst run. The camps are not different. The runs are just loud.
Control deck · set the rig
every figure below follows these two dials
↓ Run 01 · the camp that didn't matter
Twelve statically typed languages, eleven dynamic ones, two assembly targets. Same brief every time: build a zstd decoder that passes 34 hidden test cases. Forty runs a language, cost and wall-clock logged per run.
The static camp passed — cases. The dynamic camp passed —. Dynamic languages also cost more per run, which is the opposite of the folklore. Neither gap survives contact with the noise.
Static camp · 12 languages
—
cases of 34, at $— a run
Dynamic camp · 11 languages
—
cases of 34, at $— a run — dearer, not cheaper
The entire argument
—
cases between the camps — one twentieth of a single run's own spread
Fixed at zstd medium · these three do not follow the benchmark dial
Head to head · your picks only
ranked against the best of them
↓ Run 02 · the folklore audit
The token-efficiency claims that circulated all year were counted on Rosetta Code puzzles — twenty-line problems where terse syntax wins by construction. This rig asks for a working zstd decoder and a Pandoc feature, then runs the hidden tests.
Under that load the ordering inverts. Terse array languages come last. PHP, the punchline language, places third on Pandoc. Only one piece of folklore survives, and it is the boring one: don't write your agent's code in assembly.
Fig.03 · claim register
both benchmarks, full effort · fixed, not dial-driven
| Claim you have probably seen | Verdict | What these runs show | Source |
|---|---|---|---|
| Dynamic languages are cheaper to run | Not supported | Across all 25 languages the dynamic camp cost slightly more per run ($5.07 vs $4.63) and scored within one case of the static camp. | Google AI summary |
| Clojure needs 2.6× fewer tokens than C | Measured something else | That counted tokens in short Rosetta Code puzzles. Asked to build a real zstd decoder, C cost less ($3.80 vs $5.26 a run) and got more right (29.3 vs 23.0 of 34 cases). | Alderson |
| Array languages like J are the most token-efficient | Measured something else | Same puzzle-sized measurement. On zstd, J came last of 25 for correctness and second-costliest per run. | Alderson follow-up |
| Languages with lots of bad training code, like PHP, do worse | Not supported | PHP came 3rd of 25 on Pandoc and sat mid-pack on zstd. | folklore |
| Pick a powerful language like Haskell, since rewriting is cheap now | Not supported | Haskell came 5th of 25 on zstd but only 14th on Pandoc — no better than average overall. | folklore |
| Stick to a popular, mainstream language | Weak support | Only assembly clearly suffers: the two assembly targets passed 11% of Pandoc's tests, against 19–31% for the other 23 languages. | folklore |
↓ Run 03 · two rigs, two leaderboards
Two problems, two leaderboards. If a language were genuinely better for agents, its rank would travel. Python, JavaScript and C# do travel — they finish near the top of both. So does the floor: assembly is last on both, and nothing rescues it.
The middle does not travel at all. C++ finishes 2nd on the zstd decoder and 17th on Pandoc. Rust does the reverse, 13th then 2nd. Cost is the stable signal here: a language that runs expensive on one problem runs expensive on the other.
Every rank on this page rests on an average. Open the averages up and the same language, given the identical brief forty times, lands anywhere across a band of — cases. Python's own runs cost between $1.36 and $8.71.
All twenty-five language averages fit inside — cases. The spread within one language is about twice the spread across all of them — so a single run tells you nothing, and neighbouring ranks are indistinguishable.
Within one language
—
cases between its own worst and best run
Across all 25 languages
—
cases between the best and worst language average
Runs behind this figure
—
zstd runs, at the effort the control deck selects
Pandoc publishes no per-run figures, so this chapter always reads zstd — ultra when the control deck is set to Zstd (Ultra), medium otherwise. Blank cells mean the case never failed. Clojure is the one language with a failure signature of its own — it misses a set nothing else misses. Swift and both assembly targets ran twenty times, not forty.
↓ Run 05 · the lever that actually moves
The rig ran one more experiment. Instead of a single expensive run, let the cheap model loop eight times over the same problem — keeping its context between rounds, or wiping it each time. This is the strategy that is supposed to make the expensive model unnecessary.
It is not. One high-effort run beat the eight-round loop in — of — languages, and did it while spending $— less per language. If you do loop, keep the context: wiping it lost — times out of —.
Fig.08 · spend against result
averaged over all 25 languages · zstd
Effort buys
+—
cases of 34, one medium run to one ultra run
Language buys
—
cases, static camp over dynamic — five times smaller
The loop's record
—/—
languages where one ultra run beat eight cheap ones
Pick the language your team already knows. Spend the argument on the model.
One honest caveat from the source's own analysis: three cheap runs, with tests to pick the winner, reach about 30.5 of 34 cases for roughly 70 percent of a single expensive run's cost. That is the best value on the board — if you have the tests. The expensive run still scores higher in 18 of the 25 languages.
Every language on both benchmarks. The last five columns come from the loop experiment: the cheap model run eight times on the same problem, keeping or wiping its context. The final column divides the ultra run's cases-per-dollar by the loop's — above 1.00 means the expensive run was the better buy.
Fig.09 · full run table
click a header to sort · scroll sideways for more