Company
A number needs a witness
When an industry sells on a number, someone outside the sale ends up producing it. For AI models, that is the company we are building.

It has happened before
Other industries reached the point AI has reached now. The answer was the same each time, and it was never a better test.
1894
Underwriters Laboratories
William Henry Merrill was sent to inspect the wiring of the 1893 World’s Fair in Chicago. He stayed, and opened a laboratory that tested electrical goods for the insurers who paid when they burned. The maker of a lamp stopped being the one who said it was safe.
The laboratory’s history1988
The Transaction Processing Performance Council
Database vendors had spent three years publishing figures from a test each ran its own way. Eight companies formed a council. It left the test alone and changed the run: every result came with a full disclosure report, a price beside the figure and, from 1993, an auditor’s approval.
The council’s own history1988
SPEC
Workstation makers founded it the same year, on a line that still holds: an ounce of honest data was worth more than a pound of marketing hype.
SPEC, on its founding2015
Volkswagen
The other way it ends. The cars’ software turned full emissions controls on only during the test, and on the road they emitted up to forty times the limit. A number made where nobody else can look is worth what its maker says it is.
The EPA’s notice
AI is there now
The tests are public and good. The runs are not accountable. Three measurements, none of them ours.
6points
How far the sandbox’s resource limits alone moved scores on Terminal-Bench 2.0. At strict limits, 5.8% of trials ended in an infrastructure error, which most harnesses count against the model.
Anthropic, February 202637%at one host
One open model scored 36.7% served by one host and 93.3% served by another, on the same test in the same week. A score that does not say who served the model describes nothing you can buy.
Artificial Analysis, August 202516,500dollars in tokens
The dearest single run on the current Terminal-Bench leaderboard, by model. Each of its 66 tasks is a machine an agent works in for up to eight hours, and the leaderboard asks for five trials of each.
The Terminal-Bench leaderboard
The run, made accountable
The test is somebody else’s and stays as its authors wrote it. What we add is everything around it. Six rules, each one the product enforces.
Everything is disclosed
A run ends in a record: the benchmark and its version, every setting and every departure from the official ones, the model, the host that served it, and how many times the benchmark was tried on it.
An error is not a failure
A sandbox that would not start is not the model’s fault. Errors are left out of the score, counted beside it by kind, and can be tried again.
A score has an interval
It is taken over tasks, not attempts, so a reader sees how far the number can be trusted before they repeat it.
We are not paid by the thing measured
Your own key pays your provider. We take no margin on tokens, train no model and rank nobody, so we have no stake in the answer.
The tests stay out of training data
We never show the content of a published benchmark’s tasks, on this site or in a record. Their authors ask that of everyone.
Nothing is claimed that cannot be shown
A run that never happened says Demonstration beside it. We name no customer we do not have.
Where it stands
Said the way we ask a result to be said: with what was left out.
True today
- The product runs end to end: a benchmark launched from the dashboard, a sandbox for every trial, every model call recorded, and a score with its interval.
- Every finished run has a record its owner can publish at a public address. The reader’s browser checks its digest, and a link to it unfolds into the score with what it rests on.
- 11 of the benchmarks on the board can be launched today, Terminal-Bench 4.1 and Terminal-Bench-Science 0.1 among them, each under its own rules.
- Any model behind any OpenAI-compatible base URL, on the customer’s own key. Your own tasks run beside the published ones.
- The service is in pilot. Access is by request, and a person answers.
Not yet
- A full run of a published benchmark on a model we may name. The site plays a demonstration in its place and says so.
- Customers we can name.
- Runs of OSWorld 2.1, FrontierCode 2.0 and 13 more on the board. Each is made ready to launch before access opens.
- Card payments. Credits are granted, and plans arranged, by a person.
- The company
- Berkway, a Wyoming company
- Address
- 1309 Coffeen Avenue, Sheridan, WY 82801, United States
- Stage
- In pilot. Access is by request
- Write to
- hello@berkway.com