Log
4 postsWhat the numbers in the table turn out to mean once you look at where they came from. Written from the catalog, not around it.
latest 5 min
Most models are beaten outright
A ranking is a line. It can tell you that one model is better than another and nothing else — not what the difference costs, not whether it was worth paying for. Put cost on a sec…
read →Earlier
7 min Spending fewer output tokens The last post established that the answer is the bill: about 98% of a typical call's cost is what the model writes back, not the prompt you sent. This one is the practical follow-… 5 min You are not paying for your prompt There is a new page on this site: paste a prompt, pick the model you use, and it tells you what the call costs and what else that money would have bought. Building it turned up th… 5 min The scaffold is half the result SWE-bench Verified has an official leaderboard, maintained by the people who built the benchmark. It runs every submission through mini-SWE-agent, their own reference harness: the…