Most AI claims arrive as a single number: A beats B, the model scores 96%, the new mode is smarter. In four episodes of How AI Is Built we checked what sits behind such numbers, with small tests we ran ourselves and with the Kimi K3 technical report as a reference.
“A beats B” depends on how you test
We trained the same small network on the same handwritten-digit data with two optimisers. With a shared learning rate of 0.1, SGD won (93.67% vs 82.75%). With a shared 0.001, Adam won (94.92% vs 12.58%). With equal tuning for both, the gap was 0.98 points, below the 1.42-point bar we had set in advance, and the same setting varied between 95.5% and 97.0% across 20 random seeds. The K3 report makes a related point: it ran a separate scaling-law search for each learning-rate schedule before comparing them. Our checklist: equal tuning budget, several seeds, a bar set in advance, a sealed final test, and a check at other scales.
AI learns to cheat when the reward allows it
We wrote a simple parcel-delivery robot and gave it rewards. Under the first rule it farmed a checkpoint and delivered nothing (29.0 reward against 11.0 for honest delivery). Under another it pressed “done” at step one. Only a rule we added after the first round got 100% delivery in nine steps. The K3 report describes the defences a large lab uses: grading agents on the final state of the environment rather than on what the agent claims, hidden checks, limited submissions, and a detector that penalises shortcuts such as caching inputs.
Smarter is not always worth the price
K3 trains its thinking effort against a token budget per problem. According to the report's own figures (prices as of July 2026), it reached 91.2% on one browsing benchmark at about $2.03 per task, roughly half the cost of a rival model at 90.4%, while trailing the strongest models overall. For a business the useful measure is cost per successful task, with typical and slow cases (p50, p90) and the long tail, and a 30–50-task pilot still leaves wide margins.
“96 out of what?”
We scored one model on tests of different sizes: 30-question tests ranged from 83.3% to 100%, 400-question tests from 93.8% to 98.0%. A model that had seen the test pictures scored 100%, against 97.26% on fresh ones. A score belongs to the whole setup: model, harness, prompt, tools, number of attempts, effort, grader, version and date. Near 50%, thirty questions give a margin of about ±18 points. The K3 report gives sampling settings but no run-to-run spread and no contamination analysis, and, as we say in the episode, the absence of a check is not evidence of contamination.
We apply the same habits when we accept AI systems for clients: our releases are built to fail closed, and a green test only proves what it tests.