Building a large model is less like one invention and more like raising a student: decide the job, gather the reading, find the classroom, design the brain, teach broadly, then teach manners, test, release safely and keep serving. Our series How AI Is Built walks through these nine stations using one public case study: the Kimi K3 technical report and open inference code from Moonshot AI. (Independent educational content, not affiliated with Moonshot AI.)
What the report tells us
- Per the report, K3 has about 2.8 trillion parameters in total, of which roughly 104 billion (about 3.7%) are active for each token.
- It is a mixture-of-experts model: 896 routed experts, 16 picked per token, plus 2 shared experts; 92 of its 93 layers are expert layers.
- Context grows from 8K to 64K tokens in pre-training and from 256K to 1M in the cooldown.
- Nine teacher experts (three domains, each at low, high and max effort) are distilled into one model, and thinking is always on.
Just as important is what the report does not give: the full data recipe or the data itself, the total number of pre-training tokens, and the full size or cost of the main run. And because these are open weights, a model cannot be recalled once it has been downloaded.
Three questions that explain any model
Any model design answers three questions: how it looks back over the text (sequence), how its layers talk to each other (depth), and how it shares the work (width, or experts). With full attention, doubling the length costs roughly four times as much. K3 mixes a fixed-size memory (Kimi Delta Attention) with conventional attention layers. As a course illustration, keeping full notes in every layer at a million tokens would need around 4.8 TB of memory, against roughly 26–29 GB for K3's layout.
The data nobody shows you
The report names its text domains (web, code, maths, knowledge) and its kinds of vision data, but not sources, sizes, dates, licences or the order of its filters. Two practical lessons carry over to any project. Removing a sample from the data does not automatically remove its influence from the trained weights. And labelling has maths: two annotators who are each 80% accurate agree only about 68% of the time, while a three-way majority vote reaches about 89.6%. A small, trusted “gold set” keeps machine labels honest.
When the chip changes the model
One of K3's smallest rules shows how hardware reaches back into the maths. The report sets a floor of −5 on a decay value, so that inside a 16-token tile the numbers stay below e80, which fits in the BF16 number format. That lets every tile use fast matrix units. In our own small lab test, without the floor BF16 overflowed in some entries; with it there was no overflow and the error stayed within about 0.4% of a full-precision reference. The rule protects one step; it does not make the whole model overflow-proof.
Why this matters for business AI
Most companies will never train a frontier model, and neither do we: our team usually starts from a ready-made model and builds the system around it. But knowing the stations helps you ask better questions of any vendor: what data, what tests, what it costs to run, and what happens when the numbers get large.