My first modded-nanogpt attempt: 34.9 seconds?

The speedrun goes back to Andrej Karpathy's nanoGPT and his llm.c reproduction of GPT-2. Karpathy is now at Anthropic. Keller Jordan, now at OpenAI, turned it into modded-nanogpt, a community speedrun. It trains a GPT-2-scale language model on FineWeb until it reaches a validation loss of 3.28, as fast as possible on 8 NVIDIA H100 GPUs. Since 2024, many contributors have taken the record from 45 minutes (the llm.c baseline) down to under a minute. The current record (#92, by @DevenPzak) stands at 39.9 seconds.

This weekend I made my first modded-nanogpt attempt: 34.9 seconds on a rented 8×H100 node from runpod.io, now in review as PR #391. Until the maintainers verify it on their own hardware, it's only an attempt. It's the first one I've submitted. I tried once before, but didn't beat the record, so I never submitted it. As you'll see below, most of the credit for this one belongs to others.

Standing on other people's work

As a first-timer, I didn't bring a new modeling idea. The attempt combines two open pull requests by people who did:

• #379, CPLM, by Nathan Godey. The model's next-token distribution mixes its normal prediction with a "pointer" that can copy tokens that appeared earlier in the same document.
• #375, by Daniel Monroe. The model's large n-gram lookup table treats tokens like " The", "THE" and " the" as the same.

The repo encourages exactly this ("incorporating open PRs into a new record is strongly encouraged"), so a combination can help both PRs get validated and merged. The two merged without conflicts. My part was modest: combining them, re-tuning the number of training steps (955 instead of 978), and a careful certification. Mostly, it was learning how this community measures and verifies records.

The numbers

All runs were on one 8×H100 node, interleaved, with seeds fixed in advance:

The rules require showing, at p < 0.01, that the mean loss is at most 3.28. Here p ≈ 10⁻⁷, and all 18 runs were below 3.28. On the same machine, the attempt is 5.3 seconds (13%) faster than the current record's code. Every run counts. I planned 14 runs and extended to 18, to match the current record's run count, after the first four and before seeing the rest. That decision, all the logs, and a script that recomputes the statistics from them are in the PR.

One small finding of my own

While timing the runs, I noticed something about the stopwatch itself. The official timer runs on one of the 8 GPU processes. It starts as soon as that process has finished its untimed setup. If another process is still finishing its setup, the first synchronization makes the timer wait for it, and that wait gets counted.

On our node this was usually zero, but once it added 0.83 seconds to a run. A three-line fix makes all processes start the clock together. I've proposed it separately as PR #390, because it changes how time is measured, not how the model trains. The record attempt doesn't include it and doesn't need it: 34.86 s with or without.

What's next

• The retrieval-hybrid track. A different branch of the speedrun mixes the neural network with exact counts from the training data and reaches ~9.6 seconds (#380 by Deven). We're testing a combination of #380 with Daniel Monroe's #381 there. A first certification of that combination is running as I write this, and I'll update this post with the result.
• A systems idea for the 34 s path. Profiling showed that in late training, every fourth step takes about 20 % longer. A periodic update of the model's large n-gram table queues its GPU-to-GPU traffic behind everything else. Rearranging that without changing the training math could be worth a few tenths of a second.

Thanks to Keller Jordan for the speedrun, and to Nathan Godey, Daniel Monroe and Deven, whose work this first attempt builds on. If it doesn't hold up in review, I'll have learned a lot anyway.