AI that makes AI inference faster

+24–32% on AMD MI355X, +9% on NVIDIA B200, with code

November 2, 2026

TLDR:

• We pointed Elise, the optimization system behind our recursive self-improvement (RSI) work, at the software that serves AI models. With her help, inference engines got faster on three models, two engines, and both AMD and NVIDIA hardware: Kimi K3 by 24–32% on eight AMD MI355X GPUs, Nemotron 3 Ultra by 9% on four NVIDIA B200 GPUs (and 12–22% faster per user), and GPT-OSS-120B by 13–64% on one AMD MI300X, each against its own stock setup.
• Why that matters: AI companies, frontier labs included, make most of their money from inference.¹¹ Anthropic’s IPO filing (Form S-1, October 2026) says 80% of its revenue in the first half of 2026 came from usage “metered in tokens”, with subscriptions making up the rest (p. 139). It also says its inference optimizations let it “generate greater revenue from our compute base”, because “customer demand exceeds available compute” (p. 136). Early this year, a leading open-model provider was running at around 10% gross margins.²² Dylan Patel of SemiAnalysis, on the Latent Space podcast in October At that margin, a 10% speedup nearly doubles profit.
• On GPT-OSS-120B, Elise helped AMD’s MI300X overtake NVIDIA’s H200 per GPU, against an untuned H200 running stock vLLM 0.20.2, the release we started from in May: at each chip’s best load, the MI300X now serves 17% more tokens per second. This is significant because on paper the MI300X has more memory, more bandwidth and more compute, yet at low and moderate load it had served the model more slowly. On today’s vLLM (0.32), a stock H200 is level with our MI300X engine at 8–20 users and ahead at 1–4.
• We took on the challenge to reproduce our results on a tight timeline, with an engine, GPU and model we had never worked with (TokenSpeed, MI355X, and Kimi K3, a 2.8-trillion-parameter model). Two of our TokenSpeed changes are already merged upstream: sharing KV-cache reads across the draft positions verified together in one speculative step, and a faster MoE output reduction.
• Not every change is an improvement. In one earlier run, a kernel that scored 21% faster in its own test made the whole engine 1% slower, and our end-to-end check threw it out. Every change we kept matched reference outputs within numerical tolerances and was measured in the full engine.
• Want a feel for how hard this is? Run your own small inference business in our simulator: rent GPUs, serve models, and try not to go broke with Elise’s help.

1. Why start with inference?

Inference is serving an already-trained large language model (LLM) to the people using it. When you ask your favorite AI whether people’s forearms are generally as long as their feet (they are!), its answer comes out as tokens (word chunks). How quickly those tokens come out, at the same quality, is what inference is about. The faster they come out, the faster an AI can think and act.

MakerMaker works on recursive self-improvement (RSI), so why start with inference? RSI requires AI models to do things, and AI models work by producing tokens. Every hypothesis an AI proposes, every line of code it writes and every result it reads comes out of an inference engine. If you can make inference engines faster, any AI running on them gets more thinking out of the same GPUs.

That’s one of our loops: the more tokens per second an AI system gets, the more experiments it can run, hypotheses it can test and results it can check each day, and the sooner it can improve the whole stack, engines included. That’s true even if it never gets any smarter. Making it smarter is, of course, just as much a part of RSI, though in this report we will focus on speed ;)

      ┌──────────── a faster inference engine ────────────┐
      │                                                   ▼
 an AI system                                  more tokens per second
 improves the engine                                      │
      ▲                                                   ▼
      │                                     more experiments, hypotheses
      │                                          and checks each day
      │                                                   │
      └──────── finds the next improvement sooner ◂───────┘

Inference also has the rare and beautiful property of being hard to argue with because throughput is easier to verify than benchmark scores. But it can still be gamed (the blooper reel in section 10 has examples), so every supposed improvement has to pass correctness checks within defined numerical tolerances. Inference engines support hundreds of models, with new ones shipping every week.

2. Why it matters beyond inference

Making inference engines faster does two things at once:

1. Everyone serving these models gets faster inference: more tokens out of the same GPUs.
2. Any AI that runs on these engines, systems like Elise included, gets faster when the engines do. An AI that improves the software AI runs on is a small, concrete piece of recursive self-improvement.

History has many examples of tools improving tools,³³ Computers help design better chips; lathes make more precise lathes. but in all of them, the human brain did most of the improving and was the “gap” in the improvement loop. Elise shows progress toward closing that gap: she both does the improving and is the kind of system being improved.

There’s a ceiling, too. A chip can only do so much arithmetic and move so much memory per second, and each gain gets harder as an engine gets closer to that limit.

3. Inference components

Imagine you want to serve an open-weight model on your own. Once you have downloaded the weights, you need two things:

• Hardware: the chips AI models run on, such as NVIDIA B300 or AMD MI355X GPUs, Google TPUs, Cerebras chips, or Apple’s M-series.
• An inference engine: the software that produces the tokens. It schedules and batches different users’ requests, and runs the model’s computations on the chip (the kernels). Examples include vLLM, SGLang, TokenSpeed, and DeepSeek’s DeepInfer.

The challenge is that there are a whole lot of combinations of engines, AI models and hardware to search through when optimizing how tokens are delivered. Harder still, the best combination can depend on how the AI is used: a setup that quickly answers a one-line question in a chat app may not be optimized for a multi-turn coding session, because different kinds of traffic, which we call workload shapes, can push the engine, model and hardware in different ways. There are far too many combinations to try them all, and good settings are rare.

4. Why is this hard?

Optimizing an inference stack is hard, and it’s one of the most closely guarded secrets in the AI world, because it lets the billions of dollars invested in AI infrastructure do much more.⁴⁴ In frontier labs these are called “compute multipliers”: improvements worth as much as having more compute (Dario Amodei, “On DeepSeek and Export Controls”, January 2025). Until recently, optimizing inference was the job of a small group of human experts. These experts are in high demand: every improvement they find can make billions of dollars of hardware worth more. But there are few of them, and their backlog grows every week as new models, engines and hardware are released.

These experts don’t have time to optimize every combination. Instead, they write a general engine that works well enough for most cases, and hand-tune only the few combinations worth the effort. The best settings and kernels for one combination can need retuning for another, and can even make it slower.

A combination covers everything from the model, chip and engine version down to the kernels and dozens of engine settings. Some of these choices come packaged as recipes: published starting setups for running one model on one chip. A vLLM recipe typically names the minimum engine version, the numeric precision, how to split the model across GPUs, the attention backend, the KV-cache page size, the speculative-decoding setup and a few environment flags. A recipe doesn’t change the engine’s code: it runs the kernels the engine already ships with.

GPT-OSS-120B
Kimi K3
Nemotron 3 Ultra
DeepSeek-V4-Pro
DeepSeek-V3.2
Qwen3.5-397B
Qwen3-Coder-480B
Kimi K2.6
GLM-5.3
MiniMax-M2.7
Llama 4 Scout
Gemma 4 31B

NVIDIA                             AMD                  Google TPU
  H100   H200   B200   B300  GB200  MI300X MI325X MI355X  v6e     v7
   ■      ■      ■      ·      ·      ■      ■      ■      ·      ■
   ·      ■      ■      ■      ■      ·      ·      ■      ·      ·
   ·      ·      ■      ·      ·      ·      ·      ·      ·      ·
   ·      ■      ■      ■      ■      ·      ·      ■      ·      ·
   ■      ■      ·      ·      ·      ·      ·      ·      ·      ·
   ·      ■      ·      ·      ■      ■      ■      ■      ·      ·
   ·      ■      ·      ·      ·      ■      ■      ■      ·      ■
   ·      ■      ·      ■      ■      ■      ■      ■      ·      ·
   ·      ■      ·      ■      ·      ·      ·      ·      ·      ·
   ■      ■      ·      ·      ·      ■      ■      ■      ·      ·
   ■      ·      ■      ·      ■      ■      ■      ■      ·      ·
   ■      ·      ·      ·      ·      ■      ■      ■      ■      ■

■ verified recipe   · none
all 133 text models: 339 of 1,330 model–chip pairs (25%)

5. Try it for yourself in this immersive simulator

Want a feel for how many choices are hiding in this problem? Try running a (fun) simplified inference service yourself.

Fair warning: it's harder than it looks. (If you do well, we're hiring!)

6. Our optimization engine: Elise

Elise⁵⁵ Named after Elise “Lise” Meitner, who co-discovered nuclear fission; the Nobel Prize for the discovery went to her colleague Otto Hahn alone. is our AI optimization system: given a problem, she proposes changes, implements them, tests them, and keeps whatever works.

Elise started by tuning settings (backends, KV-cache size, speculative decoding, and so on), but soon moved on to changing the inference engine’s source code.

Under the hood, Elise works from three things: a problem description, an optional starting solution, and an evaluator that measures how well each new solution performs. For GPT-OSS-120B on the MI300X, they were:

• Problem: speed up GPT-OSS-120B inference on one AMD MI300X.
• Starting solution: vLLM’s source code.
• Evaluator: component tests that check outputs against reference results within numerical tolerances, and time them on the GPU.

The engines that came out of this served more tokens per second on all three open-weight models in this report (section 9 has the details). Elise helped propose, implement and test the changes.

  problem description ──┐
  starting solution ────┼──▸ Elise proposes and writes a change ◂──┐
  evaluator ────────────┘                  │                       │
                                           ▼                       │
                         checks: is it correct? is it faster?      │
                                           │                       │
                              yes ◂────────┴────────▸ no ──────────┘
                              │
                              ▼
                         kept, and built on

7. Big Feedback Loop

What matters is not only knowing how to improve part of a complex system, but also knowing which part to improve. So we built a pipeline around Elise, the Big Feedback Loop (BFL): it profiles the whole system to find where the time goes, asks Elise to help cut that time, and folds the improvements back in.

The most visible bottleneck is often a symptom rather than the cause. When you’re stuck in traffic, you might think more lanes would help, but the real cause may be a badly timed traffic light two blocks back.

Here is a real case. In one BFL round on the MI300X, profiling showed the mixture-of-experts step taking about 80% of GPU kernel time. Part of that time went to generic routing bookkeeping that GPT-OSS never uses. Skipping it, sorting small decode batches in a single kernel and fusing a scatter step gave about 10% more throughput than the routing fix alone.

BFL also benchmarks the whole engine, end to end, because improvements to one part often don’t carry over to the whole, and it checks that output quality and correctness haven’t dropped while throughput improves.

Bringing Elise and BFL together gives us one system: provide a model and specify your hardware, and we obtain a faster inference engine.

The rest of this report is for readers who want the details: how we measured, and the results for each model, with code.

8. How we measured

Every chart in this report uses AgentX, a benchmark from InferenceX that replays the request lengths, timing and prefix reuse of real agentic coding sessions, filled with synthetic text. We chose it because its benchmark tools are public, so anyone can rerun our curves with the reproduction guides in section 9, and because, unlike a single fixed prompt, it lets an engine reuse cached prefixes the way it would with real users.

Because the text is synthetic, a speculative-decoding draft model can’t predict it the way it would predict real text, so each run fixes how many draft tokens are accepted: for Kimi K3, 3.75 at one user (InferenceX’s value) and 3.0 at 4–16 users; for GPT-OSS-120B (2.31) and Nemotron 3 Ultra (3.69), values we measured the same way. These charts measure engine speed, not drafter quality.

Context windows limit which sessions can run. GPT-OSS-120B’s 131,072-token window fits 82 of AgentX’s 393 sessions, Nemotron 3 Ultra’s 262,144-token window fits 220, and Kimi K3 runs all 393. The GPT-OSS sessions that fit are the shorter ones: their median request is about 67,000 input tokens, against 142,000 for the full set.

Each chart shows a trade-off. The vertical axis is output tokens per second per GPU; the horizontal axis is the decode speed that 90% of requests meet or beat (P90 interactivity, in tokens per second per user, excluding time to first token). Each point is one concurrency level, a fixed number of users served at once: more users means more total output, but a slower experience for each of them. Further up and to the right is better.

Each point is a single one-hour run after warmup. Repeated runs of the same GPT-OSS setup differ by up to about 3.5%, so we don’t read much into gains smaller than that. GPT-OSS stops at 20 users because beyond that all three setups saturate. A few requests failed in the stock GPT-OSS runs (17 of 6,514 on the H200 and 17 of 6,151 on the stock MI300X, both under the benchmark’s 1% limit), and none in the optimized run. Each chart’s caption names the software versions, and each section links its reproduction guide.

9. Results

9.1 GPT-OSS-120B: closing the gap between AMD MI300X and NVIDIA H200

For the first experiment, we chose GPT-OSS-120B because it’s a good model that fits on a single H200 or a single MI300X, so the comparison is one GPU against one GPU. To set our control, we ran stock vLLM 0.20.2, the current release when we started doing inference engine optimization in May, on one MI300X and on one H200, each with vLLM’s recommended serving configuration for its chip (the reproduction guide has every flag) and the same speculative decoding: EAGLE-3 with three draft tokens. At low load the H200 was up to 40% faster in output tokens per second per GPU, despite the MI300X having more memory, higher memory bandwidth, and higher peak FLOPs.

From this baseline, we applied Elise to the problem. With Elise’s help, the same MI300X went from trailing the H200 to beating it per GPU: at each chip’s best load, it produced 17% more output tokens per second (334 against 286), and it was ahead by more than 4% at five of the seven loads we measured; at 4 and 8 users the two are within run-to-run noise. Against stock vLLM on the very same chip, the gain is 13% to 64%. This is the same GPU and the same vLLM version with only the code having changed.

Both baselines are vLLM 0.20.2 from May, and the world of inference optimization is moving quickly. A GPT-OSS fix for NVIDIA GPUs (vLLM #43135, released in 0.22.0 in late May) makes a stock H200 faster, and vLLM 0.30.0 changed which MoE kernels the MI300X uses by default for GPT-OSS.

Two of the changes Elise found on the MI300X in May and June, a sort-based routing shortcut and smaller decode tiles, resemble changes that landed upstream about four months later (vLLM #58897 and AITER #6688, in October). Against a stock H200 on today’s vLLM 0.32.0, our MI300X engine, built on May’s release, is level at 8–20 users and behind at 1–4.

The GPT-OSS optimization began with an implementation produced by earlier BFL campaigns. Elise could modify the implementation, including GPU kernels, supporting operations, and the conditions that select which implementation runs. The evaluator compared candidate outputs against reference results within defined numerical tolerances and measured execution time on the target GPU. Performance scores combined relative speedups across the test cases using geometric means. The test cases, correctness requirements, and scoring rules remained fixed during search. AgentX requests were excluded from these component test suites.

After search, the retained candidates were combined and refined, with Elise’s help, into an integrated implementation. Different candidates could be useful for different input conditions, so the integration also decides when each implementation runs, and the combination was checked against the evaluators. The optimization explanations below describe those decisions and their corresponding source changes.

We evaluated the resulting engine with AgentX to measure how the component improvements translated into serving performance. These measurements exercise the complete engine, including scheduling, batching, cache management, and communication between the client and server. Each optimized GPT-OSS point represents one hour of measurement after AgentX warmup.

9.2 Multi-GPU: Nemotron 3 Ultra on 4× B200

Nemotron 3 Ultra let us test whether Elise could help improve a 550-billion-parameter NVIDIA model running across four B200 GPUs. We chose this model because it was made by NVIDIA and optimized for their hardware.⁶⁶ NVIDIA’s model card says “Our AI models are designed and optimized to run on NVIDIA GPU-accelerated systems” (Nemotron 3 Ultra model card), and NVIDIA’s serving team describes kernels tuned to “map the hybrid architecture efficiently to NVIDIA Blackwell GPUs” (NVIDIA Technical Blog, September 2026).

We followed the same optimization workflow described for GPT-OSS: computations to be optimized were selected and correctness and performance tests defined, then Elise helped develop candidate implementations, combine complementary improvements, and refine the result. We evaluated the integrated engine across all four GPUs using AgentX. Each point represents one hour of measurement after AgentX warmup.

The optimized engine delivered higher p90 interactivity at every tested concurrency, by 12–22%. It also delivered higher output throughput everywhere, by more than 3% at five of the eight loads; the smaller gains are inside run-to-run noise. At 48 simultaneous users, where both configurations reached their highest measured throughput, the optimized engine raised output throughput by 9%, that is, 9% more output from the same four GPUs, and p90 interactivity by 17%.

One of the changes raises the batch-size limit for running the shared expert alongside the routed experts from 256 to 512 tokens, which stock vLLM can also do with a setting. So we reran the baseline with that setting: stock throughput moved by under 1% (+0.4% at 48 users), so the gain is not from this setting.

9.3 A second engine: Kimi K3 on TokenSpeed

With AMD’s help, we took on the challenge to run our system on an engine, GPUs and a model we hadn’t worked with: TokenSpeed, eight MI355X GPUs, and Kimi K3, a 2.8-trillion-parameter model.

Against TokenSpeed commit 06a63156 (August 24, 2026), with the same support patches in both engines, the optimized engine produced 24–32% more output tokens per second per GPU at every load we measured (1–16 users).

Six optimization tasks were selected: prompt processing and speculative-token verification for each of K3's two attention mechanisms, mixture-of-experts computation, and the mixing of representations saved across layers. These tasks included the surrounding projections, metadata preparation, and state updates, giving Elise room to speed up complete operations as well as individual GPU kernels.

Correctness and performance tests were defined for each task. Each component test ran on one MI355X using the weight slice that GPU would hold in the eight-GPU deployment. The tests varied prompt chunks, history lengths, batch shapes, and speculative-verification widths. They compared outputs against fixed references within defined numerical tolerances and checked that later calls correctly consumed the implementation's own cache or recurrent state. Workload cases and scoring rules remained fixed during search.

Elise helped develop implementations that fuse preparation steps, reduce intermediate memory traffic, and choose different kernels for different batch sizes. After search, the candidates were combined and refined into the optimized TokenSpeed engine. The explanations below show the resulting optimizations and their implementation.

10. The blooper reel

It wouldn’t be fair to write this report without sharing some of the issues we hit on the road.

1. Don’t trust the recipe blindly. vLLM’s GPT-OSS recipe suggests seven speculative draft tokens in its NVIDIA EAGLE-3 configurations. On our test traffic, three worked better, so both chips use three. The right number can depend on the workload.
2. Faster kernel ≠ faster engine. One kernel Elise made 33% faster changed end-to-end throughput by 0.2%: the bottleneck simply moved. Another file had a literal fast_mode flag. It was fast, but only in isolation, as it doubled the kernel’s speed by skipping a compilation step that’s only safe for some input shapes. Our isolated test happened to use those shapes; real traffic didn’t, and the full engine slowed badly.
3. Tune for one traffic shape, and the gains can shrink on others. A Nemotron change that gained 14% on our standard test traffic gained 8% on input-heavy traffic and 4% on output-heavy traffic.
4. If it looks too good, check the scorer. Has our agent ever cached the whole dataset to respond faster? Yes. Has it ever “borrowed” other GPUs? Yes (our gates caught it). Has it ever made throughput look better by rejecting most requests, so the average over the few that finished soared? Yes. It took a while to get it right.

11. Inference is just the beginning

This report shows the first turn of the loop: an AI system helped make inference engines faster, the same kind of engines our AI systems run on. Inference is just the beginning.

If you want to help AI recursively improve itself, we're hiring MakerMakerMakers!

[↑ top]