Prime Mega MoE: Faster MoE Training, Fact-Checked

Prime Intellect, the open-AI lab and compute platform best known for decentralised training runs and its Prime-RL reinforcement-learning stack, published a research post on 29/10/2026 titled "Prime Mega MoE: Fast Fused MoE Kernels for Training". Written by Mario Sieg, it describes a set of GPU kernels that fuse the routing, communication and expert computation of a mixture-of-experts (MoE) layer into a single overlapping operation, for both the forward and backward passes.

The post leads with two numbers: up to 3.11x faster MoE kernels than the default PyTorch-based implementation, and up to 1.34x faster end-to-end training than the Prime-RL baseline, which Prime describes as 34% higher throughput. A third and larger number, 55%, appears later on a chart comparing Prime-RL with Mega MoE against TorchTitan, and that is the figure that has travelled furthest on social media and video channels.

All of these are real, sourced figures, but they answer different questions and use different baselines. Mixing them up is the easiest way to over-sell the result. In this article we separate them, reproduce the test set-up, explain why the technique works, say what it does and does not change, and map it onto the wider MoE landscape, including DeepSeek's own models, which our readers follow closely in our DeepSeek V4 Pro review.

The short verdict: this is a genuine, well-documented engineering win for a specific audience (teams training or post-training large MoE models on multi-node Hopper or Blackwell clusters), it builds openly on DeepSeek's code, and the honest headline is a roughly one-third throughput gain against a strong baseline rather than a doubling of anything.

A conventional dense transformer sends every token through every parameter of every layer. A mixture-of-experts model replaces the large feed-forward block in many layers with a bank of smaller feed-forward networks called experts, plus a small router that chooses, for each token, a handful of experts to use. Only those experts run for that token. The model can therefore hold a very large number of parameters while spending the compute of a much smaller model on each token.

Names such as Qwen3-30B-A3B and Qwen3-235B-A22B encode exactly this: roughly 30 billion total parameters with about 3 billion active per token, and roughly 235 billion total with about 22 billion active. Those two models are the test subjects in Prime Intellect's benchmarks. The same sparse design underpins many of the strongest open-weight releases of the past year, including the DeepSeek V4 family covered in our DeepSeek V4 KV cache explainer and large open models such as those in our Qwen 4 coverage.

The catch is that sparsity moves the cost somewhere else. A 235-billion-parameter model cannot fit on one GPU, so its experts are spread across many GPUs, a technique called expert parallelism. Once experts live on different devices, a token routed to expert 17 must physically travel to whichever GPU holds expert 17, be processed, and travel back. Do that for every token, in every MoE layer, in both the forward and backward pass, and communication rather than arithmetic becomes the bottleneck.

Prime Intellect's post states this directly: tokens are sent to the GPU that holds their expert, the expert computation runs, and the results are sent back, and this happens twice per layer on every step, once forward and once backward. That repeated shuffle is the target of Mega MoE.

In a standard implementation these stages are separate operations. The GPU sorts tokens by destination, hands them to the communication layer and waits; then it runs the expert matrix multiplications, hands the results back to the communication layer and waits again. Each wait is a bubble in the pipeline during which expensive silicon does nothing useful.

Mega MoE, originally part of DeepSeek's open-source DeepGEMM kernel collection, replaces that chain with one kernel invocation that handles computation and communication together, without round trips from the GPU to the CPU. In a single pass it routes tokens, runs the expert MLP maths and combines the results, overlapping compute with communication throughout. While one group of tokens is in flight, the kernel is already computing on a group that has just arrived.

A fused kernel has a second benefit that is easy to miss: intermediate tensors can be skipped. Because the kernel uses GPU-only buffers as workspace and scratch memory, the framework does not have to allocate and move as many temporary tensors, which helps memory pressure as well as speed.

The important detail is what DeepSeek did and did not ship. Its Mega MoE kernel provides only a forward implementation, optimised for Hopper and Blackwell GPUs. That is perfectly adequate for inference but useless for training, which needs gradients to flow backwards through the same layer. Prime Intellect's contribution is the fast, overlapping backward kernel that completes the pair. Combining the forward and backward kernels and replacing the standard MoE layers with calls to them is what produces the speed-ups below.

Supported data types are bfloat16, MXFP8 (Blackwell only) and blockwise FP8 (Hopper only). That hardware split matters: the numbers Prime published were measured on H200 (Hopper) GPUs in FP8, so MXFP8 on Blackwell is supported but is not what the headline charts show.

Speed claims are only as good as their set-up, and Prime Intellect documents theirs unusually clearly, which lets us judge how far the results generalise.

• Full-step benchmark: 8 x H200 per node; Qwen3-30B-A3B on 4 nodes (32 GPUs) and Qwen3-235B-A22B on 16 nodes (128 GPUs); sequence length 16,384 with one packed row per GPU and no gradient accumulation.
• Parallelism and precision: expert parallelism of 8 within a node plus FSDP across all GPUs; FP8; full activation checkpointing; fp32 master weights with AdamW; torch.compile with full graph; no offloading.
• Data: real data and real routing, 20 steps, reporting the mean of steps 6 to 20 because steps 1 to 5 are compile and warm-up.
• Per-layer benchmark: one node of 8 x H200, a single MoE layer (dispatch, experts and combine, no router), 16,384 tokens per GPU, 128 experts with top-8 routing, random routing identical for both sides, full activation checkpointing so the cost is two forward passes plus one backward pass.

Three observations follow. First, this is a throughput measurement on a fixed, fairly aggressive configuration, not a convergence study; it says how long a step takes, not whether training is better or worse. Second, the baseline is not a straw man: the Prime-RL baseline already uses torch all-to-all with DeepGEMM FP8 experts, compiled. Third, the per-layer benchmark uses random routing, which is a fair like-for-like test of the kernel but is not how a trained router distributes tokens; the full-step runs use real routing, which is why they are the more trustworthy guide to real gains.

The first chart isolates a single MoE layer. For the Qwen3-30B-A3B layer shape (hidden size 2,048, expert feed-forward width 768), the baseline forward pass takes 12.0 ms versus 3.9 ms with Mega MoE, a 3.11x speed-up. The backward pass falls from 15.4 ms to 8.2 ms (1.87x). Taking the full-activation-checkpointing cost of two forward passes plus one backward pass, the layer goes from 39.3 ms to 15.9 ms, a 2.47x speed-up.

For the larger Qwen3-235B-A22B layer shape (hidden size 4,096, expert feed-forward width 1,536) the gains are smaller: forward 20.7 ms to 9.5 ms (2.17x), backward 30.5 ms to 24.6 ms (1.24x), and 71.9 ms to 43.7 ms (1.64x) per layer with full checkpointing. The 3.11x headline is therefore the best case, forward pass only, on the smaller model's layer shape.

Notice the trend: the backward pass benefits least, and the bigger layer benefits less than the smaller one. That is what you would expect when expert matrix multiplications grow with hidden size while the communication share shrinks. The technique helps most where communication is the largest slice of the layer.

The step-time chart is the one Prime Intellect itself leads with. On Qwen3-30B-A3B across 32 H200 GPUs, a training step falls from 2.78 s to 2.08 s: 34% higher throughput, or 25% less time per step. On Qwen3-235B-A22B across 128 H200 GPUs it falls from 12.72 s to 9.65 s: 32% higher throughput, or 24% less time per step.

Be careful with the language. A 34% throughput increase is not a 34% reduction in time: the same work finishes in about 75% of the time. Prime Intellect prints both framings on the chart, which is commendably transparent.

The third chart changes the baseline. TorchTitan is PyTorch's reference large-scale training framework. Against it, Prime-RL with Mega MoE finishes a 30B step in 2.08 s versus 3.07 s (48% faster; 5,337 to 7,886 tokens per second per GPU) and a 235B step in 9.65 s versus 14.98 s (55% faster; 1,094 to 1,697 tokens per second per GPU).

This is where the viral 55% comes from. It is a real number on a real chart, but it compares two different frameworks, so it bundles the Mega MoE kernel with every other difference between Prime-RL and TorchTitan. The cleaner isolation of what the kernel itself contributes is the 32% to 34% from the previous chart.

The honest summary is a ladder. At the kernel level the gain is up to 3.11x. At the full-step level on the same framework it is 1.32x to 1.34x. Against a different framework, it is 1.48x to about 1.55x in step-time terms. Each rung answers a different question, and conflating them is how a 34% improvement becomes "twice as fast" in a thumbnail.

It is tempting to assume the maths dominates, but in sparse models the arithmetic per token is deliberately small. A GPU computing on a few experts' worth of tokens finishes quickly and then has to wait for the next batch to arrive across the interconnect. If the pipeline is serial, the machine alternates between bursts of work and gaps of waiting.

Overlap converts those gaps into useful work by double-buffering: the next tile of tokens is being transferred while the current tile is being multiplied. The theoretical ceiling is the larger of compute time and communication time rather than their sum. When the two are comparable, as they often are in expert-parallel MoE training, the saving approaches half of the layer time, which is consistent with the 2x-to-3x per-layer speed-ups measured on the smaller layer shape.

It also explains why the 235B layer benefits less. Larger hidden and expert dimensions make the matrix multiplications heavier relative to the data movement, so there is simply less waiting to hide. And it explains why the full-step improvement (about a third) is smaller than the layer improvement: a training step also contains attention, normalisation, optimiser updates, FSDP all-gathers and everything else outside the MoE layers, none of which Mega MoE touches. This is Amdahl's law in action, and it is the right lens for judging any single-component optimisation.

Forward-only kernels are comparatively easy to reason about: data flows one way. The backward pass has to compute gradients with respect to the inputs, the expert weights and the routing weights, and it has to send gradients back to the GPUs that originally sent the tokens, reversing the dispatch. Overlapping that without race conditions, while using the same fused workspace buffers, is why DeepSeek's release stopped at forward.

Prime Intellect's backward kernel is, by their numbers, less dramatic than the forward one (1.87x and 1.24x versus 3.11x and 2.17x per layer), which is plausible: the backward pass does roughly twice the arithmetic of the forward pass, so there is more compute to hide communication behind but less communication left to hide. What matters commercially is that it exists at all. Without it, the forward kernel could speed up rollouts and inference but not the gradient updates that make up the bulk of training cost.

Prime Intellect's blog does not publish numerical-equivalence or convergence curves for the new kernels in this post, so we cannot confirm that training loss matches the baseline step for step. Teams adopting it should run their own loss-parity checks before trusting a long run, which is standard practice with any custom kernel.

Prime Intellect positions itself as an open lab and compute provider, with a model factory, compute marketplace, inference and sandboxes under one roof. Its Prime-RL framework is the asynchronous reinforcement-learning trainer behind its recent INTELLECT models, and the blog's related posts list a Prime-RL 0.10.0 release with a new default RL loss called IPO, an acquisition of the Moondream vision-model team, and a rewrite of its agent tooling in Rust. In that context, Mega MoE is infrastructure plumbing for a lab that needs to post-train large sparse models cheaply.

This matters because the economics of frontier-adjacent training are increasingly an engineering contest rather than a pure capital contest. Open labs that cannot outspend the largest closed labs can still compete by wringing more out of each GPU-hour. Our coverage of Weco's AIDE 2 and recursive self-improvement and of open releases like Nanbeige4.3-33B shows the same pattern: smaller teams winning on efficiency and method.

The open-source lineage is also worth underlining. The kernel starts life in DeepSeek's DeepGEMM collection, is extended by Prime Intellect, and is benchmarked on Qwen's open models. Three different organisations' open work stack together. Our readers who follow DeepSeek's tooling, including its agent work in the DeepSeek harness explainer, will recognise a familiar pattern: infrastructure releases from one lab becoming building blocks for another.

What will you actually experience if you adopt Mega MoE? Several things temper the headline.

• Hardware gating. The kernel targets Hopper and Blackwell. Clusters of A100s or consumer cards are outside scope, and the published numbers are H200 only.
• Precision gating. The benchmarks are FP8. If your recipe is bf16 or relies on a different quantisation scheme, expect different gains.
• Scale dependence. The improvement shrank from 34% to 32% going from 32 to 128 GPUs. Prime does not publish larger-scale results, so extrapolating to thousands of GPUs is guesswork.
• Sequence length and routing. Everything ran at 16,384 tokens per GPU. Shorter sequences change the compute-to-communication ratio and therefore the benefit.
• Single-source evidence. All figures come from Prime Intellect's own post. At the time of writing we found no independent reproduction, so treat the numbers as credible but unverified.

The practical advice is to benchmark on your own model shape. A 24% to 25% reduction in step time on a long run is meaningful: if a training job would otherwise consume 1,000 GPU-hours of steps, you would expect to save roughly 250 of them, subject to the caveats above. That is an illustration of the arithmetic from the published step-time figures, not a promise for your workload.

Prime Intellect does not publish pricing for the kernels, which are open software, so the cost story is about GPU-hours. Using the sourced step times, the 30B run drops from 2.78 s to 2.08 s per step and the 235B run from 12.72 s to 9.65 s. Across 100,000 steps, that is the difference between about 77.2 hours and 57.8 hours on the 30B configuration, and between about 353 hours and 268 hours on the 235B one, a saving of roughly 19 and 85 wall-clock hours respectively, before any difference in queueing or checkpoint overhead. Multiply the 235B saving by 128 GPUs and you have roughly 10,900 GPU-hours.

At any plausible rental rate for H200-class capacity, that is a meaningful sum, although we have deliberately not attached a pound figure because rates vary widely by provider and commitment. Readers can drop their own hourly rate into the arithmetic. The more strategic point is iteration speed: a post-training experiment that finishes a quarter sooner lets a team try more ideas in the same week, which is where open labs can punch above their weight.

No quality data. The post reports speed only. No independent replication yet. The results are first-party. Narrow hardware window. Hopper and Blackwell only, with data-type support split between them. Limited scale tested. Up to 128 GPUs. Dependence on upstream. The kernel builds on DeepSeek's Mega MoE, so its long-term maintenance, licence terms and compatibility depend on a project Prime Intellect does not control; check the repository licence before commercial use. Complexity. Fused kernels are harder to debug and customise than composable PyTorch ops, and a custom router or an unusual expert layout may not be supported.

There is also an interpretive risk: because the headline 55% compares frameworks, a reader could wrongly assume that adding Mega MoE to any existing training stack yields that gain. The cleaner expectation is the one-third figure against an already well-optimised baseline.

Against stock PyTorch all-to-all MoE, the per-layer gains are the largest, up to 3.11x. Against DeepGEMM FP8 experts with torch all-to-all, which is the Prime-RL baseline and already strong, the end-to-end gain is about a third. Against TorchTitan, the full-step gain is 48% to 55% on the published set-up, with the caveat that the frameworks differ in more than the kernel.

Other communication-overlap approaches exist in the research literature and in vendor libraries, but Prime Intellect did not publish comparisons against them in this post, so we make no claim about relative merit. For inference-side MoE efficiency, our readers may prefer our coverage of model-level approaches such as the Nemotron 3.5 Lightning review and the memory-focused DeepSeek V4 KV cache explainer.

Adopt and benchmark it if you train or post-train MoE models on multi-node H100, H200 or Blackwell clusters, use FP8, rely on expert parallelism and see all-to-all time in your profiler. Wait if you are on older GPUs, train dense models, run small single-node jobs, or cannot afford to validate loss parity. Ignore it if you only consume models through APIs: it will lower the cost of building models for the labs that adopt it, but you will not see a button for it. For wider tooling options, browse our AI tools directory.

Prime Mega MoE is a solid, openly documented systems contribution: DeepSeek's forward-only fused kernel made trainable, with a clear test set-up and charts that show their working. The defensible headline is up to 3.11x at the kernel level and about 1.34x end to end against a strong baseline, rising to 55% only when the comparison switches to TorchTitan on a 235B model. If you run large MoE training on Hopper or Blackwell, it deserves a benchmark this week. If you saw "55% faster AI training" on a thumbnail, now you know which chart it came from and what it does not mean.