NVIDIA's Nemotron 3.5 report details memory, decoding and compression gains for agents
NVIDIA's Nemotron 3.5 report says hybrid layers, draft-token decoding and compression speed up agent models, citing 2.43x throughput for the four-bit Super version.
NVIDIA's Nemotron 3.5 technical report claims that a set of architecture and serving changes speeds up agent models. The report covers two models. Super has 120B parameters with 12B active per token and accepts text, images and video. Lightning has 30B parameters with 3B active per token and handles text. NVIDIA also releases a compressed NVFP4 version of Super. The memory argument rests on how attention and state-space layers grow differently. NVIDIA says the design combines attention with Mamba-2 layers. Attention keeps a cache tied to earlier tokens, so it grows as a conversation lengthens. Mamba keeps a running state whose size stays fixed, so the memory cost of a long session grows more slowly. For an agent that rereads files and tool output across dozens of calls, that difference bears on how many sessions a server can hold at once. Active parameters address a separate cost. Each token passes through only part of the model, so each generation step does less computation, although the server still has to store all of the weights. The decoding gain comes from speculative decoding. A smaller prediction component drafts upcoming tokens, and the main model checks several of them together, keeping the ones it accepts and replacing the rest. When many drafts pass a single check, generation moves ahead without another full step of the main model. According to the report, the size of the saving depends on draft quality, so NVIDIA gives the prediction component extra training against the finished model to keep its drafts close to what the main model would produce. The quality questions center on compression. Many weights in the NVFP4 version of Super use four-bit storage, which reduces memory and the amount of data moved during generation. Because compression can change answers, NVIDIA follows it with training meant to bring the compressed model back in line with the original. The evaluation also checks response length, since a model that starts writing unnecessarily long answers can erase its speed advantage by generating extra tokens. The 2.43 figure comes from a serving test. NVIDIA used a 20-turn workload that reuses earlier conversation content while adding new input on each turn. The server had to sustain at least 50 output tokens per second for each user. Among configurations that met that bar, the four-bit version delivered 2.43 times the throughput of BF16 under the evaluated conditions. That measures how much work a server completes across many requests. NVIDIA notes that it does not measure how much faster an individual agent finishes a task.
What changed
NVIDIA's Nemotron 3.5 technical report describes Nemotron 3.5 Super, Nemotron 3.5 Lightning, and a compressed NVFP4 version of Super that uses hybrid attention and Mamba-2 layers with draft-token decoding.
Type: Research · Applies to: NVFP4 version of Super
Why it matters
Companies running multi-call coding agents may see lower serving cost per request, but the reported throughput gain does not directly translate into faster completion of agent tasks.
What you can do
Measure how long a complete agent run takes, including the wait after each tool result, and check whether those waits grow as history gets longer.
Original source
We have no link to the original source for this story yet.
Where we saw it
AI Engineering
AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works