QDA: Closed-Loop LLM Quantization Design with Agents

Xingyang Li*, Tianlang Zhao*, Song Han, Ligeng Zhu
MIT    NVIDIA    *Equal contribution

TL;DR

QDA brings a coding agent into LLM quantization design. For each model and precision, the agent designs the post-training quantization recipe on its own, with no human in the loop. Its checkpoints beat the strongest public recipe on four reasoning models, by 0.9, 1.4 and 2.2 points on average at W4A16, W4A8 and W3A16, and run on existing kernels with no inference overhead.

The agent works the way a quantization expert does: it proposes a recipe, builds it, measures it and decides what to change next. Two things make this loop fast enough for up to 36 candidates in one eight-hour search: a KL score that takes minutes instead of dozens of GPU-hours, and a library of 21 PTQ methods that stack in any order. We find that no method wins everywhere: the best recipe differs for every model and precision, and the agent finds it by searching.

On the 320B-parameter GLM-5.3-Flash at NVFP4, one 6.5-hour, $69 search recovers 98.3% of the FP8 model's accuracy, against 96.9% for NVIDIA's release. The tools matter too: given NVIDIA ModelOpt's tools instead of ours, the same agent falls 0.5–1.3 points behind on Qwen3.6-35B. A search costs $19–75 of agent tokens.

1An old question

In 2024, Yujun Lin, Xingyang's wonderful mentor at MIT HAN Lab, wrote DeepCompressor, the quantization toolbox behind QServe and SVDQuant. He once told us that the results sometimes puzzled him: a new method that looks strong in its paper often adds little once it is stacked on a few basics, such as AWQ's smoothing, a Hadamard rotation and GPTQ. Moreover, nobody knew which methods work well together, because each is tested on its own.

Answering that by hand means trying combinations model by model, which no one has time for. Today, however, LLMs can act as optimizers[1, 2], and agent harnesses such as Humanize[3] keep a coding agent working for hours with no one watching. So QDA lets an agent run exactly these experiments, for every model and precision.

The question is real. Across four reasoning models and three precisions, no public recipe is the most accurate everywhere (Figure 1). The newest method wins three of the four models at W3A16, yet plain GPTQ is still the best on R1-Distill-Llama-8B at every precision.

No public recipe wins everywhere
The most accurate public recipe for each model and precision; hover a cell for all scores
GPTQAWQnewer method: ParoQuant, QuaRot+GPTQ or GSQ
Figure 1. Mean accuracy over math, GPQA-Diamond, LiveCodeBench and IFBench. Each precision has its own candidates: GPTQ and AWQ everywhere, plus ParoQuant at W4A16, QuaRot+GPTQ at W4A8 and GSQ at W3A16. The small number is the winner's lead over the runner-up; a split cell is a tie.

Stacked well, however, known methods go far. On Qwen3.5-9B at W4A8, GPTQ alone sits 17 points below BF16. A Hadamard rotation wins back 9.9 points, and the agent's recipe wins back another 5.2 with self-generated data, an FP8-aware scale search and scale distillation. None of these steps is new; the gain comes from choosing them for this model.

2Why the loop is slow

An agent designs a recipe the way an expert does: propose a recipe, build the checkpoint, measure it, and decide what to change next. The more rounds it fits into a budget, the better the recipe, but two steps of this loop are slow (Figure 2, left).

The design loop, without and with QDA
The agent's loop is the same; QDA replaces the two slow steps
Figure 2. Left: building means porting methods across codebases, and measuring means a full evaluation, so a search fits only a few rounds. Right: QDA builds from one library and scores by KL in minutes, so a search fits dozens of rounds.

Measuring is slow, because reasoning models think for thousands of tokens per answer and each problem must be sampled many times, so a full evaluation of one checkpoint takes dozens of GPU-hours. Building is slow, too, because most PTQ methods live in separate codebases. Left to port them itself, the agent wrote 3.1k lines of code per search and still ended up less accurate. QDA makes both steps fast (Figure 2, right). Figure 3 shows how the pieces fit: the agent works in a sandbox and reaches the library, the evaluator and the GPUs only through four commands. The harness around it is built on Humanize, our agent flow system, which fences the agent's files and network, restarts a stalled turn and keeps a long search running unattended.

System architecture
The agent reaches the rest of the system only through four commands
recipe or checkpointscoreGPU job
Figure 3. The agent calls qda to build, kl to score, qsched for other GPU jobs such as long generations, and handin to submit. Reference traces and test benchmarks stay outside its sandbox, so it cannot tune on them, and every GPU job goes through one scheduler that keeps the search within budget.

2.1Measuring: a score in minutes

QDA scores a candidate in minutes, 350–990× faster than a full evaluation, by how far its next-token predictions drift from the original model's. A quantized model is good when it behaves like the original one, so instead of grading answers, QDA measures the KL divergence between the two models' predictions on reasoning traces that the original model wrote. The original model's side is computed once and cached, so a score needs only one pass of the candidate. The traces stay outside the agent's sandbox, so it cannot tune on them.

The score ranks checkpoints the way accuracy does. Most PTQ methods minimize the error of each layer on its own, which cannot see errors that build up across layers. On Qwen3.5-9B at W4A16, this layer-wise error ranks AWQ last of three public recipes although it is second in accuracy, whereas KL gets the order right (Figure 4). Over 44 pairs of checkpoints, KL picks the more accurate one 82% of the time, against 73% for the layer-wise error, though it does not catch every failure (§3).

Three ways to rank three public recipes
Qwen3.5-9B at W4A16; lower is better in every panel
ParoQuantAWQGPTQ
Figure 4. Layer-wise error puts AWQ last even though it is second in accuracy, whereas KL gets the order right.

2.2Building: parts that snap together

QDA's library implements 21 PTQ methods behind one interface. Because every method takes a checkpoint and returns one, the agent can stack them in any order, and a method written once runs on new architectures. Each method changes one part of the math behind a quantized layer, which Figure 5 lets you assemble.

Assemble a recipe
Pick one option per row or start from a preset; lines that differ from plain GPTQ are highlighted
Figure 5. A recipe as math, with a few of the library's 21 methods.

It is also fast enough for the loop. On one H100, a GPTQ build of Qwen3.8-27B takes 191 s against 501 s with llm-compressor, and the 35B MoE builds in 181 s, where llm-compressor runs out of memory.

3One search, start to finish

Each search runs Claude Opus 5.5 in Claude Code at high effort, driven by Humanize, on eight GH200 GPUs for eight hours, with no human in the loop, for $19–75 of agent tokens per search (37–122M tokens, mostly read from cache). Figure 6 replays the one on Qwen3.6-35B at W4A16.

Replay of one search
KL of every candidate the agent scored, Qwen3.6-35B-A3B at W4A16
Figure 6. Grey dots are the 26 scored candidates; the red line is the best so far.

The agent first rebuilds GPTQ with different clip and solver settings, none of them better than the public checkpoint. At 1.5 hours, it calibrates on the original model's own reasoning traces instead of generic text, and the KL drops by 23% in one step. Its notes, where the “anchor” is the original model, spell out both the bet and the risk:

Calibrating GPTQ on the anchor's own generated thinking traces is the biggest lever so far (…), but it raises ref KL: it specialises on anchor-style text.

Its KL looked fine, but KL scores the model on text it is given, not on text it writes. So at 3.1 hours the agent had this candidate write full answers to 191 hard problems of its own choosing, and 12 were cut off at the length limit, most of them code answers stuck in a loop, against 3 for the original model. The agent starts watching truncations (“the truncation count is the signal I watch”) and turns to distillation on the quantized model's own long answers. Its hand-in scores 0.0088 against 0.0153 for the best public recipe and cuts off only 6 answers.

4Results

We ran one search per model and precision on four reasoning models (dense, linear-attention hybrid and MoE) and evaluated the final checkpoints on math, GPQA-Diamond, LiveCodeBench and IFBench.

4.1Over the best public recipe

QDA is more accurate than the best public recipe on 11 of the 12 pairs of model and precision, by 0.9, 1.4 and 2.2 points on average at W4A16, W4A8 and W3A16 (Figure 7). The largest gains, +5.2 on Qwen3.5-9B at W4A8 and +6.1 on R1-Distill-Llama-8B at W3A16, come where the best public recipe is far below BF16, by 7.1 and 11.7 points.

Gain over the best public recipe
Points of mean accuracy over the winner of each cell in Figure 1; hover a cell for all scores
Figure 7. Darker means a larger gain. Below each gain: the public recipe it is measured against. The Qwen3.6-35B cells are means of three searches.

4.2A 320B model at NVFP4

On GLM-5.3-Flash, a 320B MoE that none of the main searches saw, one 6.5-hour search for $69 recovers 98.3% of the FP8 model's accuracy, against 96.9% and 96.3% for the NVFP4 checkpoints that NVIDIA and Red Hat AI released (Figure 8). It used the same agent, library and evaluator as the main searches, with no public recipe to start from.

Figure 8. Mean accuracy over math, GPQA-Diamond, LiveCodeBench and SuperGPQA, as a share of the FP8 model's 77.7. All three checkpoints are served by the same engine on the same GB300 GPUs, with the same sampling settings.

4.3Over an agent with NVIDIA ModelOpt

A strong agent alone is not enough. NVIDIA's ModelOpt ships a recipe-search skill for coding agents, so we gave the same agent ModelOpt and that skill instead of our tools, with the same 64 GPU-hours. It falls 0.5–1.3 points behind QDA, and even below the best public recipe at W4A8 and W3A16 (Figure 9), although it picks its candidates on the test benchmarks themselves.

The same agent, with ModelOpt or with QDA
Qwen3.6-35B, mean accuracy, the same 64 GPU-hours per search
best public recipeagent + NVIDIA ModelOptQDA
Figure 9. The agent with ModelOpt picks its candidates on the test benchmarks; QDA never sees them. QDA is the mean of three searches.

4.4Free at inference

QDA's checkpoints decode as fast as GPTQ's (Figure 10). ParoQuant, the strongest public recipe on Qwen3.5-9B, earns part of its accuracy with a rotation applied to each layer's input at serving time, a sensible trade that comes with its own kernel; at batch sizes 16 and 64, QDA's checkpoints decode 1.3–1.6× faster. QDA only uses steps that fold into the weights, so its checkpoints keep the standard formats and kernels.

Decode throughput, Qwen3.8-27B at W4A16
Output tokens per second on one GH200, vLLM
ParoQuantGPTQQDA
Figure 10. QDA runs on GPTQ's kernel and decodes within 0.2% of it. Checkpoints for Qwen3.5, Qwen3.6, Qwen3.8, Qwen3-Coder, Gemma 4, Nemotron 3 Nano, R1-Distill-Llama and GLM-5.3-Flash are on Hugging Face, ready for vLLM, SGLang and llama.cpp.

5What we learned, and what's next

PTQ methods are parts, not rivals. Papers pit one method against another, as if the field were a contest with a single winner. Yet each method fixes one kind of error, and which error dominates depends on the model; so the best checkpoints came from stacking known methods, and a method that loses in isolation can still be the step a recipe needs.

Agents are smart enough already; they need fast feedback and good tools. With a score that takes minutes, each search tried 5 to 36 candidates, using only 4% of its GPU time on scoring. Without the library, by contrast, the agent spent its effort on porting code and never tried a rotation at W4A8, the step that gave QDA its largest gain there.

What's next: agents that make their own tokens cheaper. A search spends tokens to make tokens cheaper: $19–75 of agent tokens buys a checkpoint that serves every later token with about a quarter of the weight memory. Pointed at the open models that agents themselves run on, this becomes a loop that feeds itself, and we plan to close it in three steps. First, day-one quantization: every new open model gets searched recipes for every format on release day. Second, a library that grows itself: our agents already invented steps of their own, such as training on the quantized model's answers and averaging checkpoints, and the ones the evaluator confirms can join the library, so each search starts from more than the last. Third, the same loop beyond weights, for KV caches, sparsity and new low-bit formats co-designed with their kernels, where coding agents such as NVIDIA's Kernel Design Agents[4] already write faster kernels. We hope QDA answers the question DeepCompressor raised, with a tool anyone can run.

Acknowledgements

We thank Han Cai, Yi Pan, Haocheng Xi, Yujun Lin, Ziyi Xu, Mark Kim and Enrico Shippole for their feedback. This research was supported by SK Telecom and the MIT Generative AI Impact Consortium (MGAIC). Jenia Jitsev and LAION provided access to compute, and the Gauss Centre for Supercomputing e.V. (GCS) provided computing time on the JUPITER supercomputer at the Jülich Supercomputing Centre (JSC).

References

[1] Yang et al. Large Language Models as Optimizers. ICLR 2024.

[2] Novikov et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery. 2025.

[3] Humanize: the agent flow system. github.com/humanfia/humanize.

[4] KDA: Kernel Design Agents, an agent-centric workflow for high-performance CUDA kernels. github.com/NVlabs/kda.

Citation

Please cite this work as:

Xingyang Li, Tianlang Zhao, Song Han and Ligeng Zhu. “QDA: Closed-Loop LLM Quantization Design with Agents.” arXiv:2611.23388, 2026.

Or use the BibTeX citation:

@article{li2026qda,
  title   = {QDA: Closed-Loop LLM Quantization Design with Agents},
  author  = {Li, Xingyang and Zhao, Tianlang and Han, Song and Zhu, Ligeng},
  journal = {arXiv preprint arXiv:2611.23388},
  year    = {2026}
}
MIT HAN LabBack to top ↑