Plain reranker fine-tuning beats decision fine-tuning for Qwen rerankers up to 9B
Researchers from Passau and IT:U find that plain reranker fine-tuning wins by 5 to 29 nDCG@10 points from 0.6B to 9B parameters.
Plain reranker fine-tuning beat decision fine-tuning at every model size up to 9B parameters, by 5 to 29 nDCG@10 points. That is according to a paper by Michael Dinzinger, Kanishka Ghosh Dastidar and Michael Granitzer of the University of Passau and Interdisciplinary Transformation University Austria. At 27B, the two approaches tied. The authors tested six Qwen backbones ranging from 0.6B to 27B parameters on ten collections from TREC DL and BEIR. Each backbone was compared three ways: the untuned model asked a yes-or-no question zero-shot, a decision fine-tune, and a reranker fine-tune. The authors trained the last two themselves.
Decision models, sometimes called System One models, are trained to answer a question about an input with a calibrated probability rather than with generated text. The paper notes that they were used as rerankers within weeks of their first release in September 2026, and that gap is what the authors set out to test. A reranker takes a list of candidate passages, already pulled by a cheaper first-stage search such as BM25, and reorders them by relevance. The paper's experiments reranked the same BM25 top-100 list for every model.
From 0.6B to 9B, reranker fine-tuning won on every backbone. The advantage held on the zero-shot collections as well as on the ones used in training. Up to 9B, decision fine-tuning added nothing over the untuned model asked the same question. The extra training bought no ranking quality at those sizes.
The calibration result is more complicated and depends on wording. Asked whether a passage answers the query, decision models were better calibrated than the published reranker fine-tunes from 4B upward. When the authors asked a slightly different question, drawn from prior work, decision models lost that advantage. The calibration edge, then, depends on the question asked. Calibration did not make up for a worse ranking, though. Up to 9B, a reranker fine-tune kept the better passages at a fixed score threshold. The authors put the practical advice in one sentence: "If you need a reranker, fine-tune a reranker."
What changed
Plain reranker fine-tuning beats decision fine-tuning by 5 to 29 nDCG@10 points on Qwen backbones up to 9B parameters, and the two approaches tie at 27B.
Type: Research · Applies to: Qwen backbones ranging from 0.6B to 27B parameters
Why it matters
Teams building reranking for search or retrieval should not assume decision-style fine-tuning improves ranking quality. In these tests, plain reranker fine-tuning was the stronger option up to 9B parameters.
What you can do
Benchmark plain reranker fine-tuning against decision-style alternatives on your own queries before adopting either.
Original source
Where we saw it
DAIR.AI (Elvis Saravia)
AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works