Label-Free Steering: Compressing Test-Time Reinforcement Learning into Bias-Only Subspaces

Abstract

Test-time reinforcement learning (TTRL) enables models to improve their reasoning without relying on labeled training data, but existing approaches typically optimize a large fraction of the model parameters. This raises a natural question: can effective test-time adaptation emerge when both the reward signal and the optimization space are severely restricted? We answer this question with label-free bias-only TTRL, which uses majority-vote pseudo-labels as rewards and optimizes only ~100K bias parameters while keeping the pretrained backbone frozen. On MATH-500, our approach reaches 76.67% accuracy with Qwen2.5-7B, slightly exceeding our own labeled bias-steering reproduction while optimizing 76,000× fewer parameters than full-parameter TTRL. The same training procedure improves performance across vision-language and audio reasoning tasks, including MathVista, AI2D, LogicVista, and MMAU. We further show that the learned steering vectors transfer to 4,500 held-out MATH problems, indicating that the adaptation is not limited to the problems used during test-time optimization. Finally, we analyze why this highly restricted adaptation can work, showing that majority-vote reliability improves with rollout consensus and that bias subspaces with greater accessible gradient energy exhibit stronger downstream trainability.

Method

Label-free bias-only TTRL sits at the intersection of two restrictions applied together for the first time: it removes the need for labeled rewards, the way TTRL does, while also restricting optimization to a tiny per-layer additive bias subspace, the way bias-only steering does — rather than requiring either a labeled reward or a full-parameter update. For each unlabeled problem, the frozen backbone generates multiple rollouts; the majority-vote answer across rollouts defines a pseudo-label reward, and GRPO updates only a ~100K-parameter set of per-layer bias vectors against it — the backbone itself never receives a gradient.

Two theoretical results motivate why a signal this restricted can still work — one about the reward, one about the subspace:

Key Findings

Main Results

The same ~100K-parameter training procedure is used across all tasks and modalities, with only the task-specific prompt and reward/grading function adapted to each benchmark.

Mean ± standard deviation over 3 independent training seeds. Full comparisons against self-consistency, LoRA (r=1 exact-match, r=16, r=64), and full-parameter fine-tuning are in the paper (Table 1).

Held-out transfer: bias vectors trained only on MATH-500, frozen and applied without further optimization to 4,500 verified-disjoint MATH problems.

Training Recipe

All training and evaluation scripts, organized per benchmark, share the same recipe:

python src/train_ttrl_steer.py \
  --model_id Qwen/Qwen2.5-Math-7B \
  --dataset math --num_steps 300 --G 64 --lr 5e-4 \
  --use_vllm --output_dir outputs/math500_math7b

See the repository README for the full set of launch commands across all five benchmarks. Trained bias vectors are on Hugging Face, one repo per benchmark and base model, gathered in the Label-Free Bias-Only Steering collection.

Resources

@article{vakada2026labelfree,
  title   = {Label-Free Steering: Compressing Test-Time Reinforcement Learning into Bias-Only Subspaces},
  author  = {Vakada, Naveen and Li, Mingyuan and Ji, Shaoxiong},
  journal = {arXiv preprint arXiv:2609.18587},
  eprint  = {2609.18587},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url     = {https://arxiv.org/abs/2609.18587},
  year    = {2026}
}

Funding

HAIF is co-funded by the European Union's Horizon Europe research and innovation programme's Marie Skłodowska-Curie Action. The authors also acknowledge CSC — IT Center for Science, Finland, for computational resources, including access to the Roihu and LUMI supercomputers.