Label-Free Steering: Compressing Test-Time Reinforcement Learning into Bias-Only Subspaces

Authors Naveen Vakada, Mingyuan Li, Shaoxiong Ji
Venue Preprint
Date September 2026
Read Paper arXiv Code Models Cite

Abstract

Test-time reinforcement learning (TTRL) enables models to improve their reasoning without relying on labeled training data, but existing approaches typically optimize a large fraction of the model parameters. This raises a natural question: can effective test-time adaptation emerge when both the reward signal and the optimization space are severely restricted? We answer this question with label-free bias-only TTRL, which uses majority-vote pseudo-labels as rewards and optimizes only ~100K bias parameters while keeping the pretrained backbone frozen. On MATH-500, our approach reaches 76.67% accuracy, matching the labeled bias-steering baseline while optimizing 76,000× fewer parameters than full-parameter TTRL. The same training procedure improves performance across vision-language and audio reasoning tasks, including MathVista, AI2D, LogicVista, and MMAU. We further show that the learned steering vectors transfer to 4,500 held-out MATH problems, indicating that the adaptation is not limited to the problems used during test-time optimization. Finally, we analyze why this highly restricted adaptation can work, showing that majority-vote reliability improves with rollout consensus and that bias subspaces with greater accessible gradient energy exhibit stronger downstream trainability.

Method

Label-free bias-only TTRL sits at the intersection of two restrictions applied together for the first time: it removes the need for labeled rewards, the way TTRL does, while also restricting optimization to a tiny per-layer additive bias subspace, the way bias-only steering does — rather than requiring either a labeled reward or a full-parameter update.

Optimization space ~100K additive bias parameters per model (one vector per MLP layer), backbone frozen.
Reward Majority-vote pseudo-labels from the model's own rollouts — no ground-truth labels.
Modalities Text (MATH-500), vision-language (MathVista, AI2D, LogicVista), audio (MMAU).
Models Qwen2.5-7B / Qwen2.5-Math-7B, Qwen2.5-VL-7B-Instruct, Qwen2.5-Omni-7B.

Key Findings

Main Results

Model Benchmark Baseline Bias-only TTRL (ours)
Qwen2.5-Math-7B MATH-500 56.0 79.50 ± 0.70
Qwen2.5-7B MATH-500 45.9 76.67 ± 1.57
Qwen2.5-VL-7B-Instruct AI2D 73.58 78.29 ± 0.03
Qwen2.5-VL-7B-Instruct LogicVista 37.50 43.38 ± 1.01
Qwen2.5-VL-7B-Instruct MathVista 61.60 65.40 ± 1.77
Qwen2.5-Omni-7B MMAU 57.40 60.87 ± 0.37

Mean ± standard deviation over 3 independent training seeds. Full comparisons against LoRA and full-parameter fine-tuning are in the paper.

Training Recipe

All training and evaluation scripts, organized per benchmark, share the same recipe:

python src/train_ttrl_steer.py \
  --model_id Qwen/Qwen2.5-Math-7B \
  --dataset math --num_steps 300 --G 64 --lr 5e-4 \
  --use_vllm --output_dir outputs/math500_math7b

The repository README lists the full set of launch commands across all five benchmarks. Trained bias vectors for each benchmark are released on Hugging Face.

Resources

Paper

arXiv Preprint

Read the full paper on label-free bias-only test-time reinforcement learning.

Open Paper
Models

Hugging Face Checkpoints

Trained bias steering vectors for each of the five benchmarks covered in the paper.

View Models

BibTeX

@article{vakada2026labelfree,
  title   = {Label-Free Steering: Compressing Test-Time Reinforcement Learning into Bias-Only Subspaces},
  author  = {Vakada, Naveen and Li, Mingyuan and Ji, Shaoxiong},
  journal = {arXiv preprint arXiv:2609.18587},
  eprint  = {2609.18587},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url     = {https://arxiv.org/abs/2609.18587},
  year    = {2026}
}

Funding

HAIF is co-funded by the European Union's Horizon Europe research and innovation programme's Marie Skłodowska-Curie Action.