Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPO

The problem formulation and theoretical grounding are rigorous. The distinction between permutation-blind standard GRPO and the proposed permutation-aware approach is clearly articulated, and the ablation studies in Table 2 validate that both the Cross-Permutation Advantage ($A_{PA}$) and Consistency-Aware Reward ($r_{con}$) contribute meaningfully to performance. The evaluation protocol is commendably thorough: they test on Llama-3.1-8B, Qwen3-8B, and Qwen3-32B, using full permutation expansion ($N!$) at test time rather than random sampling, ensuring unbiased consistency estimates. The results show substantial improvements in Consistency (e.g., MT-Bench consistency rising from 80.6% to 88.0% on Llama-3.1-8B) and Consistent Accuracy without catastrophic drops in standard accuracy.

First, the method is strictly limited to discrete-choice settings (MCQ and pairwise comparisons) where permutation groups are well-defined; extending PA-GRPO to open-ended generation remains unresolved, as the authors acknowledge in the Limitations section. Second, training computational cost is significantly higher than standard GRPO: for MCQ tasks, the method requires $P \times N$ samples per instance (with $P=5$ permutations and $N=8$ samples each), yet the ablation in Figure 3 shows $P=24$ (full permutation) yields only marginal gains over $P=5$, suggesting the structured subset is a necessary but potentially suboptimal approximation. Third, the Consistent Accuracy (CA) metric relies on majority voting across permutations, which can mask systematic errors—a model consistently choosing the wrong answer across all 24 permutations would score perfectly on Consistency and CA. Finally, Appendix H reveals residual sensitivity: the model is most robust under coupled label-order permutations (the training condition) but less so under isolated label-only or order-only perturbations, suggesting it learns training-set specific biases rather than true semantic invariance.

The comparison against baselines is generally fair but mixes methodologies: inference-time methods (PriDe, CalibraEval, UniBias) require no training but add computational overhead at inference, while PIF (supervised fine-tuning) and GRPO (RL) are training-time methods like PA-GRPO. The comparisons with PIF and GRPO are the most relevant, and PA-GRPO consistently outperforms these, particularly on consistency metrics. However, the comparison with inference-time methods should acknowledge that PA-GRPO's gains come at the cost of expensive RL training with group sampling. Table 1 shows PA-GRPO achieves superior Consistent Accuracy (e.g., 75.0% vs 73.0% for GRPO on TinyMMLU for Llama-3.1-8B), though some benchmarks show minor accuracy trade-offs (e.g., RewardBench drops from 76.3% to 75.8% for GRPO vs PA-GRPO on Llama-3.1-8B). The evidence supports the claim that PA-GRPO reduces selection bias, but the magnitude of improvement over strong baselines like PIF is sometimes modest (e.g., 1-3% absolute on CA).

The paper provides substantial implementation detail that aids reproduction. Training uses the open-source verl RL framework with LoRA ($r=32, \alpha=64$), and hyperparameters are specified in Appendix F: AdamW optimizer with learning rate $10^{-5}$, KL coefficient $\beta=0.001$, and batch sizes of 40 (MCQ) or 32 (Judge). The authors describe their data filtering protocol—retaining only instances where the base model shows inconsistent predictions across permutations—which is critical for reproducing the training set but introduces a dependency on the specific base model's initial behavior. The paper states code will be made available on GitHub, but as of submission, it is not yet accessible. Full permutation evaluation at test time (using all $N!$ permutations) is clearly defined and should be reproducible, though computationally expensive (24 evaluations per MCQ instance).