Poly-EPO: Training Exploratory Reasoning Models

Dimensions overall score 7.0

• Ifdita Hasan Orney∗ Jubayer Ibn Hamid∗ Shreya S Ramanujam Shirley Wu Hengyuan Hu Noah Goodman Dorsa Sadigh Chelsea Finn Stanford University ∗Equal contribution. Correspondence to{ifdi1101, jubayer}@stanford.educlaim 82%
• Figure 5:Training dynamics on synthetic domains. The two left plots correspond tomulti-digit multiplication, andthetworightplotscorrespondtopolynomialsolving. Wemeasurestrategydiversitybyclusteringgenerationsand reporting the number of distinct clusters among correct and incorrect responses for each promptclaim 82%
• Furthermore, as test-time compute has become a standard paradigm for performance gains, recent research has explored training objectives aligned with inference-time metrics [TZS+25; CTG+25; CQW+25; WK25]claim 82%
• overhigh-levelstrategiesduringLMpost-trainingremainsunclear. Similarly,UCBandcount-basedbonuses[SKM25;claim 82%
• drivingasynergybetweenexplorationandexploitation. Othermethodsimplicitlyinduceexplorationthroughcuriosity- driven techniques [DSL+25; GPW+26] or novel objectives independent of diversity measures [TZZ+26; CQW+25;claim 82%
• More closely related to our approach are works that promote exploration via objectives targeting the semantic diversity of generations [LZY+25; YCW+25; HWH+26]. However, while these distribute probability mass across multiplecorrectgenerationsclaim 82%
• the full benefits of polychromic objectives. 8 Conclusion Inthispaper,wepresentedPolychromicExploratoryPolicyOptimization(Poly-EPO)claim 82%
• 2025.url:https://hkunlp.github.io/blog/2025/Polaris. [BBK+24] M.Besta,N.Blach,A.Kubicek,R.Gerstenberger,M.Podstawski,L.Gianinazzi,J.Gajda,T.Lehmann, H. Niewiadomski, P. Nyczyk, and T. Hoefler. “Graph of Thoughts: Solving Elaborate Problems with Large Language Models”claim 82%
• maxθ Ey∼πθ(·|x) [r(x, y) +λd(x, y)], whereπθ is the parameterized policy,r(x, y)is the task reward andd(x, y)is an exploration bonus, such as an entropy bonus, UCB bonus, or semantic diversity bonus. Howeverclaim 82%
• .(2) We can use the policy gradient, whereA(x, y)is the advantage of generationy, to optimize this objective: ∇θEx∼DEy∼πθ(·|x)claim 82%
• =E x∼DEy∼πθ(·|x) [∇θ logπ θ(y|x)A(x, y)].(3) 2.2 Set Reinforcement Learning Setreinforcementlearning(setRL)[HOX+26]isaframeworkthatgeneralizesstandardRLbyassigningrewardto setsofsampledactionsorgenerations,ratherthantoeachoneindependently. Importantlyclaim 82%
• begins to outperform them as early ask= 32. This improved diversity under repeated sampling is especially valuable in domains such as mathematics, where one can pair generation with strong external verifiers such as Lean. Training Dynamics.We now analyze the training dynamicsclaim 82%