aispin-dev/llm-as-a-Verifier-dsh

README 摘要

LLM-as-a-Verifier for dsh — Best-of-N (Bo5) conversation mode English 中文文档 Give DeepSeek V4 Flash test-time scaling: V4 Flash + Bo5 self-verification reaches Fable-5-level scores — 88% on Terminal-Bench 2.1, frontier-model accuracy at a fraction of the cost (≈11× cheaper). An independent dsh-native implementation of the test-time selection method from LLM-as-a-Verifier (arXiv:2607.05391, MIT). Method by the paper's authors; this implementation by Aispin. The paper's idea, in one minute Cheap models can generate great answers — they just can't recognize which one is great. LLM-as-a-Verifier closes that gap: 1. Sample N candidates from a cheap model (DeepSeek V4 Flash): slightly different attempts at the same task. 2. Grade with a fine-grained verifier — the same cheap model, asked to grade pairs of candidates on an A–T letter scale. The score is not the sampled letter: it is the expectation over the grade token's logprob distribution , Σ p(token)·φ(letter) — the model's full belief, not one dice throw. 3. Both orderings per pair cancel the verifier's position bias; repeated evaluations alternate slots. 4. Select the best — the paper's core result: V4 Flash sampling 5 candidates + se…

分类