DevRico003/dsh-verifier

README 摘要

dsh-verifier An LLM-as-a-verifier plugin for DeepSeek Harness ( dsh ). It adds a quality gate that runs at the end of every agent turn, plus three tools the agent can call to check its own work. The scoring method is a port of llm-as-a-verifier by Kwok et al. (project site llm-as-a-verifier.com, paper arXiv 2607.05391, MIT). That repo selects the best of N agent trajectories. This plugin takes the same math and wires it into a running harness. Why a verifier The method comes from the llm-as-a-verifier authors. Their framework (source: llm-as-a-verifier.com): probability over the logits instead of a sampled token, a fine-grained scoring token, repetition, and decomposition into simpler criteria, aggregated as R(x, tau) = 1/(C K) sum over criteria, repeats and scale values of p(v x, c, tau) phi(v). What that buys with the same model this plugin runs on (chart from the llm-as-a-verifier README; Terminal-Bench 2.1, mini-swe-agent, DeepSeek V4 Flash as generator and verifier, costs at OpenRouter prices of 2026-08-17): best-of-3 lifts DeepSeek V4 Flash from 78.7% to 86.5%, best-of-5 to 88.0%, at roughly a quarter of the cost per task of GPT-5.6 Sol in Codex. The gate in this plugin is th…

分类