1105623876/dsh-bench

README excerpt

dsh-bench English · 中文 Run one task across several models, tool surfaces, or context configurations, and get back a controlled-experiment table: which arm actually passed, what each cost, and where a slower arm's time went. When the experiment's design or its result is unsound, the report says so to your agent directly — a task every arm solves along the same trajectory measures typing speed, not capability. Built for DeepSeek Harness. The plugin registers three tools: bench run to measure, bench history to compare against past runs, and bench gc to reclaim the session logs the trials leave behind. Each trial is a fresh one-shot spawn subagent that sees only the task text — no parent history — so the arms are comparable. Why The plugin ecosystem has plenty of tools that observe a single session — usage dashboards, context breakdowns, trajectory diagnostics — and none that compare two. Without a comparison there is no way to answer whether a cheaper model, a smaller tool surface, or a shorter output budget actually changes the outcome; there are only opinions about it. Every number here already exists in the harness. The plugin drives the matrix and folds the results: what where it …

Category