Morriaty-The-Murderer/dsh-plugin-abtest

README 摘要

English · 简体中文 DSH Plugin A/B Test Test a DSH plugin change on the same tasks before you ship it. DSH Plugin A/B Test runs your current plugin (Control) and proposed change (Candidate) in isolated DSH environments, pairs their results case by case, and produces evidence you can review before release. It helps answer three practical questions: - Did the Candidate improve task success? - Did that improvement come with a meaningful regression in tokens, latency, or tool errors? - Can someone else reproduce the result from the same inputs? Every experiment ends with one of four deterministic outcomes: PROMOTE , REVIEW , REJECT , or INCONCLUSIVE . Even PROMOTE is an offline recommendation only—this project never changes your real DSH profile or publishes a plugin for you. See the decision first This is a real result from the repository's offline example, with unrelated fields omitted: Alongside the decision, you get raw session evidence, assertion results, pair-level deltas, and reports in JSON, Markdown, and HTML. The model does not choose the outcome; deterministic rules from the experiment manifest do. Quick start The current MVP runs from source and requires Node.js ^22.19.0 =24.0.0…

分类