uson1x/dsh-plugin-llm-verifier

README excerpt

dsh-plugin-llm-verifier A plugin for DeepSeek Harness that adds an LLM verifier : it grades candidate solutions with a model and returns scores between 0 and 1. Based on LLM-as-a-Verifier (paper). The headline feature is verify rollout : ask for something once, and the plugin runs several independent agent attempts in parallel, grades them, and gives you the best one. Install Requires Node 20+ and a working DeepSeek Harness profile. Install into the profile your dsh command actually loads — web for dsh web , headless for the CLI: Then append this entry to that profile's cordis.patch.yml (the file is a YAML list and usually already exists — add to it, don't replace it; examples/cordis.patch.yml is a copy with the most useful options commented in): provider and model name the LLM route that does the grading — replace them with a provider and model id your profile registers (the same names dsh's model picker shows). The plugin refuses to load without them. Restart dsh (patch files are read at boot). If the web app was already open in a browser tab, reload the tab once so it picks up the plugin's UI bundle. Smoke test: ask the agent to "use llm as a verifier to write a haiku" . You sho…

Category