yan77-h/dsh-agent-evaluator

README excerpt

dsh-agent-evaluator An agent evaluation plugin for DeepSeek Harness: import test sets, run agents, score results, and generate reports. Chinese Install If dsh is installed globally, use it instead of pnpm dsh . Cloning the repository or using a git URL also works. Usage Web UI: Settings → Agent Evaluation, or run /eval commands: Subcommands: import , list , drop , model , models , judge , run , resume , kill , runs , progress , report . Headless / CI: Features - Imports JSONL / JSON / CSV / TSV and directories, adapting common benchmark field names. - Runs each case in an isolated agent with concurrency, retries, and timeouts. - Scorers: exact , contains , llm . - Persists reports under $DSH EVAL HOME/eval/reports , with checkpoint/resume support. - Web panel provides model pickers, live progress bars, and pass/fail charts. Documentation - INSTALL.md / INSTALL.zh.md — installation - USAGE.md / USAGE.zh.md — full usage - AGENTS.md / AGENTS.zh.md — architecture and maintenance Development Integration tests and client builds require a deepseek-harness checkout; see AGENTS.md. MIT License, see LICENSE.

Category