renjianguojinqianfan/dsh-skill-eval

README 摘要

dsh-skill-eval Skill-trigger evaluation plugin for DeepSeek Harness (DSH). An LLM judge recreates the exact DSH skill catalog prompt and decides, for each test query, whether the target skill should be triggered. The plugin reports accuracy, precision, recall, false-positive/negative rates, and a confusion matrix — a reproducible measure of how reliably a skill description routes matching queries (and how often it over- or under-triggers). Install From the repo root: Then configure the judge model route in your profile/overlay cordis.patch.yml : The provider must be registered in the DSH LLM runtime (the same one your profile uses for chat). The plugin validates the route at startup and warns if the provider is not yet registered. Usage Slash command: Model-callable tool: test-file is optional; it defaults to examples/dsh-plugin-eval.json inside the plugin package. Relative paths resolve against the plugin package directory. Test-case format A JSON array of { query, should trigger } objects: category is optional and reserved for future use. How it works 1. Enumerate the session's model-invocable skills ( ctx.skills.snapshot ). 2. Recreate the official catalog message verbatim ( + +…

分类