xiaozhengdeng/dsh_omnivision

README excerpt

dsh omnivision OmniVision — OmniParser-powered GUI agent plugin for DeepSeek Harness English 中文 English OmniVision is an OmniParser-powered GUI agent plugin for DeepSeek Harness. It turns the desktop or any image into structured elements (text + icons with pixel coordinates), so the model can see and operate the screen without multimodal vision . Screenshots Vision view (智能识别) Recognition history (识别记录) Features Feature Description 🖥 Screen recognition Capture the desktop (or parse an image) and get interactive elements with pixel coordinates via OmniParser 🖱 Desktop automation Click, double-click, right-click, drag, type (unicode-safe), key, hotkey, scroll — by element id or raw x/y 👁 Live vision dock Real-time recognition view with SOM-annotated overlay, hover highlight, click-to-zoom 🕘 Recognition history Thumbnails per capture, diff vs latest (added / removed / moved) 📋 One-click summary Sends the recognized elements to the session for the model to summarize 🖼 File parsing Parse local images from the dock, bypassing model multimodal limits 📊 Call log Tracks every gui tool call and action for inspection Tools The plugin registers these gui tools into the shared tools regi…

Category