harness-lab

harness-lab is an evaluation harness for DeepSeek Harness. It benchmarks agent loops, compaction, skills, and tool configurations on reproducible offline tasks. The project runs deterministic offline tasks through a minimal ReAct baseline and, in progress, DeepSeek Harness, then reports completion, tokens, cost, steps, and failure classes. The repository includes a DeepSeek V4 client with a token cost model, sandboxed workspace tools for list/read/write/shell, a ReAct baseline and DeepSeek Harness adapter preview, a task loader and disposable workspace factory, a deterministic grader runner, and aggregation and markdown reports. It provides 6 deterministic offline tasks across file-ops, data-clean, mini-code, and long-qa. Every task has an immutable fixture workspace, a deterministic grader, and per-task budgets. Each configuration runs N attempts and reports completion, pass@1, token usage, USD cost, steps, tool calls, and errors. The roadmap lists M1 task framework + ReAct baseline + first task suite, M2 DeepSeek Harness adapter (pinned preview version), M3 24 tasks, configuration matrix, failure classifier, and M4 Streamlit dashboard, benchmark report, demo. The project is licensed under MIT.