Clean-room coding-agent infrastructure

Compare coding agents across DeepSeek, Qwen, Claude, OpenAI, and local models.

OpenCode Harness runs the same coding-agent workflow through a shared agent loop, permissioned tools, MCP extension points, JSONL traces, reproducible eval suites, trace-aware failure-mode diagnosis, and before/after reliability comparison.

View on GitHub v0.1.0 Release

Explore

Why it exists

Agent demos are easy. Comparing agents is hard.

Coding agents often ship as model-specific demos with hidden prompts, inconsistent tools, and weak audit trails. OpenCode Harness focuses on the runtime and evaluation layer: one task surface, many providers, reproducible traces, and reports you can inspect.

What is inside

Built like agent infrastructure, not a toy prompt wrapper.

Model-neutral runtime

Provider presets for DeepSeek, Qwen, Claude, OpenAI, local OpenAI-compatible endpoints, vLLM, SGLang, Ollama, and mock mode.

Permissioned tools

File reads/writes, search, patches, shell, git diff, repo maps, context packs, todos, and finish events with conservative policy gates.

MCP-compatible extensions

Stdio MCP tools, resources, prompts, per-server approvals, lifecycle diagnostics, and namespace collision handling.

Traceable evals

JSONL traces, provider transcripts, replay summaries, Markdown/HTML reports, trace-aware diagnosis, before/after comparisons, terminal trace viewer, and eval dashboard.

75-second demo

See the harness story before reading the docs.

Product surface

Run an eval, inspect the trace, publish the dashboard.

OpenCode Harness eval dashboard showing model pass rates and reports — Eval dashboard for comparing runs and failure types.

OpenCode Harness trace viewer showing model responses and tool calls — Trace viewer for auditing model responses, tool calls, and transcripts.

python -m opencode_harness eval examples/mock-suite.json --preset mock --max-steps 2
python -m opencode_harness tui runs/latest.jsonl
python -m opencode_harness dashboard eval-runs --output eval-runs/dashboard.html
python -m opencode_harness diagnose eval-runs/path-to-run/report.json --output eval-runs/diagnosis.md
python -m opencode_harness diagnose-compare --before eval-runs/before/report.json --after eval-runs/after/report.json --output eval-runs/before-after.md

Model Labs

One harness, multiple model families.

DeepSeek Qwen Claude OpenAI Local Models

DeepSeek case study

Failed evals become a reliability map.

The first DeepSeek-only benchmark set is published as a diagnostic artifact: smoke 1/4, long-context 1/4, and repair 0/2. The value is the failure signature: marker drift, tool-loop overrun, long-context synthesis gaps, and repair finalization gaps.

Open DeepSeek Case

v0.1.0 released

Open-source, packaged, and ready for benchmark runs.

The v0.1.0 release includes wheel and source artifacts, GitHub Actions CI, a release workflow, a model-eval workflow example, provider benchmark guide, changelog, and a reproducible local demo flow.

Download v0.1.0