Benchmark snapshot
DeepSeek-only diagnostic benchmark
Turning failed evals into an agent reliability map.
OpenCode Harness ran DeepSeek through smoke, long-context, and repair suites. The results are intentionally published with failures because the point is diagnosis, not a polished leaderboard.
What the harness exposed
The score is less important than the failure signature.
Marker-following drift
Cases missed required completion markers such as ZH_MODEL_SUMMARY or reached the step limit before closing.
Tool-loop overrun
Several summaries ended with intent to inspect more files instead of finishing the requested answer.
Long-context synthesis gap
The focused safety explanation passed, while broader repository mapping and eval-flow synthesis were less stable.
Repair finalization gap
Repair cases did not reliably close with the expected markers after editing copied fixture workspaces.
Public evidence
Reports are committed. Raw traces stay local.
| Suite | Passing Case | Main Failures | Report |
|---|---|---|---|
| Smoke | patch-proposal-no-write |
expectation_mismatch=3 |
Smoke report |
| Long context | security-and-permissions-context |
expectation_mismatch=2, max_steps=1 |
Long-context report |
| Repair | None | expectation_mismatch=2 |
Repair report |
.\scripts\run-deepseek-benchmark.ps1 -SuiteSet all
python -m opencode_harness diagnose eval-runs/path-to-run/report.json --output eval-runs/deepseek-diagnosis.md
python -m opencode_harness diagnose-compare --before eval-runs/before/report.json --after eval-runs/after/report.json --output eval-runs/deepseek-before-after.mdNext reliability work
Use the diagnosis to improve the agent loop.
The harness now includes finish-marker reminders, a final-step guard, trace-aware failure-mode diagnosis, before/after diagnosis comparison, and repair verifier feedback. The next iteration should rerun DeepSeek and publish the reliability delta.
Open Full Diagnosis