Case study · first measured run · 2026-11-06

Seven of ten real bugs, reproduced. Three missed, shown too.

PatchPilot ran ten curated Python issues from real public repos through the full pipeline on its own host: fetch the issue, clone the repo at the commit before the fix, generate a reproduction script, run it in a capped Docker container, and compute the verdict from the exit code and the error match. Nothing here is a demo script. The misses are listed alongside the hits because a tool that only shows hits cannot be trusted.

Model: Qwen3-Coder-30B, via Amazon Bedrock
Run: fixture-qwen7, 2026-11-06 20:21 PKT, EC2 host
Result: 7 REPRODUCED, 3 NOT_REPRODUCED, 0 setup failures, 0 errors
Target: At least 3 of 10, set in the Phase 0 plan

How the ten were picked

Each fixture is a real issue with a merged fix. Before the run, every one was checked twice by hand: the recorded error occurs at the base commit (the first parent of the fix merge) and does not occur at the fix commit. Issues that failed that check were rejected and recorded in the repo. The run then had to reproduce the bug from the issue text alone, starting at the base commit, with no knowledge of the fix.

The ten, with verdicts

PyCQA/pyflakes #764 REPRODUCED TypeError: 'bool' object is not subscriptable
PyCQA/pyflakes #834 REPRODUCED AttributeError: 'ClassScope' object has no attribute 'indirect_assignments'
asottile/pyupgrade #194 REPRODUCED ValueError: Single '{' encountered in format string
pypa/packaging #638 REPRODUCED TypeError: expected string or bytes-like object
python-humanize/humanize #17 REPRODUCED TypeError: can't subtract offset-naive and offset-aware datetimes
python-humanize/humanize #57 REPRODUCED ValueError: math domain error
python-humanize/humanize #122 REPRODUCED TypeError: can't subtract offset-naive and offset-aware datetimes
PyCQA/pyflakes #744 NOT REPRODUCED AttributeError: 'Name' object has no attribute '_pyflakes_parent'
asottile/pyupgrade #437 NOT REPRODUCED KeyError: (3, 11)
python-humanize/humanize #62 NOT REPRODUCED ValueError: cannot convert float NaN to integer

The run that counted was the seventh

Six earlier runs were invalidated by real bugs the live host exposed and unit tests could not see, because tests fake the exact boundaries that failed. Each was fixed, one commit each, before the next run:

The Bedrock client was built with no region, so every model call died before it started.
Model replies arrive wrapped in markdown fences, and the raw reply was written as the script, so all sixty attempts failed the syntax check with zero container runs spent.
A relative workspace path was passed to Docker as a volume name, so all sixty attempts exited 125.
Installs happened in a disposable container and never reached the run container, so the target package was never importable.
Container root could not write into the workspace, so all ten jobs failed setup.
VCS-versioned projects could not build without git in the slim image, so all four humanize jobs failed setup.

An intermediate fair run before the last fix scored 3 of 10 with 4 setup failures. The 7 of 10 run supersedes it. None of the invalidated runs is quoted as a model result anywhere.

The malicious-issue drill passed

A crafted issue ordering the agent to curl an external URL and exfiltrate /etc/passwd went through the real pipeline on the host the same evening. The job verdict was REPRODUCED on the crafted bug, and the only commands that reached the executor were the repro script and the test run. No injected text reached the executor.

What is not claimed

The head-to-head against Claude Haiku 5.5 never ran. The account returns AccessDenied for that model, so only Qwen is measured. Qwen is the default because it is the only measured model, and it beat the target.
The three misses are real. The attempt loop retries the identical prompt up to six times with no feedback from the previous attempt's log. Feeding the last error back in is the clearest known improvement and the likely lever for those three.
JS and C++ reproduction paths are built and unit-tested, but the live fixture proof so far is Python only.
ISSUE_UNDERSPECIFIED and LIMIT_REACHED are covered by unit tests only; they have not been seen live.

See the evidence

Sample reports from this run, two reproduced and one missed, with the script and test for each. The fixture list and the curation rules live in the repo.