UQO Working Paper No. 29· v1.1· November 2026· 121 pages
Protocol-Guided Autonomous Research with Claude Code
A Controlled Comparison of Instruction Regimes at Ceiling
On 24 bounded scientific tasks that the agent already completes 0.903 of the time under a one-line instruction, a full research protocol changes nothing detectable in correctness (+0.014, interval [−0.028, +0.056]) and costs 1.7 to 2.6 times more per run.
288 live Claude Code runs compare four instruction regimes with the agent, tools, inputs and budget held fixed and hidden oracles on a separate host. The suite is at ceiling, so the test has no power against plausible effects. A launcher defect truncated every user prompt, so the gate-feedback and review-repair mechanisms were not delivered in the frozen runs; a replication with the corrected launcher delivered them and found they had no occasion to act.
Simon-Pierre Boucher — contact@spboucher.ai
0.903 vs 0.917
Intervention-free verified completion under the open instruction and under the full protocol text, 72 runs per arm on 24 tasks
+0.014 [−0.028, +0.056]
Paired task-level difference, task-cluster bootstrap 95 % interval (exact sign-flip inversion [−0.033, +0.060]); only 3 of 24 tasks are discordant, so the preregistered test cannot go below p = 0.25
0.000
Difference on the 20 tasks whose oracle rules are uncontested: the arms tie exactly
0.24 → 0.64 USD
Mean inference cost per run from the open instruction to the full regime (client list-price estimate), the only endpoint that moves: +0.398 USD [+0.357, +0.441]
0 vs 7 and 9
Deliverables failing the post-hoc contract check: none in the two gated arms, 7 under the open instruction and 9 under the protocol alone
0 of 144, 72 of 72
Replication with the corrected launcher: runs in which the controller gate failed its first check, and reviewer verdicts that were accept; the mechanisms were delivered and had no occasion to act
25 on 4 tasks
Frozen non-successes, all on four tasks whose oracle rules the author contests after unblinding; under the strict post-hoc rubric completion is 0.958, 0.958, 0.958 and 0.944, and the null is unchanged
Abstract
Instruction files that steer coding agents through research tasks range from one sentence to several thousand words of phases, gates and integrity rules. We ask whether that structure changes what one agent delivers on bounded scientific tasks, and at what cost. In 288 live Claude Code runs (client 2.1.314, model claude-opus-5-5) on 24 tasks in six families, 3 fresh sessions per task received one of four regimes injected through the system prompt: an open instruction; a structured protocol; the protocol plus an instruction to run an executable, oracle-free gate; and the same plus a separate reviewer session. Deterministic hidden oracles on a separate host scored every run; design, tasks and analysis were frozen with hashes before the confirmatory runs. A launcher defect (amendment A-006) truncated every user prompt to one word, so the gate-feedback and review-repair mechanisms were not delivered in any frozen run. Intervention-free verified completion was 0.903 under the open instruction and 0.917 under the full text; the paired task-level difference is +0.014 (task-cluster bootstrap 95% interval [-0.028, +0.056]; exact sign-flip inversion [-0.033, +0.060]), with 3 of 24 tasks discordant, so the preregistered test could not reject at any effect compatible with the observed base rate: no detectable difference at ceiling. Integrity violations did not differ either. The protocol text bought full compliance with the deliverable contract at 1.7 to 2.6 times the inference cost per run. A quality-control replication of the two gated arms with the corrected launcher (144 confirmatory and 12 transfer runs, 2026-11-09, 93.05 USD) delivered the gate-feedback and review-repair prompts in full, left every frozen number unchanged and reproduced the frozen pattern (Crep − C = +0.000, 24/24 ties; Drep − D = −0.014, one run below on a contested task); the controller gate passed at its first check in every confirmatory run and every reviewer verdict was accept, so the mechanisms had no occasion to act (Drep − A = +0.000 [−0.042, 0.042], 22/24 ties). The 25 frozen non-successes lie on four tasks whose oracle rules we contest after unblinding; under a versioned post-hoc rubric the null is unchanged. We read the result as evidence that, on tasks at ceiling, what limits delegated research is the oracle rather than the organisation of the work; that reading is an interpretation, not a measured endpoint.
JEL: C18, C63, C88, O33
Key Results
A delegation-contract formalisation and reference implementation
A small, auditable harness (controller, durable state, isolated bare-mode executors, oracle-free runtime gate, separate review, deterministic hidden evaluators) in which every claimed invariant is stated with what its checker actually verifies.
A benchmark of bounded scientific tasks with hidden oracles
24 confirmatory and 7 development tasks with seeded validation sets and provenance for 14 real-data and 5 source-document tasks, plus three transfer mini-projects, a success-side audit of the oracles and a versioned post-hoc rubric.
A controlled four-arm comparison at ceiling
Instruction regimes compared with the executing agent, tools, inputs and budget held fixed and the design frozen with hashes. The result is no detectable difference in completion or integrity; the paper states that the suite cannot detect a benefit or a harm of a few percentage points.
Structure moves compliance and cost, not correctness
The protocol text bought full compliance with the deliverable contract at 1.7 to 2.6 times the inference cost per run, in line with controlled comparisons on software-engineering tasks that the paper cites as closest prior work.
A funded replication of the gated arms
144 confirmatory and 12 transfer runs with the corrected launcher, reported separately and never pooled with the frozen sample: the frozen pattern is reproduced and the hypotheses about gates and review remain untested in substance.
A reflexive account
An instrumented description of how the paper itself was produced under a mandate of the same kind, kept separate from the confirmatory evidence, including the harness defects found along the way.
Quality control
This paper went through a multi-stage quality control: 7 quality-control reports and 85 points checked. The version presented on this page is the revised manuscript. Scores are on a 0–100 scale and are a diagnostic of the manuscript, not an acceptance probability.
Score after revision
73.8/100
64.8 before quality control · +9.0
Quality control dated November 9, 2026
- Quality-control decision
- claims not supported as written → reframed and revised
- Scientific integrity
- PASS
- Reproducibility
- R1reproduced from documented analysis data
- Maturity
- Bone revision cycle from submission
- Quality-control reports
- 7
- Points checked
- 85
Scores by dimension
Before quality controlAfter revision
- Originality & contribution69.1 → 66.0
- Scientific rigor59.9 → 74.0
- Identification & methodology56.3 → 66.0
- Data & reproducibility79.0 → 88.0
- Robustness & validation59.1 → 74.0
- Writing67.3 → 75.0
- Figures & tables64.6 → 84.0
- Literature positioning68.0 → 80.0
Revised manuscript121 pages8 figures34 tables3 equations119 references46,292 words
What quality control changed
Quality control found that a launcher defect had truncated every user prompt, so the gate and review mechanisms the paper claimed to test were never delivered, and that the task suite was at ceiling. The paper was reframed from a precise null on protocols to no detectable difference between instruction texts at ceiling, and the author funded a replication with the corrected launcher that is reported separately.
- Subtitle "Scientific Verification, Reproducibility, and the Cost of Delegation: A Controlled Study" → "A Controlled Comparison of Instruction Regimes at Ceiling"
- "A precise null" → "no detectable difference at ceiling": with 3 discordant tasks the preregistered test cannot reject below p = 0.25, and the exact sign-flip inversion set [−0.033, +0.060] is reported next to the bootstrap interval
- Gated and reviewed arms described as executable gates and review → described as delivered: texts only, with 0 gate-feedback and 0 repair sessions in the 288 frozen runs
- Added: replication of the two gated arms with the corrected launcher (144 confirmatory and 12 transfer runs, 93.05 USD); the gate never failed its first check and every reviewer verdict was accept, giving a difference of +0.000 [−0.042, 0.042] against the open instruction
- "Every non-success was an oracle defect" → the 25 non-successes lie on four tasks whose oracle rules the author contests after unblinding; frozen scoring stays primary and a versioned rubric is a labelled sensitivity (strict 0.958, 0.958, 0.958, 0.944)
- "Two to three times the cost" → 1.7 to 2.6 times the inference cost per run, with intervals on the cost contrasts; post-freeze edits to the frozen manifests are now ledgered
Limitations stated in the paper
- The suite is at ceiling for this model, so the comparison is uninformative about effects of a few percentage points; a suite calibrated below the ceiling was not funded and remains future work.
- The hypotheses about gates and review are untested: the mechanisms were not delivered in the frozen runs and were inert in the replication.
- Tasks, oracles, validation sets and treatments were authored within one organisation with no human check of any reference value, and the adjudication of the four contested oracle rules was made after unblinding; a blind adjudication and a hand verification were not performed.
- The estimand covers one model, one client version, one resource envelope and a provider state of a few hours; nothing licenses statements about other models or about research in general.
Data
Frozen confirmatory runs
288 live runs (362 sessions) of Claude Code 2.1.314 with claude-opus-5-5 in bare mode, 3 fresh sessions per task and regime, under an envelope of 4.00 USD, 1,500 s and 80 turns per session, executed on 2026-10-30 between 03:32 and 06:05 UTC.
Task suite
24 tasks in six families: data audit, estimation and inference, forecasting under information constraints, source-grounded synthesis, theory and numerical validation, and integrated mini-projects. Tasks, oracles and treatments were authored within one organisation, with no human check of any reference value.
Replication runs
144 confirmatory and 12 transfer runs of the two gated arms on 2026-11-09 with the corrected launcher, for 93.05 USD of inference, against 171.76 USD for the original sessions.
Corpus of researcher-written mandates
Eleven distinct instruction files previously issued by the author for earlier working papers, from 1,092 to 8,809 words, coded in a single pass to motivate the protocol arms; one author's practice, not evidence about mandates in general.
Methodology
Four instruction regimes through the system prompt
An open instruction; a structured protocol with phases, ledgers, stopping and recovery rules; the protocol plus an executable, oracle-free gate; and the same plus a separate reviewer session. As delivered in the frozen runs the gated arms are texts only: 0 gate-feedback and 0 repair sessions in 288 runs.
Hidden deterministic oracles on a separate host
Executors ran on nodes that held no copy of the reference answers; every run was scored by a task-specific evaluator checked before the freeze against seeded correct and incorrect artefacts.
Preregistered analysis at the task-cluster level
The primary endpoint is intervention-free verified completion within a shared resource envelope and the primary contrast is the full regime against the open instruction, with task-cluster bootstrap intervals, sign-flip permutation tests over the 24 task clusters and an exact inversion set.
Frozen scoring primary, rubric as sensitivity
The frozen oracle scoring is kept as primary; a versioned post-hoc rubric in strict and permissive variants re-scores the four contested rules as a labelled sensitivity analysis.
Controls
A length-matched unstructured instruction, interruption-and-resumption experiments, stale-input, duplicate-job and missing-fixture faults, and three held-out end-to-end mini-projects.
Reproducibility
- Every estimate, table, figure and number of the paper is regenerated from the committed state store and run directories; a clean-copy replay on a second host found every regenerable estimate file identical byte for byte.
- The repository holds the run records, deliverables, gate receipts, reviews, evaluations, 478 session transcripts, the task bundles, the protocol documents with their freeze manifests and the analysis code.
- The hidden oracles stay outside the repository, hash-listed and available for verification on request, so a third party can replay every number but cannot re-execute the scoring.
- A fresh live replication needs an API key, executor hosts and the oracles; hosted-model stochasticity is not seedable, so it reproduces the procedure, not the realised outcomes.
- Infrastructure tests and the seeded evaluator validations need no provider access.
Cite this paper
Boucher, S.-P. (2026). Protocol-Guided Autonomous Research with Claude Code. UQO Working Paper No. 29, Université du Québec en Outaouais.
@techreport{boucher2026wp29,
author = {Simon-Pierre Boucher},
title = {Protocol-Guided Autonomous Research with Claude Code},
institution = {Université du Québec en Outaouais, Département des sciences administratives},
number = {29},
type = {UQO Working Paper},
year = {2026},
month = {nov},
url = {https://www.spboucher.ai/research/wp29},
}