AutoResearch Evidence Pack

01 / CONTEXT

The problem

A polished score can hide a broken input path, a distribution shift, repeated access to a public test set, or a competitive search baseline. Credible application evidence must expose those failure modes and its authorship limits.

02 / OWNERSHIP

My contribution

Requested an AI-assisted effort to strengthen the application evidence and preserve the protocols, artifacts, corrections, and negative results for review.
The experiments, analysis, and copy were produced with AI assistance; personal understanding, independent reruns, code explanation, and timed trial work remain to be demonstrated.

Evidence status: AI-assisted execution; personal independent rerun and explanation remain pending.

03 / ENGINEERING JUDGMENT

Decisions & tradeoffs

Withdraw results built on a broken tokenizer path

The choice: After an audit found a zero-merge tokenizer backend and mismatched encodings, withdraw the earlier scores, switch to a verified tokenizer path, and retrain from scratch.
The tradeoff: Correcting the input invalidated a more convenient earlier result, but preserved the integrity of every later comparison.
How it is checked: The corrected matrix contains 20 formal training runs plus one from-scratch repeat, with tokenization, predictions, checkpoints, and selection records checked in the full evidence package.

Test in-domain success against a word-order stress set

The choice: Evaluate the six fixed 12-epoch full/LoRA checkpoints on PAWS only after MRPC model selection, and report balanced accuracy and negative-class recall alongside F1.
The tradeoff: Twelve-epoch LoRA averaged 0.9074 F1 on MRPC, but 0.5009 balanced accuracy on PAWS was near the constant-classifier baseline; the public stress test is exploratory, not a blind confirmation.
How it is checked: The proof PDF summarizes the result; the LoRA repository holds the tables, correction, error analysis and audit scope. Its reviewer snapshot has a passing public CI audit.

Keep the agent result even when preset search wins

The choice: Run three fresh-context agent trajectories with six development calls each, freeze their selections, then compare them with the preset six-point search on the public NFCorpus test split.
The tradeoff: All three trajectories chose the same configuration and scored 0.307044, below the preset search at 0.307294. They produce one unique test ranking, not three independent generalization results.
How it is checked: The public research-agent repository contains the task code, hypotheses, six-call logs, selection lock, reports, and a source-data rebuild route. Its r2 snapshot adds a paper-to-task index and a CI-tested offline Docker audit of traces and a synthetic demo. The container check does not rerun NFCorpus or demonstrate the applicant's personal Docker skill. Earlier access to the public test makes this an exploratory extension.

What the work shows

The package records a corrected 20-run LoRA matrix plus one repeat, a near-chance PAWS stress result, and three fresh six-call agent trajectories that did not beat the preset search. Its strongest signal is the preserved correction and negative evidence, not a claim that either method won.

FROM CLAIM TO SOURCE

Follow the evidence

What this does not establish

AI-assisted execution and documentation do not establish the applicant's independent research or coding ability; that still requires personal reruns, explanation, and live trial work.
The post-release replay is one AI-agent-run, same-machine repeat of one frozen configuration using retained assets. It is not a fresh asset download, a cross-machine test, a full 21-run reproduction, or proof that the applicant personally trained or debugged the model.
MRPC validation, PAWS labeled test, and the NFCorpus public test were used in an exploratory setting. The results are not a blind benchmark, a general LoRA verdict, or evidence that research agents outperform search.
The one-page PDF and application addendum alone omit complete raw predictions, model weights and rank caches. The linked repositories expose source, logs and aggregate results; third-party data and weights require separate upstream access. Personal reruns remain pending.
The experiments cover one language-model backbone, two sentence-pair datasets, one retrieval task, and small numbers of training seeds and agent trajectories.