The problem
Evaluation becomes unreliable when reviewers interpret a criterion differently, rounding changes a verdict, or reference answers reach the model being evaluated.
My contribution
- Led the task-design requirements and selected the checks worth making explicit.
- Used AI-assisted drafting and implementation, then reviewed synthetic examples, scoring boundaries, and validation results.
Working approach: product-led, AI-assisted implementation, followed by review and iteration.
Decisions & tradeoffs
01
Write observable criteria before grading
- The choice
- Separate hard requirements from graded quality dimensions and define concrete scoring anchors.
- The tradeoff
- The template is one scoring design; a benchmark’s own rules still determine the rubric.
- How it is checked
- The rubric linter flags vague wording, and the synthetic task shows criteria beside an answer.
02
Represent tolerances as data
- The choice
- Use explicit absolute, relative, percentage-point, interval, or set specifications.
- The tradeoff
- A validator can enforce the stated band but cannot decide whether that band is appropriate for the task.
- How it is checked
- Passing, failing, and boundary cases exercise the tolerance checker.
03
Check the package as well as the answer
- The choice
- Declare agent and judge assets separately, then check required files and references together.
- The tradeoff
- Filename and table checks catch known packaging mistakes; they do not detect every semantic leak.
- How it is checked
- The package checker and synthetic example run in CI alongside the automated tests.
See the work
Explore the work
See why an answer passes or fails
SYNTH-SaaS-001 is fully fictional. This public teaching example shows judge material openly; real evaluation packages keep it separate from model inputs.
Build the revenue bridge
Calculate next-quarter monthly recurring revenue (MRR), identify exactly two cohorts driving the most churn in dollars, and state when expansion is booked.
| Cohort | MRR (USD) | Churn | Expansion |
|---|---|---|---|
| C1Enterprise | 1,800,000 | 0.03 | 0.02 |
| C2Growth | 1,200,000 | 0.04 | 0.025 |
| C3MidMarket | 900,000 | 0.08 | 0.03 |
| C4SMB Legacy | 700,000 | 0.14 | 0.01 |
| C5SMB New | 250,000 | 0.06 | 0.04 |
| C6Startup | 100,000 | 0.09 | 0.05 |
| C7Edu | 30,000 | 0.05 | 0.02 |
| C8Nonprofit | 20,000 | 0.04 | 0.01 |
What the work shows
The v0.2.0 release contains methodology chapters, reusable templates, three command-line checks, and a complete synthetic SaaS task. Its 17 automated tests run in CI on Python 3.9, 3.11, and 3.12.
Follow the evidence
What this does not establish
- All public examples are synthetic; client tasks, rubrics, and private feedback records are excluded.
- Passing local checks does not establish benchmark quality, judge calibration, or model capability.