Case study · Live in production since July 2026
Hiring Automation
My company screened every inbound operations application by hand. I replaced that with a four-stage pipeline where a language model evaluates candidate work product against an explicit rubric and decides who advances.
The interesting part is not the automation. It is that the rubric is an eval set, the grader is the harness that runs it, and getting one right does not get you the other. A design decision that survived review broke in production six weeks later, because the harness misread its own test content as data.
01The four stages
-
Intake
Parses inbound applications from email, extracts structured fields, applies rule-based filters.
-
Questionnaire
Scores and summarizes free-response answers through the OpenAI API. Adds location and compensation gates.
-
Portal exercise
Candidates complete a simulated task in a school portal. GPT‑4o vision evaluates the screenshot against a rubric and returns a pass or fail with reasoning.
-
Spreadsheet grader
A server-side grader evaluates a spreadsheet submission against an explicit rubric, returns category-level feedback, and runs a two-round retry loop for candidates who can improve.
02The eval design
The core of stage four is not the code. It is the rubric an automated grader enforces, which is the same discipline as designing an eval set for any AI system.
Hard gates versus soft gates
I defined which criteria must pass for a candidate to advance, against which are recorded but not disqualifying. That boundary is what keeps the grader from rejecting good candidates or passing weak ones. Both failure directions are expensive, and they are not symmetric: a false reject is invisible and permanent, a false pass surfaces at interview.
A deliberate data-fidelity trap
The exercise seeds a flaw in the source data. Reading the spec against the grader's actual behavior surfaced a contradiction — the spec said to forgive the flaw, but the grader correctly rewards a candidate for catching it. Forgiving it would have rewarded the wrong behavior in a hiring exercise. Caught in the design document, before it reached code.
The trap broke the grader six weeks later, and that is the more useful half
The design decision held. The implementation did not. In August, the seeded flaw turned out to be splitting two lookup keys into separate buckets, so the grader read its own test content as real data and false-failed candidates who had done the work correctly.
Nothing in the design review would have caught it. It took working backward from a live candidate whose numbers were nine of nine correct and who still received a retry email.
Getting the rubric right is not the same as getting the grader right. An eval is only as good as the harness that runs it, and the harness needs its own adversarial pass.
Fixing the harm, not only the bug
A code fix does not un-reject anybody. After deploying, I swept the back catalog for grader-caused failures, rewrote the recorded reasons, manually advanced the confirmed false fail, and queued two candidates for re-grade under the corrected logic. I verified with a negative fixture — a candidate who never catches the seeded flaw still fails — confirming the fix did not quietly lower the bar.
Calibration under a real cost function
- The location gate was set too aggressively at first, then recalibrated after I confirmed East Coast candidates were still viable.
- Raising the compensation gate exposed a parser that had silently defeated every threshold it ever had. The old helper mis-read comma and abbreviated formats, so a six-figure answer parsed as zero. Every high earner had passed every threshold since the gate existed. I wrote a replacement with 14 tests and verified it against production.
- Feedback categories expanded from 3 to 12, each tied to a specific instruction step, after the coarse version produced feedback candidates could not act on.
03Silent-failure QA
The recurring theme across four months. Each of these fails by doing nothing, so no alert fires and no log line appears.
| Failure | What it would have cost |
|---|---|
| Retry pool never fires unless timestamp columns are datetime-formatted | Every candidate's retry silently vanishes |
| A standalone script cannot install a form-submit trigger from the UI | The same, caught on deploy day |
| Uncatchable permission error on files not shared with the service identity | Real runs marked "failed" in production |
| Dev running a stale build missing the core resolver | Testing a system that is not the one shipping |
| Dev and prod sharing one hard-coded live spreadsheet | The grader acts on fake candidates the moment it goes live |
| Name-search fallback matching the bot's own alert emails | It replies to itself, marks the row processed, and the candidate is never retried |
| Form response limits binding to an account while the pipeline keys on a typed address | One candidate produces two rows and gets double-emailed |
| A questionnaire accepting free text where the pipeline needs numbers | A qualified answer parses as NaN and auto-rejects |
| Reply recipients resolved from our own last message in the thread | Every candidate-facing reply addressed to the wrong recipient — correct content, wrong envelope, no error |
The last one shipped with form validation plus a hold-for-human path that flags unreadable answers instead of guessing, followed by a sweep of past responses for false rejections. Validation stops the next bad input. Refusing to guess stops the pipeline from making a decision it has no basis for.
04Scope discipline
Two decisions where the sophisticated version was worse, not merely more expensive.
Killed a fuzzy-matching reconciliation mid-build
School-name matching had grown overlap scoring, edit distance, and tie-breaking. I replaced it with a ten-line rule that acts only when exactly one name is unaccounted for on each side. The simple version had the stronger safety argument: fuzzy matching can silently pair the wrong two schools, while a one-to-one rule either has an unambiguous answer or does nothing. Correctness argued the same way as cost.
Parked a six-file remediation plan
For a bug seen once in months, I shipped the four-line piece instead — which also fixed an unrelated class of rejected threads. The larger plan stays available if the failure recurs.
05Engineering practice
- Full spec → plan → build → review → deploy loop, with per-task and whole-branch code review.
- The plan caught what the spec missed: a spec-versus-grader contradiction, a second retry round, and a feedback token added at plan stage.
- Safe deployment — diff live against local before pushing, draft-mode burn-in before flipping the flag, and verify every fixture in production under the correct service identity rather than trusting dev.
- Local development under version control with clasp and git, adopted as the team standard.
- Bounded automation scope — new-template submissions auto-grade, legacy submissions stay manual by design. Making the boundary explicit prevents the grader from acting on input it cannot correctly parse.
06What it demonstrates
End-to-end ownership from spec to production. Eval-set design and calibration. Judgment about silent failures — the class of bug that throws no error and writes no log line. Willingness to kill my own sophisticated build for a simpler one with a better safety argument. And the habit of fixing the harm a bug caused, not only the code that caused it.