Every AI pipeline that runs without a human watching eventually produces something wrong. The model has an off day, an upstream source changes shape, an instruction gets misread — and the pipeline keeps going, because nothing in it knows the output is bad. The fix is not a smarter model. It is a dumber component: a machine gate, a small script that checks every output against an explicit contract and refuses to pass what fails.
We run gates on every automated pipeline we operate, including the one that publishes this site. This guide is the full method: what to check, how to wire it, and the two failure modes that turn gates into decoration. Everything here is model-agnostic and survives every model upgrade — which is precisely the point.
Why a script, when a model could check?
Using a second AI to review the first AI’s output is popular, and it has real uses — but it is a reviewer, not a gate. A reviewer has judgment and, with it, variance: the same output can pass on Tuesday and fail on Thursday. A gate is deterministic. Same input, same verdict, every time, and when it fails you can read the code and know exactly why.
The deeper reason is failure correlation. The conditions that make your generator produce garbage — a weird input, an ambiguous instruction — are often the same conditions that confuse a model-based checker. A script does not share the model’s blind spots. Its stupidity is the feature: it cannot be argued into leniency by fluent prose. Use model reviewers for quality judgment; use script gates as the floor beneath everything. The floor is what this guide builds.
Step 1: Write the contract before the checks
A gate enforces a contract, so the contract comes first. Write down what every valid output must satisfy, in terms a script can test. Good contract lines come in four families:
- Existence and freshness — the file exists, in the right place, dated from this run, not a stale survivor of yesterday’s success.
- Structure — required sections present, parses as valid JSON or frontmatter, within expected size bounds. Minimum length is the workhorse: quality failures in AI systems very often arrive as short output, because a confused model produces less, not more.
- Forbidden content — patterns that must never ship: placeholder text like TODO or lorem, empty links, leaked prompt fragments, banned claims. Each entry usually memorializes a real past incident, which is what makes this list valuable.
- Cross-references — every internal link resolves, every referenced file exists, counts match between summary and detail. This family catches the “the reference is there but the thing is not” class of rot that looks fine until a reader clicks.
Resist the urge to encode taste (“the writing should be clear”) — a gate checks facts about the artifact, and taste belongs to the model reviewer above the floor. If you cannot express a rule as a check that either passes or fails, it does not go in the contract.
Step 2: Wire it so failure blocks, loudly
A gate that logs a warning and lets the pipeline continue is not a gate. The wiring rule: the gate runs after generation and before anything irreversible — publishing, sending, deploying — and a failure must physically stop that next step, not merely note an objection. In CI this is an exit code failing the job; in a scheduled pipeline it is the output staying out of the “approved” location so downstream steps find nothing to ship.
Loudness is the other half. Report every violation with file, line, and a message that says what to fix — a gate that says “FAIL” without saying where teaches people to bypass it. And report all violations in one run rather than dying at the first, because fix-rerun-fix-rerun loops make people resent the gate, and resented gates get deleted.
Step 3: Break it on purpose — the step everyone skips
Here is the trap that makes this section the most important one: a gate can fail silently in the passing direction. A typo in a regular expression, a path that no longer matches anything, a condition inverted during a refactor — and the gate begins approving everything, while every report still says all checks passed. Nothing looks more like a healthy pipeline than a broken gate.
We know because it happened to us: a one-character escaping bug turned one of our checks into a rubber stamp that had never once fired. We found it only because of the practice this step prescribes — negative testing. For every check, construct an input that must fail it, and verify the gate actually fails. Feed it the empty file, the stale date, the document with a forbidden phrase, the link pointing nowhere. Watch each one die. A check you have never seen fail is unverified code guarding your reputation.
Then keep the breakage as a permanent regression suite that runs whenever the gate changes, and make the suite import the same functions the production gate runs — test a copy, and the copy is what your tests certify while production drifts. Finally, apply fail-closed to the gate itself: if it crashes or cannot read its input, that run is a failure, not a pass. A gate that shrugs on error lies at the worst possible moment, and the “no news is good news” pipeline design is how broken gates survive for months.
Step 4: Grow it by autopsy
Do not build the grand gate up front — over-engineered contracts block legitimate output, and every false rejection burns trust. Start with existence, freshness, length, and one forbidden-pattern list. Then let incidents write the rest: every bad output that reaches a human becomes a new check, added the same day, with its negative test. Each check earns its place by pointing at a real failure, the contract stays honest, and the gate compounds — the same property that makes it worth building at all. After a few months you will trust the boring green line, which is the entire product: attention spent only where something is genuinely wrong.