Stop Judging AI Agents on Vibes: Give Them an Acceptance Test

Most people judge their AI agents the same way: they read a few outputs, nod, and decide it feels good enough. That is not quality control. That is vibes.

Vibes are fine when you are babysitting every result. They fall apart the moment you want an agent to do real work without you hovering over it. If you cannot say, in concrete terms, what "good enough" means, you can never safely hand the agent the keys. So you stay the bottleneck forever, approving every draft, every email, every post.

I got tired of being that bottleneck. So I built my agents an acceptance test.

The idea: earn autonomy, do not assume it

I think about agent autonomy as a ladder. At the bottom, the AI just observes and reports. A rung up, it drafts and I approve. Higher still, it acts and I confirm. At the top, it runs unattended inside guardrails and only pulls me in on exceptions.

The mistake is letting an agent climb that ladder because it "seems" ready. The fix is to make every rung something it has to earn by passing a test.

I took the pattern from A2ASphere, a marketplace I built for a hackathon, where money only moved when the delivered work passed a hidden acceptance test. No human opinion, no disputes. The test passed or nobody got paid. That same mechanism works beautifully for autonomy: an agent climbs a rung only when it proves, against a real test, that it can.

What an eval gate actually is

The gate has two halves, and you need both.

The first half is hard rules. Deterministic checks that are pass or fail, with no model in the loop. For my content agent those are the brand rules I never want broken: no em dashes, no links buried in the body of a post, none of the tired AI-tell phrases. A check either finds a violation or it does not. Cheap, fast, and not up for debate.

The second half is judgment. Some quality is not a yes or no. Voice, clarity, whether the writing actually sounds like me. For that I use a separate model as a judge, scoring each candidate on voice and correctness. The judge never sees the rule checks, and the agent never sees the judge.

A run scores every test case, combines the hard rules with the judged scores, and only clears the gate if the aggregate beats a threshold. If a candidate trips a hard rule, it scores zero on the spot, and the judge never even gets a turn. A real violation should sink the result no matter how pretty the prose is.

The first time I ran it on myself

Theory is cheap, so I pointed the gate at the content-production function inside NxtOS, the agent operating system I use to run my business, and tried to promote it one rung up the ladder. Two runs, same gate, different inputs.

The first run included a deliberately bad draft: an em dash, a link stuffed in the body, and a "game-changer" for good measure. The gate scored that run 0.667 against a 0.85 bar and refused the promotion. The bad draft scored zero on the rule checks and dragged the whole batch under the line. Exactly what I wanted. The gate bit down on the brand risk and would not let it through.

The second run used clean drafts that actually followed my voice rules. It scored a 1.0 and earned the promotion. Both verdicts, the block and the pass, got logged to my observability stack with the scores attached, so there is a paper trail for every decision the gate makes.

That is the whole point. The agent did not get more freedom because I felt good about it. It got more freedom because it passed a test I wrote, and I can show you the score.

What broke, because something always does

Two honest dents, because a build log that only reports wins is marketing.

I tried to have a local model auto-write the test for me. It produced malformed garbage and even invented a random topic about urban gardening. My first instinct was to blame the model and reach for a bigger one. That instinct was wrong, and I only found out because I bothered to check.

The prompt was the problem. I had described the shape I wanted in prose instead of showing it, and I never told the model what the function actually does. So it guessed, and it guessed badly. I gave the same model a prompt with a worked example and fed it the function's own description, and it produced a valid, on-topic test on the first try. Same model, better instructions. The fix was my words, not its weights.

The one real save was the safety net. When the first attempt came back malformed, the system validated it, caught that it did not fit the required shape, and failed loud instead of quietly saving something broken. That is the part I actually care about. A tool that fails honestly beats a tool that guesses and hides it.

The judge was also generous. It handed my clean drafts a perfect score, which tells me the rubric is too soft and needs tightening before I trust the judged half as much as the rule half. Worth knowing now rather than later.

Why this is the difference that matters

An agent that drafts your work is useful. An agent you trust to ship it without you is a different animal entirely. The distance between those two is not a smarter model. It is a gate.

Build the gate, and autonomy becomes something measured and earned, with a number you can point at. Skip it, and you are stuck reviewing everything by hand, calling it judgment, and hoping the vibes hold.

I would rather have the test.