Learn · Article
Give the Agent an Exam Before You Give It a Promotion
Historical exam suite → baseline → change → re-run → ship only when you can measure what improved and what broke.
Your helper already works.
It drafts follow-ups. It parks money guesses. It waits for your yes before anything leaves the shop.
Then someone offers a new model. Or a sharper prompt. Or a new skill. Or more access to inbox and files.
That is a promotion — more responsibility for a seat that already has keys.
Give the agent an exam before you give it a promotion.
Steal the spine from @Michael_Fenech_ (source): build a test set from real historical decisions with known correct outcomes → baseline how today’s production bot scores → change the model, prompt, skill, or decision layer → re-run the same suite → ship only when you can see what improved and what broke. Swap *change → production → wait → discover problems* for *change → test → measure → improve → ship*.
This sits next to Friday’s sister piece — *Don’t Trust the Charter. Trust the Failure Tests.* (https://x.com/SideShopAi/status/2103518668248215845). Failure tests prove the seat before first go-live. This piece proves the change after the seat is already live. Sister habit, not a remake. Different door. Same desk discipline.
Why vibes-based upgrades fail
Small shops feel it when a “smarter” follow-up bot starts inventing ship dates it used to leave blank. Individuals feel it when a new writer model gets wordier after a prompt tweak that “felt” clearer. Everyday people feel it when more calendar access turns a careful drafts-only helper into something that almost sends.
The upgrade looked like progress. The score was vibes. Production became the exam.
That is expensive. You pay in broken drafts, quiet meter burn from clean-up loops, and trust you have to rebuild. A promotion without a gate is just hope with extra tools.
Run the promotion exam
Treat every upgrade like a desk exam the seat has to keep passing:
1. Pull real historical decisions.
Last month’s tickets, follow-ups, refunds, schedule asks — cases the bot already faced. Plain pile. No invented demos.
2. Define the correct outcomes.
What should have happened: draft only, ask before inventing a number, park the irreversible move, use the house refund window. Known answers, not “good enough vibes.”
3. Turn them into a repeatable suite.
Same cases. Same grading. You can re-run it next Tuesday without rewriting the story.
4. Baseline current production.
Score the bot that is live *today* before you touch anything. You need a before, or “better” is theater.
5. Make one change.
New model, prompt edit, new skill, or more access — one layer at a time when you can. Name what you changed.
6. Re-run the exact same suite.
Same historical cases. Same correct outcomes. No moving the goalposts because the new version “sounds smart.”
7. Ship only when you can see improve / broke.
What got better. What got worse. What stayed the same. If you cannot name the break, you are not ready to promote.
Same seat. Different gate. Exam-driven desk — not vibes-driven launch day.
Everyday example: the inbox follow-up upgrade
Before the exam
Your shop follow-up bot is boring in a good way. Drafts only. Never invents a price. Stops when you say stop. A new model drops. You swap it in because the demos look sharp. By Thursday it fills thin threads with confident dates, and a draft that used to wait for you starts feeling “done enough.” You spend the afternoon correcting live customers. The promotion happened in production. The exam never ran.
After the exam
Same upgrade itch. First you pull twenty real past threads with known correct outcomes — when to ask, when to leave a blank, when to park. You score today’s bot. Then you swap the model (or the prompt, or the skill). You re-run the same twenty. You see: tone got cleaner; two money-edge cases got worse; one “ask before send” case flipped. You fix the broke lines, re-run, and only then open the live mailbox to the new version.
Same promotion. Different door. One version discovers problems on customers. The other measures first.
A “fix” that never hits the exam
If a chat correction never lands where the next run will read it, you already know that habit from *Correcting Your Bot Is Not Teaching It.* A “fix” that never touches the historical exam suite is the same wrong-home move — you feel responsible in the moment, and the desk still has no lasting gate. Teach the house. Then make the house pass the exam before you promote it.
Where Grok Bot fits
Week-one setup is not “swap the model and hope Tuesday is quieter.”
It is: name the job, keep the stop lines, build a small historical exam from real decisions, baseline the live seat, change one layer, re-run, ship only when improve and broke are visible. Grok Bot is a solid home for that habit — named helpers, drafts before sends, skills and routines you can version, and a human exit when something can leave the shop. Soft start when it fits: get on Grok Bot from about $20/mo (confirm live plan pricing before you treat that number as fixed).
Setup over spectacle. Fail the seat safely before go-live. Exam the change before you promote.
Imitate the historical suite. Iterate until improve and broke are boringly clear. Innovate only after the promotion earned the score.
One line to steal
Give the agent an exam before you give it a promotion.
Historical decisions. Known outcomes. Baseline. Change. Same suite. Ship only when you can see what improved and what broke.
Your side shop for AI that actually helps the business run.