At Flip Education, the AI product for lesson planning that I co-founded and ran with one other person and a team of AI agents, I set a simple rule. No AI model goes into production without a written test on our own work, run through the real product, with the pass rule fixed before anyone sees a result. Twice, the model that ranked higher in public lost on our own lessons. That mattered, because one model writes every lesson in 11 languages.
AI operations · Flip Education · Mar to Sep 2026
All work- About a ninththe cost of each page check, after a cheaper model scored higher, making checks in all 11 languages affordable
- About halfas many lessons in other languages going out with quality errors, after the tested switch
Why this matters now
The number of AI models has exploded, each good at different things. It also means businesses probably pay too much for their models right now, or use too little of them, or both. We saw it ourselves. We once tested models for an automatic checker that reads our web pages before they go live. The cheapest model scored higher than our default model, which cost about nine times as much per check, and at the cheaper cost checking every changed page in every language became affordable. So we built a way of choosing. Many jobs in the company, each with its own model. A full test on our own work that a model must pass before it goes into production. People, who keep the judgment calls, taste included. And a daily check that warns when a vendor retires a model.
What a switch used to cost
In the previous world, changing a model was changing a team. Weeks to months, with real risk of failure. Now a tested switch took about two and a half hours, because the test lessons already existed from an earlier test. A switch this fast is tempting to make on strong public rankings alone.
The challenge
We ran many jobs on AI models, and each job needed the right one. The job that mattered most was the model that writes every lesson, because it sets the quality, the speed and the cost of every lesson in every language, including the printable handouts. In August, Google's own page marked that model as being retired. Staying on it wasn't an option, and a wrong switch would show up in every lesson, in every country, at once.
How a full test runs
First, I decide how big the check must be. Routine work gets a lighter check, so it stays cheap; anything going into production, changing an important workflow or meant to last gets a full test, a written test that runs the same fixed set of our own work through each model and scores it. Then I ask the question and widen it, because a model can improve English and harm the other languages or the handouts. Before anything runs, we write down the pass rule, what a model must do to pass, and I approve a spending limit, so the bar can't move to fit the results. After that, AI agents run the same lessons through the real product once per model, so the model is the only thing that changes. An automatic check counts the errors and an AI judge, a model that scores the work, grades each lesson. The result is read against the pass rule, quality first, speed second, cost third, and I give the final go. After the switch, we measure again on real use.
Me
AI agents
Automatic checks
I decide how big the check must be
Routine work gets a lighter check, so it stays cheap. Anything going into production, or meant to last, gets this full test.
Model choice method
I ask the question and widen it
Every language and the handouts, not only English. That caught a cheaper model putting English role cards into Portuguese and Spanish test lessons, before any user saw them.
Before anything runs, we write down the pass rule and I approve a spending limit
Fixed first, it can't move to fit the results, and the test's cost is known before it starts.
AI agents run the same set of our own lessons through the real product, once for each model
So the model is the only thing that changes.
Claude Code
An automatic check counts the errors, and an AI judge scores the work
The model that writes never grades its own work, so the measure can't move with the work it measures.
scripts
The pass rule decides, and I give the final go
Quality first, speed second, cost third
A model that fails is held back, however high it ranks in public.
The switch goes live through the normal release
The product's analytics check the result again on real use
So the test's result is checked on real lessons. A daily check also warns when a model we use is being retired.
PostHogscripts
The test lessons are kept and reused, so every later switch is quicker to test.
Human decisions
-
Could
Switch to a newer model on the strength of public rankings and a much faster start, or simply wait for a bigger model from Google.
ChoseHold it back. It failed the pass rule written before the test ran.
BecauseIt made nearly twice as many errors per lesson, and the AI judge couldn't separate the two models, so the counted errors decided.
-
Could
Stay on the cheaper old model until Google retired it.
ChoseA newer model at a low setting, at about 35% more cost per lesson.
BecauseIt halved the errors per lesson and made lessons about a third faster. Quality first, speed second, cost third.
What changed on real lessons
Lessons in languages other than English went out with quality errors about half as often. A quality error means a lesson still broke one of our written quality rules after the automatic repair, so it went out with a warning. Handouts came out fuller, and the typical wait for a finished lesson was shorter by about a third. We told users about it. The same release also changed the instructions the model is given, so part of the gain may come from that rather than the model alone.
Which model does which job
Each job gets its own test. In September I asked for a test of a newer model that ranked higher in public, and we kept the model in use. Earlier we paid about 35% more per lesson for fewer errors, and this time we turned down 27% more per lesson for a model that was no better and slower. A week later, the same newer model won a different job. Several models wrote the same pages for our new website from the same facts, I read the drafts blind, and my first pick was that model.
Taste stays with people
When we redrew the cover pictures, the pictures at the top of the website's curriculum and blog pages, an AI reviewer and I each judged the new versions against the old ones. It called many more of them better than I did, picking blind, without knowing which version was which. So since then, anything judged on taste is decided by a person's blind pick, and an AI reviewer may only reject plain faults.
What I would keep, what I would change
I would keep the full test for anything that goes into production or is meant to last. I would change one thing. Make the AI judge prove it can tell a good lesson from a bad one before trusting its scores, because in these tests it couldn't separate the options, so the counted errors decided. The call I got wrong came three weeks after the switch. When the model wrote a whole lesson in one long answer, which is how most real lessons are made, it sometimes gave an answer the product couldn't read, and those lessons failed. The test had run a different path through the product, so it never saw this. We wrote a repair for it.
What I would tell the team
Write down the pass rule before you see any result, and test on your own work through your real product. There are many models for all kinds of tasks, and the best ones no longer come only from the West. Strong, cheap models now come from many places, including the East. Choose the provider by the data involved, so sensitive data only goes where it isn't kept or used for training. Public rankings point the way, but even a ranking built for your own field differs from your use case. The first test takes real work. After that it can be reused cheaply for every later switch. And not everything needs one.
Sources
Receipts. Lesson model, Google's Gemini line. Gemini 3 Flash (preview, retired by Google) replaced by Gemini 3.7 Flash at a low thinking setting in September 2026, after Gemini 3.6 Flash failed the August 2026 test. Gemini 3.8 Flash turned down for lessons in September 2026, then picked blind for the new website words the same month. Also ruled out in August 2026, Gemini 3.5 Flash Lite and Xiaomi MiMo V2.5 Pro, an open model from China. Web-page checker, March 2026, Gemini 2.5 Flash Lite over Gemini 3 Flash. Public index, Artificial Analysis. After-switch figures from PostHog, 14 days before against 8 days after the release. Tests built and run by AI agents through the real product, each under a spending limit I approved first.
- About a ninth: Flip Education's test of models for its automatic web-page checker, March 2026, on real pages broken on purpose in 11 languages: the cheapest of the models that bill per use scored 95% against 89% for the company's default model, at about a ninth of its cost per check, and it went into production.
- About half: Flip Education's production records, lessons in languages other than English: 25.7% went out with a quality warning before the early-September 2026 release, 13.1% in the 8 days after (12.6% on a later read). A lesson carries a warning when it still breaks a quality rule after the automatic repair. The same release also changed the instructions the model is given.