AI watches session replays automatically and turns each finding into a proven fix

One weekly view of what users do

Watching session recordings is one of the main ways product and UX teams find bugs, learn about the user experience and improve the product. The cost is time. Someone has to sit through the videos, note what they see and turn it into tickets. AI can now do this reading. At Flip Education the whole thing runs as one line of work. The recordings the website already collects come in. A free count runs across all of them every week, with no AI, and a short AI step decides how much reading the week deserves. AI then reads the screens of the visits worth reading, and each finding is proven outside the AI, on the live site, in the code and in the data. AI agents that each look after one part of the product fix what is proven, and I make the decisions that are mine.

The challenge

At SEOmonitor, a software company that tracks search rankings, where I worked before, I set out to automate the watching. We had a fairly large user base and wanted the top insights for the product engineering team without spending hundreds of hours watching. Nobody had the hours to watch them, and it still mattered, because the users' experience shows up in sign-ups and subscription income. All we had to work with was telemetry in text form, the list of pages visited and the actions taken inside the app. I tried the best AI agent then available that operates a web browser. It was meant to collect what our recording tool wouldn't export, and even the best models of the time couldn't do it reliably. At Flip Education the same problem came back, and watching the recordings by hand would take many hours a week. There was no staff to hand it to. The product team was me, so it would have eaten into the strategic time I needed for other things.

Numbers, but no screen

The first version at Flip Education ran on the same recording data the website already collected. A separate AI, run on a schedule, read only the numbers behind the recordings and emailed me a summary. It never looked at a screen. About a dozen product fixes in our code history came from that daily summary, and those fixes went live for every user. It showed a user searching the lesson maker for a Year 4 maths topic that the curriculum didn't have, and we added the topic. The recordings confirmed that every sign-up from outside the US was being saved as a US user, which we then fixed.

Better models, a better system

By September the AI models had got a lot better, both at visual understanding and at doing competent coding and reasoning work, so I upgraded the system. And the screens show things the numbers can't, like what a page actually displayed and what the user was trying to do, which means better fixes. PostHog, the analytics product behind our recordings, lets them be exported and read by software.

How a week runs

Before anything ran, I set what a finding is worth to the business and the bar, what the new findings must be worth together before a full review pays off, because the system is judged by what it is worth, not by how many problems it finds. Every week a script counts the signals across all the recordings, with no AI, among them new errors, where people left the lesson maker and what the last release changed. That covers every recording and costs nothing. A short AI step then reads the count and decides on a full review, a small one or none, and writes down why, so the costly reading runs only when it can pay off.

Scripts remove anything a visitor typed before any AI sees a recording, then rebuild each chosen visit into the screens the visitor saw, so the AI sees what the visitor saw, never what they typed. AI analysts write down what went wrong, and an agent that didn't write a finding sets aside the ones already known, reads the code itself and, for anything on the live site, sees the problem happen there, so only new, proven problems reach a fixer. A small proven problem is fixed by the agent running the week and goes out in the normal release, and anything bigger or risky goes to the agent that owns that part of the product, so small problems don't wait and risky changes stay with their owner. I hear about it only when a decision is mine, so my time goes to the decisions, not to reading reports.

A fair test, then proof

On the first evening of the upgrade, I tested whether reading the screens adds enough to be worth a weekly run. The bar was that the new findings together had to touch at least a tenth of a week's sign-ups, first lessons (the first lesson a new user makes) or signs of lesson quality. We counted lesson quality in lessons a user printed, since printing means the lesson goes to class, and users who made a second lesson within a week. Three AI analysts, the same model each time, read the same recordings, each given more to go on than the one before, and only the last also saw the pictures. The first analyst, the one reading only the numbers, already found most of the problems. The pictures added the ones numbers can't show, like what the lesson page looked like. Then 14 findings were checked on the live site, in the code and in the data. What the AI saw was fully or partly right every time, never wrong. Why it happened was fully right in only 5 of 14. A separate checking AI, a different model told to disprove the findings, had approved all 14, including the explanations that were wrong or only partly right. So in this test, a checking AI wasn't enough. Since then, every finding is checked in the code before anyone is asked to fix it. The new findings were real, but together they came in under the bar, so the test recommended running it only when needed, after a release or for a named question.

What the system caught

On the lesson page, the AI inside our product writes measurable learning objectives for every lesson, and those objectives were written correctly. The problem was the page itself. It looked for them in the wrong place, so it showed filler lines where the real objectives belonged. Only the analyst with the pictures noticed. The picture below shows what users saw. The code check found the cause, and the fix was one line.

On the sign-up box, from mid-September a visitor without an account who pressed Generate first met a sign-up box. Generate is the button that creates a lesson. The box had no close button. The numbers showed visitors leaving it without signing up, and they couldn't show how. The recordings answered it the same evening. Every time, the mouse moved up to the browser's Back button. Checking that path on the live site found a crash behind it. After going Back, the page froze and the visitor couldn't continue. One fix repaired the crash itself, so the next page change works again after going Back. The other shows the sign-up box again at once when the visitor presses Generate a second time.

On the curriculum pages, visitors clicked links again and again and nothing happened. A scripted browser, software that clicks through the site like a visitor, showed why. A loop, code that kept running over and over in the background of those pages, blocked every link, and switching it off made the same click work at once. The recordings showed what the loop had been costing visitors, people clicking again and again with nothing happening. The loop itself was already gone, removed by another agent to make the pages faster, without knowing it blocked clicks.

That first test, run the same evening I started, led to four fixes in the code. Three came from the catches above, the learning objectives on the lesson page and the sign-up box problems. The fourth was a smaller wording fix. Every finding, fixed that evening or not, went into the plan of the agent that owns that screen.

What changed for the business

About 25 hours passed from my first request to the version of the system I approved for production. I spent under 6 of those hours working on it myself. Those four fixes have since gone out to users through the company's normal release. The first full run of the upgraded system, after the test, found its problems, checked them in the code and sized them in 57 minutes.

What I'd tell a product team

AI that reads your session recordings gives a small team a direct view of what users actually do and what they want. Aim it at a named question, or run it after a release. A weekly run is worth it only if a cheap first step decides whether anything new justifies a full review; without that step, stay with the named question or the release. In my test, reading recordings with no question in mind found real but small problems. You need recordings you can export, typed text removed before any AI reads them, and proof of each finding before anyone fixes it.