80% of support conversations resolved, above Fin's average

Why the resolution rate is the business

Any software company that serves users in many languages faces the same arithmetic. It needs a high resolution rate, because whatever the AI can't resolve needs a person who can answer in the user's language. That's either someone who speaks it or, now that translation is good, an English-speaking team replying later rather than live. So the share of conversations the AI resolves on its own decides how many people a company needs behind it. That share isn't set by the model, which anyone can buy. It is set by what is built around it, what the assistant knows, what it may promise, who sits behind it, and what happens to each failure.

The challenge

Two of us ran Flip Education, and nobody was hired for support. In my previous role I managed a customer support team, and we used Fin, the AI support agent made by Intercom. It was a good tool, but operating it was still complex with a small team, because we read messages one by one to find what to improve, turned that into the bot's settings, and wrote help articles by hand. That was just before AI agents could do much of this work. So I wanted to see how much of running a good support assistant AI agents could carry, answering users and writing and shipping the fixes, with me deciding what to fix.

One assistant, human limits

A user writes in the chat window, and Flo reads it with what that signed-in user just did, from pages used to whether they started to pay. Flo also sees their country and the real topics of that country's curriculum, so it never asks for what the account already says. Plans, prices and features come from the product's own code, so they can't drift. Flo answers at once, in the user's language. Flo never promises deadlines or new features, and never claims to do something it can't do, rules we wrote after public failures of other companies' support bots. Every conversation also lands in the team inbox, where a person can reply. When a person joins, Flo goes quiet. Hard questions still reach a person.

Two checkers, plain scripts with no AI, read the conversations afterwards, one for wrong facts, one for unanswered users, so nothing depended on a user complaining. I decided what to fix, and AI agents shipped each change. Every change must pass the tests, and the facts must match the code, before it can go live. A fixed failure stayed fixed, and three times a fix was live for every user before our reply had reached them to say so. Twice, that was about an hour after they first wrote.

Knowing what the user just did

The way to better answers was context. The example below is a user asking how many lessons they can still make on the free plan, which the product calls Spark and which allows 5 lifetime missions, the product's word for the lessons it generates. Without the user's context, Flo can only give the plan's general rule and send the user to their dashboard for their own number. With context, it already knows the user has generated 2, so it answers with their exact count at once.

What broke, and what we made permanent

When Flo met real users, it failed in ways we hadn't foreseen, and every failure led to a lasting fix. In April Flo told a paying user about a "Pro" plan that doesn't exist, and that same evening Flo's facts started being generated from the product's own pricing code, with a first checker reading answers for wrong facts. Later, a user sent seven messages in a few minutes and got no reply, which that checker couldn't see because it only judged answers that existed, so a second checker, an hourly watch for unanswered users, followed. An Italian user got a reply in Spanish, one of several wrong-language answers that led to a test set of real user sentences, which every change to the language check must pass. The Hindi case is below.

A language we didn't serve

In August a user wrote five messages in Hindi, typed in Latin letters. Flo answered all five in English, most with the same standard line. I saw the ticket about ten minutes later. The real gap was that Hindi wasn't one of the 11 languages Flo was set up for. We also tested a ready-made language detector and rejected it, because it read a plain English support question as Venetian, a language spoken in parts of Italy. So my decision was that Flo answers in the user's language, even one the product doesn't serve, and says in that language that the lessons come in the 11 languages Flip Education serves. The fix was live for every user about two and a half hours later, with no extra spend.

What I would keep, what I would change

What I'd keep is the user's context on every message, and Flo's facts generated from the product's own code, so they can't drift. The speed of fixing came from AI agents that wrote and shipped the code changes, with access to the product's code, and users' reports often became product changes within days. What I'd change is measurement. Users could never rate an answer, so afterwards I brought in two AI judges. They read every conversation from April to September.

Intercom's way of counting is that a conversation is resolved when it ends on the AI's answer and no person is needed. Counted that way, Flo resolved 80%. Intercom publishes 76% for Fin, its own AI support agent, so Flo sits above Fin.

What it cost

The first version took about three and a half hours of my own time, over one evening and the next day, and it went live the second day. Reviewing and improving it afterwards took about 3 to 4 more hours. All the coding work was done with Claude Code, an AI agent that writes code, on a flat-fee plan, so it added no metered spend. Flo's AI model charges work out at about 8.5 cents per resolved conversation. Intercom's price list starts at $0.99 for each conversation Fin resolves. On Intercom's own plans, the seats, a monthly fee for each person on the team who uses the inbox, come on top of that. Both figures leave something out. Flo's leaves out my build time. Fin's leaves out the seats.

What it could mean for a software company

Here's the estimate. It uses Flo's 80%, counted the way Intercom counts resolutions for Fin. Say a company gets 1,000 conversations a month. At Flo's rate, 200 are left for people, so the AI resolves 800. Resolving them costs about $6,400 a month with people, at the average cost per conversation from a Gartner survey of customer service leaders. With Fin at its list price, about $790. Flo's AI model charges come to about $68. There's also speed for the user. Companies without AI took 41 minutes on average to respond in real-time conversations, according to the customer service benchmark from Freshworks. Flo answers at once, at any hour.

What's next

The next step is one model running the whole improvement process. A more capable model could run this process on its own. It could review every conversation, or a sample, or only the ones that weren't resolved, decide what went wrong, improve Flo, put the change live and measure the result. The pieces all exist now.

When to build your own

If nobody on your team can read and change the code quickly, a bought bot is safer. Test any bought part on your own customers' messages before you adopt it, because the ready-made language detector we tested failed on real messages in ways no demo showed.