Courage Horizon

Guide

How many times should I adjust an AI tool before deciding it does not work?

Courage Horizon · Published


Ten corrections on real jobs over about two weeks is our rule of thumb, and it is our own view, not a measured finding. Plenty of owners stop at three, which in our view is about the point where a tool has only just stopped making the obvious mistakes. If you have tried an AI tool, corrected it a few times and given up, this article is for you.

Why do people stop after about three tries?

The first attempt is judged against a picture in your head of how the job should be done. The tool has never seen that picture. So the first output is wrong in ways that feel like stupidity, and you correct it. The second output fixes that and is wrong in a new way. By the third, the owner has spent half an evening on something meant to save time, and the sensible conclusion seems to be that it does not work.

That conclusion mixes up two things. The tool may be able to do the job, and the tool may not yet have been told how your business does the job. The second is only learned by being corrected on real cases. This is our opinion from how the work goes, so test it against your own experience.

What are the corrections actually about?

Most of them concern rules that have never been written down.

Say the job is chasing unpaid invoices, and the AI agent drafts a polite reminder to a customer. It looks fine. Then you notice you would never send that one, because that customer has an open complaint. The next draft goes to a customer who always pays on the fifth, three days before the agent thought they were late. The one after that uses a tone you would only use with a long-standing client, and goes to someone new.

None of those is a fault in the software. Each is a house rule that lived in your head or in the head of whoever does the job now. Here is the kind of list that builds up over a fortnight:

  • Never chase anyone with a complaint or credit note open.
  • Wait until the seventh day after the due date for the first reminder, not the first.
  • Use first names for customers of more than two years, surnames for the rest.
  • Copy the office manager into anything over a set amount.
  • Leave out the late-payment interest sentence unless a person has approved it.

Five rules like these, written down and given to the agent, will change the output more than rewording the original request.

What should I do on each correction?

The habit that does the most good is the cheapest one: say why. “Not this one” teaches nothing. “Not this one, because they have an open complaint” teaches a rule.

  1. Look at the draft and decide whether you would send it.
  2. If not, write down the reason in one sentence, in the words you would use to a new member of staff.
  3. Give that sentence back to the agent as a standing rule, not as a one-off edit to this draft.
  4. Check the next three drafts for that rule before you move on.
  5. Add the rule to your log with the date.

The log is worth copying. A simple version is a table with four columns: date, what the agent did, what you would have done, and the rule in one sentence. At the end of the fortnight you have a page that describes how your business actually handles that job, which is useful whatever you decide about the tool.

How do I know a correction has worked?

Look for the same mistake not coming back. A correction has worked if the rule you gave still holds a week later, on a case you did not mention. Suppose you told the agent about the customer with the open complaint, and it now also leaves out a customer who has asked for a statement first. It is applying your rule to a new case, much as a new colleague would.

Count the repeats, not the total. Say you correct ten things in the first week. If in the second week you correct three, and they are new situations rather than old ones returning, the setup is settling. If you keep correcting the same thing, the rule is either not reaching the agent or is too vague. Say it more plainly, with an example of a case where it applies and one where it does not.

When is it right to give up?

Sometimes it is. These are the signs that the job, and not the effort, is the problem:

  • The job has no consistent rules, and you cannot explain to a person how you decide. If you cannot write the rule in a sentence, the agent cannot follow it.
  • The mistakes are always different and never repeat, even after a fortnight of corrections on the same kind of case.
  • One wrong output would cost enough that you would check every single one anyway, so no time is saved.

Those are fair reasons. “I corrected it three times” is not one.

Owners also tend to judge a tool by its worst output. One embarrassing draft is remembered for months, while the ordinary ones are forgotten. An agent that asks before it sends anything changes that: the worst draft costs you a few seconds of reading and the customer never sees it. Asking first also means your corrections come from real drafts on real jobs, which is where the useful ones come from.

Where does the Hub fit in?

We set up AI agents for UK small businesses, watch them and fix them when they break, and nobody technical is needed on your side. The Hub shows every agent on one screen, what each did today and what is waiting for your approval. It also has a box to ask for a change in plain words, and a sentence like “not this one, because they have an open complaint” is the sort of thing that goes in it.

If you are trying a tool on your own, the same habits apply. Keep the log, give a reason with each correction, turn each reason into a standing rule, and hold your verdict until the second week.


Courage Horizon sets up AI agents for UK small businesses: software that does one job, such as chasing unpaid invoices, and asks you before it sends anything. Get a free report on your own business.

All guides