Courage Horizon

Guide

How do I change what my AI agent does without breaking it?

Courage Horizon · Published


Treat every change to an AI agent’s instructions like any other change to how your business works: one change at a time, checked against real past cases, with a way back.

Change one thing at a time, check it against about ten real past cases before it goes live, and keep a note of what the instructions said before. That is the whole discipline, and it is the same one you would use before changing how your team sends quotes or chases payment.

An AI agent is software that does one job for your business, such as chasing unpaid invoices, and asks you before it sends anything. What it does is set by a written set of instructions, which technical people call a prompt. Changing those instructions is changing how part of your business behaves, even though it feels like typing a sentence.

Why does a small change to the instructions cause trouble?

Because the agent reads every instruction every time, a new line affects jobs you were not thinking about when you wrote it.

Say your invoice-chasing agent is writing polite reminders and you notice one reads as too soft for a client who is 60 days late. You add a line: “Be firmer with late payers.” The next morning the agent is firmer with the 60-day client, and also with the client who is three days late and has paid on time for six years. You fixed the case in front of you and quietly changed the other cases.

People who have looked after these agents for a while learn this quickly. The change you can see is rarely the one that causes the problem. The problem turns up in a job you were not looking at, a week later, when a customer replies.

How do I test a change before it goes live?

Keep a small set of real past cases and read the agent’s output on all of them before and after the change.

Pick about ten real situations from the last few months, including the awkward ones. For an invoice-chasing agent that might be: a first reminder, a long-standing client who is a few days late, a client who disputes the invoice, a client who has already paid but whose payment has not been matched, and one who is 60 days late. Write down what a good message looks like for each.

Then, when you want to change the instructions, have the agent draft messages for those same ten cases with the new wording, put them next to the old drafts, and read them. If nine improve and one gets worse, you have learnt something before a client did. This takes an afternoon the first time and much less after that, because the ten cases are already written down.

In the Hub, you can do this as often as you like. Every agent has a test button: press it and the agent drafts its messages against your cases, with nothing sent to anyone. Test the old wording, test the new, change a word and test again. There is no limit on how many times you try, so there is no reason to guess.

Because the agent also asks you before it sends anything, you get a second check on live work. That check catches individual messages. It does not tell you the new wording has changed the tone across the board, which is why the ten cases come first.

How do I make a change I can undo?

Keep a dated copy of the old instructions with a one-line reason before you edit anything, and change only one thing per edit.

Say you make three changes on a Monday: firmer tone, a new line about late fees, and a shorter sign-off. On Wednesday a client complains about the tone of a message. You now cannot tell which of the three caused it, and going back means guessing what the old wording was. If you had made one change on Monday, one on Wednesday and one on Friday, each with a dated copy of what came before, you would know exactly which edit to undo.

You do not have to keep those copies by hand. The Hub keeps a version history of every agent, so each earlier set of instructions is still there, and if a change makes things worse you can go back to the version before it. You also ask for changes in the Hub in plain words, in a box on the same screen as the agent’s job and its history. The habit still matters: one request, one change, then test it before you leave it live.

One more thing worth knowing. The instructions can stay exactly the same and the agent can still start behaving differently, because the software underneath it can be updated by its maker. So the ten cases are not a one-off. Read them again now and then, perhaps once a quarter, and whenever a client mentions that a message sounded different. With the test button that is a few minutes’ work.

What does an untested change cost?

It costs the clients who receive the wrong message, which is easy to underestimate because the damage is spread thinly.

Say your agent drafts 60 reminders a month. You make the “be firmer” change and it makes about 10 per cent of them too blunt, which is 6 messages a month (60 x 0.10 = 6). You approve most of them because you are busy and they look fine at a glance. Say one of those six goes to a client who is worth £3,000 a year to you and who quietly decides to move supplier. That is £3,000 lost against a change that took one minute to type. The ten-case check would have cost an afternoon once.

You can put your own figures into that sum: how many messages go out, how many might be affected, and what your clients are worth to you. The free report does this for your business. It gives three jobs an AI agent could take off your team, our estimate of the hours each takes, and what that time costs a month at your own hourly figure. There is a full example for an invented plumbing and heating firm at couragehorizon.com/free-report/example, and you can apply at couragehorizon.com/free-report.

What is bloat, and why does it matter?

Bloat is what happens when instructions only ever get added to, until nobody knows why half of them are there.

Every awkward customer produces a new line. Say your agent started with 5 instructions and a year later has 40, because each complaint added one. Some now contradict each other: one says “keep messages under 80 words” and another says “always explain the late fee and how to dispute it”. The agent has to choose, and it will not always choose the same way twice. You see this as inconsistency and assume the agent is unreliable, when the instructions are simply asking for two things at once.

The fix is the same as for any set of office procedures. Every few months, read the whole set from the top. Remove anything that nobody can explain. Where two lines pull against each other, decide which one wins and delete the other. Then test the result against your ten cases. Short, clear instructions that someone has read recently almost always beat a long list nobody has looked at.

What should I do first?

Write down ten real cases for the one job you would hand to an agent, and what a good result looks like for each. You need them whether you change the instructions yourself or ask us to, and they are the first thing we would ask you for. If you want to see which three jobs in your business would be worth handing over, take the free report at couragehorizon.com/free-report.


Courage Horizon sets up AI agents for UK small businesses: software that does one job, such as chasing unpaid invoices, and asks you before it sends anything. Get a free report on your own business.

All guides