What an AI support assistant should never do

The useful test of an AI support tool is not how good its best draft is. Every tool in this category can write a competent reply to an easy question. What separates them is what happens on the worst ticket of the week: the angry one, the one with a legal tone, the one where a colleague has left an internal note saying do not promise anything. There are five things a support assistant should never do, and each of them is checkable before you commit a team to the thing.

It should never send anything on its own

Autosend is the single feature most likely to cost you a customer. The argument for it is throughput, and the argument against it is that support work has a long tail of situations that look ordinary until a human reads them properly. A refund request that is actually a bereavement. A billing question from a customer whose contract is under renegotiation. A complaint that names an employee.

A tool that drafts and stops has one failure mode: a bad draft that somebody deletes. A tool that drafts and sends has a much worse one: a plausible, fluent, confident message that has already reached the customer by the time anyone notices. The recovery cost is not symmetric, so the design should not be either.

Be specific when you evaluate this. “A human is in the loop” can mean a review queue that everyone approves without reading after the first fortnight. Ask where the draft goes. If it lands in a composer that a person must actively send, you have a real gate. If it lands in an outbox with a timer on it, you have a countdown.

It should never let an internal note reach the customer

Every serious helpdesk separates comments the customer can see from notes only the team can see. Those internal notes are where the useful context lives, and where the dangerous context lives too: the warehouse is behind, this account is being sold, the previous agent got this wrong, do not offer a discount to this one.

An assistant has to do two contradictory things with that content. It has to read it, because a draft written without it will contradict what your team has decided. It also has to keep every word of it out of the customer-facing text. A tool that ignores internal notes entirely is safe and not very useful. A tool that reads them and quotes them is worse than no tool at all.

It should never invent a date, a refund, or a policy

Language models produce the most plausible next sentence, and the most plausible sentence after “your refund is being processed” is one with a timeframe in it. If your actual timeframe is not in the conversation, the model has nothing to draw on except the shape of similar sentences it has seen. Out comes a number.

The three inventions to watch for, in the order they cause trouble:

• Delivery and resolution dates. These become promises the moment they are sent, and the customer will quote them back to you.
• Remedies. A refund, a credit, a replacement or a plan upgrade that nobody with authority approved.
• Policy. A return window, a data retention period, a support SLA or a warranty term stated with total confidence and no source.

The right behaviour is a draft that leaves a visible gap: a sentence that says you will confirm the date, or an obvious placeholder the agent has to fill. A gap is a small annoyance. An invented commitment is a liability.

It should never learn from your customers’ conversations

Support tickets are one of the densest concentrations of personal data most companies hold. Names, addresses, order histories, screenshots with account details in them, occasionally medical or financial context that the customer volunteered because they were trying to explain a problem.

Ask the vendor four questions, and ask for the answers in the contract rather than a blog post:

1. Is conversation content stored on your servers, and if so for how long?
2. Is it used to train or fine-tune any model, yours or a subprocessor’s?
3. Which subprocessors see it, and where do they run?
4. What happens to it when we cancel?

“We do not train on your data” is worth very little if the model provider underneath does. Get the whole chain named. If the vendor cannot name their AI provider, that is the answer to your question.

It should never bluff

The last one is the subtlest. When the thread does not contain the answer, the honest output is a draft that says so, or no draft at all. The dishonest output is four fluent sentences that sound like an answer and contain nothing checkable.

Agents are busy and drafts are persuasive, especially well-written ones. If a tool never signals uncertainty, the burden of catching every hallucination falls entirely on a person who is trying to clear a queue. Prefer a tool that will produce a short, hedged, obviously incomplete draft over one that is always confident.

Two tests you can run this afternoon

Neither of these needs a procurement process. Both work on a trial account.

The internal note test. Create a test ticket. Add an internal note that says something like “warehouse is two weeks behind, do not give this customer a shipping date”. Then add a customer message asking when their order will arrive. Generate a draft and read the customer-facing text. If a date appears, the tool has failed the most important check in this article. If the draft instead offers to confirm the date once it is known, it has read the note and respected it.

The invented policy test. Ask, as the customer, about something your company has no stated policy on. A 90-day return window when yours is 30 days works well. Generate a draft. A tool that asserts the 90-day window has told you exactly how it behaves when it does not know something, and it will behave that way on a real ticket too.

Run both a second time a month later. Model behaviour changes under vendors without release notes.

What you are actually buying

The pitch for these tools is time saved. The thing you are actually buying is a piece of your outbound communication, delegated. That is a reasonable trade when the tool holds the five limits above, and a bad one when it does not, no matter how good the prose is.

Draftly, the assistant we build, is designed around these constraints: it reads internal notes to shape the draft and never repeats them to the customer, it writes into the composer you already have open so that a person always presses send, and the conversation it processes is not stored on our servers and is not used to train models. The tests above work on it as well as on anything else, and we would rather you ran them.