Before you build an AI assistant: seven questions to answer
Most AI assistants that get built never get used, and it is rarely the technology. It is that nobody decided who it is for, what it is allowed to say, and how anyone would tell whether it works. Seven questions to answer before anyone writes code, in this order.
1. Who will use it, specifically?
Not "our customers" or "staff". Which role, in which situation, instead of doing what? An assistant for support agents who otherwise search three systems is a sharp problem. An assistant "for everyone" is not a problem at all.
2. What should it answer from?
Name the documents. If the answer is "everything we have", the project is not ready. Start with the bounded set covering the most common questions, often a hundred pages rather than ten thousand. Expanding later is easy; cleaning up an index built on everything is not.
3. Is the source material actually correct?
The uncomfortable question. An assistant answering faithfully from outdated documents spreads errors faster than a person would. If your guidelines were last updated in 2019, that is where the project starts, not at model selection.
4. What must it not do?
Write an explicit list. No pricing, no legal advice, no commitments on delivery time. The list is part of the requirements and belongs in place before the build, because it determines how the flow is constructed.
5. What happens when it does not know?
This step is almost always skipped and it is what decides trust. Where does the user go next? Is the question logged so somebody can add the answer? An assistant that declines and routes correctly is better than one that guesses.
6. How will you measure that it works?
Decide the metrics before launch. Share of questions resolved without escalating to a person, share of answers the user flagged as wrong, number of questions with no match. Without measurement the evaluation becomes "it feels pretty good", and that does not survive six months.
7. Who owns it in a year?
Content changes, questions change, models get replaced. If nobody is responsible for adding new documents and reading the log, the assistant will quietly degrade until it is unusable, and nobody will notice until then.
Common questions
How long does it take to build an AI assistant?
A prototype on real data usually takes a few weeks. What takes time is rarely the build but the answers to the questions above: naming which documents apply, discovering half of them are out of date, and agreeing what the assistant must not say. Projects that skip that step finish quickly and then go unused.
What does an AI assistant cost to run?
Two lines: model calls per question, and indexing the content. Calls dominate and are driven by how much text is sent along, not by the number of users. A flow sending five retrieved passages costs a fraction of one sending whole documents. Set a per-user daily ceiling from the start.
What do we do if the source material is out of date?
Then the project starts there. An assistant answering faithfully from outdated documents spreads errors faster than a person would, because it answers the same way every time and sounds certain. Clean the bounded set covering the most common questions first, often a hundred pages rather than ten thousand, then expand.
More in the same area
RAG explained: making AI answer from your own documents
How retrieval augmented generation works in practice, and why retrieval decides the quality rather than the model.
How to measure whether AI search actually got better
A method for evaluating search quality with numbers instead of opinion, and why hybrid search almost always wins.
