How to measure whether AI search actually got better
To know whether a search improvement actually helped you need an evaluation set: real queries from your own search logs, each with the result that should rank first. Run the set before and after every change. Without it, every discussion about search quality becomes a discussion about how it feels, and whoever speaks loudest wins.
How do you build the evaluation set?
Take the hundred most common searches from the last three months. Add the twenty most common searches that returned nothing, because that is where it hurts most. For each query: which page should rank first? It takes an afternoon and it is the one investment that makes everything else measurable.
Which metrics are enough to start with?
You can measure search quality a dozen ways. Two of them carry most of the value:
- Share of queries where the right page ranks first. The blunt measure, and the one users notice.
- Share of queries where the right page is in the top five. Catches improvements that have not yet reached the top.
Which search method suits which query?
The three methods are layers rather than competitors. Hybrid almost always wins on the measurement because it covers both extremes.
| Query type | Keyword | Vector | Hybrid |
|---|---|---|---|
| Part number | Exact match | Often misses | Exact match |
| Product name | Good | Good | Good |
| Rephrased question | Misses | Matches | Matches |
| Typos | Misses | Usually copes | Usually copes |
| Synonyms | Misses | Matches | Matches |
| Cost | Lowest | An index to maintain | Slightly higher |
Why is pure vector search not enough?
Vector search matches meaning and is therefore good at questions phrased in words other than yours. But it is weak on exact strings: part numbers, model codes, people’s names. A customer pasting a part number wants that exact product, not something that means roughly the same. That is why hybrid search almost always wins on the measurement: classic keyword search and vector search run in parallel and the results are blended.
Why change one thing at a time?
How content is chunked, the weighting between keywords and vectors, how many results are passed along, which embedding model is used. All of them move the numbers. Change two at once and you will not know which helped, and you will keep both.
What does the set not capture?
Two things from production the evaluation set misses: share of searches returning nothing, and share of searches where the user clicked anything at all. Zero-result searches point at missing content rather than broken search. Searches without a click mean the results looked wrong, which is often a presentation problem rather than a ranking one.
How do you roll back if it got worse?
Put the new search alongside the existing one rather than replacing it outright. Then you can compare on real traffic and switch it off without drama if the measurement shows no improvement. Search projects that cannot be rolled back rarely get evaluated objectively.
Common questions
How many queries does the evaluation set need?
Thirty is enough to see a difference, a hundred gives steadier numbers. More important than the count is that the queries come from your own search logs rather than being invented, and that the zero-result ones are included. A set of made-up queries measures how good search is at questions nobody asks.
Do we have to replace our current search?
Rarely straight away. Vector search is usually placed alongside the existing one and the results blended, so exact matches on part numbers keep working as before. It also means the change can be rolled back if the evaluation shows no improvement.
What does keeping the index current cost?
Less than most people expect. The cost is in creating vectors, and that only has to happen for content that actually changed. The search itself runs against the index without model calls, so traffic volume barely affects the running cost.
More in the same area
RAG explained: making AI answer from your own documents
How retrieval augmented generation works in practice, and why retrieval decides the quality rather than the model.
Before you build an AI assistant: seven questions to answer
The questions that decide whether the project becomes useful or becomes a demo nobody opens twice.
