Why AI search won't fix a legacy knowledge base

What does AI search actually change?

Most knowledge tools now offer some combination of three AI features. Semantic search matches the meaning of a query instead of its exact words. Generated answers read the top few articles and write a short response. Copilots and agent assist tools listen to a conversation and suggest content without anyone typing a query.

All three improve the same step: getting from a question to the content that already exists. That is a real gain. Agents who used to give up after two keyword searches now get something useful on the first try. Customers in self-service get a sentence instead of a list of links.

What none of them change is the content. Every answer they produce is assembled from articles somebody wrote, at some point, for some purpose. The AI layer sits downstream of that work. It can only be as complete, as current and as consistent as the articles underneath it.

Why can't AI search tell you what's missing?

Search, of any kind, ranks what exists. A missing article has no text to embed, no vector to compare and no passage to summarise. When the answer isn't there, AI search does what it was built to do: it returns the closest thing it can find.

Take a fictional energy retailer. A customer asks whether their concession carries over when they move house. The knowledge base has a general article on concessions and a separate one on moving house, and neither answers the combined question. Keyword search might have returned nothing, and an agent would at least know they were on their own. AI search returns both articles, and the generated answer stitches together a plausible response from two passages that were never written to answer this question.

Nothing in that exchange tells anyone the article is missing. The search succeeded. The answer was delivered. The gap is still there, and now it is hidden behind a fluent paragraph. Finding it requires a different kind of evidence: the questions customers actually asked, checked one by one against what the knowledge base can answer. Search activity can't supply that, which is the core argument of failed searches vs knowledge gaps.

What happens when AI retrieves stale or contradictory content?

AI retrieval has no built-in sense of time or authority. An article written before last year's pricing change matches the query just as well as the current one, because the words are nearly identical. If both exist, retrieval may return either, or both.

Stale content is the more common failure. Policies, forms, fees and eligibility rules change far more often than articles are reviewed. An old article that keyword search buried on page two can rank first under semantic search, because it happens to describe the topic in plain language. Ranking well and being correct are separate properties, and retrieval only measures the first.

Contradictions are worse. When two articles give different answers, a person reading both might notice the conflict and ask. A generated answer has three options, all of them bad:

• Pick one, which gives different customers different answers depending on small changes in how they phrased the question.
• Blend them, which can produce a statement that neither article actually makes.
• Hedge, which gives the customer an answer that sounds careful and resolves nothing.

None of these outcomes is visible in a search dashboard. They show up later, as repeat contacts and complaints. Finding them means checking the content itself for validity against your authoritative sources and for conflicts between related articles.

Why does a fluent summary make bad content more dangerous?

An agent reading a full article picks up cues. The article is dated three years ago. It mentions a product you stopped selling. It contradicts what the team leader said this morning. Experienced agents catch a surprising number of errors this way, because the article looks wrong.

A two-sentence summary removes those cues. It is written in the same confident voice whether the source was reviewed last week or never. It drops the caveats in paragraph four. It turns "in most cases" into a flat statement. The agent, or the customer in self-service, sees a clean answer with no reason to doubt it.

Scale amplifies this. A stale article in a keyword-search world was read by the agents who happened to find it. The same article feeding a copilot is proposed to every agent on every matching conversation, and in self-service it is delivered straight to customers with nobody checking in between. One bad source now produces many bad answers, quickly. Why no vendor can promise otherwise is the subject of can AI answers be hallucination-free?

Why do the dashboards look better after an AI rollout?

AI search tends to improve every metric that measures search behaviour, whether or not answers improved:

• Zero-result searches fall, because semantic search almost always returns something.
• Search success rates rise, because a result was returned and opened, or a summary was shown.
• Deflection appears to rise, because a customer who reads a generated answer and leaves is counted as helped, even if they phone the next day.

These numbers report on the AI layer, which genuinely improved. They say little about the knowledge, which may not have changed at all. The measures that reflect answer quality sit downstream of the conversation: whether the contact resolved, whether the customer came back on the same topic within a few days, whether it was transferred, and whether sentiment dropped after the answer was given. If those measures don't move after an AI rollout, the content is the constraint.

How should content be written for AI retrieval?

AI search is a multiplier. Content that is complete, current and consistent becomes much easier to use. Content that isn't becomes much easier to misuse. Those three properties apply to every knowledge tool, and we set them out in a knowledge tool is only as good as the knowledge in it. AI retrieval adds a fourth requirement that keyword search never had: every passage has to make sense on its own.

Retrieval works on chunks of a few paragraphs, not whole articles. A chunk is lifted out of its page and handed to the model with no headings above it and nothing after it. Content written for a person scrolling a full page often fails that test. Before switching AI on, check your highest-volume topics for these patterns:

1. Conditions far from the answer. "This applies to business accounts only" in the introduction, three headings above the fee it qualifies. Put the condition in the same sentence as the answer.
2. Pronouns and references that need context. "As above", "this form" and "the new process" mean nothing in a retrieved passage. Name the thing.
3. Several jobs in one article. A single page covering eligibility, applying and cancelling produces passages that answer half a question. One topic per article, or at least per clearly titled section.
4. Titles in internal language. Retrieval weights titles and headings heavily. Phrase them the way customers ask, which the conversations show you.
5. Retired content still in the index. An article removed from the menu but left searchable is still retrievable. Retire it from the index, and merge near-duplicates so only one passage can answer a question.

None of this needs a new platform. It needs a reliable way to see which topics fail each test, ordered by how many conversations they affect and how hard those conversations are.

How should you test AI search before you trust it?

Vendor demonstrations use questions the knowledge base answers well. A fair test uses the questions your customers actually ask, including the ones it can't answer. A simple evaluation set looks like this:

• Draw questions from real conversations, in customer language, across your top topics. Include multi-part questions such as the concession-and-moving example above.
• Include questions with no answer in the knowledge base. The right response is to say so, or to hand over to a person. A confident answer to an unanswerable question is a fail.
• Include questions where you know an article is stale or two articles conflict. Check which version the answer used.
• Check citations. Every answer should point to the passage it used, and a reviewer should be able to confirm that the passage supports the answer.
• Re-run the set after content changes, so you can see whether fixing an article fixed the answer.

Expect the results to say more about your knowledge base than about the AI. That is useful. The failures form a prioritised list of content work.

Do you need to replace the knowledge base?

No. A new platform, with or without AI, inherits the same articles through migration, along with the same gaps, stale pages and contradictions. The work that improves AI answers happens in the content, and it can happen in the knowledge base you already run; adding gap detection to your knowledge base shows what connects.

Knowledge Quest does that work alongside your existing tools. It reads every conversation from your contact centre platform or service desk, works out what each customer wanted, and checks it against your knowledge base, so the missing answers become visible. Gaps are ranked by how many conversations hit them and how hard those conversations were. Six quality checks look for invalid, duplicated, contradictory, hard-to-read, overly complex and off-brand content. Knowledge Copilot drafts fixes from the conversations, with the evidence attached, and your team approves them before they go back to your knowledge base. Your AI search then has better content to retrieve.