Beyond the Audit: Refining Copilot with User Intent and Theme Analytics

Standing up a Copilot agent is the easy part. The harder question arrives a few months later: what are people asking it, where is it failing them, and what needs to change? Most organisations can prove the agent gets used. Far fewer can turn months of accumulated questions into an agent that answers better than it did on launch day. That accumulated history is the most valuable asset a deployed agent produces, and it almost always sits unread.

The Audit Log Will Not Give You the Questions

Start with the thing that derails most improvement efforts.

Microsoft Purview Audit records who interacted with Copilot, when, which app they were in, and which files Copilot referenced to build its response (Microsoft Learn). What it does not record is the prompt text or the response text. Reaching those requires Content Search or eDiscovery, because Microsoft 365 Copilot prompts and responses are stored in the user's mailbox.

Teams assume the audit log is a searchable archive of every question anyone asked. It is not. It is a compliance record of activity, not a corpus of user questions. Auditing proves the agent was used. It will not tell you how to improve it.

The audit log proves usage. Analytics is where the improvement signal lives.

So where does the improvement signal actually come from?

Where the Search History Actually Lives

For a Copilot Studio agent, it sits in Analytics, and it arrives already clustered rather than raw.

Themes. Copilot Studio groups similar user questions into named clusters automatically, suggesting fresh themes weekly and classifying new questions daily (Microsoft Learn). This is your plain-language readout of what users genuinely expect the agent to handle, as opposed to what the project team assumed at build time.

Answer rate. The split between answered and unanswered questions per theme, and how it is trending (Microsoft Learn). A high-volume theme with a poor answer rate is failing a lot of people at once. That is where effort pays back.

Worth planning around: Copilot Studio does not currently offer a native export of every prompt submitted to a custom agent, and there is no API that hands you the raw query history (Microsoft Q&A). The clustered view is what you get. That is a constraint, but it is also a mercy. Nobody was going to read nine thousand individual prompts anyway.

Read the Questions, Not Just the Numbers

Here is the part most teams skip.

A theme with a poor answer rate is not a verdict. It is a symptom, and very different causes produce the same symptom: users and the knowledge base using different vocabulary, questions arriving without the context the answer depends on, three questions crammed into one sentence, or a genuine gap where the information simply does not exist in any source the agent can reach.

Only that last one is fixed by adding content. The rest are fixed by changing instructions. Diagnosing which you are looking at, before changing anything, is the difference between an improvement cycle and an expensive loop of prompt tweaks against an input that was never going to work.

From Pattern to Prompt

Once a pattern is identified, the change is usually specific and small. The examples below are illustrative, showing the shape of the reasoning rather than reporting any particular agent's results.

Pattern: users never specify what they are asking about.

A cluster of questions arrives in the form "can I share this?" with no indication of what "this" is, who it is going to, or why. Individually these look like failed conversations. Collectively they are telling you something specific: the agent is being asked to answer with information it was never given.

The fix is not a better answer to that question. The fix is an instruction change:

Before answering any information sharing question, confirm the type of information involved, the intended recipient, and the purpose of the sharing. If any of these are missing from the user's question, ask for them before attempting an answer.

The agent stops guessing and starts eliciting. The answer rate for that theme moves, because the questions arriving at the answering step are now answerable.

Pattern: users and the knowledge base speak different languages.

Users repeatedly search for "contractor access." The policy documents consistently say, "external service provider arrangements." The knowledge is present. The retrieval is failing on vocabulary.

No document needs rewriting. The instruction does:

Treat "contractor," "external staff," and "third party" as equivalent to "external service provider" when searching knowledge sources.

That is a two-line change that can lift an entire theme, and you would never have found it without reading the actual questions.

Pattern: compound questions scoring as answered.

Users ask three things at once and get one answered. The metric says the agent responded. The user's experience says otherwise. An instruction to identify and separate multi-part questions, answering each in turn, addresses a failure the answer rate alone would never have surfaced.

The common thread across all three: the user query history did not just tell you that something was broken. It told you what to write. That is what makes it training data rather than a usage report.

Running the Loop, Step by Step

What follows is the full cycle executed on a representative dataset for a general employee self service agent, generated to match Copilot Studio's documented analytics schema exactly, including quality sampling, Poor-response reasons, and multi-theme classification. The numbers demonstrate the method, not a specific client's results. The mechanics are identical when the data is yours.

1.   Pull the themes. The agent's 364 classified questions cluster into six named themes plus an Unclassified bucket. Already this is a readout of what users genuinely expect the agent to handle, as opposed to what the project team assumed at build time.

Figure 1. The themes list. One theme is not like the others.

2.   Rank by volume and answer rate, carefully. Two traps sit in this table. Response quality is scored on a sampled subset, so dividing Good responses by total questions would misreport this dataset's overall quality as 20% when the sampled figure is 62%. And one question can classify into more than one theme, so summing the volume column overstates activity. Read past both and the target is unambiguous: Payroll & Benefits, 52 questions at 65% answered, 38% quality, and 20 thumbs down against 4 up. Not the loudest theme. Leave & Time Off has more volume and is fine. The loudest theme is rarely the broken one.

3.   Read the underlying questions. This is the step most teams skip, and it is the entire game. The questions under Payroll & Benefits are ordinary and well formed: “Where can I download my payslip?”, “How do I change my superannuation fund?”, “Why was my pay different this fortnight?”. Nothing wrong with the asking. So the failure is on the agent's side, and the drill-down says where.

Figure 2. The diagnosis in two charts: why answers were Poor, and what they cited.

4.   Diagnose before changing anything. The dominant Poor reason is Incomplete knowledge use, and the sources cited on those Poor answers skew toward Expense Guidelines and the HR Handbook rather than the Payroll Policy. The knowledge is present. Retrieval is reaching past it, because users say “pay”, “payslip” and “super” while the authoritative document speaks in payroll administration language. This is a vocabulary and grounding problem, not a content gap, which means the fix is an instruction change, not another document.

5.   Change the right thing. The instruction added to the agent:

For questions about pay, payslips, salary, superannuation, tax declarations or bank details, treat these terms as payroll matters and ground the answer in the Payroll Policy first. Cite other sources only where the Payroll Policy does not cover the question.

Two sentences. No knowledge source touched, no topic rebuilt.

6.   Re-measure next cycle. The same theme, one cycle later:

Figure 3. One targeted instruction changes, re-measured.

Answered moves from 65% to 83%. Sampled quality moves from 38% to 71%. Reactions invert. And the residual Poor reasons shift from Incomplete knowledge use to plain Incomplete, which is the next, smaller problem to work on. That last detail matters: the re-measure did not just confirm the fix, it handed over the next diagnosis. If the numbers had not moved, the conclusion would be a wrong diagnosis, not wasted effort.

Why the Structure of the Fix Matters

Notice what the change was not. It was not a broad instruction to “answer payroll questions better”, and it was not a rewrite of the whole system prompt. The analytics named a theme, the theme named a failure reason, and the failure reason named the sources being wrongly cited. A prompt change written at that level of specificity, mapping the user's vocabulary to the authoritative source, is small enough to be safe and targeted enough to move a whole theme at once. That is what the search history buys you: it does not just tell you something is broken, it tells you what to write.

The Loop

1.      Pull the themes. Start from what people asked, not from a hypothesis about what needs fixing.

2.      Rank by volume and answer rate. High volume plus poor answers equals highest return.

3.      Read the underlying questions. Diagnose the pattern. Do not skip this. It is the entire game.

4.      Change the right thing. Instructions for vocabulary, context, and structure problems. Knowledge sources for genuine content gaps.

5.      Re-measure next cycle. If the answer rate did not move, the diagnosis was wrong, not the effort.

Moving Forward with Confidence

Auditing proves your agent was used. Search history tells you how to make it worth using.

The gap between those two is where most Copilot deployments stall. The analytics are already collecting. The themes are already clustered. The questions your users have been asking for months are sitting there, already grouped, already scored, waiting for someone to read them as instructions rather than as a report.

If nobody has opened the theme analytics on your agent, start there this week. Read the questions, not just the numbers. The agent will tell you exactly what to fix, in your users' own words.