Forgetting without warning: can we rely on AI to report research insights?

We recently looked into using AIs to embody users, customers (synthetic panels). Now we’re going one step further: we’re looking at how well AI can interpret, can embody user insights. The question we’re asking is what the best technical approach is for an AI to use user insights to answer questions. The fantasy behind it is still the same: talking to users when they’re not there.

And the technical question is what the best way to do it is. Is it better to give only well-targeted excerpts? That’s what’s called RAG: before each answer, a program searches the interviews for the excerpts that most resemble the question asked, and the AI only gets those excerpts. Which RAG approaches work best? And by the way, do we even need a RAG? For this comparison, we used made-up interviews, which means we know the right answer to each question. Our own data, anonymised, is the subject of a second article.

What we found is that, obviously, if you give few excerpts, even well-targeted ones, the AI misses a lot (25 points out of 53 with 6 excerpts), and the more excerpts you give, the better it gets. The problem is that these omissions aren’t documented. Roughly, you get a perfectly worded answer, which uses the user’s words, but which contains omissions you’re not told about, and which repeat. If you give the full interviews instead of excerpts, there are very few errors (2 out of 53), and we didn’t observe a single hallucination.

How we set up the test bench

The data we had was three fictional users of an online service, with ten interviews each, so about 14,000 tokens per user (a token is the unit of text the model reads and that you’re billed for). To query this corpus, we had 30 questions, broken down into 77 specific points. For 53 of them, we knew the right answer, and even the interview sentence that proved it. The other 24 were traps: either the topic, a plausible one, was absent from the interviews, or it concerned another user, because we were also asking ourselves whether things got attributed to the right user.

We tested Anthropic models, going up in size: first Haiku, then Sonnet, then Opus, all three in version 5.5. Each question was asked five times, to measure repeatability. At the end of the chain, a judge, another AI (Sonnet), scored the answers without having the right answer. To check this judge, we slipped it 48 inventions: it spotted 46.

Statistical tests

With 53 points, a gap of a few units can come down to chance. So for each rate, we give a 95% interval, calculated with the exact Clopper-Pearson method: really, it’s just the set of rates still compatible with what we observed. To compare two ways of doing things applied to the same points, we use an exact McNemar test, which only looks at the points that are right on one side and wrong on the other, and gives a p-value, the probability of observing a gap at least as large if the two ways of doing things were equivalent. And since we make five comparisons of this kind, we tighten the threshold to 0.01 (Bonferroni correction) so we don’t find a gap by chance.

If the AI has the info, it uses it, if not, it doesn’t guess it

Some of our results seem to push at an open door, others a little less so. If you provide few excerpts, even well-targeted ones, the AI clearly makes a lot more errors: 25 points missed out of 53 with 6 excerpts, 18 with 12, 7 with 24. If you load everything into the model’s context window, it’s even better, 2 out of 53, but again, because there isn’t all that much information.

These are inferential statistics: we measure a gap, and we check that it doesn’t come down to chance. With 6 or 12 excerpts, the difference with the full interviews is significant. With 24, it isn’t: we can’t say it does worse, or that it does better. So from a third of the corpus onwards, we can no longer tell the two apart.

The open door is the cause: if the AI has the info, it knows how to use it, and if it doesn’t have it, it doesn’t guess that it doesn’t have it. With 6 excerpts, when the information is among the excerpts, the AI answers correctly 28 times out of 29, and when it isn’t there, it never finds it, 24 times out of 24 (we had first counted 4 correct answers without the right sentence: the information was actually in another excerpt). The search doesn’t give it the right piece, so of course the AI gets it wrong. The less obvious case is one of our fictional users who says, at the start of his interviews, that a feature doesn’t exist, then discovers it in his last interviews. The search only brought back the first ones, and the AI answered with the opinion he had at the start. So with excerpts, you can lose the thread from one interview to the next. With the full interviews, the AI reads the whole story. And a stricter prompt or a bigger model don’t produce demonstrably better results.

Getting it wrong, here, means forgetting: no answer states the opposite of an interview, and on our 24 traps the AI made nothing up. Nor is it going to say, for each question, “Here’s my answer, but there may be other things in my corpus I don’t know about”. So you get a perfectly worded answer, which uses the user’s words, but with omissions you’re not told about, and which repeat. With 6 excerpts, of the 24 points missed that called for a specific detail (the 25th only checked that the AI abstains), the AI warns that something is missing in 13 cases, and nothing in the answer signals the gap in 11 cases out of 24. For 17 of the 24, the same omission comes back in all 5 runs.

What’s less obvious is the gap with common settings: by default, LangChain and LlamaIndex, two libraries used to build a RAG, give the model 4 and 2 excerpts respectively, fewer than our smallest run. We didn’t test it, but at 6 excerpts we were already missing nearly one point in two.

And our corpus isn’t a crazy size: one user’s ten interviews represent about 10% of the threshold where Anthropic summarises the context by default (150,000 tokens). We should still add that giving the full interviews is only possible on a corpus of limited size. If the corpus is huge, you’re back to the needle-in-a-haystack phenomenon (finding an isolated piece of information in a very long text), which the model cards of frontier models hardly document anymore: Anthropic stopped publishing it, OpenAI and Google only keep a harder version of it. And Anthropic recommends compaction (the model summarises what it has read to make room) beyond 150,000 tokens, while acknowledging that it loses context: beyond a certain amount of information loaded into the context window, there’s loss of data, of quality, of precision. We didn’t measure it, but it’s obvious: if it’s the best solution, it isn’t a solution that answers every problem either.

Does the judge get it wrong?

We scored the answers with a judge, so the judge had to sit the same exam. We gave it three.

First, we looked at its stability. Out of 154 judgements repeated 3 times, none changed.

Next, we looked at its ability to spot inventions. As in software testing, you deliberately build wrong answers, and you count the ones the judge catches. A fresh AI agent, meaning one that hadn’t seen our data (Opus), built 48 answers that deliberately state a false fact, at four levels of subtlety, from blatant invention to a single false sentence buried in an honest abstention, plus 12 sound answers.

The judge detected 46 inventions out of 48, the two it missed being at the subtlest level, and it didn’t accuse any of the 12 sound answers. The lower bound of its detection rate is 85.7%, treating the 48 inventions as independent when they actually come in fours on the same trap. The two it missed are false product rules slipped into an answer that abstains.

Finally, we looked at its plain accuracy, and there the judge gets things wrong too. On a question that checked whether the answer borrowed a word specific to another user, the judge answered “yes” when no answer used that word: they all used a neighbouring one. Run three times on the same five answers, it said “yes” 8 times out of 15, with a verdict that changes from one run to the next. We think it confused the two words, but it doesn’t explain itself. A corrected version of its prompt brings that number down to 0 out of 15.

What can this be used for?

The first conclusion is that vendors have stopped publishing the simple needle-in-a-haystack test, a sign that this simple test no longer gives models any trouble, even if the risk remains on a very large corpus. On a small corpus like ours, the model does very well. If the corpus is small, clearly, you should give it in full. On a large corpus, RAG is probably necessary, but it isn’t a cure-all either. It’s a term that has been very fashionable lately, there’s an element of hype, but when you measure, RAG doesn’t necessarily give good-quality results: with 6 excerpts, the AI misses nearly one point in two.

The second point is that you need to look at rules tied, probably proportionally, to the size of the corpus (how many excerpts to give for what corpus size), because if you don’t give an AI a piece of info, it can’t guess that the info exists elsewhere. You can’t assume a technical solution is the right one because it’s very popular. On research topics, where data volumes can be large, even very large, it’s probably reasonable to keep a critical eye on this approach, to make sure the LLM really covers the content exhaustively. Otherwise, you’re interacting with a partial version of your research material, and you can only notice it more or less by chance, when the LLM doesn’t answer something you were sure it should.

So this argues for systematically citing the sources used, and that applies here as much as to any other topic where you let an LLM go and fetch information, bring it back, or even synthesise it.

Limits of this study

We know nothing about an AI fed with real interviews, nor about open questions, since our 77 points are closed, nor about a corpus of several hundred interviews. On top of that, our RAG figures depend on a chunking choice, an excerpt here being one interviewer question with the answer that follows. Each of our three users is a single fictional person, interviewed ten times, whereas in a real project you interview different people who don’t all say the same thing: that’s the subject of the second article. And everything rests on very few pairs of eyes, because a single author wrote the right answers, a single judge scored all the answers, this judge is one of the three models tested, which all come from the same vendor, and the program that checks our right answers itself let through, before correction, 9 of the 39 defects we had deliberately slipped into it.

The illustration in this article was generated by AI (see our AI use statement).