Can an AI embody a persona?

As a preliminary to this small piece of research: for us, personas are always built from real data, but they’re archetypes, caricatures, not real users. So a persona is a patchwork of customers, the barycentre of a set that holds together. The other way round, a customer is a patchwork of personas. We’re wondering how to make this approach hold up once you put AI into it!

We’re going step by step. The first step was to ask whether AI can answer in place of real customers. We devoted a Sandlab article to it, and the answer was rather no, at least not with the level of quality you expect from user research. The second step is what a designer or a researcher might want to do: an LLM plugged into a repository or a user research database. What can we do with it?

One of the fantasies people have shared with us on persona projects in recent years goes like this: “Great, now that we have user data and personas, we’ll be able to plug the personas into that data.” It can be user data, the CRM, anything: you plug the archetypes, the caricatures that personas are, into these sources, as in a classic workshop, but with all the dynamic, interactive side that the LLM offers.

Because on this point, we can only agree: a persona is static. You can work on its look and its organisation so that the team takes ownership of it and uses it day to day, which actually sometimes works not too badly, but it remains a static medium, and like any static medium, it’s a bit of a pain. So the ambition of bringing things to life with LLMs is perfectly understandable, and to be honest fairly legitimate: keeping alive both the user knowledge and the personas we took the trouble to build. That’s the ambition of this study.

So the question we’re asking is this: if we plug personas into the data, routing it to them correctly, does a persona manage to use the data attached to it to give a relevant answer? A persona is only ever the barycentre of a set of users, summed up by a certain number of dimensions, and necessarily not all the dimensions of all the users: the combinatorics would make that impossible. Does this barycentre manage to reflect the diversity, or does it average it out? If it averages, does it do so in an interesting way, or do we lose a lot of information? If we decide not to average, can it reflect the group it represents, with its subtleties? Is it useful, relevant, reliable? And if we answer no to both questions (does the persona average well? does it reflect its group?), don’t we give up the idea of a persona embodied by AI, so that the AI embodies the diversity of the panel instead? And then, how precisely?

A persona speaks for several customers, and these customers don’t agree with each other: in our notes, they diverge on 34 topics. So we test three hypotheses:

1. The “character” persona takes a stance that, somehow, averages the group it represents.
2. The “arbitrating character” persona picks up trends here and there, and tries to render the subtlety and the diversity of the group. It still speaks in the first person, but it cites the notes it uses and flags disagreements between customers.
3. The “panel” drops the character, arbitrating or not, and goes purely with a panel logic. It no longer speaks in the first person: it presents what the customers say, how many notes touch on the topic and which positions emerge, with the codes of the notes behind each position.

Test bench: seven personas plugged into their notes

On the protocol side, we use personas built rigorously, the way we know how to, from qualitative and quantitative data. Each persona receives its sheet and, attached to that sheet, all the notes from the customers who resemble it: between 6 and 15 notes depending on the persona, so roughly 4,000 to 8,000 tokens (the unit of text a model reads). A note is the summary of an interview written by the researcher, not a transcript. The previous article showed that RAG (automatic retrieval of excerpts) wasn’t an effective solution on a corpus this size. So this time we drop the idea: there’s no excerpt retrieval, the persona reads all its notes in full.

We ask three types of questions: 67 points where the right answer really is in the notes, 56 traps where the persona has to say it doesn’t know, because the topic isn’t in its notes, and the 34 topics mentioned above, on which we know the customers diverge.

Haiku and Sonnet, two of Anthropic’s three models (Opus is the biggest), play the persona, and each question is asked 5 times. We set Opus aside for budget reasons: it’s clearly too expensive. On the other hand, Opus is the one judging the answers, and another model re-judged 100 cases completely blind. It agrees with Opus on 95. This wasn’t checked by hand, but with 5% disagreement, we’ll settle for it for this study.

A very important point: no customer note or quote is published here, only counts. Everything is anonymised.

How we tell a gap from chance

With 67 substantive points and 34 disagreements, the counts are small. So for each rate, we give a 95% interval, calculated with the exact Clopper-Pearson method: it contains all the rates still compatible with what we observed. To compare two instructions applied to the same questions, we use an exact McNemar test, which only looks at the points that are right with one instruction and wrong with the other. Below 0.05, we usually consider the gap demonstrated.

No inventions across 56 traps

First point, and it’s a real concern: hallucinations, inventions. Whatever the scenario, across the 56 traps, those questions on topics absent from the notes, we don’t observe a single invented answer, and that holds whatever the model, Haiku or Sonnet. We only have 56 traps. What we can say is that the hallucination rate is below 6.4%, with 95% confidence, but we really did get zero occurrences on these 56 traps.

We also tried feeding the persona in three other ways. Its sheet alone, as a control: without any notes, does the model invent from the stereotype in the sheet? Its sheet with only the notes from the customers who most clearly resemble it: more homogeneous groups, with fewer disagreements. All the project’s notes, including the salespeople’s, each marked as such: this is the realistic case of the whole file dropped into a tool, and the only one where we check that the persona doesn’t claim what a salesperson said. Here again, no inventions. We have few cases, and these trials mostly ran on Haiku, but it’s fairly reliable.

The character forgets one point in two

Second aspect of the quality and reliability of the approach: after hallucinations, the question isn’t what gets invented, but what gets forgotten. Out of 67 points whose answer is in the notes, Haiku forgets 56 when it strictly embodies a character, 49 when it arbitrates and 21 when it represents the panel. The character’s 56 omissions are fairly logical: since it averages, it’s inevitably not very exhaustive. Sonnet does very clearly better, but it remains very unreliable. It forgets one point in two when it embodies a character, one in three when it arbitrates, and a bit more than one in four when it represents the panel.

So even with a panel, we lose many more user insights than in the previous study. Reading all the interviews, the AI made only 2 errors out of 53 there, whereas here we get 19 out of 67. With Sonnet, the gap is significant between the character and the panel, and also between the character and the arbitrating character (McNemar test, p below 0.0001 in both cases). Between the arbitrating character and the panel, we can’t call it (p = 0.76).

The AI is, on the other hand, consistent in its unreliability. With Sonnet, as a panel, it changes its verdict across the 5 trials on only 12 questions out of 215 (those where a stable answer is expected, disagreements excluded), against 49 when it embodies the character.

Finally, we tested Opus as a panel, on just 24 points. It misses 6, which Sonnet had already missed. It’s a much bigger and much more expensive model, and it doesn’t wipe out these omissions.

So how do we explain it? Honestly, we don’t know. It’s not the amount of information: each persona only has 4,000 to 8,000 tokens to read, less than in the previous article, where the AI only missed 2 points out of 53. So we’re not in a needle-in-a-haystack situation. On a big corpus, on the other hand, we would be, and we haven’t compared a RAG with an overloaded context window.

We looked at what we could measure with the data we have. It’s not the number of customers: with Sonnet, a persona built on 15 customers doesn’t forget more than one built on 6. Nor is it the position of the note in the stack, we don’t see any particular forgetting in the middle. And the missed points aren’t harder facts than the others: with Sonnet, the panel and the character miss 12 in common, barely more than the 10.5 we’d expect if each missed its points at random.

With Sonnet, two trends come back with all three instructions. Notes at the start of the stack are retained a little better. And facts from customers the persona shares with other personas are forgotten more often than those from customers who mainly resemble it: as a panel, it’s 12 out of 37 against 3 out of 23. This second trend is consistent with our starting point, but none of these gaps is statistically demonstrated, and with 7 personas, we can’t go any further.

One last finding: of the 67 points, 7 come from the persona’s sheet rather than from a customer note. With Sonnet, the panel forgets 4 of these 7 facts, whereas the character forgets none. That’s fairly logical, since the panel reports what the customers say, and the sheet isn’t part of that.

When customers don’t agree

That leaves the particular situation of customers who belong to the same set, represented by the same persona, but whose opinions diverge. We had identified 34 topics on which the customers associated with the same persona diverge. On these 34 topics, with Sonnet, the character only flags the disagreement on 5. On 14, it gives a single opinion, always the same one, without saying that other customers think the opposite. It changes its mind from one trial to the next on 3 topics, it invents an intermediate position that no customer expressed on 2, and it mostly answers “I didn’t talk about that” on 10.

The arbitrating character and the panel fully play their role, since that’s what their instruction asks of them: they flag the disagreements 34 times out of 34. The flip side is that we had also set 17 topics on which the customers agreed. The panel sees a non-existent disagreement there in 7 cases out of 17, and the arbitrating character in 10 cases out of 17.

Opinions that change

Another topic: stability over time. The character seems to change its mind from one trial to the next on 5 topics with Haiku and on 3 with Sonnet. Of these 8 topics in total, there are only 2 where the model really alternates between one customer’s opinion and another’s (one with each model, with 4 trials for one opinion and 1 for the other). The other 6 mix a customer’s opinion with an “I didn’t talk about that” answer, or with an intermediate position that no note contains. So confusion between two opinions present in the notes only happens 2 times out of 8.

Character or panel: neither makes the cut

If we have to draw an overall conclusion, whatever the setup, “character”, “arbitrating character” or “panel” persona, no solution gives good-quality results. We tested 15 configurations: 3 instructions, 2 models, several ways of supplying the notes. None meets all the thresholds we’d set before the trials, for example no more than 3 omissions out of 67 points. Even the best one, the panel with Sonnet, is still a fair way off: 19 omissions when we accepted 3, 12 points where the answer changes from one trial to the next against a threshold of 10, and 7 disagreements seen where the customers agree, when we only accepted one. These thresholds were arbitrary, admittedly, but we’re still very far off.

The root of the problem is how the problem is framed. A persona that would embody the diversity of user opinions will never be able to embody it in a single voice. It will inevitably say “some think this, others think that”. In the best case, if it’s faithful, it’s no longer a persona, it’s a panel report. And if you force it to be a persona, it has to pick a side. That’s what Sonnet does on 14 of the 34 disagreements, and it does it without warning.

The conclusion is that we can look at using an LLM as a front end to a database of insights, of user research. It’s possible. It’s probably not possible through personas, as quite a few people are apparently trying to do right now. And when you think about it, it’s obvious. On top of that, and this is a more general caveat, the panel isn’t reliable, it isn’t faithful, not to mention that it isn’t a character. Yet it’s the only alternative that offers a minimum of quality, and it doesn’t even reach the level we’d set ourselves.

Limits of this study

We worked on a single project and seven personas, with notes summarised by researchers rather than transcripts, and closed questions. The right answers were written by a single person. On one of the criteria, the sorting of notes under each position, a review by someone else found one expected answer wrong and two debatable. The judge is a model, as is the one that re-judged 100 cases blind. For cost reasons, we stopped at Sonnet: Opus only ran on 24 points. And with seven personas, we can only offer leads to explain the omissions.

What can it be used for?

Fundamentally, it has the merit of questioning the value of the persona method all over again. Right from the start, it was created to build empathy. There’s always this request, perfectly understandable and legitimate, to tie personas to tangible data. Throughout the agency’s life, on the projects where we designed personas, we’ve very regularly, not to say systematically, seen teams ask us to what extent these personas could be linked to CRM data, for example. We already talked about it in a blog article on personas and CRM. Quite honestly, whether you want to link them to the CRM, to user insights in general or to a research repository, the issue is exactly the same. It’s legitimate to want to make the link with data, but that was never what personas were about: they were never meant to be simply a gateway to data.

In Alan Cooper’s The Inmates Are Running the Asylum, the persona is first of all a way to project yourself: “We become character actors, inhabiting the minds of our personas.” And he embraced the caricature: “Stereotypical personas are more effective if the stereotyping lends more credence to the persona […] I am shooting for believability, not diversity.” He was aiming for believability, not diversity, whereas it’s the diversity of its customers that we’re asking the AI persona to render here.

A persona is supposed to create empathy, rely on fuzzy logic, let you project yourself. User research and qualitative methods don’t deliver truth, and neither do personas. When you caricature an approach that by its very nature gives a fuzzy view of reality, you’re not going to consider that you have a better way of getting at the truth, whether you put LLMs in the loop or not. You can’t hope to create caricature and, by way of that caricature, reach fidelity. There’s something of a contradiction in that.

If you choose to have AI embody these personas anyway, the best lead may not be to link real data to the personas, but to give them more depth, a background. You stay vague, in the average, you don’t go looking for the precise insight, but maybe for a slightly different kind of fuzzy, one that goes a bit further. You stay in caricature, and the insights will remain caricatural: they don’t reflect the subtlety, the singularity of what was collected. But you can probably go a bit further than PDF sheets, by embodying, by giving a voice.

What it comes down to is that you can imagine an LLM being an interactive, conversational, generative front end to collected data, and being a relatively faithful translator of it, but not a perfectly faithful one. It’s then no longer really a persona, more a conversational front end to a research repository, which has to be properly tagged. And if you consider that fidelity lies in qualitative approaches, it’s quite likely that you can’t replace fidelity on small samples with pure invention, which would be a kind of average. So the question is whether it’s really reasonable to keep making personas plugged into data talk.

Another angle would be to use the indexing and information retrieval capability to fetch information from a database. But there too, we get errors: we run the risk of information that really does exist simply being forgotten. An LLM’s main capability is generating content, not indexing correctly.

One last point, it could be useful to check whether quality improves by tagging more and better. But tagging more and better is more painstaking work, and you might wonder whether we wouldn’t be introducing new biases by giving in to the temptation, a very legitimate one too, of handing this labelling over to an AI. That’s a topic for a future article.

Or we go back to the users.

Text and illustration produced with the help of AI, under the supervision of the team (see our AI use statement).