Smoke, two-way mirrors and fakery: the cult of the qualitative guru

30 January 2015

I’m going to be a little controversial in my views about the roles of moderators and qualitative researchers in the insight process. However, I’d also like to raise some serious points around the nature of data collection, data analysis and data interpretation.

I like data! When fieldwork is complete on a project, I like to see my desk overflowing with transcripts, pictures, post it notes and ‘stuff’. This means that I can begin the uncomfortable, but ultimately rewarding process of assimilation and looking for meaning. I hate having poor quality data or insufficient data. The process is very different. Instead of having to sift and edit, one finds oneself dissecting each and every sentence and extrapolating from the data in order to produce something meaningful.

I wonder sometimes whether the difference is obvious. Is there a discernible difference between a presentation deck which includes 10 verbatims because these were about the only usable quotes in the dataset, and one with a similar number that have been carefully selected from the hundreds of similar comments available? I hope the difference is obvious. Hopefully there is something universal that ‘rings true’ in something constructed from large dataset and something that is ‘a bit fake’ and cobbled together from poor data.

So, starting from the need for good data, and lots of it, where does this lead us in terms of methodology?

I’m an ‘entertainer’ by training (I studied classical voice), so I quite enjoy the theatrical dimension to groups. Viewed groups are a form of performance art where the moderator is playing to two audiences; the respondents and the people in the back room. It’s a bit like one of those Shakespearian speeches where there is one message for the groundlings and a ‘hidden’ subtle subtext for the more educated audiences. But I am sometimes doubtful over both the quality and quantity of data that comes out of a focus group, and in particular from larger group sessions.
I’ve learned over the years that when a client asks for focus groups, I do my level best to dissuade them from having more than five or six respondents at the most. I’ve heard a number of different explanations as to why eight should be the ‘standard’ number. Apparently it’s possible to get ‘more and varied input’ if you have between eight and ten participants’. Actually if you look online to find the reasons for this ‘magic number’ you’ll find an awful lot of comments along the lines of ‘focus groups tend to work better with around 8 people’, and very few comments that finish that sentence’; because…., the reasons are…..

I’ve heard it suggested that if one runs six groups with eight people the results will be more ‘robust’ than from six groups with five people; forty eight is better than thirty! I’ll leave my quantitative methods colleagues to discuss the merits of that one. Another example of circular reasoning: ‘we have to have eight people because that’s the standard way of doing things’. Here we have an example of a ‘standard’ that has dubious provenance and which creates a whole series of practical issues.

Let’s return to the data issue and use a ‘standard’ focus group with eight participants as an example. Assuming that the moderator is able to get each respondent to speak for an equal length of time (a small miracle in itself), they would have about eight minutes each. The moderator inevitably talks for longer as they’re having to read all the questions as well as describing any stimulus involved. Let’s also assume that I am the moderator or that I watch the video rather than rely on the transcripts. I have eight minutes to find out who this person is and why they are saying what they say. Inevitably any reading of this person is at a surface level. People are complex and nuanced. The danger in a group situation (especially a larger group) is that we observe a collection of apparently one dimensional characters and form opinions about them and their views accordingly.

For some qualitative researchers, a significant part of their status is tied up with their abilities as a group moderator; it’s one of their key skill sets. This is compounded by the fact that many experienced quantitative researchers quake at the thought of running a focus group. However, as far as I can make out there is no absolute measure of what makes a good focus group moderator or a bad one. I’ve seen highly empathetic and confident moderators. I’ve seen moderators who phrase questions cleverly, either to shock or to demonstrate their understanding of Neuro Linguistic Programming, Transactional Analysis or some other psychological model. However, many of these models and techniques are more focussed on delivering ‘nuggets’ of interesting data than they are on producing sustained quantity of data.

I was recently talking to a client about the role of the moderator in research. I suggested that, if moderators are relying on their skills, instincts, experience and sector knowledge rather than the data obtained from the research, in reality they are more consultants than researchers. They are effectively ‘reading between the lines of the transcript’ and extrapolating from there. Her opinion was that the best moderators are consultants and that she was comfortable with this. My feeling is that I am uncomfortable when presenting findings to clients without the data to back this up. Even if a finding is based on a view expressed by one or two respondents, I like to be able to anchor my findings back to these views and to be able to explain the context.

So where do you get great data from? Other than focus groups, you get greater understanding and emotional depth from a depth interview when you employ a creative device such as ZMET. I’ve also seen really good data from bulletin boards, where people have the time and space to respond to questioning. These work particularly well when respondents are highly engaged with the topic. In my experience, people provide much more honest and useful data in group situations where you’re asking them to do something rather than respond to a series of questions. So, the insight comes from what they do rather than what they say. I guess what I’m saying is that where we’re asking respondents to interact, we should let them get on with it as far as possible and simply observe; become more like ethnographers than psychoanalysts.

A senior colleague once introduced me to a client as the agency ‘qualitative guru’, but seriously I hate this label. I hope that I’m a confident and engaging presenter, but I honestly want my clients to look beyond me and appreciate the data I present to them, whether it’s on the PowerPoint slide or coming out of my mouth.

One of the things I was taught early on at music college, is that there are essentially two types of singer: There are the ones that you listen to and you think, ‘what a beautiful voice’, and there are the ones you listen to and think, ‘what a beautiful song’. I’d much rather be seen as one of the latter.