Take your time

Editor's note: Daphne Dodson is managing partner of Qual-Smart LLC, a Cary, N.C., research firm.

I recently completed two separate projects that prompted my hyperawareness of the importance of a good screening. Though there may always be hiccoughs that arise during any recruit, preparing for and addressing common challenges helps to minimize significant issues. I’ll address some of those challenges here and some strategies for working around them.

Value the screener. The qualitative screening process is imperfect. The end client has the “right” respondent in mind and conveys this target to his internal market research partner who, in turn, informs her qualitative market research firm, who communicates this to the recruiting firm’s project managers, who train the actual recruiters, who screen potential respondents. Remind you of the game of telephone?

The screening tool is the document that should keep all project members searching for the same person. Thus, taking the time and effort to write, review, modify and then re-review the screener is invaluable. However, screener preparation, review and recruiter training are some of the most rushed components of the qualitative research process. Not only do the key members of the research team not take the time, we often don’t put in the effort to carefully review the screener to ensure that who we want to interview is exactly who we will interview based on our screening questions. We all own the issue but the ultimate accountability heavily weighs on the end client, who has a very firm mental grasp of the “perfect respondent.” If she truly understands this, my experience has been that she is far more likely to dedicate the effort to screener review.

Vet the screener. A well-established screener makes the screening process a great deal easier because in all likelihood bugs (such as poorly-written questions or impossible criteria) were worked out in the first few screener generations. When you are seeking a new target, you don’t have the luxury of relying on an established screener. You might have starting places – screeners that appear right for the job after minor edits. But beware: a “close screener” may create even greater issues because it allows for complacency.

I’ll offer a simplistic fictional example. Say the marketing team wants to begin to gather customer reaction to a prototype child car seat. You have conducted qualitative research with moms in the past for your infant car seats, so you dig out your existing screener as a starting place and adjust for the fact that this study seeks to interview moms of children (not infants). The first interview begins and the end client informs you this respondent is all wrong: “We didn’t want moms who drive a coupe; our child car seat won’t fit in the back seat of a typical two-door car.” (The infant seat never had this issue.) Interview two begins and you become painfully aware that the prototype has a weight limit – and there are some toddlers who apparently exceed it. The end client was aware the car seats were too big to fit in the back of a coupe and too small to accommodate larger toddlers but never thought about screening out moms based on their cars or their children’s weights because, honestly, the marketing team assumed that most moms drive minivans or four-door sedans and toddlers are about the same weight. Of course, this newly-gained awareness is in and of itself good learning but if the study objective is to improve the prototype, the sample of qualified recruited respondents dramatically dwindles.

Two of the best solutions to manage this potential issue are:

  1. Informally vet the screener. In this example, we could have thought of the moms we knew and called them to go through the screener and then to talk about the product with us. Informal “friends and family” interviews help us to see holes that are difficult to anticipate and assumptions that are inaccurate. This is especially helpful in medical or business recruits where our personal knowledge of the new target is even more limited.
  2. Formally vet the screener. This takes a little more money and time. In this case, you would screen respondents and then invite them to a short telephone conversation. The researcher would ask a few questions to ensure they are the right target and can be invited to the actual research interview. This technique helps ensure quality screening questions and is also a nice way to prescreen for respondent articulation.

Be mindful of elusive algorithms. Qualitative research offers a rich opportunity to better understand market segments, digging deeply to bring segments to life in a way that quantitative research simply cannot. To find respondents who fit the segments, algorithm questions provided by the quantitative firm are added to the screener. In most cases, they are questions in the quantitative questionnaire that drove the segmentation solution. The challenge is that these algorithm questions are often based on attitudes and reported behavior and segments are often developed based on degrees or skews of behaviors or attitudes rather than on absolute distinctions.

Let’s take another fictional example. A cosmetics company has an algorithm that includes the following questions:

On a scale of 1 to 7, how strongly do you agree with the following statement?

I feel very self-conscious if I leave my house without first applying makeup.

How long does it take you to apply makeup before you go out in the evening?

The difference between a response of a 5 or 7 on the first question and 15 minutes to 30 minutes on the second question might move a potential respondent from being segmented as “Easygoing Glamour” to a “Cosmetics Connoisseur.”

Consequently, respondents recruited to one segment may shift in their attitudes or reported behaviors between initial screening and interview and wind up in a different segment. Again, if we think of the example above, imagine that an “Easygoing Glamour” respondent walks into the group with seriously “smoky eyes” because her new, beautiful friend just introduced her to a how-to YouTube video and our respondent is trying to perfect the technique (which, by the way, took her an additional 15 minutes of application time).

This phenomenon proves not only frustrating but quite dicey when trying to draw conclusions from the research. To minimize the issue:

  1. Be aware of this when working with your quant firm to develop segments and segment algorithms. Algorithms that are heavily based on skews of attitudes and reported behaviors can be quite challenging when put into practice.
  2. Rescreen before the interview or at the beginning of the interview so that the qualitative researcher and listeners are not mistakenly forming perceptions based on incorrect information. 
  3. Ensure that the design of the study takes into account that a certain number of interviews are likely to shift. If the sample size is robust, interviews that shift can be removed or adjusted before analysis without compromising the study.

Beware of mini-quants. It is not uncommon for end clients to look at the screening process as an opportunity to do a bit of quantitative assessment. Consider all those information-only questions just to get a record of how many said what. Of course the operative term of “how many” exposes the issue – we are using a qualitative screening process to seek a quantitative metric. But why bother to care?

  1. These questions bog down a screener, impacting both the behavior of the recruiter and the potential respondent. The longer the recruiters are on the phone and the more questions they are asking any potential respondent, the greater the risk of quality degradation. It is simply reality that sometimes long screeners result in sneaking in a “bad recruit” just to get the project complete. 
       Picture a doctor, a mother or a businessperson on the phone answering a myriad of questions to see if they just might qualify for the actual paid interview. At a certain point during the screening process, many potential recruits wane in actually listening to and thoughtfully answering questions – even the questions responsible for screening them in or out of the study.
  2. The sample size, sample base and screening methodology for qualitative recruiting cannot offer quantitatively-sound findings.

Information-only questions can easily be added to the discussion guide and if they are not important enough to ask during the interview, arguably, they should be stripped from the screener. Occasionally these questions are added to the screener to guide future studies (gaining directional findings regarding incidence rates of responses before making them recruit specifications). While this can be an appropriate use of such questions, a better approach (than risking the current recruit) is to ask recruiting partners to e-mail blast their databases to pretest a specification.

Quality often requires flexibility. Some clients hold the perception there is nary a reason to not find a respondent. It often becomes shocking to clients when they come to discover that the man who rides a unicycle to his job in the staple-removal department of an allergen-free peanut factory is difficult to find. Okay, I’m being cheeky but my sentiment is rooted in reality. The market, methodology and logistics must suit the screening criteria.

A great deal of my work is spent in health care and I’m always surprised when clients think that because there are 60 specialists in the city we are heading to, we will have no problem getting the eight we want. I try to help them become more realistic when I explain that:

  1. We are asking them to leave their practices, commute 60 minutes round-trip and arrive on time for a specific hour and date (which we’ve deemed the best for us to listen to them, not best for them to attend).
  2. We only want them if they meet a certain size of practice, years in practice, prescribing behavior, etc. 
  3. The rate we are paying is barely, if at all, fair compensation for their time and effort.

This isn’t just a health care client issue. Consumer packaged goods and service industry clients are also challenged to remember that incidence rates of attitudes, behaviors, socioeconomic and even demographics differ greatly by geography – a national incidence rate is just an average. Couple this with our strict recruiting criteria, available interview times and our compensation rates and our large number of potential recruits significantly shrinks.

The best way to avoid this potential issue is to be thorough and open-minded during the design phase. The more the end client, market research client and qualitative researcher know who they want going in, the better we can design the study and select the best recruiting partners. This might mean choosing cities not based on where the client and agency are located but where the category development index is the highest. It might also mean a willingness to spend two days in one market to allow for evening slots, with no interviews during the day. It can also mean a mixed design (adding telephone and Web-assisted interview methodologies) to round out a needed sample size. It definitely means listening to the recruiters and qualitative researchers who are willing to stretch themselves but also recognize the importance of setting realistic expectations.

Invest a bit more

Consider the three factors that impact all projects: quality, cost and time. If we want a quality recruit, we might have to invest a bit more time and financial resources to ensure we have just who we need. If our timeline or budget is inadequate, we might need to accept less-than-ideal quality. While I like to believe that my skills as an interviewer and analyzer are essential to sound findings and actionable results, I humbly recognize that whatever talents I bring to the table are compromised or complemented by the person or people who I am interviewing. A good interview begins with a good recruit.