CDC turns to qualitative research to address fears of bioterrorism

Assessing the threat

Editor’s note: Mark Herring is senior moderator for Market Directions, Inc., Kansas City, Mo. He also is president of Mark Herring Associates, Inc., Malvern, Pa. Richard Tardif is a senior scientist at the Oak Ridge Institute for Science and Education, Oak Ridge, Tenn. This report is based on work performed for the CDC through an interagency agreement with the U.S. Department of Energy.

Since the terrorist attacks of September 2001, many agencies and organizations have worked to prepare for other attacks on the United States. The Centers for Disease Control and Prevention (CDC) is the agency primarily responsible for protecting public health in the event of a bioterrorist attack. CDC is made up of 16 centers, each of which has responsibility for addressing a high-priority public health concern (for example, infectious diseases). Each center does extensive research in its area of responsibility. Based on the data they collect, these centers develop guidelines and recommendations that they distribute to physicians, nurses, public health officials and other interested parties.

Almost immediately after the terrorist attacks, the National Immunization Program (NIP) at CDC began to address the possibility of a future smallpox outbreak. NIP selected smallpox as the focus of its market research because smallpox is the worst-case scenario among the biological agents that might be used for bioterrorism - the disease is contagious, it kills approximately 30 percent of those who contract it and many of the remaining 70 percent have permanent disfigurement or other health problems.

NIP had three general goals in mind for this market research project. First, the organization needed to identify those areas where key stakeholders - health care professionals and the general public - were uninformed or misinformed about smallpox. Second, NIP wanted to use this feedback to craft messages that would effectively educate stakeholders in advance of an outbreak and to create other messages for use during or after an actual smallpox crisis. Finally, this study was designed to test the effectiveness of all messages with their respective target audiences in advance of an actual health crisis.

Fascinating and unusual

As soon as we began working on this project, we discovered that dealing with smallpox was a fascinating and unusual topic for market research, with many unique elements. One challenge was to complete research on a disease that no longer exists in the natural environment. The last case of smallpox worldwide was in Africa in 1972. The last case in the United States was in Texas in 1949. Most doctors and nurses have never seen a case of smallpox and have received no training in the recognition and treatment of the disease.

Another initial learning was the importance of using qualitative research techniques for this project. We decided to rely on qualitative for several reasons:

• Many times during these conversations, we needed to drill down beyond the cognitive, intellectual responses to get at the rich emotional undertones. In fact, respondents expressed strong emotions throughout this project.
• This project was fast-moving. We were in the field collecting data less than two weeks after receiving approval for Phase II of the project. As one CDC staff person remarked, “For a federal agency, we were moving at the speed of light.”
• Several issues, for example balancing the risks of being vaccinated with the possible benefits of inoculation against a disease that might never appear again, were stunningly complex. Having the flexibility to probe for understanding was crucial.
• Both phases of the project involved distributing pictures or written materials and getting real-time responses to them. As a practical matter, we needed to meet with people face to face.

Specific methodology and outcomes

Three organizations were involved in this research: NIP at CDC; the Oak Ridge Institute for Science and Education (ORISE), an organization that provides technical assistance to the CDC and other federal agencies; and Market Directions, Inc., a market research firm selected by ORISE and approved by CDC to help design the research project and perform the research.

Phase I – April, 2002
The goals of the initial phase of our research were to:

• measure physician and public knowledge/beliefs about smallpox disease and the smallpox vaccine;
• assess whether these perceptions varied by racial/ethnic group;
• gather data about vaccination strategies that physicians and the general public perceive as viable – both pre-outbreak and post-outbreak;
• determine respondents’ interest in receiving the smallpox vaccine.

We completed this study using one-on-one interviews with physicians and focus groups with members of the general public. Ultimately, we interviewed 17 physicians. Each physician was board-certified in either family practice, infectious disease, emergency medicine or pediatrics - the medical specialties most likely to deal with cases of smallpox during an outbreak. We completed these interviews in Philadelphia, Chicago and San Francisco.

Table 1 shows the composition of the focus groups. Same-race moderators were used for all focus groups, except the four Asian-American groups, which were completed by the European-American (Caucasian) moderator.

Key findings:

• Neither physicians nor the general public knew much about smallpox disease, the smallpox vaccine or smallpox immunization strategies.
• Physicians and members of the public questioned whether ring vaccination (promptly identifying people with smallpox, then vaccinating them and all the people they have recently come into contact with) would be sufficient for controlling an outbreak, given today’s mobile population.
• People had little enthusiasm for being vaccinated before an outbreak of smallpox.
• Respondents believed that, in the event of an outbreak, the health care system (particularly hospital emergency rooms) likely would be overrun with panicked people wanting to be vaccinated.
• Members of racial minorities, particularly African-Americans, distrusted the federal government and questioned the social equity/social justice of any smallpox immunization program.

Phase II – December, 2002
During the fall of 2002, the Bush administration announced plans to again begin vaccinating American citizens against smallpox. The first wave of vaccinations would include military personnel and members of state-based smallpox response teams. On short notice, CDC was asked to collect the following information:

• provide an update about the knowledge of smallpox and the smallpox vaccine among physicians, nurses and hospital service workers (people working in ancillary areas such as maintenance, housekeeping and cafeteria);
• determine the factors that would encourage health care workers to be vaccinated for smallpox, if asked, and identify the barriers that would discourage them from doing so;
• test a Smallpox Q&A information sheet and the text for an informed consent document that might be distributed to health care workers prior to vaccination.

As with Phase I, we collected data from physicians through one-on-one interviews. In contrast to the earlier study, we recruited only emergency-room physicians, since we had learned previously that they were most likely to come into contact with smallpox victims, in the event of an outbreak. Because the agenda was shorter, more focused and potentially more emotionally charged than a typical focus group agenda, we asked nurses and health care service workers to participate in mini-groups - one-hour groups involving no more than four persons. Table 2 offers details about the respondents who participated in this phase of the project. The screener for this project insured broad representation by race, gender and age (over/under age 35).

Key findings:

• Knowledge about smallpox had increased markedly. In December, hospital services workers were as knowledgeable as several of the physicians interviewed in April.
• Many respondents said they would not volunteer to be vaccinated, once they discovered that they might not be covered by health insurance or worker’s compensation if they had an adverse reaction (got sick or died) after receiving the vaccine.
• Health care professionals saw the decision to be vaccinated as very important, an event that could ultimately have real consequences for family members and others, not just themselves. They said they would need time to consider their options thoughtfully.
• Most immunizations or health care treatments are assessed according to a risk-reward algorithm: Is the risk I am taking in receiving this treatment outweighed by the potential benefits of the treatment? Because no one knows the likelihood of a smallpox outbreak, this algorithm could not be applied.
• There were concerns among health care professionals about the safety of the smallpox vaccine.
• Overall, we predicted a lukewarm reception to requests for health care professionals to be vaccinated for smallpox.

Project impact

Data from both phases of this project have been reported widely. In May, The New York Times drew heavily on our work in a substantial article about smallpox. While Phase II was underway, CDC staff members were in contact daily with the Office of the Director of CDC and key aides in the Department of Health and Human Services (HHS). As a result, a few of our preliminary findings were reported in USA Today even before the project concluded. Recent policy statements and press releases from HHS and CDC have referred explicitly to the data collected during this project.

Because smallpox is an ongoing policy issue at the national level, it is too early to fully determine the final impact of this project. Clearly, the data we collected is one of many data sources on this topic. However, we can point to several actions at the federal level that are consistent with our findings:

• There is legislation now pending in both houses of Congress to insure health care professionals have personal financial protection, if they should have an adverse reaction to the smallpox vaccine.
• This study provided an early indication that there would be significant obstacles to overcome to achieve widespread voluntary vaccination of health care professionals. In fact, recent media analysis by the CDC shows that of the 261 health departments and hospitals mentioned in print articles from December 10, 2002, to February 28, 2003, 61 percent are participating in the vaccination effort, 25 percent have declined to participate and 14 percent are undecided.
• The Advisory Committee on Immunization Practices, a national scientific advisory committee to the CDC and HHS, received the findings from Phase I and used this data in the formulation of its recommendations about how and when to offer smallpox vaccination. These recommendations received extensive national media attention.
• The CDC has developed a variety of education materials about smallpox, particularly a brochure that helps physicians identify smallpox in its early stages and distinguish this disease from other maladies that have similar initial symptoms.
• Acknowledging concerns about vaccine safety, CDC has commissioned several studies to test the efficacy of the vaccine and attempt to predict more accurately how frequently adverse reactions will occur.

Risk communication

One of the most powerful and interesting uses of this data is in the area of risk communication. Generally speaking risk communication has two purposes: to calm the general public if they are unjustifiably alarmed (for example, in the case of a chemical spill) or to encourage the public to take action if they are too apathetic (for example, using seat belts or condoms). In the case of smallpox, the CDC has used this data for both purposes. It has prepared for a smallpox emergency by developing message maps that succinctly but accurately describe what the public should do, while hopefully reducing the public’s sense of alarm or panic from irrational levels. The CDC and the Oak Ridge Institute for Science and Education are now offering training sessions and disseminating this information to federal, state and local health officials and emergency planners throughout the United States. Because of interest and demand, they are now offering these materials to health officials around the world.

Lessons learned:

• “Scientific” methods. Many key leaders at the CDC are scientists. As such, they have a strong bias toward research that generates numbers, percentages and is subject to computer analysis. Over time, these decision-makers have come to some appreciation of qualitative research. However, we have to be constantly aware of their need for hard data. In some situations, this means we must resist overtures to quantify qualitative research. In other circumstances, it means we need to adapt our qualitative design to more closely approximate the methodology used in “hard science.”
• Projectives. Experienced qualitative researchers do not need to be sold on the value of projective techniques. In this project, projectives proved invaluable. Rather than asking people to identify the assets and liabilities of voluntary smallpox vaccination, we used psychodrawing (in this case, a stick-figure exercise) to elicit this information. We modified the conventional stick-figure exercise by placing a second, smaller stick figure in the upper right-hand corner of our worksheet. We instructed respondents to describe what this person was thinking and feeling as he/she watched the primary stick figure deliberate about being vaccinated. In this study, the secondary figure soon became “my pregnant wife,” “my daughter” or some other key person. The exercise elicited emotional content that was richer and deeper than could ever have been gleaned from an “I ask, you answer” approach.
• The geography of terrorism. Based on our research, we believe the United States is divided into three regions when it comes to public attitudes on terrorism and its perceived risks. People on the East Coast, our first region, believe strongly “it will happen again here.” Most of these people were intimately affected by September 11, 2001. The attack on the World Trade Center is an event they take personally and they discuss their feelings about terrorism in very emotional terms. The second group, the West Coast, sees terrorism as more abstract, because they have not experienced it personally. They feel little urgency about preparing for future acts of bioterrorism, since they believe future terrorist acts are likely to occur first on the East Coast “and we would have plenty of notice.” As one San Franciscan said, “We are patriotic Americans. But for us, on a practical basis, New York City is the other side of the world.” The third region, which we call “the big middle,” is midway between these two psychological extremes. Terrorist acts seem real to them and provide motivation to take personal action, but the feelings are less extreme than those held on the East Coast and less disengaged than those on the West Coast.
• Predictive value. Dr. Pete DePaulo (see “Sample size for qualitative research,” Quirk’s, December, 2000; enter QuickLink number 636 at www.quirks.com) and others argue persuasively that by having in-depth contact with 30 individuals, we can uncover 90-95 percent of the important ideas on a given topic. One goal of this project was to identify the most significant issues the public and the media would latch onto about smallpox. According to Alan Janssen, health communications specialist, National Immunization Program, CDC, “We’ve had no surprises. As I do media tracking, all of the major issues being addressed in the media are issues we identified during this project.”
• Race/ethnicity of moderators. We found it helpful to have an African-American moderator for the African-American focus groups. While we have no means of comparison, we suspect that respondents were much more forthcoming about their smallpox concerns and their distrust of government because an African-American moderator was facilitating the discussion. We also learned the value of having a Spanish-speaking moderator for the Hispanic-American groups. In several instances when emotions were high or English vocabulary was low, respondents used Spanish to make key points. On one occasion, our moderator simply said, “Say it to me in Spanish,” when the respondent was at a loss for words.
• Need for decompression. Many elements of this project were emotionally charged. In Phase I, we showed respondents graphic photographs of individuals who had suffered adverse reactions to the smallpox vaccine as well as a photograph of a smallpox victim. For many participants, talking about the events of September 11, 2001 was a painful process. Early in Phase I, we decided to conclude each group or IDI by having a CDC representative speak with the respondents, providing context for the project (i.e., reassuring them there was no known threat of a smallpox attack), answering questions and allaying unreasonable fears.

In summary, applying research tools and techniques to bioterrorism has been a learning experience for each of us. Working on this project has had permanent impact on how we think about qualitative research and how we will use it in future projects.