Editor's note: The following article is an excerpt from a U.S. Dept. of Commerce report, "Approaches to Developing Questionnaires."

The National Health Interview Survey (NHIS) is a repetitive survey which collects health and demographic information from a national sample of about 40,000 households each year. Field operations for the survey are performed by the Bureau of the Census under specifications established by the National Center for Health Statistics (NCHS).

With the objective of fielding a revised NHIS questionnaire in the early 1980's, a series of field tests was planned to test an evaluation version of the NHIS questionnaire. The evaluation version, or experimental questionnaire, was designed to eliminate redundancies, define health concepts more explicitly, present topics in a more logical order and enable interviewers to use the material efficiently and smoothly. In conjunction with the results of a statistical analysis of the test data, the feedback from an extensive program of observation provided the basis by which to judge whether the objectives of the redesign had been achieved.

The testing was conducted in three phases.

1. Phase I (June). The first version of the evaluation questionnaire was administered in 250 households in Springfield, Ohio, by 15 Bureau of Census interviewers. The primary purpose of this informal test was to form a qualitative or subjective assessment of the draft instrument.

2. Phase II (October-December). This phase of the testing was designed as a formal (split sample) test to quantitatively assess the revised evaluation questionnaire by comparing selected estimates produced by the standard NHIS document and the experimental document. The control group, consisting of the fourth-quarter NHIS sample (10,500 households), received the standard questionnaire. The experimental group receiving the evaluation questionnaire contained 5,000 households selected in the same manner as the control sample. Randomization of questionnaire versions among interviewers was not possible because of the risk that the fourth-quarter estimates from the continuing survey (the control group) could be affected by interviewer confusion of the two complex sets of rules and procedures. Instead, a separate group of interviewers administered each questionnaire version, the groups being matched as closely as possible on years of experience with the NHIS. The interviewers that had to be hired to meet the 50 percent increase in overall sample size were equally distributed among control and experimental groups.

3. Phase III (August of the following year). Based on the outcome of the Phase II experiment, the evaluation questionnaire was again revised and used in an informal test in York, Pa. Like Phase I, the purpose of the test was largely qualitative. The size of the sample and interviewing staff were also similar to those in Phase 1.

Programs of observation: Phases I and III

Since the design and objective of the Phase I and III informal tests were similar, their observation programs can be described together. Because both NCHS and the Census Bureau are involved in conducting the NHIS, observers from both staffs took part in the tests. The NCHS observers represented all of the disciplines involved in the survey's development, including questionnaire design, data analysis, and methodological design. The Census Bureau sent field supervisors and persons responsible for writing the training material and the interviewers' manual. Such a large and diverse observation team allowed for broad coverage of interviewers and a range of professional experience by which the adequacy of the training and questionnaire could be judged.

The test site and sample of households were selected by Census Bureau specialists in accordance with demographic, budgetary, and other procedural requirements. The households to be observed were determined indirectly by pairing observers with interviewers so that all interviewers were observed for at least one day, but not more than one day, by the same observer. Observations were conducted throughout the five-day field test period. Approximately half of the test interviews were observed. The interviewer training session and the interviewer debriefing were also observed.

Observers from NCHS relayed their impressions in three ways: (1) observation forms-observers were asked to time major sections of the interview, pay particular attention to new or difficult questions and concepts and indicate whether questions were understood, needed elaboration, or were difficult to ask (some of these observations could be tallied to give an indication of how frequently each problem occurred); (2) observer debriefing-lead by one of the questionnaire designers; (3) written reports-specifying problems and solutions.

Census Bureau observers attended a separate debriefing which focused on the training materials, training session, interviewers' manual, and questionnaire.

Program of observation: Phase II

Organizing a program of observation for the national split sample test phase posed many more logistical difficulties than the single-site tests in Phases I and III, since interviews were spread out geographically and over time. Only the experimental group interviews were spread out geographically and over time. Only the experimental group interviews using the evaluation questionnaire were observed.

1. Interview Observations. At least one interviewer in each of 12 regions of the country was observed. An effort was made to observe both experienced and inexperienced interviewers. About 12 to 16 interviews were observed for each interviewer.

For each interviewer, observers completed a brief observation sheet. This form obtained times for the many questionnaire sections and provided space for comments. In addition, observers were given a detailed memo about potential problems in the questionnaire. It should be noted that all observers were extremely familiar with the data collection instrument and its underlying concepts and objectives. Based on their accumulated observations, observers were asked to submit a written report.

2. Interviewer Debriefing Sessions. After data collection had been completed, interviewer debriefing sessions were held in each regional office. These sessions were observed by NCHS staff and Census Bureau staff. Their written reports, summarizing interviewers' comments, were submitted to NCHS questionnaire designers.

3. Interviewer/Supervisor Evaluation Forms. Every interviewer and interviewer supervisor was asked to fill out a lengthy questionnaire evaluating the adequacy of the training materials, training session, interviewers' manual and the NHIS questionnaire.

4. Regional Supervisors' Debriefing Sessions. NCHS survey planners conducted and observed a debriefing session of the Census Bureau regional supervisors at the end of the data collection period. Because supervisors had conducted the interviewer training sessions and had observed all interviewers in their region, their comments on the adequacy of the training materials and questionnaire were valuable.

In conjunction with the results of the quantitative data analysis which compared estimates of key health variables obtained from the two NHIS questionnaire versions, the results of the more subjective field observations led to important revisions in the experimental questionnaire. This version was then tested in Phase III.

Results of the Observation Program

The questionnaire currently used in the National Health Interview Survey is the product of this multistage test in which observational feedback was as important as statistical analysis of the data. The evolution of the questionnaire during the phases of testing is illustrated by the series of questions asked to elicit reporting of visits to doctors during the two week period preceding the interview.

The NHIS concept of a doctor visit is defined as a consultation with a physician in person or by telephone for examination, diagnosis, treatment, or advice. This service may be rendered directly by the physician or by a nurse or other assistant acting under the physician's supervision or authority. The standard NHIS questionnaire used three probes to elicit reporting of doctor visits. They were:

"During the past two weeks, how many times did you see a medical doctor?" (Do not count doctors seen while a patient in the hospital.) Parentheses around parts of a question indicate to the interviewer that the statement is to be included conditional upon circumstances reported earlier in the interview. In this case, the statement is read only if the individual has previously reported a hospitalization.

"During that two-week period, did anyone in the family go to a doctor's office or clinic for shots, x-rays, test or examinations?"

"During that period, did anyone in the family get any medical advice from a doctor over the telephone?"

NCHS analysts suspected that the concept of physician visits was not being fully understood by respondents. Of particular concern was the under- reporting of visits to certain types of medical specialists, such as ophthalmologists and psychiatrists. Also, visits in which the patient saw a physician's assistant rather than the physician, phone calls made to obtain prescriptions, advice or test results, and visits occurring in places other than the usual doctor-patient settings were overlooked by respondents.

1. Phase I Version
The first version of the experimental questionnaire was designed to communicate the comprehensive definition of physician visit to respondents. The new questions were worded as follows:

"These next questions determine whether anyone has recently received health care from any kind of medical doctor-including general practitioners and any types of specialists, such as pediatricians, psychiatrists, ophthalmologists, and so forth. Also include health care received from a doctor's assistant or a nurse working under a medical doctor's supervision."

1. "During the two-week period outlines in red on that calendar, how many times did see or talk to a medical doctor or assistant? (Do not count times while an overnight patient in a hospital.)"

2. "(BESIDES THOSE TIMES) During that two-week period, did anyone in the family see a doctor or assistant for any surgery or operations, shots, X-rays, medical tests or treatment, or physical or mental examinations? (Do not count times while an overnight patient in a hospital.)"

3. "(NOT COUNTING THE TIMES YOU HAVE ALREADY TOLD ME ABOUT) During the two-week period, did anyone in the family receive health care at home or make any (other) visits to receive health care at a hospital, or doctor's office, a clinic of any kind, or any other place?"

4. "During that period, did anyone in the family get any (other) medical advice from a doctor or an assistant over the phone?"

Observers attending the informal Phase I test reported that the experimental questions were much too verbose. Respondents frequently interrupted the introduction to answer, "No," and would then become irritated at being asked the remaining questions. Instead of communicating the scope of the doctor visit concept, the wordy definition and qualifications seemed to badger the respondent.

2. Phase II Version
For the national split sample test, the introduction was shortened so that it became a transition statement between questionnaire sections while the function of defining the doctor visit concept was distributed among the follow-up probe questions. The probe about the nature of treatment received was eliminated entirely, while the types of telephone calls to be included were stated more explicitly. The questions were:

"These next questions are about health care anyone in the family may have recently received."

1. "During the past [the two weeks outlined in red on that calendar] how many times did see or talk to a medical doctor? [Include all types of medical specialists, such as dermatologists, psychiatrists, and ophthalmologists, as well as general practitioners.] (Do not count times while an overnight patient in a hospital.)" Statements in brackets were read the first time the interviewer asked the question in the household.

2. "We are also interested in the number of times anyone received health care from a nurse or anyone else working with or for a medical doctor. (Besides the time(s) you just told me about) During those two weeks did anyone in the family receive care at home or go to a doctor's office, clinic, or hospital to receive health care?"

3. "(Besides the time you already have told me about) During those two weeks did anyone in the family get any medical advice over the PHONE from a doctor, nurse, or anyone else working with or for a medical doctor? Include calls to get prescription or test results."

A comparison of the estimates yielded by the control group questionnaire and the experimental questionnaire showed that the experimental questionnaire produced the desired reporting patterns. Major changes in the questions were not deemed necessary; however, some awkwardness was noted during the field observations. Observers reported that the questions were still too wordy, that respondents often gave a negative response to the introduction and that respondents answered Question 3 before the instruction to "include calls to get prescriptions or test results."

3. Phase III Version
To remedy these deficiencies, further revisions were made in Questions 2 and 3 for Phase III, informal test. Question I remained unchanged.

2. "(Besides the time(s) you just told me about) During those two weeks, did anyone in the family receive care at home or go to a doctor's office, clinic, hospital, or some other place to receive health care? Include care from a nurse or anyone working with or for a medical doctor."

3. "(Besides the time(s) you already told me about) During those two weeks, did anyone in the family get any medical advice, prescriptions or test results over the PHONE from a doctor, nurse, or anyone working with or for a medical doctor?"

4. Final NHIS Version
Following the Phase 111 test, the experimental or "evaluation" questionnaire was revised for the last time before becoming the standard core NHIS instrument. Consensus among observers and interviewers was that the questions were still unnecessarily verbose. Although the basic structure and concepts were not changed, the final version of the questions reflects the effort to reduce them to their essential elements.

"These next questions are about health care received during the two weeks outlined in red on that calendar."

a. "During those two weeks, how many times did you see or talk to a medical doctor? [Include all types of doctors, such as dermatologists, psychiatrists, and ophthalmologists, as well as general practitioners and osteopaths.] (Do not count times while an overnight patient in a hospital.)"

b. "(Besides the time(s) you just told me about) During those two weeks, did anyone in the family receive health care at home or go to a doctor's office, clinic, hospital or some other place? Include care from a nurse or anyone working with or for a medical doctor. Do not count times while an overnight patient in a hospital."

c. "(Besides the time(s) you already told me about) During those two weeks, did anyone in the family get any medical advice, prescriptions or test results over the PHONE from a doctor, nurse, or anyone working with or for a medical doctor?"

In this example, the repeated qualitative assessments made by observers (and interviewers) resulted in a more efficient series of questions. Statistical analysis of the formal test data in conjunction with observers' evaluations indicated at what point the benefits of a thoroughly defined concept were outweighed by the costs of a verbose questionnaire.