A U.S. Department of Commerce report, "Approaches to Developing Questionnaires," includes a section on the results of testing three questionnaire techniques to obtain information on "health events" in a given time period. A summarized excerpt from this section appears below.

Introduction

The National Center for Health Statistics has undertaken numerous studies to improve the reporting of health events in household surveys. The example of a split sample test presented here was selected for three key reasons:

First, it illustrates the use of a small and unusually homogeneous sample, showing the strengths and weaknesses of such an approach. Second, it tested three questionnaires reflecting different strategies of questionnaire design. Third, it illustrates the successful application of hypotheses developed in another field-cognitive psychology-to survey research.

The concrete problem facing the designers of this test was the under-reporting of their key dependent variable, "health events," in a given time period (e.g., the number of dental visits in the last fourteen days). Especially likely to be under-reported were health conditions of low impact to the respondent and those occurring considerably prior to the interview.

The test was developed using a cognitive model of how people learn, store, and retrieve information. Methodologically, the aim was to determine whether reporting can be significantly increased by focusing on and aiding the recall tasks facing respondents.

The model suggests that an event is reported only if the researcher can design a survey question/stimulus that can spark the memory during an interview. For example, a single event number of dental visits may be recalled by the respondent in terms of money, pain, or lost work time, and a direct question on dental visits may not get an accurate answer.

The Questionnaire

To test some hypotheses generated by the model, three questionnaires were developed for a split sample test: an extensive questionnaire, a diary with a follow-up procedure, and a control questionnaire. All relied on personal interviews, although the diary follow-up was partially self-administered.

In the extensive questionnaire there were many questions aimed at providing respondents "with multiple and overlapping frames of references and cues." The strategy rested on the assumption that respondents could more easily recall health conditions through "some specific behavior implications" (e.g., activity restrictions, medicines, diet, visits to the doctor) than through a conceptual or general framework.

For example, previous field work showed that questions about operations usually resulted in reports of major surgery, but questions about stitches elicited reports of minor surgery as well. Therefore, standard questions included additional probes, and general medical terms as well as more popular language were used. Finally, the pace of the interview was designed to be more relaxed by allowing more time for recollection and reporting and by use of transitions between sections.

A "body review" of aches and pains and a series of questions on symptoms (e.g., "Have you had any aches or pains in your joints?") opened the extensive interview. When respondents reported a symptom, interviewers asked "Do you have any idea what causes it?" in an attempt to help the respondents better define and isolate the underlying health condition. Next, questions about the respondents' medical history specified various time dimensions (e.g., childhood, last week) as another approach to uncovering events stored in the memory. Behavioral implications were referenced in the next questions. Two checklists of chronic conditions provided a direct items-recognition approach to conclude the interview.

Diary Procedure

Reviews of previous research on health diaries and informal tests of various procedures led to the second experimental approach. It utilized a diary procedure, in which the respondent kept a health record every day for a week in an eight-page booklet containing seven simple questions on health events. A short personal interview took place at the end of the diary week. The design operationalized two major ideas which were to facilitate the respondents' task of remembering and to consider this daily recording activity as a sensitization device for health thinking and reporting.

In the follow-up interview, each diary question was carefully reviewed, answers were clarified when necessary, and a short structured set of questions--the chronic condition lists and items on present effects of past accidents, injuries, dental sits, and hospitalizations--were asked.

The control questionnaire used a single direct question for obtaining information on each major health item. To sensitize the respondent, the interview opened with a checklist of nineteen symptoms. Questions were then asked on recent health events, including restriction of activity, and on present effects of past injuries or illnesses. Then, the chronic conditions checklists, identical to those used in the two experimental questionnaires, were administered. The interview concluded with questions about recent visits to the doctor and hospitalizations or dental visits in the past year.

In addition to the chronic conditions checklist, items on hospitalizations, dental visits, demographic characteristics, and a general health rating were identically worded in all three approaches. Other questions were similarly worded across the instruments. Then, at the conclusion of the health questions in every interview, interviewers asked a standard series of questions about each reported condition. The resulting "condition table" was designed to separately record the first report of any health problem mentioned by respondents. The purpose of these standardized questions was to allow comparative evaluation of the three experimental collection methods.

Sample Design

Previous research on health reporting had shown that "characteristics of the respondent are not nearly as consistent, nor as strong as their influence on under-reporting, as are the characteristics of the event." For this reason, and because of the experimental nature of this test and the desire to minimize costs, a geographically concentrated and relatively homogeneous sample was selected. Specifically, all cases were in Detroit, and a modified area probability sample with clustering was used to locate "low-middle and middle socioeconomic groups, English-speaking, native-born white females between 18 and 65 years of age." The three questionnaires were randomly assigned to households within each sample block. The design yielded 462 occupied dwelling units, containing 356 dwellings with eligible respondents. Only one respondent per unit was interviewed, and 305 interviews were completed.

Evaluation

Because under-reporting of health events was a known problem, comparisons among the questionnaires focused on the amount of reported information. The assumption was that the more health information reported the better; no outside records were used for validation. There were two types of dependent variables: (1) the number of health conditions reported and (2) the impact level (i.e., the amount of medical care, psychological concern, and other indexes of salience to the respondent) of reported information.

The overall response rate of 88% was quite similar among the three questionnaires. Demographic characteristics were also similar with the exception of education, which was highest in the diary group. However, correlations within the treatments between education and the key dependent variables were not statistically significant.

Mean Number of Conditions Reported Per Person By Condition and Collection Procededure

  Collection Procedure
Column Reporting
Variable

Extensive Control Diary
Extensive
Control
Diary
Control
Extensive
Diary
 
  Mean number of conditions per person Difference between means
 

Total

7.88

4.42

5.08

3.46*

0.06

2.80*

Chronic conditions on recognition lists

3.54

3.25

3.29

.29

.04

.25

Other chronic conditions

2.75

.74

.58

2.01*

-.16

2.17*

Illness in last 14 days

.58

.28

.69^

.30*

.41*

-.11

Injuries in the last 14 days

.24

.05

.39^

.19*

.25*

-.06

Other unclassified conditions

.76

.10

.22

.66*

.12

.54*


*P .01
^These figures in the diary technique refer only to the last seven day period, a restriction which enhances the observed differences between diary and other techniques.


The evaluation first examined the mean number of health conditions reported per person in each of the three questionnaires. As seen in the table, results supported the hypothesis that the multi-stimuli approach of the extensive interview increased reporting: the 7.9 reported conditions in the extensive interview were significantly greater than the 5.1 reported in the diary or the 4.4 in the control.

The hypothesis that the diaries would also increase reporting received less clear support. The difference between reported health conditions in the diary and control questionnaires was statistically significant only at about the 10% level.

Five Types

To learn more about the source of these differences, conditions were classified into five types, and the table also shows the number of reports of each type, by questionnaire version. Again, the extensive questionnaire achieved higher reporting than the control among all types, although, as the authors pointed out, "whenever the control questionnaire uses an extensive recognition type of approach, such as the recognition lists of chronic conditions, a reduction of the gap between the two techniques can be observed. An increase in the amount of information reported still exists in the extensive technique but is no longer statistically significant."

Compared with the diary follow-up approach, the extensive questionnaire also achieved higher reporting except for acute conditions. This particular strength of the diary procedure was expected, but since the reporting of chronic conditions did not significantly differ from reports in the control interview, doubts were raised about the general sensitization function of the diary.

When reported conditions were dichotomized into those first noticed less than three months ago and those first noticed three months ago or longer, reporting of both recent and older conditions was significantly higher with the extensive questionnaire than with the control questionnaire. But compared with the diary follow-up questionnaire, the extensive questionnaire got significantly higher reporting only for longer term conditions. The authors believed these results are not surprising since older reported conditions are more likely to be chronic and recent reported conditions more likely to be acute.

Level of Impact

The second key dependent variable was the level of impact on the respondent of the reported health conditions. It was hypothesized that the extensive and diary follow-up questionnaires would improve reporting of low impact conditions but have little, if any, effect on high impact reporting. Thus, the predicted result was a lower mean level of impact reported using these questionnaires compared with the control.

For testing the hypothesis, an impact level was constructed for every eligible condition, using for example, evidence of frequency (or levels) of discussions with doctors, medications taken, days in bed, and pain. Results supported the hypothesis and further showed that the extensive questionnaire produced more complete reporting of serious (i.e., high impact) conditions. Differences among the questionnaires in the mean level of impact according to whether conditions were chronic or acute were also uncovered.

Summary

By emphasizing various ways of encouraging respondents to recall health events, this small test produced extremely encouraging results. The extensive questionnaire with multiple probes and cues significantly increased reporting in all groups of health conditions. Compared with the control, the extensive and diary questionnaires also produced higher reporting of health conditions of low impact to the respondent. The diary follow-up procedure resulted in more reported acute conditions, although hypotheses about the sensitization function of the diary were not generally supported.

Because of the special demographic characteristics of the sample, generalizing the results to other groups cannot be done with any certainty. The test was instead part of a larger and long-term research effort aimed at achieving greater understanding of survey techniques for better reporting of health events.

Methodologically, the improved reporting was "interpreted as the result of a greater correspondence between the questioning procedures and the manner in which respondents organize health information in memory," although the authors caution that motivational factors were not controlled in the study. Rather, "the major outcome was a pragmatic one; techniques designed in a cognitive framework to facilitate recall have proved effective in increasing reported information."