Help us help you

The California Public Employees Retirement System (PERS) contracts with 25 health plans to provide coverage for over 360,000 active and retired state and public employees. Adding family members, that total comes to over 850,000 people.

By state law, part of the responsibility of PERS' Board of Administration is to periodically evaluate the level of member satisfaction with the various health plans. This serves three purposes. One, it helps PERS evaluate the health benefits program and assists in rate negotiations with health plans. Two, it identifies service areas needing attention. Three, it aids PERS members in choosing a health plan that meets their needs. (Each May, during an open enrollment period, members may select a different health plan.)

The most recent evaluation effort was the 1991 Consumer Experience Survey, which was sent in September of 1991 to over 20,000 health plan enrollees.

The cover letter included with the questionnaire stated the purpose of the survey: "PERS has recently completed this year's open enrollment period for health care plans. If you considered changing plans, you know the choices can be difficult. We try to provide members with as much comparative information as possible to help you make the best decision. We believe it would be even more valuable for you to know how other enrollees judge these health plans based on their actual experience. In order to do this, we are asking you, and other PERS enrollees, what your health plan experience has been during the last year."

The questionnaire touched on several topics, including:

  • reasons for selecting the plan they chose
  • satisfaction with dealings with the health plan for any administrative problems
  • satisfaction with prescriptions
  • opinions of telephone interaction with doctors and/or nurses
  • time required to get an appointment
  • communication/interaction with doctors and satisfaction with treatment
  • satisfaction with the plans' health improvement programs (smoking cessation, weight loss, exercise, etc.)
  • hospital stays
  • emergency services
  • family usage of service
  • the importance of additional services that the plan could offer

Design and testing

The questionnaire design and testing for the 1991 Consumer Experience Survey was lengthy and thorough, due in large part to the scope of the project.

The survey's final report states: "In designing this study, one important factor was to create a series of questions which would best capture the experience and opinions of PERS health care consumers. The topic areas covered by these questions are fundamental to health care services, measure performance around activities over which providers have some control, and directly ask consumers about their satisfaction with those services they actually experienced."

The testing included public meetings with active and retired enrollees and members of employee associations, a focus group, telephone interviews, intercepts, and suggestions and review by the PERS Health benefits Advisory Council and the State of California Department of Personnel Administration.

"PERS is very sensitive to the needs and opinions of its members," says Craig Hartung, chief, Information and Program Development division, California PERS. "And in order to determine what would be the most valuable questions we held focus groups. We wanted to determine areas of concern for our members because sometimes you're just guessing about what they peed to help them make decisions on the plans."

Focus groups

Throughout the project, PERS worked with Freeman, Sullivan & Co., a San Francisco-based research firm. For the focus groups, a dozen PERS members recruited from a nearby health plan filled out the questionnaire and then discussed what they liked and disliked about it, says Charles DiSogra, senior consultant, Freeman, Sullivan & Co. "The approach that we took was that if we went to the health plan and we took a random sample of public employees that we'd get people from different areas and that's how it worked out. We had some retirees as well as some active employees in the group."

Some of their suggestions included changes to skip patterns and a clearer definition of the time frame covered by the survey, DiSogra says. "One of the instructions said: 'Over the last year...' Well, a year is interpreted very differently depending upon what kind of job you have. Some people from government service thought in terms of fiscal year. Some individuals thought in terms of calendar year, and some, such as school teachers, thought in terms of the school year."

To further test the questionnaire, 50 PERS members belonging to another health plan were recruited by phone and sent a copy of the questionnaire. They were asked to fill it out and then discuss it afterwards with a researcher from Freeman, Sullivan.

Another version of the questionnaire, reworked using information obtained from the focus groups, was tested through intercepts at a state office building. "We had people fill out the questionnaires on breaks or during their lunch hour. They were real helpful. We were able to ask them about the changes to the questionnaire. All of the testing we did was helpful. It contributed quite a bit to the success of how we approached this instrument," DiSogra says.

Health plans gave input

Input was also sought from the health plans themselves. Involving the health plans in the design process, DiSogra says, increased the likelihood that the health plans would view the results of the survey as valid. When it came time for rate negotiations they might be less likely to dismiss the results as invalid because they had a hand in designing the survey instrument.

"We kept the health plans informed along the way and sought their input so they couldn't come back and say, 'We didn't have a chance to say anything.' If they wanted to set up a conference call with their marketing people, their researchers and us during the development stage we were happy to answer their questions."

Craig Hartung: "We wanted them to be fully informed as it moved along so that there wouldn't be any questions at the end. At the same time, some of them were able to provide valuable input to the development of questions and the overall balance of the survey by pointing out, for example, where their services differed from the other plans. It was quite helpful."

Nearly 70 percent response rate

Much to the delight of all involved, the survey achieved a response rate of nearly 70 percent. Both DiSogra and Hartung have theories on why the survey earned such nice numbers. First, the survey benefited from the fact that health care was at the time of the survey (and still is) an issue of great interest to many people. "I think a lot of it has to do with the fact that the PERS enrollees consider their health benefits extremely important," DiSogra says.

Second, postcard follow ups and duplicate surveys were sent to those who failed to respond by two suggested dates. The postcard reminded enrollees that: "This timely information is intended to benefit all PERS enrollees in the months to come." The letter accompanying the duplicate survey stated: "Your participation in this survey is very important, but we have not received your completed questionnaire. In case you may have misplaced it, we are enclosing another one... The information you provide will help all PERS health plan enrollees make informed choices about their health care plan. Please help us help you."

Third, Hartung says, public employees have in general shown more interest in participating in surveys and other calls for their opinions. "Our members generally are people that respond to things at a little higher level than the average person. When we've gone out with other questionnaires we've tended to get a little higher return."

Fourth, though the questionnaire dealt with many topics and had many questions, it was designed to be easy to complete, DiSogra says. "We put a lot of emphasis on making it easy to navigate through the questionnaire. We tried to avoid questions such as 'if yes, go to...' We made sure we had instructions between each question and when the data came back we suffered from very minimal item non-response. If someone was going to answer the questionnaire they pretty much answered the whole thing."

The sixteen page survey was in booklet form and it made liberal use of color and strong graphic elements such as arrows and boxes to move the respondent through the questions.

Hartung: "Freeman, Sullivan tried to make it easier for the respondent to tackle the survey. When you're dealing with that many questions, the first instinct is to either throw it away or put it aside. I think the design created an interest in the survey."

Weighting the data

Substantial work went into weighting the data to compensate for the differences in size of the various health plan member populations. Because of size and geographic location, some of the plans had a few thousand enrollees, some had several thousand.

"We were concerned because we knew right away that there was a very disproportionate distribution of the PERS enrollees among all the health plans," DiSogra says. "Some of the health plans were more accessible; they simply covered a greater geographic area. Some of the enrollees in the rural parts of the state didn't have as many choices but there were health plans up there for them, though those plans had very small numbers.

"The idea was, if you join a plan, it is obligated to make you satisfied to a certain level, no matter what the size. I didn't want the average to be affected by the numbers of people in those plans, because the numbers were not a product of having to choose from all the plans equally.

"If, for example, all of the enrollees who used one of the large plans were tremendously dissatisfied that would skew the overall satisfaction figures because of one particular plan. So by weighting it the way we did we made it as if PERS enrollees were equally distributed among all 26 plans and then the weighted average of satisfaction would not be overly influenced by large plans or small plans. They would all come up to about the same standard."

Resolve lower rankings

The rate negotiations between PERS and the various health plans are ongoing, Hartung says. "Information from the survey will be used to try to get the plans to make some kind of commitment to resolve some of the lower rankings. Nobody came out bad, nobody got a D, for example, but some got A's. As the plan facilitators it's our responsibility to try to move the plans up into the higher levels so that the span of grades isn't so large.

"We'll talk to the plans with the lower ranking scores to make sure that they have a thorough understanding of the responses and that they have some ideas in place to do something about their score."