Editor's note: While this month's installment of Data Use doesn't deal with a marketing research application, it is a good example of how many of the statistical techniques regularly discussed here can be used. Lois Sayrs is senior methodologist with the Office of the Auditor General, Performance Audit Division, State of Arizona. She performed the data analysis on ADWM and developed the sampling plan.

Every motorist in Arizona is familiar with the small round sticker on the gas pump at their favorite gas station. That sticker shows the consumer that the pump was inspected and is accurately delivering fuel per the price indicated on the pump. And, from October to May, the sticker also means that the octane and oxygenate levels meet state environmental quality standards. Inspecting fuel pumps is only one of the variety of functions that the Arizona Department of Weights and Measures (ADWM) performs. It also counts package contents and weighs packaged meats and poultry in grocery stores. It assesses taxi meters, propane dispensers and even oversees the disposal of used oil. Virtually any device that is used for weighing and measuring, be it mass, volume, counts or time, ADWM inspects.

ADWM has two purposes: to protect the consumer in transactions and protect the public health. Such admirable purposes, however, also harbor the potential for unlimited government intervention, a growing bureaucracy to accommodate increasing numbers of inspections and an inflated budget at a cost of millions to taxpayers. In their oversight capacity, state legislators must evaluate requests for bigger budgets, more staff, and more services to the public.

The Performance Audit Division (PAD) of the Arizona Auditor General's Office is charged with auditing the performance and compliance with state laws and regulations of all state agencies and regulatory boards under legislative oversight. Although PAD performs audits at the special request of the legislature, PAD audited ADWM as part of the normal sunset review cycle.

(The sunset review cycle establishes a review cycle schedule for all agencies over a ten year period. Agencies will terminate or "sunset" at the end of the cycle unless there is specific legislative reauthorization.)

Legislators rely on performance audits to provide the needed information to make tough decisions. The fiscal reality of the 1990s is that state agencies like Weights and Measures have to do more with fewer resources.

Our audit of ADWM focused on three questions of particular interest to the legislature: better enforcement, better management, and a self supporting used oil program.1 From the beginning of the audit, management issues seemed to overshadow our other concerns. Through interviews and a staff survey, we found reported allegations of racism, sexual harassment and interagency conflict that touched the highest levels of the agency. We also found rather quickly that the used oil program was not self-supporting as the legislature had hoped. But even as we were uncovering serious management issues and determining the extent to which used oil was lagging in revenue generation, our preliminary survey of inspection and enforcement data revealed another vexing problem. We learned that the enforcement and inspection reports were not used for tracking noncompliers. Indeed, the agency had no idea what the compliance rate among vendors was. Multiple inspections, each of a different type, were collected on the same reporting sheet but the information was not summarized and analyzed. And none of the information was maintained in an electronic format. These data issues led the team to generate our own electronic file using the last three years of data as the file sample.

27,000 inspections

We entered over 27,000 inspections into a database file that contained information on vendor, inspection frequency and type, compliance and type of violation. Generating and analyzing this data set would be the most time-consuming task of the audit, exhausting about one-half of our allotted budget hours. Initially, we had several questions guiding our analysis:

Do inspections deter non-compliance?

Who were the non-compliers?

Were inspections being performed fairly and equitably throughout the state?

Even though the data would answer these questions, it raised some new ones. For example, by using simple frequency distributions, CHI-square tests of the difference of means, and analysis of variance, we found regional effects but no effect from urbanization. So, contrary to our expectations, the rural areas were inspected as frequently as the urban areas. And, although vendors in the north region were likely to be inspected more frequently than any other region, those vendors were no more likely to be in compliance. Indeed, this latter result led us to consider the relationship between compliance and inspection frequency with greater scrutiny.

By correlating inspection frequency and compliance, we found that the more ADWM inspected, the more non-compliance was found. Frequent inspections did not appear to deter violations but they did help to ferret them out. At first glance, this result may appear to support the trend we saw in the agency: an increase from 20 to 33 staff since 1987; from a $1 million budget in 1987 to $2.4 million in 1992. And, the data seemed to support a recommendation of increasing agency staff and budget (since there are more violators, they need more staff and a bigger budget to find them). But we recognized that such a recommendation would lead the legislature down a slippery slope. How many inspections does it take to find a violator? How many to deter? What kind of violations are these frequent inspections finding? How much will it cost taxpayers to find every violator?

We found that 90 percent of all inspections resulted in finding a vendor in compliance. Only 10 percent of vendors were not in compliance but surprisingly these 10 percent do not account for the vast proportion of repeat inspections of the same facility. Most of the repeat inspections were of facilities who were already in compliance. Relying on CHI-square tests, we found that those vendors inspected frequently were no more likely to be in compliance than those inspected only once annually. We concluded that first, most vendors are in compliance and, second, that non compliers are not targeted for enforcement actions. Valuable agency resources were being used to monitor a basically compliant population and non-compliers were not still not being brought into compliance.

Violators not deterred

Our results painted a picture of an agency that inspected frequently, that went out of its way to inspect small town gas stations and grocery stores in remote areas of the state but to no effect. Violators were not deterred. The missing ingredient, i.e., an enforcement mechanism that would deter, was not integrated into department philosophy or action. And repeat offenders were not targeted for enforcement action. So, even though the department was finding violations it was not tracking violators nor was it doing much more than inspecting to make them comply.

ADWM's philosophy of inspection had been based on the idea that finding violators alone would deter them. Our analysis, however, showed that this philosophy rested on a false assumption. Frequent inspections were not a deterrent. Deterrence requires two components: a verification mechanism and a credible threat of inflicting costs. ADWM relied upon verification alone when both verification and threat are necessary. Moreover, valuable agency resources were being wasted when the department 1) re-inspected compliers, 2) followed-up on violations but was unable to enforce compliance; and 3) did not target non-compliers for enforcement actions.

Our recommendations directly addressed the problems we found in management, the enforcement division and the used oil program. As a result of our audit work, we recommended to the Legislature that ADWM initiate a management training program, study the feasibility of raising fees on used oil collection and, for enforcement, use their statutory civil penalties to provide the missing component in deterring offenders. With civil penalties in place, the department could move to a sampling strategy (similar to the IRS) for annual inspections. The department would need to build a tracking system using data similar to the kind we generated to identify and track non-compliers. Our office offered to relinquish our 27,000 case data bank to the agency for its own use.

Benchmark audit

Our innovative approach has become a benchmark audit for our division. The Federal Clearinghouse on Weights and Measures was unaware of any state using a sampling strategy but informed us that most states did not have the necessary data or the analytical resources to develop and monitor a database for inspection sampling purposes. We believe that a good database, a desktop computer and a small commitment of resources to a data manager/analyst is all that is necessary to move to a sampling strategy. Most agencies have some data and some type of data manager in place already. So, improving the agency's service is only a matter of upgrading the quality of the data and the skills of the data analyst.

Some legislators feared that moving to a sampling strategy might cause currently compliant vendors to try and cheat. Or, the public, not seeing the familiar sticker on the pump might assume the pump is not operating fairly and go to another gas station. We confirmed for the legislators that deterrence requires a two-fold approach and with civil penalties in place, the sampling strategy should have a stronger effect in deterring non-compliance than the previous strategy.2 The public will still see the familiar sticker on the gas pump but now this sticker will be sent to all vendors as part of their registration process with the agency. Information on the sampling strategy will be sent to vendors in their registration packet. Our recommended changes will result in saving ADWM almost $500,000 annually in reduced staffing levels, resources will be more efficiently and effectively spent on non-compliers and the public will be better protected through a targeted inspection process. A legislative committee accepted our recommendations and they were incorporated into the 1994 budget.

Robust conclusions

The success of our audit work rested squarely on our ability to generate, manage and analyze data from a variety of sources. And, we were able to draw robust, if not definitive, conclusions about inspection activities through the use of multiple statistical techniques. Although these techniques were fairly simple (frequency distribution analysis, crosstabulations, correlations, ANOVA), they led to powerful results. But perhaps the real power of our findings lies in the recommendations they allowed us to make, and ultimately the savings to the taxpayers and the improved protection of the public. At a time when the threat of lost service is as real as a state budget deficit, it doesn't get any better than that.

Notes

1 The Used Oil Program operates in conjunction with the Arizona Department of Environmental Quality (ADEQ) to insure that the used oil sold to resellers is not contaminated with hazardous waste. A portion of the monies collected from the reseller is returned to ADWM through ADEQ.

2 However, we recommended that ADWM oversample (one-third of the total population) so as not to move too quickly for the public and vendors. After two years of oversampling the department can decide, on the basis of our three years of data and their two years, how much the sample size can be reduced.