Editor's note: Steven Heffner is acquisitions editor at Kalorama, an imprint of MarketResearch.com, Inc., New York.

Regardless of your exact title and your exact industry, almost every one of us is in the information business. If you're in publishing like me, you're in the business of selling it. But even if you're developing it, processing it, or consuming it, you will agree with this simple proposition: information has value. And in the case of market research, information has a lot of value.

Because our business is packaging and selling information, we at Kalorama Information do a bit more obsessing over the value of information than most, but that value has implications for everyone in research. There are two basic ways to assign value to information: 1) you can ask how much it costs to acquire, develop, and publish the information, or 2) you can ask how much people will pay for it (i.e., what the market will bear).

Predictably enough, these two questions often have different answers, and balancing them is the trick. However, recently the difference between the two has grown dramatically.

The gulf between reality and perception

Here is the problem: In the past few years, as a publisher I have seen that the perceived value of information is falling rapidly because of the constantly rising tide of data available for free on the Internet. As an editor and researcher, however, I see that extracting real value from that sea of information is becoming more and more difficult. The widening divide between expectations of cheap, plentiful data and the truth of labor- and skill-intensive knowledge mining is creating somewhat of a crisis for the market research industry.

The conflicting value propositions are a concern for information vendors in particular but it also has implications for market researchers in general. It is most obvious to us in the syndicated research sector; we're the canary in the coalmine because we're exposed to direct market pressures. However, this gulf in expectations has implications for in-house researchers and consultants too, as the perceived value of their work product changes within an organization.

Already in the past year, several major life science companies have downsized or even liquidated their in-house market research staff, feeling budget pressures. One can only assume that the misperception that market information is "cheap" is also a big factor. In the information business, the law of supply and demand appears to be distorted, and researchers and analysts are finding themselves on the wrong end of the lens.

Of course, you and I know that finding meaningful information on the Web is, indeed, more difficult than one thinks. Separating the wheat from the chaff turns out to be, in many cases, as labor-intensive and more skill-intensive than harvesting the wheat the old fashioned way. Here are some of the reasons why...

Herding information: "free" is a deceptive word

How many times have you heard, "But most of that is free on the Internet"? Whether you're preparing a report or justifying your bill to a client, the notion that because it is "on the Internet" it is available and thus didn't require finding or gathering can be a problem.

Finding the information is the first step and not always the easiest. Knowing where to look requires a lot of experience. There are some secondary resources to help - MarketResearch.com publishes a guide called Finding Market Research on the Web, which documents some of the best methods and sources, for example - but even guides cannot replace the applied expertise and experience of a seasoned researcher.

The second step is collating the information. Company product information, prevalence and mortality data, financial reports, patent and regulatory filings - all of these are, indeed, part of the public record and/or free for the asking. Does that mean a single file containing of all the patent filings of note in transdermal drug delivery coupled with financials on the competitors in the sector is not of value? On the contrary, this type of editorial data organizing is one of the most valuable things a researcher can create.

Filtering information: who said that?

Even when a piece of market information seems easy to find, the reliability of the source is a serious concern. It's one thing to get bad driving directions from the Web, it's another to base an important strategic decision on dated, skewed, or outright faulty data. Evaluating the source requires that you know the answer to two basic questions:

  • Do they know what they're talking about?
  • Are they a trustworthy reporter of the facts?

The first one is difficult, but most researchers know by now who and who is not a reputable vendor or record-keeper. The second is a little trickier because it spans the continuum from outright fraud to a little "harmless" spinning. Although lying is a concern, the first question should weed those out. The more prominent concern is the source's ulterior motive in shading or even lying about a data set. Investment banks and other equity analysts are prolific producers of market research reports, but their revenue forecasts are often inflated when it comes to industry sectors they're focused on and invested in. They have a clear conflict of interest. They're not lying, they're just optimistic - very optimistic.

Evaluating the source also requires that you really know the source. Just as the number of vendors and outlets has seemed to increase, the number of true sources has shrunk significantly. When you purchase a report from Reuters on a pharmaceutical market, quite often the data reported are cited as being from Datamonitor, which in turn be from IMS. In fact, it's relatively easy to see the same numbers pop up for a dozen apparently different sources, because they are all regurgitating a single source - one your organization may already have access to elsewhere.

Digesting information: what are they really saying?

Finally, you must ask the trickiest question: Do you know what they're talking about? Do you know what the data they're reporting really mean? Here is where you must understand the methodology of the data collection.

A drug company's annual report, for example, may pin the potential market for a soon-to-be-released drug at $3 billion. Do you know how they came up with that number? Let's say it's an estimate of unmet need. Is it reasonable to assume that the drug will make it into the hands of everyone who needs it, and that every one of them will be paying the drug company's asking price? I don't think so.

How about reported revenues on existing products? One former big pharma employee told me that they used to multiply the number of pills shipped by an arbitrary "cost per pill" to arrive at that number, even though some of the pills were free samples, some were sold at deep discounts to certain distributors, some were sold overseas at yet another price, etc., etc.

Less sinister, but perhaps more dangerous for the unsuspecting researcher is the misunderstanding of the information from large databases. The danger isn't that these sources are trying to spin the information like a drug company or an equity firm, but rather that the information is only as good as the method of collecting it.

Take, for example, statistics from the National Center for Health Statistics or from the Healthcare Financing Administration. These numbers can be very deceptive if you don't know where they come from. Not too long ago, a customer of ours was looking for the number of stents placed in coronary arteries in the United States. The government had a number that was nearly half that of some industry analysts. How could that be? Well, the government number was based on hospital discharge surveys, which statistically sampled ICD-9 codes (used for reimbursement purposes). Fair enough, but what one needs to know is that only the primary condition was coded; if the stent placement procedure was an also-ran, it was not included in the count. Also, if the patient had more than one stent inserted, this was counted as one. Add these basic flaws to the geographic variation of the sampling and the errors compound.

The same can be said for some pharmaceutical industry databases such as IMS. You have to know how the data are collected, and you have to account for the statistical methods used to arrive at the numbers. Is the information useful? Yes. Is it all you need to know? No. Numbers have a pedigree, and understanding that pedigree is the key to using them correctly.

Evaluating information: so what is it worth after all?

All these research steps make it clear that information is not at all getting easier to acquire and use - quite the contrary. Even when one finds relevant information from a reputable source, it takes expertise to evaluate what the source is really saying, so that it speaks in harmony with your other datasets.

The unregulated free-for-all that is the Internet has made tons of market data easy to come by, but good market research harder to come by. It is actually more difficult to find the useful information from reliable sources now that the number of vendors and outlets has grown beyond the proverbial haystack while at the same time the number of true primary sources has dwindled.

For example, our firm has seen the price of life science market research change quite a bit over the past few years. A few publishers have held the line on pricing, whereas some have increased their prices by more than 100 percent. During the same period, we've seen a fourfold increase in the number of life science research vendors in our database. More supply and still a higher cost! The reason is quite simply that meaningful market research is getting more and more difficult to glean.

In the end, the story is the same: market information is not cheap, and the advent of online sources of information has not made it less expensive to acquire. Indeed, it has made it more difficult to get solid, reliable information and to truly understand its implications. The perception that the online world represents a well of cheap, plentiful data is simply wrong, and the role of the experienced researcher with a critical eye is still very much in need.