Partnering for insights
Editor's note: David Soorholtz and Terry Grapentine are independent marketing research consultants. They can be reached at david@vetsense.biz and tgrapentine@gmail.com.
Client-side researchers (who, for purposes of this article, we refer to as “clients”) rely heavily on the research suppliers they utilize to conduct primary marketing research. Selecting the best supplier for each project and then working effectively with that supplier is critical to the success of any research project. However, in our experience, many clients receive little training on evaluating and effectively managing the research suppliers they rely on. In this article, we will use our experiences as client-side researchers and research consultants to propose best practices that clients may use to evaluate, select and manage research suppliers.
Both clients and suppliers have the same goal: high-quality research projects. Both want to conduct research that delivers on time and on budget and meets the study objectives – with no surprises. Suppliers want to make their clients “look good” by hitting these marks because it will help to ensure a long and profitable relationship with that client. Clients want the same because it will answer their internal clients’ business questions, leading to job satisfaction and (hopefully) advancement.
However, to the client, a research study is an input: an expense that delivers a product that they and their internal clients use to develop strategies and tactics. To the supplier, the research study is an output: the product of their work and a source of revenue. As a result, there is the usual tension between supplier and client. Because of this, client-side researchers must conduct adequate due diligence on potential suppliers because they know the product delivered by their suppliers reflects on them.
General considerations
Do you trust them? This may seem like a strange place to start but it’s the first question to ask when selecting a supplier. If you question your ability to trust a supplier, you can’t use them in good faith. Unfortunately, developing trust without first working with a supplier is nearly impossible. But what does trust mean in this context? In our experience, a supplier that you have come to trust does what they say they will do (meet the study objectives and deliverables) when they say they will (meet deadlines) at the cost they promised.
So, what are some keys to determining if you can trust a new supplier? One indicator is the supplier’s willingness to constructively challenge you versus taking everything you say at face value. This indicates that the supplier is honest about what they can do and how they can do it. When selling anything to a new customer, it takes some guts to not always say yes. Also, although it’s sometimes not feasible, consult references. Often, researchers know their peers in competitive companies. It’s ok (as long as your legal team oks it) to ask them about their experiences with a supplier.
Were their past projects of high quality? Although this seems obvious, if you like the quality of a supplier's projects, keep using them. As suppliers, we have always assumed that you are only as good as your last project – it is imperative that there be no “clinkers.” However, if you feel a supplier didn’t hit it out of the park on a study, don’t cast them off. First, look internally at the hand you dealt them. Were there issues with the study objectives, communication or any other aspect in which you or your team may have contributed to the study not being a winner? Also, in most cases, a study is unlikely to uncover a new “aha” but rather confirm in some way what you or your clients already believed. This is especially true when the research is conducted on in-line products or services the client knows well. Don’t blame the supplier for not striking gold – there’s not always gold there. However, asking the supplier to evaluate their processes is fair if a study doesn’t meet your expectations. What if anything changed on their side that might have contributed to this?
Who will do the work? Before commissioning a project, client-side researchers should know who at their supplier will be playing which role in the study. Most suppliers have teams of researchers and support staff participating in each project, ranging from recent college grads to people with years of experience. In our careers, we have been the fresh-faced junior researchers and are now the “old gray-haired guys.” When bidding on a research project, most suppliers are pricing the hours each person (or job title) will spend on the project. Each person has a differing hourly price, depending primarily on their experience. The final cost of a study is the rollup of these hourly costs plus any out-of-pocket costs.
Herein lies the conundrum. Ideally, the client would like only the most experienced people at their supplier to carry out their study. Conversely, the supplier would like to keep the study cost down, train new researchers and leverage the time of their experienced people while still successfully executing the project. What to do? In our experience, it is vital that a client-side researcher determines precisely who will be participating in what role in a study. If you want or need a highly experienced person playing a key role in a study – say, qualitative moderation or advanced analytics – demand them. Make sure that the old gray-haired guy or gal isn’t just involved in pitching the proposal and delivering the final report, especially if the study has major implications for your internal clients.
However, new researchers don’t gain experience without spending years participating in and then managing research projects and they often add fresh ideas. Suppliers (and ultimately the clients) can’t afford to have their highly experienced people do all the work. Be open to a blend of more- and less-experienced people working on your project.
Also, not all experience is equal. Although we are proponents of Malcolm Gladwell’s “10,000 hours” theory (“Outliers: The Story of Success”) and believe practice makes perfect, time alone is not everything. Check out their credentials (LinkedIn). Where and what did they study? Are they published, how often and where (Google Scholar)? This information alone does not mean they are good researchers but it does provide insight into their intellect and the seriousness with which they have approached their careers.
What are their competencies? Are they selling what you need? If it’s clear that a study requires a qualitative approach and the supplier spends most of their pitch describing their quantitative skills (or vice versa), it’s a sign that you should probably not use them for some types of research. Nearly every supplier has things they do better than others and no one supplier is likely the best fit for all your projects. Ask them about their areas of specialty. Look to participation in conferences and published (preferably peer-reviewed) articles by their researchers to document this.
Professional assessment
The most common way to start assessing a new supplier is via a capability presentation. Ask them to include not just their standard pitch but also pertinent case studies (with sensitive information redacted as necessary) to help you judge how they approach a research project. We have often used this approach (via redacting key parts of past studies) to show how we address a research problem from the study’s business purpose through research objectives, methods and findings.
How’s their problem-solving? The best research suppliers are good critical thinkers and demonstrate they are effective and creative in problem-solving. They are intellectually curious, interested and excited about your work. They ask good questions and are proud of what they do and how they do it. You can ask suppliers questions about how they have or would solve a challenging research issue but, ultimately, the best way to learn about how a supplier thinks are to send them an RFP. Their proposal should answer a multitude of questions. Do they offer multiple ways to address your research issue? Do they suggest alternatives that perhaps you haven’t thought of? Do they ask questions to refine their understanding of the RFP before submitting it? With today’s technology, it’s easy to make an RFP look pretty but it’s what’s inside that counts – the depth and quality of the thought.
Do they have industry experience? As the opening song of “The Music Man” goes, “You gotta know the territory.” A key component of a supplier’s competency is their knowledge of your industry, products/services and business. This knowledge is critical in every case, especially in B2B markets where the client’s products or services become part of their customers’ final products downstream. Knowing how B2B study respondents run their businesses, make decisions and the language they use is critical to providing context that makes study findings actionable. New suppliers can demonstrate this, at least in part, by what they share in their capability presentations (including case studies). Have they worked in your market and how often? Do they demonstrate knowledge of the market's inner workings, your product/service category and the terminology used? There is perhaps nothing as uncomfortable in a report out to your internal clients as a supplier who does not know the nuances of what they are reporting on. You can prevent this by doing your due diligence.
Project execution considerations
Can they consistently meet your deadlines? The only way to find out is to give them a project. This is also a good question for your peers (even those who work for your competitors) who have worked with this supplier. Missing deadlines is one of the surest ways a supplier can get themselves in hot water – and for good reason. But there is an inherent conflict. Suppliers are trying to manage their manpower to get the most out of it without overloading the system. In a supplier’s perfect world, their workload is level. However, things are rarely ideal and the workflow has ebbs and flows. It’s not uncommon for suppliers to include aggressive deadlines in their proposals – win the bid first and then work out the schedule. This is mainly because most projects are at least somewhat delayed and most of those delays are caused by the client.
A typical supplier complaint is that clients propose arbitrary and unrealistic deadlines. If you have a firm internal deadline – a meeting or a decision that must be made by a specific date – communicate this to the supplier upfront. A good supplier will work like crazy to deliver by that date or tell you that they can’t and work with you to find some middle ground, such as the delivery of the project in phases.
Do their deliverables meet your expectations? The key deliverable in almost every study is a final report, almost exclusively done in PowerPoint. The report should look professional, be free of typos (which we’ve found unconsciously cause internal client questions about the report’s validity) and hopefully not exclusively use WordArt. Many suppliers now employ graphic artists to punch up the look of their reports or report templates. However, as with proposals, while it’s relatively easy to make reports look great, it’s what’s inside that counts. It’s fair to ask new suppliers for a couple of reports (redacted as needed to preserve confidentiality) so that you can see how they approach the content and the look of their reports.
We have found it highly beneficial to talk with our clients about what they expect in each report before we begin drafting. Ask for a report outline. Doing this ensures the client gets what they want regarding content and appearance with the first draft. If your supplier does not initiate this, tell them about your expectations for the final report – length, style, order, inclusion of verbatims in qualitative reports and any other specifics you are looking for. Our reports have ranged from a very short executive summary of 10 or 20 pages for some projects to reports of over 100 slides, with a 10-slide summary. Do the groundwork in advance so you can provide whatever the client wants.
For some studies, deliverables like market simulators or the like are necessary. Again, it is key to work with the supplier to design these to meet your needs, mainly so that you and your internal clients can easily use them post-study. If your company often conducts this type of study, ask for examples as part of new supplier screening.
Finally, get clarity on what is expected for any mid-study check-ins – preferably in your RFPs. This allows the supplier to correctly budget for the added time and cost of these activities and account for them in the study timetable. Suppliers are likely to push back or seek to increase the project cost if the client wants to add these post-commissioning.
Do they use valid methods? In other words, are the methods they use well-founded in the marketing research literature? Marketing research firms often brand methods or study types as their proprietary version of “X.” This is especially true for analytical methods. While we support the concept of branding, the client must be allowed to see inside the black box and require the supplier to describe what makes their version of X different and better than other versions. We have had better success justifying methods for which we can cite references or use methods considered the gold standard (i.e., Sawtooth Software for conjoint).
Also, some suppliers rely too often on certain methods they have become comfortable or proficient with. Every research method has its uses but no one method is always the best approach. If you find your supplier recommending one method often, ask questions.
Do they have access to the right sample? Access to the “right” sample – the respondents who can provide the best insights on your research objectives – has become increasingly difficult in many B2B markets, especially those that are highly concentrated. However, even in consumer markets with large numbers of potential respondents, be sure to ask questions about the source and quality of the sample. Your supplier must have access to sample that will allow you to address your objectives. Suppliers who work in tight verticals often maintain their own databases or even panels of qualified respondents. Don’t be shocked if the sample sizes are smaller than you’d like – response rates have declined in these markets over the past 10 years, even as respondents can increasingly be reached online.
What is their post-delivery follow-up? In most cases, good answers result in more questions. Additional questions are usually raised during the delivery of the final report. A supplier should be willing to provide some agreed-upon additional support post-delivery, usually with an additional analysis of the data captured in the study. However, these requests cannot be open-ended; it’s unfair to the supplier (and unbusinesslike) to keep asking for more. This additional support costs the supplier time and, thus, money. Unless the supplier has built this into the initial study cost, at some point, there is a point at which they will ask for more money, especially when the added support causes the original study scope to “creep.” However, a new supplier will often be willing to go above and beyond the original analysis to solidify their relationship with the client. Don’t assume that this will always be the case just because they provided extensive post-delivery support in a previous study. The RFP and proposal stage is the best time at which to address the additional follow-up the supplier needs to provide.
What does it cost? Most clients offer RFPs to multiple suppliers to allow them to compare costs and approaches as a part of the selection process. A standard way of comparing cost is cost-per-complete (CPC). CPC can act as a starting point for comparison. Still, in our experience, each supplier will offer a somewhat different approach to your RFP – methods and/or deliverables – unless your RFP sets hard boundaries for each.
Our advice is to ensure you are comparing the costs on an apples-to-apples basis that takes these differences into consideration. A more rigorous methodology or deliverables may be more expensive but might be warranted to meet your study objectives. One of the benefits of getting multiple proposals is the opportunity to see different approaches to your research objectives. Finally, be careful to point out these differences in approach to your internal clients and not let them become fixated on the CPC.
All research suppliers want to know if you are less than happy with any aspect of a study – talk to them. They are your business partners and tens if not hundreds of thousands of dollars in projects are at stake in the future – for both of you. They want to ensure you are satisfied with their product. If you are unhappy, tell them, and if you decide to drop a supplier, tell them why.
The client’s responsibilities
Remember that a supplier’s proposal is not their line in the sand. There are usually multiple ways to approach a study; their proposal is their best attempt but likely one of several they could suggest (sample size, methods, etc.). If you find a supplier you feel is a good match for a study from the standpoint of their background, competence and other factors listed in this article, but their proposal doesn’t meet your expectations, talk to them. Give them a chance to resubmit. If the goal is to get the best possible work done, take a little more time.
When you are preparing to conduct research, do your prework. Prepare an RFP – even if it’s a short one. At least list the business purpose and study objectives. It’s hard for suppliers to respond to verbal or back-of-napkin RFPs because doing so usually results in an off-the-mark proposal.
Try not to pigeonhole the suppliers as only “good for” a certain type of study. We have too often heard long-term clients exclaim, “We didn’t know you did that,” even when we had discussed those capabilities many times before. We realize that it’s not the client’s job to remember what a supplier can or can’t do and it’s normal to make this type of judgment so that at least some tasks can be put in the “handled” pile. But there is almost nothing as deflating to a supplier as hearing after the fact about a study that was totally in our wheelhouse for which you used another supplier without soliciting an RFP from us.
In the RFP or during the kickoff meeting, adequately brief your suppliers. Give them as much background as possible on the situation that led to this research. Provide information on the products/services being studied and where these fit into the company's overall business. If you cannot do this, involve your internal clients (actually, we have found this is desirable in any case to help ensure their buy-in). Provide relevant prior research. Describe the hypotheses you want the study to test and their expected outcomes. There is sometimes concern that providing this information may bias or direct the research findings. Still, in our experience, it leads to higher-quality research that better addresses the study objectives and client needs.
Hold up your end of the timetable and communication. Your suppliers will do everything possible to meet the timetable's terms but they rely on you for critical reviews and approvals.
Set up for success
Effective collaboration between clients and research suppliers is the cornerstone of successful marketing research. By establishing trust, ensuring alignment on objectives and fostering open communication, clients can set their suppliers up for success, resulting in high-quality outcomes that benefit both parties. Ultimately, when clients and suppliers work as true partners, the research not only meets expectations but also drives impactful business decisions.