Editor’s note: Marianne Moran Peterson is senior research manager, and Tanya Pinto is director, market research, at Microsoft. They are based in Redmond, Washington.
COVID-19 has thrust us all into a new world of uncertainty and adaptation almost overnight. Businesses have had to suddenly work in unexpected ways and at an unprecedented scale.
Microsoft has been committed to assisting customers as they go through these transformations by helping them to enable collaboration remotely, increase productivity across devices and help workers be more creative and efficient, all while ensuring security is protected and risk managed.
As we all try to navigate these unchartered waters, the long-standing business goal remains for Microsoft; how to deliver new value to customers and how to sell those offerings – via packaging, pricing and business models. It is vital these items are priced accurately, especially in a COVID-19 world.
The challenge remains in knowing, as definitively as possible, that a new offering matches the value customers place on them.
Informing decisions
In this new reality of uncertainty, certainty in data is more important than ever.
Microsoft has moved from a world in which customers ran software on-premise to a world in which they can run it on-premises, in the cloud and those on the cusp of digitally transforming its businesses. Cloud-based services are the primary drivers behind Microsoft’s growth and due to that, companies can now empower their first-line workers with apps, features and services that improve productivity, communication and overall experience. To better serve customer needs, it has become imperative to have high confidence in the customer research data that is collected as it informs decisions made by business planning teams and drives prioritization for engineers working on future product roadmaps.
In recent years, Microsoft struggled at times to find the right methodology that most accurately forecasted interest in its offerings, customers’ willingness to pay, as well as the ability to correctly define whether these new values should be sold as add-ons or stand-alone features. It has been additionally important to be able to build upon previous research data, rather than doing a completely new study each time for efficiency and leverage. Having that evolution of history and context has helped track shifts in customers’ needs over time – not just to grab a snapshot of where they are currently.
However, when it came to predicting pricing and revenue, the “usual” research methodologies started to feel limiting.
The team at Microsoft’s Customer and Market Research organization (CMR), a central team at the Redmond, Washington headquarters, started to notice that the Discrete Choice Modeling (DCM) research methodology they used proved over and over to show an overstatement in preference for premium offerings and not an authentic assessment of what customers truly valued. The way in which the research study presented the choices by showing the best option that always contained the most expensive and fully loaded features, led respondents to choose the “best” every time. This meant they always ended up selecting the most premium and expensive option. This wasn’t realistic as not everyone truly needs the fully loaded option. Nor does everyone have endless budgets to pull the trigger and actually purchase it. There is a tendency for people who take surveys to be positive. However, this doesn’t help engineering, product development or sales and marketing teams if customers say they want everything, and everything is fantastic. The research methodology and thus the resulting output was inherently flawed. And this is a commonly-used methodology in quantitative research.
True value
Why was this such a problem for Microsoft? Because Microsoft wants to develop products that customers truly value – products that customers use to transform their business and drive their growth. It doesn’t do any good if research produces false positives. Microsoft developers, product marketers and sales managers need to develop and sell products that customers demand, will use and that shows them how technology solves their problems. Most importantly, it is imperative that engineering teams know what customers value so they can assign appropriate resources to develop it and bring it to market. Early in the product development cycle, engineering needs to know whether its investments will pay off before making decisions to resource. But how do you predict customer demand before something is in market?
Over the past year, Microsoft’s CMR team led an effort to address these challenges, proactively looking for ways to change its usage of existing DCM research techniques to find ways to yield more accurate results.
They were unsure what the outcome would be, but one thing was for certain – the stakes could not be higher. These are multibillion-dollar decisions. The magnitude of any big change at Microsoft comes with enormous scale. And they had one shot to get it right.
What’s never been done
The CMR team knew to be successful they would need to build a vast coalition consisting of vendors, external experts and stakeholder support. This would require building partnerships on an unprecedented scale, as changing how Microsoft conducts its research methodology would impact each of its business goals. The first step toward changing its research approach was to understand best practices. CMR contacted nine research vendors that pitched to them on how they could improve the methodologies they were using. These vendor reviews gave a better sense of the types of techniques that are available, especially in relation to choice models, as well as ways of framing the exercises to respondents, best practices in showing choice sets to respondents and so on.
Second, CMR reached out to GBH Insights, whose Chief Research Officer is Professor Eric Bradlow, the vice dean of analytics, chair of the marketing department, and KP Chao Professor of marketing, statistics and education at Wharton. Each provided a helpful external perspective. Professor Bradlow came out to Microsoft to conduct workshops on best practices. Business planning stakeholders attended too, so they could learn firsthand how these techniques work. Through Bradlow’s training, as well as training provided by one of its research partners, everyone was able to learn together. This allowed CMR to build the buy-in needed to implement changes in its approach.
The game changer – the benchmark survey
CMR worked with stakeholders to field a benchmark study that would enable them to understand the value of the individual feature components in one of its key offerings – Microsoft 365. By shifting the approach from presenting a static license placemat to a more dynamic, experimental approach, they could test different packaging configurations that allowed them to generate utility values for all the components in Microsoft 365. What drives conjoint analytics is variance and you need variance so you can see what moves. The outcome was that they gained a better understanding of what customers value most. It also provided a study that they could link back to, so that in the future they could more easily field mini-DCMs to test new value. This allowed them to move away from doing purely stand-alone studies, build upon past studies and become more programmatic over time.
Some key elements included:
- Employing more experimental conjoint/DCM designs that do not lead customers to view product choices under the framework of good/better/best options.
- The ability to mask the version number of the product was key in that the respondent wouldn’t just automatically choose the latest and greatest believing it was the best. This approach, under zero pre-conceived notions from the respondents, provided research that showed what customers actually valued.
- Implementing dual response questions following product choice tasks – asking respondents to assign a probability for the likelihood they would purchase the offering they selected in a task. This provided more fidelity around take-up rates.
- Implementing budget flags that notify people when selections have gone over their budget. Now they could be more realistic in not only what they needed, but what they could afford.
- Exploring two-step approaches to pull into choice exercises the elements that are most important to people.
- Better understanding of the technical issues in conjoint approaches, e.g., around non-compensatory and compensatory strategies, and around lexicographic bias, which became a critical pre-sorting process, allowing respondents to focus and prioritize first what was important to them before moving on.
Final impact
Microsoft now has greater confidence in its data and thus, can make better-informed monetization strategy decisions to drive business growth.
This is an example of the growth mind-set that Microsoft’s CEO, Satya Nadella, has emphasized that is crucial to future success: to question, to be open to ideas, to listen and to act.
“At Microsoft, we're aspiring to have a living, learning culture with a growth mind-set that allows us to learn from ourselves and our customers. These are the key attributes of the new culture at Microsoft, and I feel great about how it seems to be resonating and how it's seen as empowering,” says Satya Nadella.
By embarking on this benchmark survey and changing its research approach, Microsoft now has more clarity on the value that customers assign to features, and much higher confidence that the research reflects customers’ true preferences.
Research data now quantifies real-world customer perceptions of value.
As the director of business planning and strategy, Modern Workplace, Microsoft put it, “The Modern Workplace Business Planning team runs regular research to inform critical decisions on packaging and pricing that impact billions of dollars in revenue. In recent years, we’ve struggled to identify research methodologies that can more accurately forecast willingness-to-pay and the impact of packaging decisions on SKU mix. CMR spearheaded an effort to address this challenge with a benchmark study that will allow us to more accurately forecast business impact based on research results. As a result, we now are able to make better-informed decisions on monetization strategy to drive business growth.”
This better fidelity with customer’s decisions means they are getting the information on the product and price point that helps predict its performance in market.
The new techniques have corrected for the various problems they saw before the changes were made. The ROI from this research comes from its ability to quantify customer willingness to pay for discrete or nuanced value. Understanding customer willingness to pay is especially important as Microsoft shifts its business model from selling physical packaged goods (one copy of software installed on one physical device) to selling to less tangible services (subscription services tied to customer usage of cloud services).
This means better decision-making – not just in business planning but for other groups that leverage the research and allocate resources such as engineering, product marketing and finance – as the teams seek to grow the business.
Microsoft’s turned a well-established DCM model on its head – garnering more precise results.