How to Make Sure You’re Not Using Data Just to Justify Decisions You’ve Already Made

How can an organization tell whether it’s actually letting data inform its decision making — or if it’s merely using superficial analyses to retroactively justify decisions it has already made?

Traditionally, organizations have used data analytics as a tool of retrospection, as a means of answering questions like, “Did this marketing campaign reach our desired audience?” or “Who were our highest-value customers over the last year?” or “Did engagement peak at regular intervals throughout the day or week?” These answers are typically built around metrics — or key performance indicators (KPIs) — like click-through rates, cost per impression, and gross rating points, which companies all-too-often decide on too late in the process.

These descriptive analytics — that is, analytics that measure what has already happened — are undeniably important. But they’re just a bit player in the far more sprawling drama that is data-driven decision making. Within organizations that are truly data-driven, KPIs aren’t arbitrarily plucked out of thin air, but are generated at the start of a decision-making process. More precisely, it’s not an organization’s KPIs, but the key business questions (KBQs) — of which KPIs are an extension — that serve as the cornerstone of its success.

In their HBR article Big Data: The Management Revolution, Andrew McAfee and Erik Brynjolfsson arrived at a similar conclusion, writing, “Companies succeed in the big data era not simply because they have more or better data, but because they have leadership teams that set clear goals, define what success looks like, and ask the right questions.”

However, arriving at “the right questions” is easier said than done, as any investigation must extend beyond, “What do the data say?” At my agency, our KBQs emerge from a rigorous four-step process that forces us to leverage data throughout the planning phases of our marketing campaigns. Though its specific applicability may vary slightly from industry to industry, our process provides a highly actionable model for deploying data analytics in a proactive, transformational manner; one that guides your decision making instead of justifying it.

Step One: Define your purpose. At the start of every planning cycle, an organization should make a concerted effort to engage stakeholders from every corner of its business in a wide-ranging discussion aimed at defining the campaign’s purpose. This begins with methodically zeroing in on the challenge(s) you’re trying to solve. Are you trying to improve a customer satisfaction rating? Cultivate long-term loyalty among a specific subset of customers? Increase the number of products that ship from a certain warehouse?

Don’t hesitate to interrogate the status quo — and, when appropriate, dismantle it. A history of maximizing pageviews is not itself a compelling reason to set a renewed goal of maximizing pageviews. Take a step back, survey the landscape (both internal and external), and carefully consider whether you’ve defined your purpose in accordance with anything other than the force of habit.

Step Two: Immerse yourself in the data. Once an organization has identified its purpose, it should conduct a comprehensive survey of what it already knows to be true. This is the stage where an organization should answer, “What do the data say?” That said, it should do so with a distinctly forward-looking mindset. At this stage of the process, an organization should take little interest in evaluating — and even less in justifying — past decisions. The totality of its interest should rest with how its data can inform its understanding of what is likely to happen in the future.

Like the previous stage, stage two is highly collaborative. In pursuit of broad-based collaboration, an organization should democratize its data to the greatest extent possible, funneling it into the hands of experts and non-experts alike. Not everyone at your organization is going to have a PhD in mathematics or a professional background in data science, but this doesn’t preclude anyone from getting their hands dirty in your data — after all, one doesn’t need to understand how a tool works to appreciate and take advantage of its utility. Ensuring that stakeholders across your organization come to a mutual understanding not only of the facts, but of their importance, is critical to the success of the rest of the process.

Step Three: Generate key business questions. While the previous stage pushes an organization to the edge of its organizational knowledge, this stage sends it tumbling into the unknown. With a goal and a set of agreed upon assumptions in hand, the organization has everything it needs to start posing KBQs, or lines of inquiry that propel it from “What do we want to achieve?” to “What do we need to know in order to achieve it?”

Using the precise purpose-defining language it established during the initial stage, an organization should now challenge stakeholders to ask as many questions as they can think of, first individually, then as teams. Good questions, bad questions, self-evident questions, unrealistic questions — it matters not. The objective is quantity, not quality.

While no topic or line of inquiry should be off-limits, an organization could start with these:

Can we predict which customers are at the highest risk of switching to a competitor, and design programs to decrease that risk?
Can we predict which customers have the highest probability of trying and subsequently adopting our brand, and design cross-channel promotional strategies to reach them most effectively?
Can we identify the optimal price point for our brand in order to maximize growth at a certain level of profitability?
Can we rethink the way we communicate with our target customers across our portfolio of products by understanding the combinations of products that are most often purchased by the same customers?

In many cases, such unfettered inquisitiveness requires feigning a degree of ignorance; that is, pretending that you don’t know what you know or pretending that your data doesn’t exist. This can be something of a high-wire act, especially for organizations new to data analytics, but it pays immense dividends if executed properly. Creativity and innovation are central to this phase of KBQ generation, and hewing too closely to your existing data is a recipe for the opposite.

To a similar end, it can be valuable to take the KBQs you generate and “invert” them. Just as sketching an object upside down can help an artist more accurately reproduce its likeness, rewriting your KBQs in the negative can produce more “Aha!” moments than would otherwise arise. Consider the following hypothetical progression that a pharmaceutical company might go through:

Purpose: Increase medication adherence among patients who have been prescribed Drug X.

KBQ: Which outreach methods do non-adherent patients respond to most reliably?

Inverted KBQ: Which outreach methods do non-adherent patients not respond to?

This slight shift in perspective can be a game-changer. Like any activity dealing with human behavior, marketing is an inexact science, and the value of strategically constraining your efforts cannot be overstated. Uncertainty is far more palatable — and far less problematic — when you know precisely where it exists than when it pervades your entire operation. In business, known unknowns are preferable to unknown unknowns.

Step Four: Prioritize your key business questions. Only after an organization has compiled an exhaustive list of KBQs should it begin evaluating, critiquing, and prioritizing them. In practice, some KBQs are highly actionable but lack the clear potential for making a business impact, while others have the potential to revolutionize your business but are highly inactionable. Pipe dreams, curiosities, and incremental improvements are all situationally valuable, but focusing on the pursuit of high-value KBQs will ultimately drive meaningful results.

Transforming a defense mechanism into a change agent. It’s tempting to place data analytics at a discrete juncture in your operational processes, but the reality is that data is not something to be used periodically, nor within strict project-based silos.

To drive real results, an organization must use data analytics throughout its business cycle. Today’s descriptive analytics are the foundation of tomorrow’s KBQ-oriented planning processes, which in turn are the foundation for a forward-looking analytics brief that details how an organization is going to answer its high-value KBQs. It’s this cyclical, mutually-informing decision-making architecture that both accelerates organizational transformation and disrupts your fixation on the rear-view mirror.

As Nobel Prize-winning physicist Niels Bohr once quipped, “An expert is a man who has made all the mistakes which can be made in a very narrow field.” Nowhere is this truer than in business. A well-conceived data analytics program empowers organizations to redirect their focus from justifying past decisions to learning from past mistakes. The sooner organizations make this pivot, the sooner they will enjoy the benefits of truly data-driven decision making.