More KPI Catastrophes & How To Fix Them

“What’s your goal or target Cost-Per-Acquisition?”

— Every Marketing Platform

How does anyone answer that question? Aren’t we always striving to get as many customers as possible, for as little $ investment as possible? There’s always something hollow about answering that question — “target given what?”

Maybe a better question is this: “What’s your business objective?”

This greatly simplifies the problem. “I want to make as much profit as possible selling shoes,” you answer. Great! There’s a few easy concepts to know about how Cost-per-Acquisition plays a role in achieving that objective.

  • If we set a Cost-per-acquisition too low, you won’t acquire enough customers. Sure, the shoes you sell will generate a profit, but it will be minimal.
  • If we set a Cost-per-acquisition target too high, you’ll sell far more shoes, but at a loss thanks to acquisition costs that exceed those returns.
  • A “Goldilocks” CPA – one that’s not too low, and not too high – is required to make as much profit selling shoes, or any other business objective you might have.

“We tried a higher CPA target. It didn’t work.”

In my days as an investment analyst in media, I heard this so many times from the biggest, most strategic firms. “How much additional investment did you allocate?” I would posit.

“We didn’t raise our investment levels, because it was just a test. We kept our investment the same, changed our target CPA in the automation, and we didn’t see an improved performance. It doesn’t work.” After a few hundred of these experiences, I usually just closed my eyes and placed my head in my hands.

I have seen tests – designed like the above – more times than I can remember. The amount of professionals convinced they’d learned something meaningful was heartbreaking.

In programmatic auction media, “trying a higher CPA” or a “Lower ROAS target” literally means trying a higher budget. If you changed a CPA target in your ad automation, but didn’t increase the allowable budget, you didn’t “test” anything. You simply told the automation: “I’d prefer you lit some of my money on fire before investing what’s left of it in the market.”

“But why is that test wrong?” Sometimes someone would ask me. I would usually struggle for clear words. It seemed obvious to me that if you take 100 of something, agree to multiply it by 4, you’d get 400, and if you took 100, multiplied it by 3, you’d get 300, and that 400 was always higher than 300, so what are we even doing here but trust me — the overwhelming majority of people invested in these decisions were not thinking about things in this way. The complexity of digital measurement clouded what would otherwise be obvious, and the pressure to grow made marketers and professionals desperate to “test” things they weren’t really ready to test in an effort to appear innovative.

Accepting the new ROAS target or not, this is how a test is supposed to look: increased investment levels are tested and measured against prior results.

Let’s simplify this.

  • Testing involves risk. If you aren’t willing to risk additional loss, please do not attempt testing new KPI targets.

  • If you don’t have the stomach to test incremental marketing funds, don’t bother testing higher CPA targets, CPO targets, lower ROAS targets or anything at all akin to it. You are wasting your time. Don’t make these mistakes in programmatic auction media, or find someone who won’t make them for you.

KPI Catastrophe #2 – Margin Mishaps

You’ve had a great year, you think. You are going into your annual performance review with finance and your numbers seem great: revenue records, strong year over year growth, and a gross profit number better than before. Top and bottom line improvements – amazing, right? But you don’t get that bonus. Instead, your CFO walks you through their tables – marketing costs exceeded returns, net-loss to the business’ bottom line. You are shocked. They trim your budgets for next fiscal. No bonus. What happened? The answer is usually R & R – and no, I don’t mean taking enough vacation.

I’m talking about returns & refunds. Marketing targets rarely account for either of these factors when calculating ROI, ROAS, or setting objectives and KPIs. On the surface, it makes sense. “None of these things are controlled by marketing, so why should it be our job to account for it in marketing KPIs?” It’s true – it isn’t the CMO’s fault if the fulfillment center manager goes on extended leave. When your office internet goes out, you don’t bring in the media manager to get a tongue lashing. But like many things in life, even though you can’t control it, you still have to learn to accept it and deal with it. Your job is to grow the company, which means that the revenue must grow and the bottom line needs to eventually follow.

As a marketer or key decision maker, this is as simple as knowing “What $ figure really represents our net margins on sales or customers today, and which direction is that number going?” Having a sense of these measures before setting your marketing KPIs makes all the difference when it comes to your performance review, and your bonus.

KPI Catastrophe #3 – Taking Inventory

Marketing analytics & data science professionals on the cutting edge of media & data strategy throw around fancier terms than “cost-per-acquisition”. There’s “expected return on ads spend”, “profit-per-action”, “net profit contribution” and other mouthfuls. These measures are often available on a per-impression, per-click, or per-event basis. While they certainly are valid and sophisticated measurements, they usually obfuscate one tiny, important detail.

You bought shoes for $20 per pair, and when you sell them for $100 per pair you’ll make $80 per pair. But that only goes for the shoes you actually have in the warehouse.

“If you only have 100 pairs of shoes in the warehouse, what’s the Profit-Per-Order on the 101st pair of shoes sold?”

It’s not a trick question – the profit is $0. You’ll have to refund that order.

“Per-item” and “real-time” metrics are valuable, but just like available capital and costs, they have real, tangible limits. You might be saying “Listen, I’d love to sell out of my products, what a great problem to have!” And it’s true that you’d rather sell out, than sell nothing, but let’s think economically and pragmatically — how could this lead to a KPI catastrophe? Let’s do a short thought experiment.

Basic model inputs to answer the question: “What should be my Cost-per-Order?”

How do you begin solving this thought experiment? You simply start at the top of the Cost-per-Order table, and work your way down until you get your answer.

Which CPO projects to drive the best results, given the business objective?

I would imagine two possible reactions to this thought experiment.

#1 – “The answer is obvious – is this insightful?” Walking through the thought experiment from top to bottom, it becomes clear right away that $20 CPOs project the best possible net profits, and it’s not particularly close.

#2 – “This is a simple thought experiment? Look how many variables! Why are there so many details? This is complicated.” To that I would say, yes, it is! In reality, there are no simple examples. Even in this basic case, we see dozens of assumptions requiring research – how do we know the projected sales by CPO? Is that from a tool within our marketing platform, or our prior research? How do we know how many people are in the market for shoes? Demand is notoriously complicated to predict, and fraught with recency bias, over-fitting, and other common pitfalls. Beyond that, what even is carrying cost?

If you think there’s too much information here, try changing just one aspect of the experiment and see if you get the same answer. For instance, what if you had 250 shoes in inventory? Suddenly $5,000 in refund losses turn into profits, and $30 CPO projects to be the best option. What if $12 CPOs drove 2 sales per day? Then $12 CPOs would be the most profitable target.

The complexities don’t stop there. For example, inventory might come in 200 case orders. you can sell 250 shoes @ $30 CPO — do you buy an extra 200 to sell the extra 50? Knowing the Economic Order Quantity means knowing enough to balance the costs of ordering more products, to fulfill the additional sales demand, and every real world retailer who’s unfamiliar with accounting for this in their projections and KPIs isn’t setting the best targets possible. And once you’ve sold all you can sell – will you have any way to liquidate the rest? What are your liquidation economics? How do pricing and offers adjust your model?

The moral here is to ask yourself the following: Am I making CPA, CPO, ROAS or other media & marketing efficiency targets without any line of sight into some of the questions above? If so, you’re KPIs might be doing your business more harm than good. Arm yourself with a better process – develop better KPIs.

About the Author

Matt Butler
Ex-Googler of 12 years, Matt was a founding member of Google’s analytical consulting team, developing analytics, statistical forecasting, auction modeling, and machine learning for companies such as Procter & Gamble, Coca-Cola, Unilever, Kohl’s, Best Buy, and many more. He went on to lead global technical partnerships before leaving to found Bonsai in February, 2020.