This article is a little bit more technical than most of our articles. It talks about a specific challenge with building MMM (media mix modeling). If you are considering MMM, and want to speak with some of our experts, feel free to reach out. Note that INCRMNTAL does not offer an MMM solution, but we are happy to advise you on the topic, as openly and honestly as we can.

MMM is getting a lot of publicity lately. Given the state of user-level attribution becoming impossible, companies search for predictive and probabilistic methods of measurement for their marketing spend. MMM is a very powerful methodology of assigning the absolute value to paid channels using a predictive analytics approach. MMM though, is not a very useful day to day operational tool, as the granularity of insights it can provide is very coarse, the resources required to run MMM models are large or even enormous, but more importantly – MMM relies on priors.

Priors, is a term in Media Mix Modeling (MMM) referring to a “prior knowledge” of the value of channels.

Where do MMM priors come from ?

In MMM, a prior would reflect the value the marketer or data scientist will assign to paid channels to form a certain “baseline”. 
i.e. The ROAS of Branded Search is 120% , while the ROAS of Paid Social is 60%

There are typically 3 ways to come up with a prior:
Conducting planned experiments
Intuition (yes, very scientific…)
Channel level attribution

Priors skew MMM directionally, to provide a working assumption of what is the value of each channel.
Priors pose a significant design flaw in MMM, which most data scientists would agree with, since it removes both the data and the science from the equation, allowing companies to paint the picture as they would like to see it.

We’ll break down the 3 main methods of getting priors below. 

Planned experiments (GeoLift tests) as a prior

Planned experiments often refer to a test where an advertiser will black out ad spend in one city, while keeping another city running, comparing what is the incremental value of marketing results in the city where ad spend was on vs the city where advertising was turned off. 
In theory, this is a good approach, creating a kind of “laboratory test”, however, in reality, the methodology relies on being able to identify users location and make sure that the test is not tainted.

To illustrate the challenges in getting users location – head over to iplocation.net from your computer browser, as well as from your phone browser (after turning WiFi off). 
My computer placed me 10 miles away from my home office, while my mobile phone IP, placed me over 60 miles away. A GeoLift test would have identified me as 3 different people:

Another common issue with GeoLift experiments, is that while stopping one channel in a certain region, other channels might claim results for themselves, given that attribution works by painting the last user that engaged with an ad as “theirs”.  

Intuition as a prior

Some marketers are better than others. So much better that they can actually feel the ROAS of TikTok by placing their hand over the vibration of each and every click. 
I’m being ridiculous here, as the notion of using intuition as a prior may just as well be going to a palm reader at a fair asking what will be the CPA of my PMAX campaign ?

But all jokes aside – Intuition is often used as a prior in MMM. Marketers and data scientists will work hand in hand, guesstimating the performance of channels, and also utilizing those when testing out with a new channel. Meaning: “as previous ROAS for Facebook was 70% - Lets assume that TikTok will also have a ROAS of 70%”.

Channel level attribution as a prior

“Our Google Ads account got hacked, and Google banned our account. When spend went to $0 I noticed that the traffic to our website stayed the same. Whatever was previously attributed to paid, was now attributed as organic” (CEO of a major D2C company)

User level attribution is and has been a fantastic method of tracking. It provides a great proxy to “who touched the user just before the conversion point”, however this is often a deceptive data point, as attribution will always do the job of attributing, even when the reported conversions were not incremental. (we wrote about this in the article titled “attribution data should be taken with a grain of salt”)

Utilizing attribution data as a prior for MMM will inherently mean that the MMM will suffer the same skewed view reported by attribution. And if your attribution data carries bots, attribution fraud, or a large discrepancy against your source of truth – it will be back to the saying:  “garbage in, garbage out”

Priors stop being priors over time

Closing with a bit of optimism , since my intention is not to bad mouth MMM – Priors are only used in MMM when deploying the first set of insights, or when introducing a new channel into the mix. As time goes by, more data is accumulated, and more spend is fed on new channels, MMM will improve as gradual changes in ad spend, and a (hopeful) change in performance will improve the outputs and insights produced by MMM to the point where the previously inserted priors are barely used. This may be a tedious and heavy resource process to go through, but there is a light at the end of the tunnel.

MMM is an awesome tool in the right hands, and a company that utilizes a good combination of measurement methodologies, answering different questions, will benefit more than those who rely on one method – any method.

While INCRMNTAL does not develop an MMM tool, we are more than happy to have an open and honest conversation about MMM if you’re currently considering developing MMM or using a 3rd party platform or agency to develop your MMM. Reach out to us, or book some time with our experts.