One of the reasons measuring “true progress” in a product is hard is that we prefer reporting good news over bad. We like charts that trend up and to the right, which isn’t all that bad by itself until we start devising charts that can go nowhere but up and to the right.
In the Lean Startup world, we label those charts as vanity metrics.
But not all charts devised to go up and to the right are bad.
First, in the interest of fairness, there is a place for vanity metrics. It is often used with great effect on marketing websites to build up social proof and ward off competition. You run into problems, though, when these same metrics are also used as internal measures of progress.
But more importantly, a subtle point that is often missed is that it isn’t the metric itself but how you measure it that makes the metric vanity or actionable. For instance, there is nothing inherently wrong with measuring visitors on a site, sign-ups, number of downloads, or revenue.
All these measurements make up the customer funnel and can be helpful to know when troubleshooting a problem area. You run into issues, though, when you either measure these metrics non-actionably (more on this in a moment) or drown in a sea of micro metrics.
Here are two simple approaches to avoid these issues:
1. Measure your metrics in batches, so they are actionable
The reason cumulative counts get you into trouble is obvious. It’s because they are numbers that can flatline but never go down. But the reason aggregate metrics get you into trouble isn’t so obvious.
Measuring your users in aggregate is problematic because the customer lifecycle is a long funnel. During that time, your product constantly changes with every feature release and marketing campaign. This makes it hard to derive long-term cause and effect which is the goal of gathering actionable metrics.
You need to be able to tie back observed results to specific and repeatable actions.
Building on the Customer Factory metaphor from last time, you can more rigorously derive causality by viewing the output of your long customer funnel in batches. The simplest type of batch is time-based.
You group your users into daily, weekly, and monthly cohorts (based on their join date) and then measure every user action (not just retention) within that cohort.
Users that join within the same timeframe experience your product similarly enough, which is why they can be grouped. Together they establish a baseline or benchmark to beat, which is uncanny.
Something I call the Groundhog Day Effect.
2. Organize your metrics, so you don’t drown in numbers
Not all metrics are created equal. It’s important to divide your metrics into macro and micro metrics, so you don’t drown in a sea of numbers.
Macro metrics are useful for learning or baselining while micro metrics are useful for optimizing or troubleshooting.
Macro metrics help you build a high-level model of your customer lifecycle, which you can use to benchmark your product. I’m a big fan of Dave McClure’s pirate metrics for this. Even though this model came out of the software domain, I find these key macro events are adaptable to a wide range of businesses.
Seeing it in action
Here is how we apply cohort-based measurement with a three-tiered hierarchical dashboard:
1. Five-up standard measure of progress view
This is our starting point. This view lets us benchmark our product using daily, weekly, and monthly cohort groups.
For clarity, here’s how to read these numbers:
“In July 2012, 1852 people signed-up. Of those users, 35% activated, 31% came back after 7 days, 23% referred the product to other people, and 2% upgraded to a paid account as of when this report was generated”.
That last part is key. The actual actions don’t necessarily have to occur within the cohort month (July 2012). For instance, most people might have upgraded two months later (in Sept 2012). Unlike aggregate metrics accounting which closes at the reporting window, cohort metrics accounting is organic as long as you have active customers within the cohort.
2. Sub-funnel view
The first view is used to identify the constraint or the key macro metric that matters. Each macro metric is, in turn, a sequence of micro metrics which you can visualize by drilling in.
Here is what an activation sub-funnel might look like:
3. User activity level view
And finally, you can drill down to the user. This Facebook timeline-like view is particularly useful for better understanding specific user behavior, such as when handling customer service requests.
So what is the Groundhog Day Effect?
After measuring dozens of products using the five-up view above, I’ve repeatedly seen a flatlining set in once steady traffic kicks in. That’s what I call the Groundhog Day Effect — from the movie Groundhog Day.
In the movie, the protagonist, played by Bill Murray, is stuck in a loop where his life repeats every day until he has a breakthrough insight that breaks the loop. If you haven’t watched the original, there are numerous plot remakes that I’m sure you can relate to.
Similarly, in a startup, once you normalize the data and account for short-term seasonality, it’s uncanny how the customer lifecycle repeats itself day in and day out until you reach a breakthrough insight that breaks this loop.
On the one hand, this can be depressing. Because once steady traffic sets in, despite your best efforts, this graph flatlines. No one wants to walk into a meeting with a flatline graph.
On the other hand, this can be highly empowering. Once you have a stable benchmark of your product, it permits you to aggressively experiment with bold new ideas to effect a spike in the flatline. If you manage to create a spike, your next course of action is to run more of whatever you did to make the line stick at the higher level.
Your goal is to do everything possible to repeatedly and systematically drive all of these lines up and to the right guilt-free of vanity.