Don’t Count Your Users Like Sheep

In my last post, I introduced the idea of establishing a standard measure of progress for your products using a handful of macro (versus micro) metrics measured in batches (or cohorts).

Macro metrics are useful for learning or baselining while micro metrics are useful for optimizing or troubleshooting.

The problem with micro metrics is that there’s still much of it to wade through. Today, I’d like to share our approach to navigating all this data without drowning in a sea of numbers.

The Traditional Approach: Declarative sub-funnels

The traditional approach is to organize your micro-metrics into one or more sub-funnels. The basic idea is that users perform a series of steps toward a particular goal (or call to action). And by visualizing the drop-offs between each step, you can pinpoint leaky buckets in your funnel.

While seemingly logical, all of this is predicated on the assumption that you can clearly define a series of steps between a starting point A and goal B.

Back in the early days of analytics software, you had to explicitly declare these funnel steps upfront and code for them. Each time you made even a slight change to your flow, you would have to redeploy new code, which was a (re)configuration nightmare.

Newer analytics products solve this problem by allowing you to define your sub-funnels on the fly, which is a huge improvement. But you still have to be aware of certain limitations which make it nearly impossible to accurately measure every user in your funnel:

Limitation #1: Users must start the funnel at the beginning or they aren’t counted

Usually not a problem if you can ensure that everyone starts at the same starting point. But the real world is full of exceptions. And the exceptions don’t get counted.

Let's look at some simple examples:

1. On a marketing website, you might explicitly separate your landing page from your pricing page so you can track each as a separate step, like I often do. But despite my best attempts to shepherd users down this path, thanks to google and people sharing links, visitors don’t always start on my landing page.

2. With a web app, you might build a wizard to force users down a certain path. First, wizards are not as trendy as they used to be from a UX perspective. But also, you might have a non-linear funnel on your hands where there isn’t a prescriptive series of steps that fits everyone (like filling out a Lean Canvas).

3. Even if you did opt for a wizard, what happens beyond your activation flow when users can truly do anything they want in the app? How do you define the funnel, then?

Limitation #2: You can’t have optional steps

Some of the best breakthrough ideas come from off-the-beaten-path. Allowing users to wander and experiment should be encouraged, not discouraged, and tracked.

Limitation #3: Users must complete the funnel within a set timeframe

Many of these newer tools leave you hanging if your funnel extends beyond 30 days. Declarative funnels work for clearly defined short-lived goals like the conversion from landing page to sign-up. But you also need to be able to tie these short-lived sub-funnels to the overall macro (like revenue), which may be several weeks or months down the road.

For example, not mentioning the price on my marketing website might increase short-term sign-ups (Yay!) but reduce end-of-month paid conversion (Doh!).

You need to measure the whole system end-to-end to not fall prey to micro-optimization (or local maxima) traps.

The fundamental problem with declarative funnels is that they presuppose we can define funnels upfront and steer our users like sheep which in practice isn’t true.

A Different Approach: Emergent funnels

Rather than declaring funnels upfront, we have been using an emergent approach to measuring our products using USERcycle with great effect, which is best illustrated by this story:

A college dean tasked with overseeing construction at a new campus asked the landscapers to only lay out the grass but not any sidewalks. When asked why, he told them to come back in a year and the paths would reveal themselves based on where students had walked.

Sidenote: I remember reading this story in a book but can’t remember which one. I found references online to both UC Irvine and Berkeley. If you know the proper source, please leave a comment below.

Irrespective of whether this story is actually true or an urban legend, this same emergent approach can also be applied to understanding your users. Rather than defining and configuring sub-funnel steps, we only define the macro goals and populate everything in between (the paths) based on what users actually do versus what we think they should do.

Let's take my earlier example on Lean Canvas and focus on the activation sub-funnel. I only need to define what qualifies as activation. For Lean Canvas, we define activation as filling out at least 6 out of the nine blocks.

We then track user actions (events) and visualize what users actually did between sign-up and activation to reveal the emergent funnel. The advantage of this approach is that there is no (re)configuration step ever required — provided you define your activation action correctly (i.e., align it with your application’s key activity). You can endlessly change and optimize what goes in between the two macros and measure progress as a function of either increasing activation rate or decreasing time to activation.

Not all users are created equal

Of course, different users will take different paths, but this is where things become interesting. In the screenshot below (click it to expand), you can see that we break this cohort of users into several different sub-populations.

First, we want to see what successful users did versus unsuccessful users. Did they take a particular step, like watching an intro video? On the other hand, did unsuccessful users take a particular step that side-tracked them away from the goal?

Next, we want to know what was the most popular success path. You can see that highlighted with a red line in the screenshot above. It’s equally important to study the outlier (less popular) activities. Do they need to be there, or are they distractions?

And finally, within the successful user population, we want to further sub-segment into fast and slow users shown in the screenshot with the box and whisker chart in the top right. You’ll see from the numbers above that the fast users (1st quartile) complete activation between 2 minutes and 9 minutes. While the slow users complete activation between 7 hours and five days. Why such a wide range?

“Happy families are all alike; every unhappy family is unhappy in its own way.”
-Tolstoy, Anna Karenina

To channel Kathy Sierra, it’s your job to take users past your application’s suck threshold and have them start kicking ass as quickly as possible.

Understanding what your fast users (power users) and slow users do differently is key to making the crossover.

What’s Next?

The activation sub-funnel is still a relatively short lifecycle goal. In a future post, I’ll share techniques for navigating even longer lifecycle goals like retention and engagement.

What do you think about using an emergent versus declarative funnel approach?
Leave your comments below.

You've successfully subscribed to LEANSTACK Blog
Great! Next, complete checkout to get full access to all premium content.
Error! Could not sign up. invalid link.
Welcome back! You've successfully signed in.
Error! Could not sign in. Please try again.
Success! Your account is fully activated, you now have access to all content.
Error! Stripe checkout failed.
Success! Your billing info is updated.
Error! Billing info update failed.