I can summarize my last post with a strong call to action:
Focus on Product/Market Fit First.
Here I’ll be laying the groundwork for how I plan to measure and optimize the process of achieving product/market fit. Metrics are, of course, the answer. But I have spent enough cycles getting lost in Google Analytics’ numbers that it’s not just about numbers, but the right numbers. The right numbers vary over time and have to be actionable.
Dave McClure has some great models for startup metrics which I’ve decided to adopt. He talks about creating a 1-page business model and identifying 3–5 actionable metrics you live by (conversion dashboard).
There are five basic steps to any conversion process that Dave defines:
- Acquisition: How do users find you?
- Activation: Do users have a great first experience with your product?
- Retention: Do users come back and use your product?
- Referral: Do users tell others about your product?
- Revenue: How do you make money?
While you should start laying the foundation for measuring all five steps from day one, you don’t have to start optimizing all of them. That said, you should have thought through all the steps and be able to state how (you think) you will address each one (Customer Development Hypotheses). It is also critical to have some customers travel through the entire conversion funnel first (Customer Validation) before starting any optimization. Lastly, for any optimization to be statistically meaningful, you must establish a few channels that drive just enough new users to enter the top of the funnel (Acquisition). These could be either through free channels (blogs, SEO) and/or paid channels (Adwords, banners). Just don’t break the bank (not yet, anyway).
1-Page Business Model
For CloudFire, I am starting with three core metrics: Activation, Retention, and Revenue. Here’s my 1-page business model that summarizes my user types, conversion events, and priorities:
Note: It’s important to frame the metrics as specific actions the user takes, which makes them both measurable and actionable.
Activation is critical to measure as CloudFire is a downloaded product, creating more friction to overcome. Retention is probably the most important metric for validating early traction and product/market fit. As I mentioned, Sean Ellis uses a 40% retention test to gauge product/market fit.
Some advocate deferring Revenue to later, but I believe getting people to keep paying for a product is the ultimate product/market validation. Since CloudFire is a premium service with no FREE plan, Retention and Revenue should follow each other closely. Unless the end user isn’t ultimately paying, you must charge from day one. You should have already asked the pricing question during Customer Discovery and acquired a few paying customers from Customer Validation. Now it’s time to test if that pricing model will scale.
Referrals, while probably the most critical channel for CloudFire adoption, will take a back seat to Activation and Retention. People don’t refer products that don’t work or solve problems. You have to be remark-able first.
Parent Conversion Dashboard
Here’s my conversion dashboard for the Parent segment:
Note: Conversion dashboards let you visualize the entire conversion process, but I am only going to *initially* focus on Activation, Retention, and Revenue.
The values on the right are just that — those that help quantify each event relative to the one real transactional revenue event. They will come in handy when I start optimizing a particular metric and want to measure the effectiveness of different approaches.
How I’m measuring Acquisition
This is the easiest metric to measure. I’m looking for unique website visitors that didn’t bounce, spent enough time on the site, and made it to the Pricing page. Most analytics tools can provide this out-of-the-box. I was using Google Analytics, but more recently have been using KISSmetrics.
While measuring overall Acquisition is easy, optimizing it is quite another story. Measuring the effectiveness across all campaigns in Google Analytics is a lot of work because the relevant data is spread across multiple screens. I am not at this stage yet, but I have come across two new services in private beta (KISSmetrics is one of them) that show a lot of promise here. KISSmetrics fundamentally uses people versus page views as the basic unit, which makes funnel and conversions visualization a snap.
How I’m measuring Activation
I define Activation as the steps: Sign up -> Download -> Create First Gallery. Since the Activation steps start on the product website and end in the downloaded application, I had already built my custom funnel analysis tool before discovering Mixpanel.
Mixpanel is real-time analytics for your application. You can use it to track any application event using Javascript/PHP, which can then be used for funnel analysis, retention, and macro-level A/B testing. So far, I have found it to be very useful and was able to replicate my homegrown funnel report in a day. The presentation of the reports could be better, though. I still use my homegrown report for other things but am increasingly depending on Mixpanel.
I have only started using KISSmetrics, and I think it might also be able to handle custom events. If it does, it would be interesting to compare the two. I like the 1-page KISSmetrics reports and would prefer a single analytics tool, if possible.
How I’m measuring Retention
For now, I’m using a custom database report to find users that have shared at least one thing per month for the last three months.
Mixpanel seems to be designed for measuring user retention around a particular action. I send an event each time an album or movie is shared, and Mixpanel tracks it and automatically generates cohort reports. The data is also available through an API which I’ll probably need for my specific metric.
Another approach to measuring Retention is to tie it to Revenue. Since CloudFire has a 14-day trial, it’s fairly easy to measure retention that way. Beyond the 14-days, CloudFire is priced annually, and I am considering A/B testing a monthly pricing plan on Nivi’s suggestion. Another approach might be to offer a money-back guarantee. The idea is to give users an easy way out. If they stay, you know it’s because of the product. There is no better measurement of Retention than a recurring payment.
I haven’t had much experience running surveys, but it might also be interesting to conduct a survey-based report using Sean Ellis’ 40% test.
How I’m measuring Referral
I am not currently measuring this metric but am using Mixpanel to track an event each time a user uses the “Tell a Friend” feature and lets Mixpanel track it.
How I’m measuring Revenue
I am using a homegrown report to measure users that convert after the 14-day free trial.
Visitor Conversion Dashboard
Here’s my conversion dashboard for the Visitor segment:
At this stage, I’ve defined but not implemented this dashboard. Since Parents come first in the sharing process, I’ve decided to implement that.
What’s next?
To reiterate, my sole focus is working toward Product/Market Fit. In Dave McClure’s model, that equates to optimizing for Activation and Retention. I’m going to measure but ignore the rest.
I am also a little weary of falling into the trap of just relying on numbers. Numbers tell you what is happening but not why. Yes, you can build yet more numbers to get to the bottom of an issue, but almost every time, it’s easier to reach out and talk to a user.
Next time, I should have some actual numbers from my Parent Conversion dashboard to share with you and start diagnosing the health of Activation and Retention.
Update: If you liked this content, consider checking out my book: Running Lean, which dedicates 50 pages alone on this topic.
You can learn more here: Get Running Lean.