“Startups that succeed are those that manage to iterate enough times before running out of resources”
-Eric Ries
Even though an iteration (in Lean Startup) is defined as a Build/Measure/Learn experiment, the iteration loop can be applied equally well to optimization and pivot experiments which have often led to Lean Startups being mischaracterized as “optimization driven.” Eric wrote a post not too long ago to address this: “Learning is better than optimization,” but I feel he didn’t go far enough to distinguish between when you should optimize versus pivot.
Unlike Eric, I do feel there is a place for optimization experiments (like the infamous Google 41 shades of blue test) — just not before Product/Market fit. Before Product/Market fit, the goal of a startup is to find a repeatable and scalable business model. Unless you are pointed in the right general direction, stepping on the gas here only gets you lost faster. You don’t “really know” if you are pointed in the right direction until Product/Market fit. Once you are, optimization is the right action.
How are Pivot Experiments Different from Optimization Experiments?
Yes, technically, you learn something in both cases, but the goal of pivot experiments is very different from optimization experiments. In a pivot experiment, you attempt to validate parts of a business model hypothesis toward finding a scalable and repeatable business model. In an optimization experiment, you attempt to refine parts of a business model hypothesis toward building a scalable and repeatable business model. The goal of the first is course correction (or a pivot). The goal of the second is efficiency (or scale).
Pivots are about changing direction, while Optimizations are about going faster.
The 3 Facets of Pivots
A pivot experiment is designed to validate parts of a business model hypothesis. I like to formulate my business model hypothesis along three questions: WHAT, WHO, and HOW:
1. WHAT: Problem
What is the problem you are trying to solve?
2. WHO: Customer
Who has the problem?
3. HOW: Product/Market
How do you solve this problem?
How big is this market?
How are you going to reach this customer?
How do you drive demand?
How will you make money?
The first pivot experiments you run are during Customer Discovery when you attempt to validate problem/solution fit with potential customers in the form of problem and product presentations. You use these results to course-correct your initial hypothesis. Beyond Problem/Solution fit, you test various aspects of HOW starting with how well you solve the problem.
4 Rules for Running a Pivot Experiment
Eric recommends focusing on the macro effect of an experiment (such as sign-ups versus button clicks), but this is required for both optimization and pivot experiments. The following additional rules help differentiate pivot experiments:
Rule 1: Maximize Learning
One of the key distinctions between pivot and optimization lies in the expected outcome.
The Principle of Optimum Failure Rate:
Fifty percent failure rate is usually optimum for generating information.
Donald Reinersten — The Principles of Product Development Flow
This means you stand to learn the most when you don’t really know what to expect. If the goal is to maximize learning, you have to pick bold outcomes versus chasing incremental improvements. So, change the unique value proposition rather than changing the color of your call to action button. Rather than experimenting with different prices, experiment with different pricing models.
Rule 2: Focus on Retention
While I agree that focusing on the macro is a good rule of thumb for running any experiment, unless you focus on the right macro metric, it is just as easy to lead yourself astray.
A startup’s life before Product/Market fit is very different from life after Product/Market fit, and so are the key metrics. Of the 5 AARRR startup metrics, Retention is the only metric that most closely tracks Product/Market Fit. Yes, I rank it above Revenue because a customer can buy a subscription, not use it, and forget to cancel. But it’s hard (if not impossible) to fake retention.
While getting paid is the first validation, getting customers to keep coming back is the ultimate validation of building a repeatable and scalable business model.
Rule 3: Minimize Waste
Where things get a little gray is that you need some initial and steady traffic to feed into your conversion funnel since Retention is towards the bottom. You can initially get by with customers you found during Customer Discovery, but you eventually need to start testing customers through other channels like your website.
I like to start experimenting with Unique Value Proposition statements and signup flows, but it’s important to watch out for the point when pivot experiments start looking like optimization experiments. Some judgment is required to know when you have “just enough” traffic to support learning. You eventually reach a point of diminishing returns which is usually a good sign of stopping and focusing further downstream. While squeezing out additional 10–20% improvements on landing page conversions or a new signup flow might sound like progress, it’s fairly easy to get sucked into the “optimization” trap, which is highly addicting and a sure way to burn both time and money.
If you don’t yet have a product customers want, leading 100 versus 10 horses to the water won’t make them drink any more.
Rule 4: Validate Qualitatively, Verify Quantitatively
Before Product/Market fit, you typically do not have enough traffic to afford to wait for statistically significant results, so I prefer to validate all my experiments in 2 phases. The first pass is qualitative, which requires talking to customers. Picking bold outcomes works quite well with qualitative testing as they typically don’t require large sample sizes for a strong initial signal. The trick is building a validated learning loop into your experiments. I routinely use a combination of customer feature validation, usability tests, and product presentations to do this.
I kill the variant immediately if I don’t get a strong initial signal. Otherwise, I set up a cohort to analyze quantitative data over time. Again the key metric I pay attention to is Retention. My products use a trial period, so retention naturally converts to revenue.
Pivot Before Product/Market fit, Optimize After
I wrote this post because I’ve experienced firsthand just how easy it is to unknowingly get caught up in the optimization trap even when you are supposedly practicing “Lean Startup” techniques. Until recently, I used to view pivots as big course corrections you make at some point X in the future when things don’t work, or you stumble into an accidental discovery — like how PayPal, Flickr, or Starbucks came to be. That’s because those were usually the best examples of pivots cited.
I now view a pivot as any iteration (big or small) toward validating a business model hypothesis. The size is usually relative to when the pivot is applied. For instance, pivoting on customer segments is easier during Customer Discovery than Customer Validation. I’ll share a case study on how (and why) I’ve been pivoting on two customer segments in parallel next time.
Words matter. The word iterate is too general, and the word optimize too dangerous (before Product/Market fit). So as a first step, I’d recommend removing it from your vocabulary till after Product/Market fit.