Hey friend 👋 After a few months of renovating our new apartment and other things keeping me busy, I'm back with some SaaS reads again – this time on a biweekly basis.

Without further ado – here are some of my best reads from the past couple of weeks:

1. False discovery rates in A/B tests
2. A new way to think about Product-Market Fit
3. Top seven self-serve onboarding mistakes
4. How ConvertKit scaled with "non-scalable" direct sales

​

#1 False discovery rates in A/B tests

Are your A/B tests producing meaningful data? Chances are, they're not. A study looked at 2,766 experiments run on the online A/B testing platform Optimizely that fit specific criteria to estimate the False Discovery Rate (FDR) and found out that more often than not, the variants chosen for A/B tests are not statistically significant. (Hands up if you're surprised.)

(This is a condensed summary for us practitioners, but anyone interested in statistical explanations can go ahead and check the study as we're not going into details here.)

What is False Discovery Rate (FDR) – and why it matters?

• Imagine the set of statistically significant effects found within an experimental context, e.g., all the A/B you've run on your site. FDR is the percentage of all the significant effects where the alternative hypothesis that you accepted was actually equal to the null hypothesis (π_0)
• Out of all the tests examined in the study, the null hypothesis was true and equal to the variant in about 70% of all experiments. In plain English, this means that people seem to be really bad at picking potential variants, often picking duds to test – whether it's because they're risk-averse or maybe testing 41 shades of blue.

What can you do to improve FDR?

• Increasing sample size (and thus power) can help a bit.
• Replicating test to verify that an effect "is real" (often unattainable due to time limitations)
• Learn to identify variants that are less likely to be true nulls and more likely to be larger (meaning that they would have statistical significance)
• This might seem counterintuitive, but one option is not to bother with significance testing and simply pick the best variant. In a straight vanilla A/B test, your FDR would be the same as π_0, but as you increase the number of variants, the authors found that the results start approaching (but never beating) the traditional A/B and two-step A/B tests.
• Sledgehammer testing. Start by smacking subjects with a ridiculously strong treatment to prove that any effect exists, to begin with. When you come out strong, you have less doubts about whether subjects noticed, understood, experienced the treatment correctly. Only once you're sure that something exists do you start targeting more complex and subtle setups. If a user deliberately doesn't click the giant blinking button taking up 50% of their screen, they'll likely never click it at a normal size. If no one understands the design during a user testing session, no few will understand it in the A/B test.

​Read article​

​

​

#2 A new way to think about Product-Market Fit

Chances are your team has a flawed understanding of what Product-Market Fit really is. The reason people get PMF wrong is an oversimplification: we reduce reality to something that is no longer useful. The result is two common and dangerous PMF misconceptions:

• Misconception 1: PMF is binary. It's easy to think you either have it and everything is incredible, or you don't, and absolutely nothing is working. In the real world, it is possible to have an initial product-market fit and strengthen it into full PMF through iteration.
• Misconception 2: PMF is a spectrum. This misconception is even more dangerous because it blinds you from a very common scenario where something is fundamentally wrong with your business. This is how teams end up iterating for years without making actual progress.

How to really think about PMF?

The good news is that both misconceptions have some truth to them. To understand how PMF behaves you'll need to imagine a landscape. This landscape has three areas, and the single thing you must remember is that you need to behave very differently in each area:

• PMF Desert: This is a big hot desert with zero PMF. Given enough time, this desert will kill you. Take extreme measures to get out of here as soon as you can
• PMF Mountain: This is a very big mountain. If you're anywhere around it, then you have some PMF. You're onto something. Take steps towards the peak, and don't expect it to be easy.
• PMF Mountain Peak: Being here means that you have full-blown PMF. Time to scale and build a real company

How to figure out where you are – and where to go from there?

• PMF Desert: It's easy to know if you're in the desert: nothing is working consistently, and everything is harder than it should be. Acquisition and retention is hard and often a stroke of luck. User behavior is all across the board with no clear patterns. The team can't articulate the value clearly. Accepting it is truly the hardest part, so congrats. Next, you need to stop refining and building features that don't matter. Then go back to the fundamentals: Is the problem I'm solving real and painful? Whose hair is on fire because of this problem? Is there really an opportunity to build a business here? How am I solving the problem in a meaningfully better way?
• PMF Mountain: On the mountain, there's some spark of consistency and clarity, something is starting to click, but it might be small. It can take different forms: You can clearly articulate the problem and find people who beg you to solve it. You are getting paid and have a way to find the next customer. However small it is, your solution makes a difference. One common mistake when you get to the mountain is to become too conservative. Resist the urge to stop iterating quickly until you're on the mountain top.
• PMF Mountain Peak: As you reach the peak, you will experience a sense of clarity and confidence. You've cracked it. You know precisely what problem you're solving and who has this problem; you have a process to find these people and get them on board; you've got users who receive tangible value and are coming for more. This also means you need a new "operating system". Instead of simply trying to find the PMF, you're now focusing on building, scaling, and leading a company – hiring, culture, sales, scale, fundraising, etc.

​Read article​

​

​

#3 Top seven self-serve onboarding mistakes

In self-serve onboarding, you have an extremely narrow window to impress your users. Why should you care? Well...

• 40-60%+ of new users never return to the product on a second day 😢
• If a user does see value on their first day, they're far more likely to become a paid customer and share your product with others.
• Improvements to new user activation tend to improve all downstream KPIs: free-to-paid conversion, retention, net expansion, CAC payback, NPS, etc. If you can nail it, you'll be on the path to a powerful business model.

The top 7 self-serve onboarding mistakes

1. Your product is too confusing without sales or success helping out. Many companies forget PLG is a mindset shift – not just a pricing model. It means designing for the end user and helping those users see value as quickly as possible. A new user is missing so much context that you might take for granted, even the basics like who your product is for and what it does. Make sure to seek this feedback intentionally.
2. You have too much of a blank slate. If everything's empty in your product (e.g., no instructions, dummy data, or pre-built templates), it's hard for users to visualize the promised land of what's possible. Instead, you should show what you want new users to accomplish on their first day, get them excited to do it, and expose additional capabilities later as they become relevant to users.
3. You don't explain "what's in it for me." You don't have a sales rep to explain why you built a feature or how that feature helps the user, so your product needs to sell itself. Use short, readable, and value-centric copy to build trust with your users and encourage them to finish the setup process. Example: Mailchimp asks users to provide a physical address (pretty invasive, huh?). But then they explain why they're asking - it's to comply with anti-spam laws, not to send you even more spam.
4. You rely too much on in-product tours. In-product tours are a band-aid and not a panacea. Your product needs to be intuitive on its own. Carve out product and engineering resources to optimize your native onboarding experience.
5. Your in-product guidance goes away too fast. You've probably seen this before: you got fed up with a long product tour and clicked out of it in haste. Then you regretted the decision because you were left entirely on your own and didn't know what to do. Make sure there are still resources to guide users on what to do after the product tour has ended. Checklists and progress bars can be useful tools here.
6. You make it too easy to sign up. Ruthless "conversion optimization" might help you generate more signups – only for you to realize later that the extra signups never translate into more customers or revenue. Solve this problem by providing more nurturing, context, and community outside of your product to turn low-intent visitors into high-intent users and/or redesign your onboarding so that you have a path for low-intent users who aren't yet ready to activate, let alone purchase your product.
7. You bombard new users with too many (and often confusing messages) that aren't helping them at all. Document every touchpoint (in-product chat, sales rep outreach, company announcements, marketing emails, etc.) that a new user receives in their first two weeks and then map those touchpoints on a timeline. Use those learnings to streamline the user journey and kill touchpoints that aren't additive. Bonus points if you can shift towards contextual, trigger-based touchpoints personalized to a given user.
8. Bonus: You aren't personalized. Users come to your product with different contexts, jobs they're trying to do, and levels of familiarity with related software tools. So why should they all go through the same onboarding path? Adding qualifying questions to your signup/onboarding flow will help you offer them the best possible experience.

​Read article​

​

​

#4 How ConvertKit scaled with "non-scalable" direct sales

In 2015, ConvertKit faced some initial traction but then stalled out at $2k/mo. The founder Nathan Barry tried everything from content marketing to partnerships but failed. Finally, he discovered the key: direct sales – and created a process to help him scale it:

• Choose a niche, make a list of people, and reach out to ask about their frustrations. Nathan first narrowed down to "email marketing for professional bloggers". Subscribe to their newsletters, follow them on Twitter, & interact with their content so when you reach out, they will recognize you. When reaching out, rather than asking for them to buy or demo my product, Nathan would ask what was frustrating them about MailChimp, Aweber, or whatever they were using. According to responses, these bloggers were frustrated with not being able to segment their list, create opt-ins to give away a free guide or incentive, creating automated email courses, & more.
• Get on a call and remove their biggest objections. Then Nathan would offer a call to give them some suggestions to improve things in their current tool and to show them what he had built with ConvertKit. People would typically like what they see but would say it's too much work to switch providers. So Nathan started helping people migrate for free and would manually move everything over to ConvertKit, one customer at a time.
• Build momentum by creating an echo chamber and ask for referrals. Each new customer they landed made getting the next one a tiny bit easier. More feedback, references, and names to drop in future emails. Influential bloggers started using their tool. Once a customer was set up and successful, Nathan asked who else they should talk to, and soon referrals were driving as many calls as cold emails. If it was unsuccessful, he would still follow their content and gradually interact. People would often come back a few months later when they hit a pain point, or the timing was better.

In July 2016 – 6 months into direct sales – they hit $15k MRR. By the end of the year, they hit $98k MRR. The momentum from incredibly unscalable work had turned into very scaleable growth. Today, their revenue has grown to $29 million per year.

​Read Twitter thread​

​

​That's it for this week. I'd love to know what was your favorite read this week!

​

Cheers, Anna

​

P.s. Did someone forward this email to you? You can subscribe to weekly updates here.
​
P.p.s. Did you know you can now find all previous summaries here?

​

P.p.p.s. Connect with me on LinkedIn and Twitter! (Warning: I mainly post memes and rants about marketing, but if that's your jam, let's be friends. I'm also Head of Growth at an EdTech startup and a freelance consultant for B2B SaaS startups, but honestly, I'm more into shitposting than personal branding.)