Wave 3

Failing Well : Avoiding Vanity Validation in your Lean Startup

After attending 3-4 Lean Startup Machine events and working with internal teams to understand and apply the Customer Development framework, I’ve begun to notice a pattern...

There is a trap hidden inside CustDev. The same passion and positive psychology required to succeed in the face of uncertainty is hindering practioners from learning fast enough to survive.

How can we take our passion, our vision, a couple “wild ass guesses”, and produce meaningful, validated learning?

Stand back... we are going to try science.

The question of how to learn as an organization and how to DEMONSTRATE learning has been explored by philosophers of science and by business theorists for years. What can the Lean Startup Community learn about creating scientifically valid experiments that create actionable knowledge?

This session will explore the theoretical relationships between scientific hypotheses and scientific experiments. We will explore the theory behind using effective questions to invalidate assumptions and following through with the learning process after an error in the hypothesis is detected. We’ll learn how to detect hypothesis and questions that can’t be falsified and therefore only lead to vanity validation not learning.

We’ll learn how to fail well and fail faster by keeping our passion focused on the vision and our dispassionate logic focused on the assumptions.

Session Leader(s) Background

Joshua (Jabe) Bloom, Chief Technology Officer, The Library Corporation is an award-winning international speaker on Lean Product development (LESS 2011), an active member of the Kanban and Lean Startup community, a CPO/CSM and a collaborator with many exploring how to make work more rewarding for everyone involved.

As a creative technologist who specializes in managing innovation, Jabe has been leading teams and companies, developing software and products, for nearly 20 years. He is interested in strange problems.

As an unrepentant philosophy major, when he has the time, Jabe writes about Lean, Agile and Complexity theory using big words. He can be found on twitter @cyetain

Session History (optional)

The information in this course has been discussed extensively within The Library Corporation, but I have never publicly presented it.

Questions Answered

How to design experiments that effectively invalidate assumptions.

Avoiding vanity validation.

Understanding different types of questions both open and closed.

What is the difference between deduction, induction and abduction? Why does it matter?

What is Single and Double Loop learning. How does Model 1 and Model 2 behavior impact a startup’s ability to be a Learning Organization

Why facts are dangerous, but continually verified tests are invaluable

Popper, Agryris and Validated Learning: Philosophy of Science, Actionable Knowledge, Vanity Validation and Customer Development

Session Basics

Session Leader Name: Jabe Bloom

Duration (60, 90)

Audience Level — Experienced - Expert : The Class will explore how theory can inform the practice of specific Lean Startup concepts. Due to the amount of material, novice level experience with Lean Startup terminology will be assumed (we won’t be starting from what is Lean Startup and what is a Pivot).

Session Style — Presentation


Submitted by

Session Leader Name(s): Jabe Bloom

Proposed Duration: between 60 and 90 minutes

Stage: Accepted

Feedback Score

14 votes

Idea Details

Vote Activity

  1. Upvoted
  2. Upvoted
  3. Upvoted
  4. Upvoted
  5. Downvoted
  6. Upvoted
  7. Upvoted
  8. Upvoted
  9. Upvoted
  10. Upvoted
  11. Upvoted
  12. Upvoted
  13. Upvoted
  14. Upvoted
  15. Upvoted
  16. Upvoted

Similar Ideas [ 3 ]


  1. Status Changed from Active to Accepted
  2. The idea was posted


  1. Comment
    Carlton Nettleton
    ( Moderator )

    I give this a 7 out of 10 (10 being perfect)

    I like that you are discussing ideas around validating experiments. I like that you are presenting something for experts. I like that you explain the questions that you are going to answer. I like that your background is unique and different from most people I meet at Agile conferences.

    What would make it perfect is a little better understanding of what will be the raw material for the session - your experiences or some idea from the audience? Also, if you could make the description much more succinct that would help - I found myself getting bored while reading your rambling preamble.

    Comments on this comment

    1. Comment
      Jabe Bloom ( Idea Submitter )

      I've edited my description to make it more concise and focused. I hope I've made it pop a bit more.

      Hopefully it is now more clear that my experience as a Lean Startup Machine mentor will be the raw material for the talk.

      Please let me know if I have left anything unclear or if I need to add more information.

    2. Comment
      Carlton Nettleton
      ( Moderator )

      I give this an 8 out of 10

      I like this much cleaner description. I like the learning objectives. I like that we are going to hear something that has not been shared publicly before.

      What would make this perfect is a time limit of 60 minutes.

  2. Comment
    ( Moderator )

    The topic and content sound great, but how will participants learn? If it's just lecture, I don't know if they will be able to learn what sounds like fairly complex material. Are there games or exercises to help participants get the concept? I'm with Carlton, my eyes glazed over a little.

    Comments on this comment

    1. Comment
      Jabe Bloom ( Idea Submitter )


      I am aware of one other speaker who has done some work integrating Argyris with Lean Startup. We've been talking about this presentation. I intend to speak with him at LSSC in Boston about his efforts to create a more experiential based exploration of this part of my presentation.

      That said, this talk is based on over a year of research into this problem, which has entailed thousands of pages of reading and hundreds of hours of work with my employees and Lean Startup Machine attendees to understand and clarify this problem. I am hoping that I can use an hour to summarize my findings well enough to spark a larger conversation that can benefit Lean, Lean Startup and Agile practitioners.

      Now that I have made your eyes glaze over yet again by mentioning 1,000 of pages of obscure technical reading... I promise to do my best to be an engaging speaker and answer as many questions as I can before during and after my presentation. I’ve recently had some luck as a speaker, winning Best Lean Product Development track award at LESS 2011. Another review of a recent public speaking event “the best talk we’ve had here in our 6 year history of inviting guest speakers.”

      I always have more to learn, hoping to have a chance to in SF.

    2. Comment

      I'm not afraid of a little lecture, particularly if it is peppered with first-person stories and examples. In our drive to make things interesting, we have pushed the pendulum all the way over to participatory learning. Which is great, but let's not forget the power of good storytelling.

  3. Comment
    Jim Benson


    I agree that current lean startup methodology leads to underdeveloped conclusions and gun-jumping. Looking forward to hearing about how you've successfully applied both lean startup techniques and sound research techniques at TLC.

    Comments on this comment

    1. Comment
      Jabe Bloom ( Idea Submitter )

      Lean Startup tends to exacerbate some of the behaviors that drive the theories of Defensive Logic (Model 1) that Argyris describes. It is difficult to engage in conversations that expose our ideas to real criticism, almost everyone unconsciously engages in behaviors designed to prevent their ideas from being challenged. I'm interested in a exploring ways to externalize ideas and make them safer to explore... In Kanban "explicit policies" play this role, as well as concepts like "No one ever yells at a sticky"

      At TLC we continue to explore this problem and work to reveal issues quickly so we can react (pivot) faster by engaging in behavior that creates actionable knowledge.

  4. Comment
    Ted M. Young

    I like how this session will use science in the context of learning, something that we know far too little about and something that many companies just assume is happening. I'd love to learn more about running the _right_ experiments, not just random experiments.

    Comments on this comment

    1. Comment
      Jabe Bloom ( Idea Submitter )

      I've been working to describe the importance of the "right" kind of experiments for the right context. Theories require ever more rigorous and challenging experiments as the cost of failure increases. I hope to be able to explore these issues with a group of practitioners who have also experienced these challenges.

  5. Comment
    Chris Matts

    Hi Jabe

    Nice subject. I think you have the content nailed with a mixture of theory and experience report.

    Some modifications I would make to the submission:-

    1. An hour / hour and a half is a long time for a chalk and talk, especially at a high energy conference. You should consider a few exercises to bring up the audiences energy level and involvement. You may want to do one big exercise and a couple of two/three minute exercises.

    2. Agile peeps love hands on learning. Think about how you could create a game / exercise around the key point of learning.

    3. Consider adding a time-line to your submission. e.g.

    15 minute intro to theory

    15 minute game

    15 minutes on experience report

    15 minutes facilitated discussion on results.

    4. The biggest mistake submitters make is to withhold because they do not want to spoil the surprise. You need to add the results of your study to the submission.

    All the best, looks like a cracking submission.


    Comments on this comment

    1. Comment
      Ted M. Young

      I disagree -- an hour is no time at all for a talk, especially if it's filled with great stories and nuggets of truth and interesting questions from the audience. While I enjoy hands-on learning, I also crave good stories and information given in a talk format. Also, I've seen more than my fair share of "experiential learning" failures, so let's not assume that just because a proposal has a hands-on learning aspect, that it's guaranteed to be great.

Add your comment