Avoiding the Perils of A/B Split Testing

A/B testing is widely used in product development, popularized as a fundamental component of the Lean Startup  framework, and providing a scientific way of validating product and business improvements. The concept is simple… put some customers in the new experience, compare the results against customers that didn’t get the new experience, and better metrics validates the improvement. In reality, this process of validation is very complicated and there is no shortage of hazards leading you to poor outcomes.

Creating Information out of Data is Hard

IMVU had a culture of data-validated decisions from almost day one, and as a result we made it easy for anybody to create their own split test and validate the business results of their efforts. It took minutes to implement the split test and compare oh so many metrics between the cohorts. All employees had access to this system and we tested everything, all the time. A paper released in 2009,  Controlled experiments on the web: survey and practical guide, reinforced that split testing was the undisputed arbiter or truth. We were clearly on the right path. 

While the ability to self-assess progress created a very empowering culture, we were largely ill-equipped to understand the nuances of what the data actually meant. Years later we would start to better understand, we don’t know how much we don’t know.

First Know Why

The first opportunity to make a mistake with split testing is deciding to test in the first place. When creating a split test has a very low barrier, it is easy to err on the side of just testing everything so that you can have the data if you need it. But every test has a lot of hidden costs than come from false-positives, clarification of data, shiny-object distractions, inconsistent customer experiences, and additional opportunities for introducing bugs.

Recognizing that being a split test packrat has a real cost, there should be some requirement for incurring this cost. Are very least, answering the question, “What are the significant changes that will be made as a result of this test?” Additional pre-test work to specify what will be measured, and what results will determine success or failure can also go a long way towards ensuring time spent testing is valuable.

Test Implementation is a Project

IMVU had a great framework to make test implementation a seemingly simple task, with a few lines of code of creating a branch for the test experience, and leaving the current experience as the control. Again, this made creating tests seem deceptively easy, and left openings for measuring the wrong thing.

Often a split test is a cross-functional effort, with an engineer handling the implementation and the customer being any combination of a product manager, acquisition team, marketing representative, revenue officer, or generally interested party. In some cases, the interpretation of test data is done by another person altogether. Correctly understanding what the internal customer wants to know, capturing the right data, and converting that data into information ends up with many points of communication that must be accurate to deliver a valid test.

For example, the acquisition team wants to test a new landing page, simply reordering the registration fields because they think it will improve the registration completion rate. The engineer realizing this is a no-brainer takes the 15 minutes before lunch to create the quick test, two paths and the test is running. However, the registration page has both manual registration and sign in with a social network account, so the test is including a lot of users that are social logins, irrelevant to the registration fields. This subtle nuance means that the impact of the registration field changes will likely be lost as the irrelevant data acts as a damper. What the customer wanted to know isn’t what the test is answering, and it’s likely that nobody on the project knows there is an error.

The ease of creating a split test should not be conflated with delivering quality results from a test. Doing it right is a project and requires investment of resources consistent with any other project.

WTF Do These Results Actually Mean?

Assuming you were diligent in your experiment design, you captured all of the relevant data, and you avoided some of the common errors of A/B testing, you now need to make sense of the data. In the best cases, you’re looking at something like “the registration landing page increased conversions from 1.83% to 2.01%”, in the worst cases you find something like “customers are engaging with messaging feature 17% longer… but their lifetime value has dropped by 4%”, and now there is work to put together a narrative that explains the perplexing results.

In 2012 I read a paper, Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained, and I had what I like to call an, “oh shit” moment. Highly controlled experiments, run by companies with world-class, dedicated analytics teams were getting perplexing results that required substantial research to understand what was actually happening. What chance did we have of getting this right when we are running 15+ experiments a week with training consisting of a one page internal wiki version of, “A/B Testing for Dummies”?

The tl;dr summary of the paper, without deep consideration for the “why” behind the change in metrics, positive results may be antithetical to what you are actually trying to achieve.

The up-front work to limit the scope of the experiment and how it will be measured / interpreted can help, assuming you have the self control to ignore the data outside of scope. Often these perplexing results require follow-up experiments to better isolate cause and effect. I also highly recommend talking to customers – often qualitative insights from hearing their experiences can often help make sense of what the quantitative results were hiding.

You’re Biased. No, Really, You Are

I’m sure there are a lot of great reasons we humans are wired to think the way we do, and this wiring probably served us very well in many situations. However, humans also come standard with cognitive biases, built-in tendencies to make irrational decisions. Unfortunately, putting a bunch of effort into building something and then getting a giant pile of metrics is a perfect enabler for a cognitive biases and craptastic decisions.

While numerous biases are working against you, with a buffet of metrics one of the most common is the Texas sharpshooter fallacy, in which the all of the test metrics that are improvements over the control metrics are used to demonstrate the success of the test. With a 95% confidence rate, 1 out of 20 metrics tracked are expected to show a false positive improvement, so even an A/A test (two separate cohorts with identical experiences) would likely show “improvements”. Before we eliminated the practice of metric-sniping at IMVU, it wasn’t uncommon to hear somebody say something like, “my pet project to streamline registration didn’t change registration, but it does deliver a 5% improvement in [the completely unrelated] customer lifetime value, so we should keep it.”

There are process controls that can help reduce the potential impact of various biases, in particular around defining and constraining each test. However, being aware of these biases and encouraging a culture consistent with the dialectical method can help make better product decisions, even beyond interpreting test results.

Talk to Your Customers!

One of the biggest risks that come from over-reliance on split testing is seeing it as a more convenient method of getting customer feedback. Why spend 30 minutes on the phone with one customer when you can simply measure the actual actions of thousands of customers?

Looking at data and sending surveys may seem like an efficient use of time, but that highly structured approach is unlikely to surface critical customer insights. Metrics and surveys will often answer the “what”, but almost always miss the “why”, the most critical driver of valuable insights. There is no substitute for talking to your customers.

In the words of Steve Blank, “Get Out of the Building.”

 

I’m interested in hearing other stories where split testing has made an impact, either positive or negative. Please share a comment if you have one!

Congratulations Successful Entrepreneur: You’re Fired

Most startup entrepreneurs understand that the odds of success are not in their favor… only about 1 in 10 startups will survive. Of course, most startup entrepreneurs don’t believe they fall into the 9 out of 10… a healthy amount of self delusion is required to go down down the startup path in the first place. But there is that 1 in 10 that does make it… and, if you are lucky enough to be the CEO that delivers that success story, the odds are you’ll be fired.

Before explaining why being fired is the most likely outcome for a startup CEO, it’s necessary to explain the startup journey…

Your Mission as a Startup

Investment-backed startups are created to discover scalable businesses, usually by inventing a new product or service that can become a large business, or by creating substantial efficiencies that take customers away from an existing large business. There is no clear, obvious path to doing either of these, otherwise success would be the expectation, not the exception. So success requires reasonable self delusion that you will succeed, as well as experimentation / rapid iteration necessary to adjust to the challenges of discovering the successful business. In practice, this can often manifest itself as the CEO coming in with the crazy idea of the day saying, “let’s try this… can we ship it by tonight?” If you like the excitement that comes from working through challenges with great uncertainty, this process can be a rewarding experience.

Through this process of discovery, a few things can happen. If the company runs out of money before a scalable business is discovered, most likely everybody loses their job, although it is possible that the board still believes in the company but sees execution or leadership as the problem, fires the CEO, and then puts in new money to support a new leader. From the CEO perspective all of these paths lead to the same place… you’re effectively fired.

But wait, Brett… those are failure scenarios… I’m that 1 in 10! I discovered product market fit! I delivered on my mission! I found the scalable business!

You’re probably fired anyway.

It’s Not Us, It’s You

You’ve done something truly amazing… you’ve lead people down a crazy path, likely engaged in some mixture of know-how, magic, luck, skill, and insanity, and came out the other side with a scalable business. It takes a particular type of person to do that successfully.

Unfortunately, that particular type of person is usually the exact opposite of the particular type of person you want growing a scalable business. Growing a scalable business is more about efficiencies and optimization, much less about discovery. That same crazy idea of the day behavior that miraculously lead to discovering the scalable business is exactly what derails the consistency a company’s organizations need, and what customers will expect. As the organization grows, process and management becomes necessary to handle the challenges that come with simply trying to get hundreds of people to work towards the same goal. The needs of operating a scalable business probably contributed to the CEO quitting their previous job and creating the startup in the first place.

The board has a responsibility to driving shareholder value (including their own investment) and, seeing how maximizing the value of the business now requires a different expertise, likely determines that it’s time to get somebody best for that job. It’s possible that the startup CEO has the rare set of skills to transition, or it’s possible that the board will bring in supporting executives to help. In these cases the same end result is usually just delayed.

Of course, getting fired doesn’t happen every time… you can look at examples like Mark Zuckerberg, Drew Houston, Jeff Bezos, and Steve Jobs and, using that healthy amount of self delusion, say “I’ll be like them” (forgetting, of course, the first run of Steve Jobs at Apple). But if you look at all of the companies in the valley that scaled successfully, you’ll find most had the founding CEO “step aside”.

Yikes! How Do I Prevent This?

Your gut response as a startup entrepreneur is likely something like, “I’m going to make sure that doesn’t happen to me.” However, I encourage looking at it a different way… this happens, you’re probably going to be replaced, and that’s probably okay. It’s better to prepare for the possibility rather than assume it can’t happen. You may find being replaced is actually be the desired outcome if you prefer building new things rather than optimizing existing ones.

The most reliable way to avoid being replaced is by not giving the board (or anybody else) the power to replace you. In practice this is usually only possible if you don’t take outside investment… venture capital investors will usually take board seats and almost always retain the ability to replace the CEO. The tradeoff you make for getting extra cash to accelerate your progress comes with the price of forfeiting some control.

Assuming you’re taking investment, the best path is likely making accommodations for a transition as part of that investment. Address things like an ongoing role post-handoff (operational and board), vesting of stock, participation in success rewards, and your treatment for liquidity events (acquisition, IPO, secondary offerings). Also account for variations to the plan… while you may want to maintain a significant operating role after a transition, it may be determined that the new CEO can’t be successful while employees still look to their founding CEO hero for direction.

Finally, if you do get to the point where you are being fired after successfully delivering on your mission, make sure you recognize your truly amazing accomplishments… you knowingly engaged in a difficult challenge, with all odds against you, and you were a success. Many people, employees and customers, will be better off because of what you built.

Congratulations.

 

This posting was greatly inspired by over 20 years of stories from many friends that have been founding CEOs, and by Steve Blank’s great presentation, Why Accountants Don’t Run Startups.

 

Have you been a startup CEO and been through this journey? I’d love to hear your story! Please leave a comment.

Joining Social Starts as a Venture Partner

My short-lived backpacking career is in jeopardy… I’m a Venture Partner at Social Starts.

Why a Venture Partner Role

Most recently I spent several years growing a company from startup to millions of customers. In each role of the company, from technical executive to CEO, I needed to spend time deeply understanding the technology and markets related to the business (social, expressive communication, VR, avatars, communities, virtual goods, scalability, digital currency, and virtual economies). While being able to get a deep understanding of subject matters was great, it left little time to explore the breadth of ideas powering innovation, and I missed that.

So, one of the ways I’ve spent my down time over the last few months is getting exposure to a wide range of companies doing things I’ve never done before. In addition to some advisory work for startups, I went to a few sources of amazing entrepreneurs with great ideas. Steve Blank generously invited me to sit in on his Lean Launchpad class at UC Berkeley’s Haas School of Business, where entrepreneurs formed and iterated businesses around IoT, energy management, and medical devices. I also spent some time at Obvious Ventures getting exposure to some really impressive companies in areas ranging from consumer packaged goods to gene therapy and wellness.

I found I really enjoyed the exposure to new companies, especially those outside of my fields of expertise… seeing how people are applying new technology towards opportunities drove my natural curiosity to research topics that were new to me.

I also found satisfaction in my advisory work, helping startups by sharing what I’ve learned, both from my successes, and my failures. Surprisingly, many companies deal with the same patterns of challenges – it’s great to help people get past those so they can move on to newer, more exciting challenges unique to their situation (I wish I could write “eliminating challenges”, but businesses just move from challenge to challenge… you’re fortunate when you’re working on the challenges with possible outcomes ranging from “good” to “great”).

When the opportunity came up to research and help fund all sorts of great startups while providing me with the flexibility to work more deeply with a few companies, I knew this role was right for me.

Why Social Starts

There are many reasons I joined Social Starts, and two factors that most greatly influenced my decision are the team and the deal flow.

In regards to deal flow, Social Starts focuses on very early stage funding, from moment of inception to Series A. The closer you are to the top of the funding funnel, the larger the pool of companies to consider. And, Social Starts considers a lot of companies… it’s been named the top fund under $100M since 2013, the 5th most active early stage fund worldwide of any size in 2015, and the 6th most active early stage fund in US tech in 2016.

For any organization I would join, it’s a requirement that I respect and appreciate the team. In my discussions with partners, I experienced many characteristics I value, including candor, humility, thoughtfulness, and pragmatism. As a bonus, the COO is a friend that is on my short list of “people I would work with at any company, any time”.

Let’s Work Together!

If you’re working at an early stage company in fields like VR / AR, health care technology, AI, work platforms, internet of things software, mobile commerce, blockchain, security, content, wellness, analytics, or human-brain interface, I’d love to hear more about your company.

I also have some availability for advisory / consulting roles for companies that need somebody with executive-level experience successfully scaling startups, helping execute through the challenges that come with growth.

Firing People Respectably

Firing people is perhaps the most unpleasant responsibility that comes with being a manager.  I’ve read many articles on “the right way” to handle firing, but my experience has taught me every case is different, and even following the best advice can result in a challenging interaction.

I’ve created guidelines for myself that feel fair (this is how I want to be fired), and I accepted that firing is unpleasant for everybody involved, so it’s ultimately about making the best out of a shitty situation.

My guidelines come from the perspective of a culture I want to see in a company, not the legal perspective (which tends to err on the side of corporate protection over recognizing the human components).

Guidelines for a Firing Manager

My guiding principle, be respectful, helping the employee retain their dignity, drives these guidelines:

  1. Always remember you’re firing a person, not a resource.  In almost every case being fired is an emotionally painful situation, and being mindful that you are firing a person, with feelings, fears, and personal responsibilities that will be compromised as a result of job loss.  People react unpredictably in emotion-filled situations.  As the firing manager it is important to be respectful through the whole process and be balanced in responses to the other person’s (re)actions.
  2. Don’t get into a detailed discussion.  A common pattern is the person being fired will want to get into the details about the decision to fire.  The firing discussion should be efficient (there is nuance in balancing not being insensitively fast vs. dragging out the pain).  The manager should absolutely provide a high-level explanation, and the next steps (ideally the company has a standard document that explains the issues that will be important to the employee), but the person being fired is very unlikely to actually hear a detailed discussion – they are too emotional to process it.  If a person being fired wants to get into details, I suggest scheduling coffee the following week, giving them enough time to figure out what questions are really important and getting past the initial shock so they can be receptive to the answers.
  3. Never discuss individual details with others.  When a person is fired, other employees frequently want to understand more details.  It can be tempting to want to bring others into the loop or calm an underlying “am I next?” fear they may have by sharing the details, but it is disrespectful to the person being fired (it’s also probably a liability for the company). Instead, have a culture that is transparent about the process (why and how) people are fired, while never discussing an individual’s specific situation.

Reasons for Firing

The reasons for firing an employee generally fall into three categories: performance, role eliminated, and violating the company relationship. Each impact the person being fired, other employees, and possible outcomes differently.

Performance Problems

When an employee is under-performing it is their manager’s responsibility to make that employee successful and, if that fails, fire the employee. An employee’s performance should be a regular discussion with their manager, and missing expectations should be made explicitly clear, along with clarity around the exact expectations and a plan to improve.  If the improvement doesn’t happen, the firing discussion should be more of a final conclusion to the mutual recognition of the problem, with both parties aligned on the shared data.  My rule is, “if the employee was surprised they were fired for performance reasons, this is a failure of their manager”.

Role Change

The role change scenario is one where the company’s requirements or constraints have changed and an employee is no longer appropriate for the role.  I’m including layoffs / downsizing in this category (not being able to pay people is a constraint).  A commonality in these firings is it includes qualified, successful employees.  This is the one firing scenario where additional insights into the decision can be shared with other employees, as the decision is not about an individual (but be sure that the role change is the real reason for the firing, otherwise it will eventually result in distrust from employees).

A role change specific to an individual feels the most personal for the person being fired and can be hardest for other employees to understand. The message of “great for previous role, wrong skills for what the company needs going forward” is easy to say, harder for employees to process, often because a good employee will be leaving, and many employees won’t have the insights into the need for the change (or may simply disagree).  The best analogy I’ve been able to come up with is sports teams, where a great player may be traded to make room for a player that has different skills that make the team better as a whole (as in Moneyball, where trading stars for players that just got on base resulted in a better team).

When a role change is impacting many people (typically driven by financial situations or discontinuing a product / service), explaining to the people impacted can be more comforting than when it is a single role, since the reasons don’t feel as personal (make no mistake, for the people being fired the impact will feel very personal, it just won’t feel like they were individually targeted).

Violating the Company Relationship

Every company has it’s own unique culture, principles, rules, and expectations in the relationship with each employee, and between employees.  I’ll use “don’t steal” as an example, since I this is probably a common deal-breaker even in the most toxic environments.

When there is a violation of the relationship, the employee needs to be fired, otherwise the company is signaling that it isn’t an actual expectation of the relationship, or perhaps worse, that enforcement is selectively applied. In this firing the employee should not be surprised, however an employee willing to violate the relationship in one dimension is likely willing to double down and deny their responsibility in the situation. Unfortunately, this is one of those nobody wins outcomes that, as a manager, you simply need to get thorough it, look for the learning opportunity, and move-on.

A particular challenge in this type of situation is the inability to offer an explanation to other employees, especially if the violation was concealed. Using the stealing example, the company could have liability is disclosing the violation to others, so employees just see somebody fired for no apparent reason.  As recommended in my guidelines above,  if your company has a (trusted) transparent culture around how and why people get fired, many may infer that it was either a performance problem or violation, which a better outcome than the firing feeling random.

Management Failures

Employment is a relationship, and the manager and company have to acknowledge their responsibility in the failed relationship, both in why it failed and the importance of properly handling the failure.

Passing the Buck

If there are other existing opportunities where the employee could be successful at the company, that can provide a solution that is both a win for the employee and the company.  However, since firing is so unpleasant, managers should be challenged to understand if they are diverting the problem to somebody else or do they really feel the employee is best for the opportunity.  Ask the question, “if the employee didn’t work here but was applying for the new opportunity, would you hire them?”  If the answer isn’t a confident, “yes”, the manager is likely passing the problem to somebody else. Another red flag is the creation of a new role for an employee that would otherwise be fired… in almost every case I’ve experienced, this is a manager avoiding a tough (and necessary) decision.

Performance Improvement Plans

Performance Improvement Plans (known as “PIPs” in HR speak) are formal documentation explaining the employee’s performance problem, the expectations, a process to improve and a success evaluation date. On the surface this is all great – issues that should have been discussed in 1:1 meetings. When used as a tool with the intention of making the employee successful, PIPs can be really helpful in providing clear expectations.

The dark side of PIPs is when they are used as an HR cover your ass maneuver, in which the employee’s fate has already been decided but, because of risk or liability, there is a desire fore the company to have ample documentation around the termination. Don’t do this.  When a firing outcome has been determined, fire the employee.  Dragging-out a process or giving false hope is disrespectful, and arguably cruel.

Learning from Failure

A firing may not reflect a failure, it might actually be the best decision for the company and perhaps even for the person being fired.  However, all firings can be an opportunity for the company to learn and improve its processes. If it was a new employee, try to understand how the interview / hiring process could have identified the issue.  With longer-term employees, look for training opportunities (for the employee or management) that could have resulted in a more successful outcome.  Understand when the firing should have happened and what should be done next time. Since firing has such a big impact to both the employee and the company, there is value in continually improving the process to reduce or avoid any firings that could have been saves.

Have you been on either end of the firing process and have suggestions for improving how it gets handled?  Please leave a comment!

In Defense of Not Doing a Startup

Recently (and quite accidentally) I talked an entrepreneur into abandoning his year-old startup.  That wasn’t my intention – we had planned an hour long meeting where I was acting in an advisory role on the product and pitch deck, but the meeting ended up taking over three hours and getting to a very hard question, “why do you want to do this?”

The pivotal moment in our discussion was when it became clear to me that the CEO saw the company as a way of obtaining some short term financial success, and that the startup demands were unlikely to be compatible with what he expressed and being important to him for his personal success.  After walking through the various likely outcomes and startup life expectations, he recognized there were better ways to achieve the personal success he wanted.  The discussion was tough – it’s hard to confront letting go of a dream, especially after sacrificing a year of sweat equity, but as we concluded our discussion he shared that he felt a great sense of relief.

All Hail the Startup

In most of the news and feeds I follow the startup is celebrated, almost so much that it can feel like the act of creating or being a startup is disproportionately more important than the significance of achieving a successful business.  More importantly, the glorification of startup life can lead people to feel discontent with a career path that may actually be far better for delivering personal satisfaction.

Startup Cheerleaders

For the most part we recognize and celebrate successful startups, and with the exception of the startups that have a prominent rise and fall, the majority of startups that exist, struggle and fail are below the radar.  It’s pretty easy to read industry news and think everybody with a startup is on the fast track to a win.

There are also several blogs and speakers working as cheerleaders for those that would take the risks to change the world.  Most respected in this group are serial entrepreneurs that have had the good fortune to have a successful exit from a previous startup, which becomes a shining example that success is possible, and the reason they continue the startup path.  These thought leaders are great for inspiration, but it is also good to have the context that the previously-successful entrepreneur risk is substantially different than the new entrepreneur, both in terms of their chance of success on their next startup, and the likelihood that they are risking a small fraction of their wealth.  If you are new to starting a company, you are likely “all in”.

A Startup is Not a Reliable Path to Wealth

It is easy to look around at the stories of the startup millionaires (or even better, billionaires) and think that starting a company is a good way to ensure a retirement in your twenties.  If the ability to retire is your goal, you’re probably better off working at established companies.  If your goal is to retire with a billion dollars then yes, a startup (or lottery ticket) provides that opportunity, with very slim odds.  Looking at my contact list, almost all of the people that are financially well-off got that way by joining companies well past the startup phase.  However, my very few contacts with obscene amounts of f-you money did obtain it from being very early at companies with large liquidity events.

As an example, one friend easily ranks in the top 5 engineers I’ve encountered in my career and any company would want him as the technical founder.  After four years of doing the startup grind of 60-hour weeks, he ended up with a lot of great experience and a bunch of stock that was worth pennies.  He made the decision to join a very large, more well-established company and forgo the dream of vast riches for continued technical growth and reasonable work-life balance.  What he didn’t understand at the time, but told me later, was how much a big company would pay for good technical talent.  For people of his caliber the total compensation is well over a million dollars a year and as a result he has a reliable path to retirement in his early forties.  His story isn’t the glamorized Silicon Valley success… you won’t see him featured in a PR-driven TechCrunch article, but you might see him enjoying life on a beach with his family.

In contrast, another friend lived the entrepreneurial startup life for 15 years, is well-known and highly regarded in the startup community (yes, you know his name), and most people assume he’s achieved financial success as a result.  Two years ago he had a company that came very close to being favorably acquired, but the acquisition fell through.  The company was later dissolved and over a dinner one evening he expressed the frustration of being in his mid-thirties, driving a 15-year old car and not being able to afford a house.  He has since joined a large Internet company, owns a house and is even able to comfortably support two children and some relatively expensive hobbies.

But wait, Brett… so you have a few friends that did better taking a traditional career path, but I see all of these Silicon Valley 20-something millionaires all over the Interwebs… what makes you think that won’t be me?

It might be you, and I am sincerely happy for anybody that is able to achieve financial success by building a company.  Let’s look at the (extremely general and simplified) math to see expected outcomes…

Some Quick Startup Lottery Math

To make things simple, we’ll assume your startup is just you and a single co-founder, so you each have 50% of a company.  And using this Quora response as reference for founder equity, after completing your Series B, you and your founder share 40%, making your ownership 20%.  The average price of successful liquidity is hard to assess (many sources suffer from survivorship bias, excluding many failed startups) but $30M at Series B would probably be considered generous (there are many examples way higher, far more examples way lower).  A $6M piece of that pie is pretty appealing.  Now we adjust for the risk… again, 90% startup failure rate is generous, especially considering Y Combinator companies representing the hand-picked cream of the crop fail at 93%.  Risk adjusted, you’re now looking at $600K as your upside, so assuming you’re able to go from zero to liquidity in three years, it’s $200K per year (of course this is on top of your well-below-market startup salary).  That doesn’t sound too bad except when you remember, you have a 90% chance of ending-up with only your well-below-market startup salary and your chair.  Again, these are generous assumptions and there are plenty of examples of successful acquisitions in the hundreds of millions where founders received substantially smaller percentages of the purchase price.

Google Director Salaries

And let’s compare that to the alternative, joining a large Silicon Valley company…  It’s fuzzy math, but I’m going to assume that the person that is capable of leading a startup with the generous odds in their favor also has the experience to get a good leadership role at one of the big companies.  According to Glassdoor, the average Director at Google has a base salary of $247K and total compensation of $399K (on a side note, most colleagues I talked to believe the Glassdoor compensation is extremely inaccurate based on first-hand observations, and Directors are frequently making 2-4x what is presented).  Using the same 3-year time frame we assumed would get to liquidity at the startup, the expected outcome is closer to $1.2M.  There are a ton of arguments to adjust these assumptions, but none are going to change the lottery-ticket nature of achieving big liquidity from a startup.

So yeah, the odds of financial success may be working against me, but what about getting to experience the glamorous life of a startup founder out to change the world?

Startups Overshadow your Personal Life

For everybody that asks me what it is like to run a startup, I tell them to read The Struggle, by Ben Horowitz.  I first read The Struggle as part of Ben’s book, The Hard Thing About Hard Things, and I immediately handed the chapter to my wife and said, “you always ask what it’s like to run a company… it’s this!”

A day in the life of a startup founder

A startup is a significant commitment and your business is typically dealing with an environment of extreme uncertainty; startups are either creating something new or believe they can do something better than an established business.  In this environment, and typically with limited resources, working longer and harder provides more opportunities to eliminate the uncertainty.  Assume working nights and weekends are sort of a regular necessity.

And as a leader in a startup, you will always have another challenge or problem driving head-on towards you.  The world owes you nothing, plenty of other companies are fighting hard to take the market that you need to succeed, and the odds of survival are very much not in your favor.  This means business will almost always be imposing on your mind share that you would normally dedicate to things like dinner, sleep, exercising, vacation, relationships, family time, and bathing (assuming you are able to work any of these into your startup life).  Your startup will permeate all aspects of your life.

Finally, there is the emotional toll of a startup.  The successes feel amazing, but they are typically few and far between the challenges and setbacks.  Failure is the expected outcome, and each failure wears you down a little bit, creating uncertainty and making you second guess your capabilities and fitness as a startup leader.  You feel the weight not just for yourself, but for the people that follow you, also making the sacrifices.  And, if you’re unlucky enough to be the CEO, you’re in the lonely position where there is almost nobody you can share your struggles with… you can’t push things down into the company and frequently the board above you is a bad choice as a counselor for issues of personal uncertainty.    

After writing all of this out, I am beginning to understand how I accidentally talked somebody out of their startup.

But… There are Many Great Reasons

I don’t hate startups.  All of my career I have either created startups or joined them at or near founding, and I expect to do it again.  I would hate to feel responsible for taking passionate entrepreneurs and shuffling them into beige and gray office spaces in corporate America.  If you understand the likely financial outcome, and you are in a place where your personal life can sustain the needs of a startup, and you are emotionally prepared for the struggle, there are great reasons to do a startup.

If you are early in your career, the economics and life impact may make more sense.  Your ability to get a job at one of the big name companies may be more difficult, and if you do you’re probably looking at the lower end of the salary spectrum.  The difference between your startup pay may not be that significant in your day-to-day life (especially if you are fine eating rats and ramen).

Startups are also a great way to learn before you earn.  Large companies have established processes and roles that have been optimized for business performance, you are less likely to get a breadth of experience or have an emphasis on innovation.  Startups frequently require everybody to have multiple roles and find innovative solutions to problems.  Learning how to deliver results with limited resources in environments with great uncertainty is a skill that will be valuable for a lifetime.

Working at a startup (even a failed one) can also often allow faster career progression than just joining a big company out of college and following the typical path of advancement.  As an example, a Software Engineer (SWE) hired right out of school at Google would be an L3.  Assuming about 3 years for each promotion, it’s 15 years until Director, L8.  If you’ve proven yourself and established solid startup experience, five years later you might be L6 material (your mileage might vary).

The Best Reason

I believe the best reason for doing a startup in the burning need to build something you are passionate about, and an organization like an established company or non-profit isn’t the best way to create it.  Maybe your passion is a product or maybe it’s a culture, but it keeps you up at night and every time you return to the idea you become more passionate about making it real.  It’s an idea you think it would be so meaningful that you would find the journey of pursuing it to be hugely rewarding.  You’re not thinking about the exit, you’re thinking about the satisfaction that comes from building the thing that drives your passion.

Do it.  Build it.  Make it happen.

 

Feedback, complaints or suggestions?  Please leave a comment!

Interviewed on #ModernAgileShow

I recently had the pleasure of being interviewed by Joshua Kerievsky on the #ModernAgileShow, where we talked about a lot of my experience working at IMVU, ranging from the early days of Continuous Deployment (without all of those fancy automated tests or cluster immune systems) to changes in experiment systems and challenges of building a culture where people feel safe.  I also provide some insights into the sausage making of The Lean Startup.

In the interest of accuracy, my title in the video should be “former CEO of IMVU“.

For more information about Josh’s work to setup agile processes and cultures independent of a specific framework, check out the Modern Agile website.

On a semi-related note, Josh mentioned that the original video of Timothy Fitz presenting on Continuous Deployment at IMVU: Doing the impossible fifty times a day was lost as the result of server corruption…. if anybody happens to have a local copy please let me know – it would be great to restore this historic presentation for the Interwebs!