Posts

Posted on

Scaling Continuous Delivery: Happiness as a Metric

A few days ago Jeff Atwood (Coding Horror) suggested a good measure of a tech company’s health is the time it takes to have a simple change become available to customers:

And while there are numerous metrics that determine the health of a tech company (see Jez Humble’s book, Accelerate, for an amazingly comprehensive overview), Continuous Delivery strongly correlates to successful outcomes.

I support Jeff’s assertion – I witnessed the value created by Continuous Delivery at IMVU, where we pioneered some of the crazy processes that would be followed by more sane practitioners. From day one, IMVU placed value on the speed of product iteration and “designed” build systems accordingly. In 2006, development was done in Windows using Reactor Server to provide the LAMP-ish stack, and the deploy process looked something like this:

 svn-server$ rcp website/* production:/var/www/

If you’re wondering why I omitted the test framework, I didn’t. Code went from a local Windows sandbox environment, to source control, to live on a Linux environment running a version of PHP different than the local sandbox. Fun! While there were numerous problems with this system, iteration velocity was amazing (those 1 word copy changes could ship in less than 5 minutes, and so could full features). This development velocity was a key component enabling IMVU to build a large, successful business in a space where the failed companies outnumber survivors 50 to 1.

And to be clear, time to get a change live to customers doesn’t in itself indicate healthy tech, but a lot of tech health comes from the corresponding systems necessary to make rapid deployment work.

Fast forward a few years to 2008 and I transition from leading the operations team to leading the engineering organization, where the build and deploy systems had matured, with reasonable test coverage, and automated deployment, with automated rollbacks when something unfortunate made it into production. It was pretty cool, even though publicly the process was mostly received with the sentiment, “that will never work , and certainly won’t scale”.

Scaling Problems

One of my first challenges as the new engineering leader was a team unhappy about their ability to get work done because it was taking hours for a commit to become live to customers. It’s astonishing when you think about it – every engineer in the company had come from companies where the commit to live process was typically measured in months, but once they experienced the value of Continuous Delivery, anything more than minutes seemed unbearable.

Digging into the problem, I came to understand that the problem was not slow builds (although that was part of it), the most significant issues were caused from the shared responsibility for build systems, combined with the desire to deliver features to customers, created a tragedy of the commons. When an engineer had a failure in the build system, the optimal solution for that engineer was to fix the problem in place, blocking the build system for anybody else in the queue, which meant the number of commits in the next build increased, which meant the chance of a failure in the that build increased, ad infinitum. The result was pushing to production could be blocked for hours, sometimes most of the work day.

Solving for Happiness

I thought the best solution was to formalize a project, have a clear success outcome, and have a single person with the responsibility for (and therefore authority over) the build / deploy systems. The first problem was determining a clear success criteria… anything time metrics I chose would be somewhat arbitrary, so instead I chose engineering happiness as the success criteria, or more specifically, pushing to production was no longer causing unhappiness. While I generally hate subjective success criteria, there were ways to assess progress through 1:1 conversations and Likert scale surveys. We also had great (highly objective) data around commit to deploy times, so we could see the correlation to the more subjective happiness index.

There was some pretty straightforward work to improve the actual test and deploy speeds, including simple things like adding more hardware and the slightly less simple sorting tests to run by speed (a surprisingly large performance gain), and fixing the slowest of the tests. But some of the most important gains came from the human parts of the deployment system… engineers were required to immediately revert code and fix the issue in their sandbox rather than blocking the build system. This was not a popular policy change as immediately engineers experienced the direct impact from a failed commit, but didn’t immediately see any gains to the overall system.  But after a few weeks the improvements were clear in the average commit to deploy time. And giving credit where it is due, Eric Prestemon was the “Buildbot Sheriff” that identified so many of the opportunities for improvement and delivered the results… many people helped, but Eric had the burden of hearing a lot of critical feedback about unpopular policy changes (eventually outweighed by the praise for the results he produced).

Eventually the build system frustration ceased being a common topic in 1:1 meetings, and it faded away as a meaningful problem in engineering surveys. 12 minutes. When the commit to live time is 12 minutes, this system is operating well. That became the new value for alerting – under 12 minutes, all is good, after that we need to actively drive improvements. In practice, deploy time was usually around 11 minutes, 8 for parallel test builds/runs and 3 minutes for rollout checks (thanks for the reminder, @jwatte).

Diminishing Returns

I have been asked why we didn’t try to make the build and deploy systems as fast as possible… why not 2 minutes? We constantly worked on optimizing these systems, adding separate hypothesis builds, automatically isolating build servers to allow diagnosing and fixing without blocking, etc. And sometimes deployment would take less than 9 minutes.

However, much like the difference between 99.99% and 99.999% uptime for a service, the difference to the customer can be negligible while the resources necessary to deliver that improvement can be extraordinary. When business requirements are being met and engineering is happy with deploy times, the resources necessary to dramatically improve were better spent delivering value to customers.

Key Takeaways

  1. Working in a (well functioning) Continuous Delivery environment is empowering, naturally encourages other strong technical practices, and is hard to retreat from once experienced.
  2. Certain problems fall into what I call the “roommates and dishes” category, where “it’s everybody’s responsibility” sounds good, but in practice actually means “it’s nobody’s responsibility”. In these cases it is better to find a results-driven person and ensure they have responsibility and corresponding authority.
  3. Hire Eric Prestemon or somebody like him.

 

Have you worked in a Continuous Delivery environment and experienced non-obvious scaling challenges? I’d like to hear about your experience – please leave a comment!

Posted on

Q&A on Digital Transformation

In August I presented The Challenges of Executing Lean Startup at Scale, generously hosted by Rangle.io in Toronto, Canada. Rangle is the premier digital transformation consultancy, founded on Lean Startup principles and achieving impressive growth – a really great success story. I spent some time with Nick Van Weerdenburg, Rangle’s CEO, discussing Digital Transformation.

Some of the topics covered in the conversation include:

  • Solving customer problems is more important than rigorously following a process
  • The challenges of being on an agile team while working with or being part of a non-agile organization
  • Successful agile transformation requiring a culture change before a toolset change… most organizations get this backwards
  • How to choose metrics that are meaningful to your business

I hope you enjoy the video:

If you watch the video I would love your feedback! Please leave a comment below telling me what you think I got it right and what you think sounds crazy. 

 

Posted on

Exposing Your Private Data – It’s Not (Just) Them, It’s You

This week the Wall Street Journal published a story about third-party Google App Developers being able to read your Gmail, which was followed by many other outlets trying to sensationalize the news. However, a huge source of the exposing personal information problem isn’t big companies providing access to customer data, the problem is customers unwittingly (or uncaringly) granting permission for their data to be accessed. And while many people are skeptical about companies like Google and Facebook handling their data, the far bigger risk is users constantly exposing their private data to relatively unknown companies in exchange for low-value benefits.

Overreaching Account Access

Many sites and applications allow you to sign-on through an account on Facebook, Google and other services. This process is known as single sign-on (SSO), and is convenient and generally secure, especially if you utilize improved security measures like two-factor authentication. However, some applications ask for more access than is necessary, and the user willingly exposes a lot of private data to a third party that they don’t really know.

This Sample Application owns you and all of your data… forever

The list of permissions presented when you first grant access can enable a third party perpetual access to your information, usually long after you forgot you granted permission.

If you are simply trying to login to a new  application using SSO, there should be very little reason to grant any special permissions. Applications that request access to private data like email, contacts, messages, or calendars will have full access to your personal data. If an application doesn’t manage your private data, it should not need access. To protect your personal data, you should only provide the absolute minimum level of access necessary and avoid applications that request more that what they need.

Untrustworthy Third Parties

Some applications legitimately need elevated permissions to provide the service they offer, like inbox management, automatic scheduling, or even shopping deal comparisons. Many of these apps only access your data in the way necessary to provide the service, but there are many that take full advantage of access to your data and leverage your data for their benefit. According to articles on CNET and the Wall Street Journal, ReturnPath scanned the inboxes of 2 million people to collect marketing data after they’d signed up for one of the free apps produced by its partners, and the company’s employees read around 8,000 uncensored emails.

Even if you trust the intentions of the company producing the application, security is a really hard challenge and even the best companies fail at it… if you are providing access to an unknown startup, you are putting an exceptional amount of trust in believing they have the resources to ensure proper security measures. Of course, when a company is acquired (or its assets are sold), the access to your private data is passed along to the purchaser, whoever that might be.

When considering trading access to your private data in exchange for an application, ask what you are really getting for the risk. If somebody came up to you on the street and offered you some coupons in exchange for letting them read all of your email (forever), would you make that deal?

It’s Your Browser, Too

In addition to granting companies access directly, web browser extensions can expose data from every website you visit. These Extensions in Chrome, and Add-Ons, Extensions, and Plugins in Firefox, provide enhanced functionality from password management to page translation, ad blocking, and simple video downloads. To provide these services, many extensions get access to everything you do in the browser. For example, a news feed reader has permission to “Read and change all your data on the websites you visit” – this means every page visited and all content on that page is accessible by the news reader extension… your web mail, your Facebook messages, your dating sites, medical issues you research… all available to some company that organizes news headlines for you.

As browser extensions potentially grant access to every account, extra care should be taken to ensure trust for the company and permissions before installing.

Clean it Up and Lock it Down!

Until we make progress on time travel, there isn’t a way for an individual to guarantee deletion of data leaked from previously granted access. There are a few steps to greatly reduce your risk going forward…

Eliminate access to every app you don’t use

Most people simply stop using an app and forget about the access they granted, which usually continues in perpetuity. Regularly review the permissions you have granted – you will almost  certainly find some surprises. Facebook has settings for Apps and Websites, Google has a great Security Checkup, and other SSO services usually have a way of reviewing apps with access to your data. Only allow access to apps you are regularly use, disable those you don’t, and review the permissions to ensure they match the access needed.

And do the same for browser extensions! If there are extensions you use infrequently, most browsers have the option to enable / disable instead of having to delete the extension, so you can easily grant access only when necessary.

Trust Before You Install

Installing applications and linked account creation on websites is simpler than ever.  The downside to this ease of access is users typically spending little time scrutinizing the application. If you are giving access to your private data, spend the time to understand who is getting access, and how they will use your data. A simple web search for the application and “security” or “trust” can reveal what others experienced. If the company doesn’t have a website with the ability to contact them, and a published policy about handling your private data, there is a good chance securing your private data isn’t a real concern for them, and it should be for you!

 

Did you actually check to see who you are sharing your private data with? If so, what is the craziest thing you found? Please share by leaving a reply, below!

Posted on

When Customers Benefit From Decisions They Hate

I’ve been looking at a lot of products recently, mostly for very early stage companies, where one typically builds a successful product by addressing a customer’s needs, and the customer is delighted. But some product decisions, while not well received by customers (or sometimes hated), end up being better for the customer in the long term.  As an example, with major version updates, customers can immediately hate re-learning a product they already knew how to use, even though the changes may result in a better experience and more customers.

A few years ago I was in the position where it was necessary to make such a product decision… I knew would be hated by my customers, it was unlikely the benefit could be communicated to them, and if the decision was wrong, it would be a disaster that could result in 100 people losing their jobs.

It was a change to the experience that powered 90% of IMVU’s revenue.

What is this IMVU Thing?

IMVU creates social products, connecting people using highly-expressive, animated avatars. A huge part of the value proposition is creativity and self expression, a lot of which comes from the customer’s choice of avatars and outfits. People are usually surprised to learn that the business generates well over $50 million annually. IMVU’s business model is based around monetizing that value proposition, as customers purchase avatar outfits and other customizations. However, IMVU doesn’t create this content, it is built by a subset of customers (“Creators”) for sale to other customers – IMVU provides the marketplace and facilitates the transactions. IMVU was the only entity that could create new tokens for the marketplace, so almost all of IMVU’s revenue was from customers purchasing tokens to buy virtual goods. This creates a true two-sided market, and one of the biggest challenges is balancing the needs of both sides of the market. Never was balancing these markets so risky as the decision to take control of the way Creators earned real-world currency through sales of their products.

But First, A Little History…

Today the idea of selling virtual goods for real money is common place, as is people getting paid for creating user generated content…. examples include YouTube, Roblox, and Twitch. When IMVU was pioneering this model, there were few examples, and a lot of uncertainty around the concept of virtual goods being converted to real currency, in particular if this process would classify the company as a bank, with all of the associated banking regulations. IMVU avoided this risk by not handling any conversion of tokens to real currency, and instead allowing third parties to engage in transactions independently.

Very quickly “Resellers” popped-up, offering customers tokens for prices below what IMVU charged, and frequently purchasing tokens, enabling successful Creators to obtain real currency (IMVU took a percentage of every transaction, so the overall supply of tokens always decreased and helped keep the economy strong). This structure created a robust marketplace, where customers loved a huge catalog of items, Creators benefitted from their success, and Resellers benefitted from arbitrage.

When there is a benefit to exploiting a system, people will try to exploit the system. Since the benefit in this system was real money, it didn’t take long for bad actors to surface. IMVU customers were being harmed by bad Resellers that would take their money and not provide tokens, or steal their accounts (and tokens). As a result, we locked-down the Reseller program to less than 20 trusted people and had requirements for them to maintain good practices to remain in the program. And things were good…

The World Changes

Fast forward to 2015 and the world has changed… selling virtual goods and making money from user-generated content are well established practices. And, perhaps related to these practices being more mainstream, financial institutions have established best practices and requirements for these types of businesses. Mobile apps were also well established, which included customer expectations for purchasing in-app content, and app store guidelines for selling virtual goods. These developments, along with recognizing opportunities to provide more purchasing reliability to customers, drove IMVU to restructure the fundamentals of the Reseller program.

The decision to go through this restructuring was highly disruptive to customers, generally unpleasant for all involved, and absolutely the right thing for both customers and the company.

The Heart Transplant

The fundamental change was eliminating Resellers altogether, with IMVU providing royalty payments directly to Creators for the sale of their virtual goods. The process of paying content creators directly is pretty straightforward if it is your starting point, but transitioning to it is painful.

The immediate pain comes from managing communication with a large, passionate community that benefits from the established system, doesn’t necessarily see the need for change, and doesn’t (and can’t) have the breadth of information necessary to understand why changes are necessary (and ultimately, beneficial). IMVU’s Community Manager made heroic efforts and did a great job with communication, but there were still massive forum threads, petitions, and doomsayers.

The next challenge is trying to do the best thing possible for the Resellers, knowing that ultimately the result is going to be eliminating their business, so all you can hope for is making the best of a crappy situation. At this point Resellers were a small oligopoly with strongly protected positions, giving them a huge advantage in both purchasing tokens from Creators and selling to customers, and many had many months of token supply in inventory. Making things even more complicated, many Creators were also uncertain about their future ability to sell tokens, and wanted a way to cash out.

The solution was to announce to the IMVU community a timeframe for the wind-down of the Reseller program, allowing Resellers a small window to purchase tokens from Creators, and a larger window to deplete their inventories. A few Resellers dumped their tokens immediately at fire sale prices, but the more savvy Resellers paced their sales, recognizing that prices would increase as supplies dwindled. A few Resellers maximized the opportunity to buy tokens from Creators at next-to-nothing prices and benefit by selling them an close-to-peak prices a few weeks later. Ultimately Resellers were able to deplete their inventories before the program end. During the two month transition, IMVU resisted discounting its own token sales as to not compete with Resellers – this choice, combined with the tokens flooding the market, had a very real impact on revenue, both in the immediate loss of token sales and in the months following, while many customers had stockpiled a large supply of discounted tokens and didn’t need to purchase from the company.

The remaining transitional work was relatively straightforward (I’d write, “simple”, but I saw several teams of people work their butts off to get everything in place and working in time). Creators needed to provide necessary documentation so they could receive payment, and IMVU needed the accounting systems and people to facilitate payments.

But it wasn’t smooth sailing yet… While IMVU was very good about tracking Reseller token supply as part of monitoring the economy, unknown was the fact that Creators had a pent-up demand to sell their tokens, and much of this demand was completely unmet by the oligopoly of Resellers. As a result, request for royalty payments were much higher than initially expected. The new process would lead to a better result for a larger number of customers, as Creators would reliably be able to receive royalties. However, this immediately meant substantially higher expenses for the company, which was already feeling the impact of lower revenue from the Reseller cash-out. No amount of spreadsheet magic could make the business results look good.

A Quick Note on Leading Through Uncertainty

I was CEO of IMVU during this transition, and I distinctly remember this period as one where I felt I may have made a catastrophic decision. Most bad decisions can be corrected if you’re responsive, and it is usually better to take action and correct if necessary vs. stagnate from analysis paralysis. However, given that the token economy was the business, getting this transition wrong was an existential problem for the company. Over a hundred employees could lose their jobs, and millions of customers could lose a product where they connect with friends.

I knew the potential impact before making the decision, and exercised a lot of diligence researching the economy and token ecosystem (to be more accurate, I had an amazing COO that did the heavy lifting and we were aligned on our understanding). There were few decisions I made where I felt as confident in the ultimate result it would produce, but the timing, and seeing the painful business results each week certainly tested my confidence internally and I would review my assumptions to see where I could have gotten it wrong. Externally I remained more confident, reassuring employees and board we would see an inflection point… soon… it’s coming… hang in there.

I’ve heard other CEOs share stories with a similar pattern… the role requires a balance of internal self-questioning while portraying confidence externally, and the CEO rarely has the ability to share that internal conflict with others.

Results!

In the third month following the changes, IMVU hit the inflection point – the transitional business pain stabilized and started producing positive results. Taking full control of the Creator and Reseller aspects of the economy meant customers could have a reliable experience, from purchasing tokens to receiving royalties for their content. As part of the better-regulated process, there were other bad actors, scams, and negative customer experiences that were eliminated. And since there were less variables in token supply and pricing, it was much easier to maintain stability in the value of the token, a huge win for Creators, the business, and customers that ultimately benefit from a vibrant Creator marketplace.

The change to a more tightly-controlled token economy, combined with other big initiatives that were engaging new customers, resulted in a significant wave of growth and record results for IMVU’s business.

Key Takeaways

  • Talking to customers is critically important! Deeply understand the core of their objectives and pain points, and make sure product changes solve for the customer’s needs.
  • Be mindful that customers won’t have the breadth or depth of information necessary to recognize the real benefit of some product decisions. Sometimes what seems to be an immediately unpopular product decision is necessary to deliver a better customer experience over the long-term.
  • Spend the time to get information necessary for confidence in decisions that can have significant impact, but also be humble, open to recognizing a mistake, and ready to adjust if the results aren’t there.
  • With a large enough customer base it becomes impossible to solve for everybody, as occasionally their needs will conflict. Be intentional in product decisions that make these tradeoffs, solving for the best long-term customer experience for the customers your business needs.
  • Unsupportive customers should be an exception… most of the time your decisions should delight your customers.

 

Do you have examples of product changes customers hated but ultimately produced a better experience for them? If so, I want to hear about them! Please leave a reply, below.

Posted on

Know Thyself – Startup or Small Business?

There are plenty of good businesses that fail because they are convinced they must be great businesses.

When an entrepreneur asks me for advice for their company, the two most common questions I end up asking are, “what do you want to get out of this?”, and some variation of “do you really want to run a Startup, or would you be happier running a Small Business?” It’s not uncommon for people to make the mistake of thinking these types of companies are basically the same.

What’s the Difference and Why Does it Matter?

When you look at all of the new companies being created, the majority of these are Small Businesses. There are a few reasons for starting these, from following your passion, to having a reliable income, to perhaps creating a family business that will provide work for future generations. These companies are generally funded with family savings, small business loans, or personal loans. In almost all cases, the goal of these businesses is to be cash-flow positive and, if there is company growth, it is usually constrained by actual cash coming into the company, not spending ahead of revenue. As such, a Small Business will have revenue very early after starting, quickly as months or weeks. Owners are typically rewarded by the longevity of the company, a share of the profits, and sometimes a sale of the company.

While you couldn’t tell from a survey of Silicon Valley, but only a very small percentage of new companies are Startups. These are companies that have a vision to discover some radical innovation, in a product, a process, or a service, that has the ability to win a huge market. Since this is an exercise in discovery, the path of a Startup is one of uncertainty and high risk, with 9 out of 10 of these companies failing. The uncertainly means Startups need risk capital (usually multiple infusions) and can take years before they have any revenue. The most common source of funding for these companies is Venture Capital. Proving a repeatable business model and massively scaling business is the goal of Startups. Owners (shareholders) are rewarded by a liquidity event where stock in the company is converted to cash, typically through an acquisition or by having an IPO, and trading stock on the public markets.

The differing goals, and the financing dynamics mean that Startups and Small Businesses operate almost opposite of each other. With cash being a critical resource in a Small Business, business decisions are typically risk adverse. In most cases the better decision will be one that keeps the business at break even rather than risk negative cash flow, even if that decision has a small chance of a huge positive change.

In contrast, since 9 out of 10 Startups fail, that last 1 has to not only deliver economic wins for itself, it has to carry the weight of the 9 others that didn’t (since investors actually want better than market returns over the several-year life of the fund, the real increase in value for a win needs to be closer to 30x). What kind of decisions lead to a 30x return on investment? Not the conservative, sane ones you want protecting the existing value of a Small Business. Investors need big returns and that means they need the company to take big risks.

Crossovers are Rare

Occasionally you will hear about company being run as a Small Business that is super successful has the outsized success of a Startup. More common is the Startup that has crossed-over to being a Small Business… in almost every case the crossover to Small Business represents a failure for investors, where the company established a sustainable business but not one that could generate liquidity. These companies are sometimes referred to as “zombies” by investors… won’t die, but the stock will never turn into cash. For Startups it is way more likely that they fail completely, burning through all cash in high-risk attempts before discovering an actual business. The lucky ones can become acquihires (where a company “acquires” the team as employees, but no real cash is spent). Acquires can be a decent outcome for some of the team, but it a failure for investors.

Know Thyself

And this gets back to my question to many entrepreneurs, “what do you want to get out of this?”

Too often an entrepreneur has shared his company with me and I’ve seen a good business – one that can pretty reliably grow at 10-15% per year, provide jobs for many grateful employees, have lots of happy customers, enable taking decent amounts of cash off the table as it grows, not require 60+ hour weeks to manage. That’s a pretty good outcome, but it is a Small Business, not a Startup.

A lot of entrepreneurs (especially in Silicon Valley), see Startup as the only option.

And, Startups are great, too! They change the world (usually with the intention of making it better), they risk death doing the crazy things that occasionally produce amazing results. And for those very few entrepreneurs that make it through the gauntlet, successfully deliver a revolutionary business, they are rewarded with substantial financial rewards and, occasionally, hero-like status. They’ve created a great business.

My advice to any entrepreneur starting the journey of building a company is understand what you want to get out of the company, from quality of life to financial reward, and understand if you want to build a Startup or a Small Business.

 

I would really like to have more great Small Business stories! If you are part of a Small Business or you know of a great Small Business, please leave a comment!

Posted on

Avoiding the Perils of A/B Split Testing

A/B testing is widely used in product development, popularized as a fundamental component of the Lean Startup  framework, and providing a scientific way of validating product and business improvements. The concept is simple… put some customers in the new experience, compare the results against customers that didn’t get the new experience, and better metrics validates the improvement. In reality, this process of validation is very complicated and there is no shortage of hazards leading you to poor outcomes.

Creating Information out of Data is Hard

IMVU had a culture of data-validated decisions from almost day one, and as a result we made it easy for anybody to create their own split test and validate the business results of their efforts. It took minutes to implement the split test and compare oh so many metrics between the cohorts. All employees had access to this system and we tested everything, all the time. A paper released in 2009,  Controlled experiments on the web: survey and practical guide, reinforced that split testing was the undisputed arbiter or truth. We were clearly on the right path. 

While the ability to self-assess progress created a very empowering culture, we were largely ill-equipped to understand the nuances of what the data actually meant. Years later we would start to better understand, we don’t know how much we don’t know.

First Know Why

The first opportunity to make a mistake with split testing is deciding to test in the first place. When creating a split test has a very low barrier, it is easy to err on the side of just testing everything so that you can have the data if you need it. But every test has a lot of hidden costs than come from false-positives, clarification of data, shiny-object distractions, inconsistent customer experiences, and additional opportunities for introducing bugs.

Recognizing that being a split test packrat has a real cost, there should be some requirement for incurring this cost. Are very least, answering the question, “What are the significant changes that will be made as a result of this test?” Additional pre-test work to specify what will be measured, and what results will determine success or failure can also go a long way towards ensuring time spent testing is valuable.

Test Implementation is a Project

IMVU had a great framework to make test implementation a seemingly simple task, with a few lines of code of creating a branch for the test experience, and leaving the current experience as the control. Again, this made creating tests seem deceptively easy, and left openings for measuring the wrong thing.

Often a split test is a cross-functional effort, with an engineer handling the implementation and the customer being any combination of a product manager, acquisition team, marketing representative, revenue officer, or generally interested party. In some cases, the interpretation of test data is done by another person altogether. Correctly understanding what the internal customer wants to know, capturing the right data, and converting that data into information ends up with many points of communication that must be accurate to deliver a valid test.

For example, the acquisition team wants to test a new landing page, simply reordering the registration fields because they think it will improve the registration completion rate. The engineer realizing this is a no-brainer takes the 15 minutes before lunch to create the quick test, two paths and the test is running. However, the registration page has both manual registration and sign in with a social network account, so the test is including a lot of users that are social logins, irrelevant to the registration fields. This subtle nuance means that the impact of the registration field changes will likely be lost as the irrelevant data acts as a damper. What the customer wanted to know isn’t what the test is answering, and it’s likely that nobody on the project knows there is an error.

The ease of creating a split test should not be conflated with delivering quality results from a test. Doing it right is a project and requires investment of resources consistent with any other project.

WTF Do These Results Actually Mean?

Assuming you were diligent in your experiment design, you captured all of the relevant data, and you avoided some of the common errors of A/B testing, you now need to make sense of the data. In the best cases, you’re looking at something like “the registration landing page increased conversions from 1.83% to 2.01%”, in the worst cases you find something like “customers are engaging with messaging feature 17% longer… but their lifetime value has dropped by 4%”, and now there is work to put together a narrative that explains the perplexing results.

In 2012 I read a paper, Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained, and I had what I like to call an, “oh shit” moment. Highly controlled experiments, run by companies with world-class, dedicated analytics teams were getting perplexing results that required substantial research to understand what was actually happening. What chance did we have of getting this right when we are running 15+ experiments a week with training consisting of a one page internal wiki version of, “A/B Testing for Dummies”?

The tl;dr summary of the paper, without deep consideration for the “why” behind the change in metrics, positive results may be antithetical to what you are actually trying to achieve.

The up-front work to limit the scope of the experiment and how it will be measured / interpreted can help, assuming you have the self control to ignore the data outside of scope. Often these perplexing results require follow-up experiments to better isolate cause and effect. I also highly recommend talking to customers – often qualitative insights from hearing their experiences can often help make sense of what the quantitative results were hiding.

You’re Biased. No, Really, You Are

I’m sure there are a lot of great reasons we humans are wired to think the way we do, and this wiring probably served us very well in many situations. However, humans also come standard with cognitive biases, built-in tendencies to make irrational decisions. Unfortunately, putting a bunch of effort into building something and then getting a giant pile of metrics is a perfect enabler for a cognitive biases and craptastic decisions.

While numerous biases are working against you, with a buffet of metrics one of the most common is the Texas sharpshooter fallacy, in which the all of the test metrics that are improvements over the control metrics are used to demonstrate the success of the test. With a 95% confidence rate, 1 out of 20 metrics tracked are expected to show a false positive improvement, so even an A/A test (two separate cohorts with identical experiences) would likely show “improvements”. Before we eliminated the practice of metric-sniping at IMVU, it wasn’t uncommon to hear somebody say something like, “my pet project to streamline registration didn’t change registration, but it does deliver a 5% improvement in [the completely unrelated] customer lifetime value, so we should keep it.”

There are process controls that can help reduce the potential impact of various biases, in particular around defining and constraining each test. However, being aware of these biases and encouraging a culture consistent with the dialectical method can help make better product decisions, even beyond interpreting test results.

Talk to Your Customers!

One of the biggest risks that come from over-reliance on split testing is seeing it as a more convenient method of getting customer feedback. Why spend 30 minutes on the phone with one customer when you can simply measure the actual actions of thousands of customers?

Looking at data and sending surveys may seem like an efficient use of time, but that highly structured approach is unlikely to surface critical customer insights. Metrics and surveys will often answer the “what”, but almost always miss the “why”, the most critical driver of valuable insights. There is no substitute for talking to your customers.

In the words of Steve Blank, “Get Out of the Building.”

 

I’m interested in hearing other stories where split testing has made an impact, either positive or negative. Please share a comment if you have one!

Posted on

Death: A Few Notes to Self

I started this post months ago after my mother passed away in May 2017, but I was hesitant to publish it. It’s deeply personal, and I feel uncomfortable sharing publicly, but as I see more friends working through their own experiences, I wanted to share in case my experience can help others. 

My mother recently died. It’s a shocking event, but this is a common occurrence… almost everybody experiences the death of their parents. At the same time, with all of this collective experience, almost nobody has a good understanding of what that experience will be like for them. And even if each person had a great understanding, it would probably only help get through the logistical aspects of death, and little of the grief. As notes to myself, and hopefully to help others, I wanted to share some of what I learned from my experience…

My mother was my last surviving parent*. When my father passed 13 years ago, I was insulated from many of the end-of-life issues, since my step mother was there to support him and handle everything when he passed. I went through a grieving process, but I wasn’t involved in the end-of life logistics, for which I am grateful.

But with my mother, I was the primary person responsible for her end-of-life care, as well as managing her post-death affairs (all of this with the tremendous support from my sister). Here are a few of the lessons I took away from my mother’s end-of-life experience…

It’s Time to Let Your Children Help

Like many people, my mother loved her home and wanted to remain there for the duration of her life. In the last few years of her life, the house was too much to manage and she needed support for the basics of daily living. I desperately wanted to respect her wishes, but eventually there were enough health and safety issues that I simply had to overrule her, which started with having in-home assistants and eventually required moving her to an assisted living facility near my home. Each of these steps were tough for her, and tough for me to make her go through. But almost everybody familiar with the situation agreed that her health and safety was greatly improved by the changes.

I thought about this and told myself I need to sit down and write a letter to my future self, with this sentiment:

“For years you have worked to steer your child in the right direction, sometimes making decisions they hated because you were trying to shape them into a better person. You did so out of love for them, and you were often correct more than you were wrong. It’s now time for you to listen to that child, as they are now trying to do the right thing for you, and they are often correct more than they are wrong.”

Possessions Are Not Worth Possessing

My mother had a lot of very nice antiques, as well as a lot of crap labeled “made in China”. While she collected many things for herself, she always felt that the valuables would be passed-down to me and my sister, so that we would have pieces of our family history, and possibly valuable antiques. As a result, instead of living in an un-cluttered house with only the nicest things around her, she was more of a pack-rat, with semi-valuable, but relatively meaningless stuff to navigate when doing things like socializing with friends.

And she would have been saddened by the outcome… my sister and I are well-established in our own homes, so we don’t need (or have room for) extra stuff. And, to get the value out of each item takes time, which neither of us had. As a result, we had an estate sale which resulted in us getting maybe 20% of the actual value of the items, not a lot of money at all, and certainly not worth the years of collecting and storing.

And, the whole process of dealing with the possessions was hard, both physically going through everything, and emotionally draining knowing how much these possessions meant to my mother, and how they would all be going to strangers at fire-sale prices.

As I get older and as my kids get more established on their own, I hope to continually reduce the material things, only passing along the few items that are truly special.

Quality, not Quantity

How one lives and ends their life is deeply personal, and I am not suggesting that my perspective is appropriate for anybody else. To put it another way, I respect any opposing views.

My mother, while in poor physical health, maintained a sharp mind, and would love to socialize and talk about her grandchildren. In the last two weeks of her life she suffered from an issue that made her disconnected, unable to communicate, and unaware of her surroundings. I consider it fortunate that she passed so quickly after this transition… while physically present, she wasn’t the person I had known, it was only her body. My mother was already gone.

As a result of both spending time with my mother and, through the environments she was in, witnessing many others in their final time, I reaffirmed that I want to make sure that my time is spent experiencing all the things. Simply extending my life isn’t important. The concept of death is scary, but the reality of fighting death only for the purpose of existing looks worse.

The most obvious way to have quality of life in later years is to take care of yourself now. As I met people that were 75 years old and confined to a wheelchair, I also had a friend tell me about her father that is 75 and bikes ten miles every day. Of course there are a lot of factors that play into quality of life, but being fit today is the best way to be fit later in life, and mobility later in life makes a huge difference in the options available to you. Eat less, exercise more. I don’t want to spend my days confined to a bed and pass peacefully in the night, I would rather be backpacking, running our my body’s warranty, and experiencing the beauty of nature until nature decides it’s time to recycle me.

 

* I recently found my biological parents, both living.

 

 

Posted on

How to Respond After Leaking Your Customer’s Data

The most recent consumer-hostile disclosure of an account breach was Uber’s leaking of 57 million accounts almost a year ago. I’d like to say this is an extraordinary event, but much like a favorite character getting killed in Game of Thrones, companies leaking customer data is just another regular occurrence we’ve come to expect. What continues to surprise me is how badly so many companies screw-up their response to a breach. The one principle that should guide companies following a breach is, “make the decisions you would want a company to make if it was your account that was compromised.”

And sure, it’s easy to point fingers when it’s not you in the hot seat, so I’ll use the breach I managed as an example… The breach I was responsible for was in September 2015, when I was CEO of a company that had over 100 million registered accounts.

Initial Response

The breach was caught around 11:00 PM at night… within a couple of hours we had a fire-team of employees in the office. The priority was confirming that the breach was indeed fully contained, and then validating we understood the full extent of the breach. We wanted to communicate to customers as quickly as possible, and we wanted to be able to accurately convey the amount of exposure. Every other project was de-prioritized and employees were working 24/7 on projects related to the breach.

Thanks to some security precautions we had in place, we were able to detect the breach in real-time, limit the data that was accessed, and understand exactly what data was exposed. Also, due to the nature of the data that was accessed, the actual customer exposure was minimal (e.g. no credit cards, social security, addresses)… assuming the attacker had planned to use the data for malicious purposes, the actual value of that data was extremely low.

As we reached morning, we contacted law enforcement and legal counsel, both of which informed us that the data exposed was insignificant in terms of risk. We were also told that, because of the type of data accessed, there was no requirement to disclose the breach.

While we had a pretty solid understanding of what happened as part of the breach, we didn’t want to be overly confident, so we continued the process of going through hundreds of servers and employee computers to look for anything that might have been missed, a process that took a little over two full days.

The Ransom

Within 24 hours of the breach I started receiving emails that threatened to release the customer data and publicly announce the breach if we didn’t pay a sum of money. My response to the blackmail was letting them know I would consider their proposal, but ultimately the damage they would do is to customers that didn’t deserve to be exploited, and to employees, good people that already feel a ton of weight from the responsibility. They gave me a few days to make a decision.

Talking to Our Customers

After we had confidence that we had contained the breach, removed any attack vectors, and fully understood the data accessed, we were ready to talk to our customers. Less than 72 hours had passed, but it felt like an eternity getting to this moment.

We posted to our forums and messaged our customers individually with the details of the breach, specific data accessed, how that data can be used, and what steps to take (on our service and others) to protect against any further attack. We also disclosed that the hacker had tried to extort money in exchange for silence.

While I can’t say that any customer was pleased that the exploit occurred, many responded very positively to our handling of it. Earlier that year credit card and health care breaches of highly-sensitive data took many months to be announced, so many of our customers appreciated how quickly we moved to keep them informed.

Evidently the hacker didn’t read our forum post, as the next day they gave me the final warning that they were about to announce the breach to our customers and the media. I informed the hacker that we would not be paying the ransom, reminded them that the people they will hurt don’t deserve it, and pointed them to the forum posting fully disclosing the breach, accessible to all of our customers and the media.

Post Breach

Through a process of many, many postmortems and follow-up action items, the company continued to improve security in several areas, projects that extended many months. We understood exactly how the breach occurred, and the human component that enabled the breach. What we explicitly didn’t do is punish or threaten anybody – throughout the whole process we made all employees feel safe, which enabled people to be fully transparent and quickly disclose their mistakes, a critical aspect of quickly understanding how the breach occurred.

The moment that sticks out in my mind the most was an email I received from an employee in response to a detailed summary of the events I sent to the company. That employee expressed that they had never been so proud to be at a company, in the integrity we demonstrated to our customers, and the unwavering support for the employees. It was one of those emails that CEOs move to their “save forever” folder. 

Key Takeaways

While there are a lot of opportunities for companies to make customer data more secure, the unfortunate reality is even the companies with the best security practices experience breaches – this is going to happen. However, a few steps can provide better outcomes for all parties:

  1. Treat your customers as you would want to be treated.
  2. Make your employees feel safe. Fearful employees will conceal critical information that is necessary to fully understand the problem.
  3. Don’t negotiate with criminals. It’s bad for your customers, there is no way to enforce the criminal’s end of the agreement, and the deception is likely to be revealed at some point. Perhaps one acceptable variation on this takeaway is, if you do negotiate with criminals in the interest of your customers (e.g. to get details about how the leak occurred), still be transparent with your customers and disclose that a transaction occurred.
  4. Do the follow-up work. After an exhausting amount of effort getting past the initial breach it’s easy to feel like your work is done… make sure all of the known exploit vectors are eliminated.

 

Have you been impacted by a company’s data breach? I’d like to hear about your experience – please leave a comment!

Posted on

My Favorite Recent Inventions

In the spirit of enjoying a lovely holiday by keeping the conversation topics away from politics, and focusing on humor and technological advancement (or, more specifically, technological advancement humor), I thought I would share some of my favorite recent inventions, and include commentary from random people in the Twittersphere.

Sheet Muffins

The innovation team at Slate discovered a sweetened bread that replaces the need for individually held muffins…

Innovations in Co-Living

Millennials invented a way to have other people leave dirty dishes in the sink and drink all but the last 1/10th of an ounce of milk in the fridge…

Underground Group Transport

Elon Musk, the Thomas Edison of our age, found a way to have subterranean vessels pick up and drop off people at regular intervals…

Street Group Transport

Not to be outdone by Elon, Uber came up with a similar concept where the transportation uses roadways to pick up and drop off people at predefined locations…

Dedicated Short Term Visit Buildings

Airbnb continues to innovate by solving the problem of not having large, multi-unit buildings that are dedicated to short-term visits…

Automated Product Dispenser

And finally, solving both the problem of human interaction and the inability to purchase goods, a startup developed a way to pay for and receive products from a mechanical device…

An Amazing Time to be Alive…

Technology has truly taken us to places that could only have been imagined 30 years ago. Embrace these advancements and marvel at how they change the world before our eyes.

 

Are you aware of other incredible innovations that are changing our lives? Please leave a comment!

Posted on

Your Agency is Hurting Your Chance of VC Funding

Early-stage venture capital firms have high deal flow and very little time to assess each company, so understanding key assessment criteria will help you get your deck from the “no” bucket to the partner discussion. A common reason many companies fail to get past “no” is they are agencies.

Is Your Company an Agency?

In an agency, value created by the company is unique to each customer. As a result, the company revenue reflects more of a work for hire relationship. The problem with this model is, while an agency can still be a very good (or even great) business, it is hard to scale and typically doesn’t improve margins when it does scale.

When asked, entrepreneurs don’t always recognize that their business model is an agency… they may see the unique customer work provided as building support in the underlying platform, or a way to help onboard early customers. While all possible, it’s unlikely, and VCs that have looked under the hood of hundreds of companies will understand the signals indicating this is an agency:

  • A majority of revenue comes from additional work provided, not from the product / service
  • Work performed is applicable to a specific customer (e.g. content creation, integration, customization)
  • Customers largely came from relationships, not from a repeatable sales process
  • The company is pivoting from a consulting business

What if My Company is an Agency?

So, what do you do if your business looks like an agency? Well, it depends on what you want for your company. If you’re happy with a potentially good (or even great) business that may grow at a reasonable rate, be a source of employment for a bunch of people, and maybe never have an exit, skip the VC and run your business (of course, you have to run cash positive or get loans to get you there). And, the lack of an exit doesn’t preclude a payout… I’ve met several owners of “lifestyle businesses” that, on top of a good salary, pull substantial amounts of money out of their company.

If you do want to go the VC route and have a VC-sized exit, you’re going to either prove your business is the exception (unlikely), or make some fundamental changes to your business to achieve some combination of the following:

  • A consistent shift in revenue away from unique customer work and towards your product or service
  • A convincing process showing the unique work for each customer is scalable (i.e. not limited on the supply side)
  • Margins improving with growth   

Pivoting to a new business model is usually easier written than done. And, if your agency model is working for you, a pivot away from a working business model can be risky. The again, if you’re the type of entrepreneur that is excited by building VC-backed businesses, you probably eat risk for breakfast.