Tag: IMVU

Posted on

Hinder, Don’t Halt: Griefing Content Thieves for Fun and Profit

The art of deterring content theft is an ongoing game of cat and mouse – generally any barrier you create to prevent theft is temporary, as thieves continue to find new ways to steal the content, so long as the value of the content exceeds the effort necessary to steal it. For this reason, it can often be more effective to hinder thieves instead of trying to stop them.

I encounter this “hinder don’t halt” pattern with others that run large services, and you can see this reflected in solutions like shadow banning. One of the most common themes I hear is the satisfaction that comes from solutions that cause frustration for bad actors, so I’m sharing one from my personal experiences…

At IMVU, customers called Creators make content that they sell to other IMVU customers. The content they create is 3D items like avatar clothing, items to decorate an environment, and ways to customize an avatar. This content creates real value for other IMVU customers, who spend real money to purchase it from the catalog of over 10 million items. While many Creators create content just for the enjoyment of creating, some do it as a business, with a few making over $100K US annually. Whether creating for pleasure or business, all Creators hated having their work stolen. And, since there is real money from the sales of content, there is real incentive for thieves to try to steal it.

At one point we discovered a site that was selling a service that would allow people to steal Creator content without paying for it. It was pretty easy to detect the service and the initial response was blocking them, which immediately broke their service completely and, not surprisingly, made the thieves quickly respond by finding a new way around the block. The block lasted less than a day and the thieves were back in business.

The next response was more fun… rather than blocking the thieves, we made their service not work… sometimes… and inconsistently. Code was added to detect thieves accessing content and randomly some content being accessed would be mildly corrupted. The corruption could be configured to occur at certain rates, on certain items, at certain times of day, and be disabled based on what appeared to be testing for the corruption. As a result, customers of the thieves started getting inconsistent results, that would sometimes lead to content failing to load and even crashes. If you are an engineer reading this, you understand why this is a nightmare scenario to debug and fix… customers are reporting different failure cases with no consistent way of reproducing the problem to understand the cause. And, since your code is working fine, the bug isn’t going to be found… you eventually have to discover that you are being served different content than is being served to legitimate customers.

The result of hindering was much more effective than blocking… it took many weeks for the thieves to understand what was happening and, during this time, we could see them getting bashed by the people that paid them because the stolen content was ruining their experience. By the time the thieves had found another solution, they had such a bad reputation that people were less willing to give them money.

If you have dealt with content thieves I would be interested in hearing your stories, successful or not. Please leave a reply, below!

Credits
Cat and mouse chase image by Jeroen Moes
Dungeons & Dragons dice by Lydia

Posted on

Rewards from Talking to Customers

Most people that build products or run companies have heard the mantra, “get out of the building – talk to customers.” It is easy to assume that talking to customers is only about building a better product. Talking to customers will help you build a better product, but more importantly, you may be rewarded by learning how your work changes people’s lives!

I recently had an experience that was so delightful I had to share it with my former employees, and they decided to share it with their millions of customers. Below is the excerpt from the IMVU blog:

You may remember a very familiar face in the photo featured in this story.  Brett Durrett is and always will be a friend of IMVU, even after his 11 years on staff and nearly 5 years as our CEO. Beyond his professional titles, or even his leadership as CEO, Brett was an active user that frequently went into chatrooms to join the conversation, answer questions, solve issues, or simply say hello. On Fridays at the HQ office, it was common to see Brett speaking from a microphone about the week’s accomplishments, and always finishing with words of inspiration, a story of encouragement, or a new product to be excited about.  Even if we didn’t hear your stories, Brett always told us your stories so that we could remember why we work at IMVU: we are here to spread the power of friendship, to help people find friends, to encourage them to express themselves, and to find an outlet for creative expression.Recently, our current Chief Operating Officer Kevin Henshaw, forwarded an email he received from Brett to the entire company about how IMVU continues to work its magic on and off our product. 

Brett’s email read like this:

On Monday I was wandering around New Orleans wearing my IMVU hoodie, as I am one to do. I went into a coffee shop and the woman at the counter asked me how I got my hoodie, to which I replied, “I used to work for IMVU”. Her eyes lit up as she proceeded to tell me how much IMVU meant to her as she was growing up.

Bea told me she used IMVU because it allowed her to connect with people without any stereotypes about who she was – she got to decide how she wanted to be seen. She also loved that it didn’t cost much to experience a fantasy lifestyle. She had a lot of friends on IMVU that felt the same. She really gushed about how important IMVU had been in her life. Her excitement went on for minutes. My traveling companion was taken aback, as I seemed to have rock star status. It was a chilly day in NOLA, but I gave Bea my IMVU hoodie (she had made me feel so warm inside that I really didn’t need it).

If you’ve talked to enough IMVU customers you know that Bea’s story isn’t unique… IMVU has helped people find their life partners, best friends, and caring families.

I thought I would use my chance encounter as an excuse to reach out to IMVU employees, say “hello”, and remind them that there are a lot of silly things than can happen on IMVU, but don’t lose sight of the really meaningful things as well! Bea’s story is a testament to what this is really about – helping people find new friends and creating something meaningful to benefit their lives. On behalf of Bea, myself, and millions of customers, keep up the great work!

Do you have a delightful customer story? I’d love to hear about it… please leave a reply!

Posted on

Scaling Continuous Delivery: Happiness as a Metric

A few days ago Jeff Atwood (Coding Horror) suggested a good measure of a tech company’s health is the time it takes to have a simple change become available to customers:

And while there are numerous metrics that determine the health of a tech company (see Jez Humble’s book, Accelerate, for an amazingly comprehensive overview), Continuous Delivery strongly correlates to successful outcomes.

I support Jeff’s assertion – I witnessed the value created by Continuous Delivery at IMVU, where we pioneered some of the crazy processes that would be followed by more sane practitioners. From day one, IMVU placed value on the speed of product iteration and “designed” build systems accordingly. In 2006, development was done in Windows using Reactor Server to provide the LAMP-ish stack, and the deploy process looked something like this:

 svn-server$ rcp website/* production:/var/www/

If you’re wondering why I omitted the test framework, I didn’t. Code went from a local Windows sandbox environment, to source control, to live on a Linux environment running a version of PHP different than the local sandbox. Fun! While there were numerous problems with this system, iteration velocity was amazing (those 1 word copy changes could ship in less than 5 minutes, and so could full features). This development velocity was a key component enabling IMVU to build a large, successful business in a space where the failed companies outnumber survivors 50 to 1.

And to be clear, time to get a change live to customers doesn’t in itself indicate healthy tech, but a lot of tech health comes from the corresponding systems necessary to make rapid deployment work.

Fast forward a few years to 2008 and I transition from leading the operations team to leading the engineering organization, where the build and deploy systems had matured, with reasonable test coverage, and automated deployment, with automated rollbacks when something unfortunate made it into production. It was pretty cool, even though publicly the process was mostly received with the sentiment, “that will never work , and certainly won’t scale”.

Scaling Problems

One of my first challenges as the new engineering leader was a team unhappy about their ability to get work done because it was taking hours for a commit to become live to customers. It’s astonishing when you think about it – every engineer in the company had come from companies where the commit to live process was typically measured in months, but once they experienced the value of Continuous Delivery, anything more than minutes seemed unbearable.

Digging into the problem, I came to understand that the problem was not slow builds (although that was part of it), the most significant issues were caused from the shared responsibility for build systems, combined with the desire to deliver features to customers, created a tragedy of the commons. When an engineer had a failure in the build system, the optimal solution for that engineer was to fix the problem in place, blocking the build system for anybody else in the queue, which meant the number of commits in the next build increased, which meant the chance of a failure in the that build increased, ad infinitum. The result was pushing to production could be blocked for hours, sometimes most of the work day.

Solving for Happiness

I thought the best solution was to formalize a project, have a clear success outcome, and have a single person with the responsibility for (and therefore authority over) the build / deploy systems. The first problem was determining a clear success criteria… anything time metrics I chose would be somewhat arbitrary, so instead I chose engineering happiness as the success criteria, or more specifically, pushing to production was no longer causing unhappiness. While I generally hate subjective success criteria, there were ways to assess progress through 1:1 conversations and Likert scale surveys. We also had great (highly objective) data around commit to deploy times, so we could see the correlation to the more subjective happiness index.

There was some pretty straightforward work to improve the actual test and deploy speeds, including simple things like adding more hardware and the slightly less simple sorting tests to run by speed (a surprisingly large performance gain), and fixing the slowest of the tests. But some of the most important gains came from the human parts of the deployment system… engineers were required to immediately revert code and fix the issue in their sandbox rather than blocking the build system. This was not a popular policy change as immediately engineers experienced the direct impact from a failed commit, but didn’t immediately see any gains to the overall system.  But after a few weeks the improvements were clear in the average commit to deploy time. And giving credit where it is due, Eric Prestemon was the “Buildbot Sheriff” that identified so many of the opportunities for improvement and delivered the results… many people helped, but Eric had the burden of hearing a lot of critical feedback about unpopular policy changes (eventually outweighed by the praise for the results he produced).

Eventually the build system frustration ceased being a common topic in 1:1 meetings, and it faded away as a meaningful problem in engineering surveys. 12 minutes. When the commit to live time is 12 minutes, this system is operating well. That became the new value for alerting – under 12 minutes, all is good, after that we need to actively drive improvements. In practice, deploy time was usually around 11 minutes, 8 for parallel test builds/runs and 3 minutes for rollout checks (thanks for the reminder, @jwatte).

Diminishing Returns

I have been asked why we didn’t try to make the build and deploy systems as fast as possible… why not 2 minutes? We constantly worked on optimizing these systems, adding separate hypothesis builds, automatically isolating build servers to allow diagnosing and fixing without blocking, etc. And sometimes deployment would take less than 9 minutes.

However, much like the difference between 99.99% and 99.999% uptime for a service, the difference to the customer can be negligible while the resources necessary to deliver that improvement can be extraordinary. When business requirements are being met and engineering is happy with deploy times, the resources necessary to dramatically improve were better spent delivering value to customers.

Key Takeaways

  1. Working in a (well functioning) Continuous Delivery environment is empowering, naturally encourages other strong technical practices, and is hard to retreat from once experienced.
  2. Certain problems fall into what I call the “roommates and dishes” category, where “it’s everybody’s responsibility” sounds good, but in practice actually means “it’s nobody’s responsibility”. In these cases it is better to find a results-driven person and ensure they have responsibility and corresponding authority.
  3. Hire Eric Prestemon or somebody like him.

 

Have you worked in a Continuous Delivery environment and experienced non-obvious scaling challenges? I’d like to hear about your experience – please leave a comment!

Posted on

Q&A on Digital Transformation

In August I presented The Challenges of Executing Lean Startup at Scale, generously hosted by Rangle.io in Toronto, Canada. Rangle is the premier digital transformation consultancy, founded on Lean Startup principles and achieving impressive growth – a really great success story. I spent some time with Nick Van Weerdenburg, Rangle’s CEO, discussing Digital Transformation.

Some of the topics covered in the conversation include:

  • Solving customer problems is more important than rigorously following a process
  • The challenges of being on an agile team while working with or being part of a non-agile organization
  • Successful agile transformation requiring a culture change before a toolset change… most organizations get this backwards
  • How to choose metrics that are meaningful to your business

I hope you enjoy the video:

If you watch the video I would love your feedback! Please leave a comment below telling me what you think I got it right and what you think sounds crazy. 

 

Posted on

When Customers Benefit From Decisions They Hate

I’ve been looking at a lot of products recently, mostly for very early stage companies, where one typically builds a successful product by addressing a customer’s needs, and the customer is delighted. But some product decisions, while not well received by customers (or sometimes hated), end up being better for the customer in the long term.  As an example, with major version updates, customers can immediately hate re-learning a product they already knew how to use, even though the changes may result in a better experience and more customers.

A few years ago I was in the position where it was necessary to make such a product decision… I knew would be hated by my customers, it was unlikely the benefit could be communicated to them, and if the decision was wrong, it would be a disaster that could result in 100 people losing their jobs.

It was a change to the experience that powered 90% of IMVU’s revenue.

What is this IMVU Thing?

IMVU creates social products, connecting people using highly-expressive, animated avatars. A huge part of the value proposition is creativity and self expression, a lot of which comes from the customer’s choice of avatars and outfits. People are usually surprised to learn that the business generates well over $50 million annually. IMVU’s business model is based around monetizing that value proposition, as customers purchase avatar outfits and other customizations. However, IMVU doesn’t create this content, it is built by a subset of customers (“Creators”) for sale to other customers – IMVU provides the marketplace and facilitates the transactions. IMVU was the only entity that could create new tokens for the marketplace, so almost all of IMVU’s revenue was from customers purchasing tokens to buy virtual goods. This creates a true two-sided market, and one of the biggest challenges is balancing the needs of both sides of the market. Never was balancing these markets so risky as the decision to take control of the way Creators earned real-world currency through sales of their products.

But First, A Little History…

Today the idea of selling virtual goods for real money is common place, as is people getting paid for creating user generated content…. examples include YouTube, Roblox, and Twitch. When IMVU was pioneering this model, there were few examples, and a lot of uncertainty around the concept of virtual goods being converted to real currency, in particular if this process would classify the company as a bank, with all of the associated banking regulations. IMVU avoided this risk by not handling any conversion of tokens to real currency, and instead allowing third parties to engage in transactions independently.

Very quickly “Resellers” popped-up, offering customers tokens for prices below what IMVU charged, and frequently purchasing tokens, enabling successful Creators to obtain real currency (IMVU took a percentage of every transaction, so the overall supply of tokens always decreased and helped keep the economy strong). This structure created a robust marketplace, where customers loved a huge catalog of items, Creators benefitted from their success, and Resellers benefitted from arbitrage.

When there is a benefit to exploiting a system, people will try to exploit the system. Since the benefit in this system was real money, it didn’t take long for bad actors to surface. IMVU customers were being harmed by bad Resellers that would take their money and not provide tokens, or steal their accounts (and tokens). As a result, we locked-down the Reseller program to less than 20 trusted people and had requirements for them to maintain good practices to remain in the program. And things were good…

The World Changes

Fast forward to 2015 and the world has changed… selling virtual goods and making money from user-generated content are well established practices. And, perhaps related to these practices being more mainstream, financial institutions have established best practices and requirements for these types of businesses. Mobile apps were also well established, which included customer expectations for purchasing in-app content, and app store guidelines for selling virtual goods. These developments, along with recognizing opportunities to provide more purchasing reliability to customers, drove IMVU to restructure the fundamentals of the Reseller program.

The decision to go through this restructuring was highly disruptive to customers, generally unpleasant for all involved, and absolutely the right thing for both customers and the company.

The Heart Transplant

The fundamental change was eliminating Resellers altogether, with IMVU providing royalty payments directly to Creators for the sale of their virtual goods. The process of paying content creators directly is pretty straightforward if it is your starting point, but transitioning to it is painful.

The immediate pain comes from managing communication with a large, passionate community that benefits from the established system, doesn’t necessarily see the need for change, and doesn’t (and can’t) have the breadth of information necessary to understand why changes are necessary (and ultimately, beneficial). IMVU’s Community Manager made heroic efforts and did a great job with communication, but there were still massive forum threads, petitions, and doomsayers.

The next challenge is trying to do the best thing possible for the Resellers, knowing that ultimately the result is going to be eliminating their business, so all you can hope for is making the best of a crappy situation. At this point Resellers were a small oligopoly with strongly protected positions, giving them a huge advantage in both purchasing tokens from Creators and selling to customers, and many had many months of token supply in inventory. Making things even more complicated, many Creators were also uncertain about their future ability to sell tokens, and wanted a way to cash out.

The solution was to announce to the IMVU community a timeframe for the wind-down of the Reseller program, allowing Resellers a small window to purchase tokens from Creators, and a larger window to deplete their inventories. A few Resellers dumped their tokens immediately at fire sale prices, but the more savvy Resellers paced their sales, recognizing that prices would increase as supplies dwindled. A few Resellers maximized the opportunity to buy tokens from Creators at next-to-nothing prices and benefit by selling them an close-to-peak prices a few weeks later. Ultimately Resellers were able to deplete their inventories before the program end. During the two month transition, IMVU resisted discounting its own token sales as to not compete with Resellers – this choice, combined with the tokens flooding the market, had a very real impact on revenue, both in the immediate loss of token sales and in the months following, while many customers had stockpiled a large supply of discounted tokens and didn’t need to purchase from the company.

The remaining transitional work was relatively straightforward (I’d write, “simple”, but I saw several teams of people work their butts off to get everything in place and working in time). Creators needed to provide necessary documentation so they could receive payment, and IMVU needed the accounting systems and people to facilitate payments.

But it wasn’t smooth sailing yet… While IMVU was very good about tracking Reseller token supply as part of monitoring the economy, unknown was the fact that Creators had a pent-up demand to sell their tokens, and much of this demand was completely unmet by the oligopoly of Resellers. As a result, request for royalty payments were much higher than initially expected. The new process would lead to a better result for a larger number of customers, as Creators would reliably be able to receive royalties. However, this immediately meant substantially higher expenses for the company, which was already feeling the impact of lower revenue from the Reseller cash-out. No amount of spreadsheet magic could make the business results look good.

A Quick Note on Leading Through Uncertainty

I was CEO of IMVU during this transition, and I distinctly remember this period as one where I felt I may have made a catastrophic decision. Most bad decisions can be corrected if you’re responsive, and it is usually better to take action and correct if necessary vs. stagnate from analysis paralysis. However, given that the token economy was the business, getting this transition wrong was an existential problem for the company. Over a hundred employees could lose their jobs, and millions of customers could lose a product where they connect with friends.

I knew the potential impact before making the decision, and exercised a lot of diligence researching the economy and token ecosystem (to be more accurate, I had an amazing COO that did the heavy lifting and we were aligned on our understanding). There were few decisions I made where I felt as confident in the ultimate result it would produce, but the timing, and seeing the painful business results each week certainly tested my confidence internally and I would review my assumptions to see where I could have gotten it wrong. Externally I remained more confident, reassuring employees and board we would see an inflection point… soon… it’s coming… hang in there.

I’ve heard other CEOs share stories with a similar pattern… the role requires a balance of internal self-questioning while portraying confidence externally, and the CEO rarely has the ability to share that internal conflict with others.

Results!

In the third month following the changes, IMVU hit the inflection point – the transitional business pain stabilized and started producing positive results. Taking full control of the Creator and Reseller aspects of the economy meant customers could have a reliable experience, from purchasing tokens to receiving royalties for their content. As part of the better-regulated process, there were other bad actors, scams, and negative customer experiences that were eliminated. And since there were less variables in token supply and pricing, it was much easier to maintain stability in the value of the token, a huge win for Creators, the business, and customers that ultimately benefit from a vibrant Creator marketplace.

The change to a more tightly-controlled token economy, combined with other big initiatives that were engaging new customers, resulted in a significant wave of growth and record results for IMVU’s business.

Key Takeaways

  • Talking to customers is critically important! Deeply understand the core of their objectives and pain points, and make sure product changes solve for the customer’s needs.
  • Be mindful that customers won’t have the breadth or depth of information necessary to recognize the real benefit of some product decisions. Sometimes what seems to be an immediately unpopular product decision is necessary to deliver a better customer experience over the long-term.
  • Spend the time to get information necessary for confidence in decisions that can have significant impact, but also be humble, open to recognizing a mistake, and ready to adjust if the results aren’t there.
  • With a large enough customer base it becomes impossible to solve for everybody, as occasionally their needs will conflict. Be intentional in product decisions that make these tradeoffs, solving for the best long-term customer experience for the customers your business needs.
  • Unsupportive customers should be an exception… most of the time your decisions should delight your customers.

 

Do you have examples of product changes customers hated but ultimately produced a better experience for them? If so, I want to hear about them! Please leave a reply, below.

Posted on

Avoiding the Perils of A/B Split Testing

A/B testing is widely used in product development, popularized as a fundamental component of the Lean Startup  framework, and providing a scientific way of validating product and business improvements. The concept is simple… put some customers in the new experience, compare the results against customers that didn’t get the new experience, and better metrics validates the improvement. In reality, this process of validation is very complicated and there is no shortage of hazards leading you to poor outcomes.

Creating Information out of Data is Hard

IMVU had a culture of data-validated decisions from almost day one, and as a result we made it easy for anybody to create their own split test and validate the business results of their efforts. It took minutes to implement the split test and compare oh so many metrics between the cohorts. All employees had access to this system and we tested everything, all the time. A paper released in 2009,  Controlled experiments on the web: survey and practical guide, reinforced that split testing was the undisputed arbiter or truth. We were clearly on the right path. 

While the ability to self-assess progress created a very empowering culture, we were largely ill-equipped to understand the nuances of what the data actually meant. Years later we would start to better understand, we don’t know how much we don’t know.

First Know Why

The first opportunity to make a mistake with split testing is deciding to test in the first place. When creating a split test has a very low barrier, it is easy to err on the side of just testing everything so that you can have the data if you need it. But every test has a lot of hidden costs than come from false-positives, clarification of data, shiny-object distractions, inconsistent customer experiences, and additional opportunities for introducing bugs.

Recognizing that being a split test packrat has a real cost, there should be some requirement for incurring this cost. Are very least, answering the question, “What are the significant changes that will be made as a result of this test?” Additional pre-test work to specify what will be measured, and what results will determine success or failure can also go a long way towards ensuring time spent testing is valuable.

Test Implementation is a Project

IMVU had a great framework to make test implementation a seemingly simple task, with a few lines of code of creating a branch for the test experience, and leaving the current experience as the control. Again, this made creating tests seem deceptively easy, and left openings for measuring the wrong thing.

Often a split test is a cross-functional effort, with an engineer handling the implementation and the customer being any combination of a product manager, acquisition team, marketing representative, revenue officer, or generally interested party. In some cases, the interpretation of test data is done by another person altogether. Correctly understanding what the internal customer wants to know, capturing the right data, and converting that data into information ends up with many points of communication that must be accurate to deliver a valid test.

For example, the acquisition team wants to test a new landing page, simply reordering the registration fields because they think it will improve the registration completion rate. The engineer realizing this is a no-brainer takes the 15 minutes before lunch to create the quick test, two paths and the test is running. However, the registration page has both manual registration and sign in with a social network account, so the test is including a lot of users that are social logins, irrelevant to the registration fields. This subtle nuance means that the impact of the registration field changes will likely be lost as the irrelevant data acts as a damper. What the customer wanted to know isn’t what the test is answering, and it’s likely that nobody on the project knows there is an error.

The ease of creating a split test should not be conflated with delivering quality results from a test. Doing it right is a project and requires investment of resources consistent with any other project.

WTF Do These Results Actually Mean?

Assuming you were diligent in your experiment design, you captured all of the relevant data, and you avoided some of the common errors of A/B testing, you now need to make sense of the data. In the best cases, you’re looking at something like “the registration landing page increased conversions from 1.83% to 2.01%”, in the worst cases you find something like “customers are engaging with messaging feature 17% longer… but their lifetime value has dropped by 4%”, and now there is work to put together a narrative that explains the perplexing results.

In 2012 I read a paper, Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained, and I had what I like to call an, “oh shit” moment. Highly controlled experiments, run by companies with world-class, dedicated analytics teams were getting perplexing results that required substantial research to understand what was actually happening. What chance did we have of getting this right when we are running 15+ experiments a week with training consisting of a one page internal wiki version of, “A/B Testing for Dummies”?

The tl;dr summary of the paper, without deep consideration for the “why” behind the change in metrics, positive results may be antithetical to what you are actually trying to achieve.

The up-front work to limit the scope of the experiment and how it will be measured / interpreted can help, assuming you have the self control to ignore the data outside of scope. Often these perplexing results require follow-up experiments to better isolate cause and effect. I also highly recommend talking to customers – often qualitative insights from hearing their experiences can often help make sense of what the quantitative results were hiding.

You’re Biased. No, Really, You Are

I’m sure there are a lot of great reasons we humans are wired to think the way we do, and this wiring probably served us very well in many situations. However, humans also come standard with cognitive biases, built-in tendencies to make irrational decisions. Unfortunately, putting a bunch of effort into building something and then getting a giant pile of metrics is a perfect enabler for a cognitive biases and craptastic decisions.

While numerous biases are working against you, with a buffet of metrics one of the most common is the Texas sharpshooter fallacy, in which the all of the test metrics that are improvements over the control metrics are used to demonstrate the success of the test. With a 95% confidence rate, 1 out of 20 metrics tracked are expected to show a false positive improvement, so even an A/A test (two separate cohorts with identical experiences) would likely show “improvements”. Before we eliminated the practice of metric-sniping at IMVU, it wasn’t uncommon to hear somebody say something like, “my pet project to streamline registration didn’t change registration, but it does deliver a 5% improvement in [the completely unrelated] customer lifetime value, so we should keep it.”

There are process controls that can help reduce the potential impact of various biases, in particular around defining and constraining each test. However, being aware of these biases and encouraging a culture consistent with the dialectical method can help make better product decisions, even beyond interpreting test results.

Talk to Your Customers!

One of the biggest risks that come from over-reliance on split testing is seeing it as a more convenient method of getting customer feedback. Why spend 30 minutes on the phone with one customer when you can simply measure the actual actions of thousands of customers?

Looking at data and sending surveys may seem like an efficient use of time, but that highly structured approach is unlikely to surface critical customer insights. Metrics and surveys will often answer the “what”, but almost always miss the “why”, the most critical driver of valuable insights. There is no substitute for talking to your customers.

In the words of Steve Blank, “Get Out of the Building.”

 

I’m interested in hearing other stories where split testing has made an impact, either positive or negative. Please share a comment if you have one!

Posted on

Less Minimal, More Viable – Creating Better MVPs

I had the exceptional luck to work with Eric Ries at both the company that was his inspiration for The Lean Startup, as well as the company that was his catalyst for the change needed to build companies differently (and I hope someday I can convince Eric to release his insightful yet unpublished manuscript “The Bloated Startup” – maybe your tweets can help #EricPleasePublishTheBloatedStartup).

One of the fundamental ideas from The Lean Startup embraced by startups is the Minimum Viable Product (MVP), a product strategy that minimizes investment while maximizing learning and market validation. And while MVP is a great and seemingly simple concept, many startups fail to execute it successfully.

There was a time not too long ago when startups regularly burned many millions of dollars in years of stealth mode, building massive projects anticipating the use cases for all of their future customers, and the concept of releasing anything that wasn’t robust being heresy. A combination of those companies spectacularly imploding, investor expectations that companies achieve validation faster,  and the embrace of accepting failure while chanting the mantra “fail fast”, made the pendulum swing the other way.

The most common criticism of MVP is too often it is actually Mvp, where minimal is emphasized and viable is highly subjective, but leans towards not viable. It’s not that MVP is a bad concept, it’s simply difficult in practice. As a result, others have looked to redefine MVP – Jason Cohen proposed the SLC (Simple, Lovable and Complete), and Laurence McCahill proposed the MLP (Minimum Loveable Product), both emphasizing the importance of delighting customers to being “viable”, and reducing the opportunity to simply ship a broken experience to customers using “learning” as an excuse.

Rather that create another TLA, I’m offering guidance to make the implementation of MVPs more effective:

  1. The MVP Delivers Your Value Proposition
  2. The MVP is a Functional Product
  3. The MVP Provides Validation or Valuable, Intentional Learning

Let’s dig into each of these a little more..

The MVP Delivers Your Value Proposition

The MVP must deliver the customer value proposition for a subset of customers that will be early adopters. Delivering on your value proposition may seem obvious, but in the interest of trying to achieve the minimum investment, it can be overlooked.

Core to IMVU’s value proposition was connecting people through expressive avatars, which was initially delivered via a 3D client on the PC. IMVU had an early mobile product that connected customers by enabling messaging from their phone, and while we called it a mobile MVP, it wasn’t. Specifically, the messaging was text-based, so it didn’t deliver on avatars or expressive communication. Since it didn’t include avatars, it also didn’t test the business model, which involved selling items to stylize an avatar. Many existing customers liked the functionality provided, enabling them to perform some basic functions while not at a PC, but nobody would become a new customer on this product – is was simply a helpful add-on.

Later IMVU built a real mobile MVP, starting with the very basic set of functionality that enabled expression via your avatar, and the ability to purchase items for customization (also important to expression). Knowing the PC offering, the mobile MVP felt pretty bare bones, didn’t include 3D (something we knew customers wanted), but the customized avatar was present, enabling self expression. We gained new customers that only knew of IMVU as a mobile experience, and we validated that the business model worked. Eventually full 3D was added with a lot of other features that did an even better job at reinforcing the value proposition, but it was a pretty humble beginning.

The MVP is a Functional Product

The need to be minimal yet completely functional is where great product design comes in, recognizing that the best products are fully functional without being complex – simplicity delights customers.

The test I’m proposing is, without adding additional functionality, does your MVP continue to deliver value to your early adopters? Asking another way, can you imagine walking away from the MVP and seeing your early adopters still using it in 24 months?

When it comes to applying MVP to new product functionality for an established product, this simple but complete requirement is even more critical. I witnessed many MVP projects that shipped in half-done limbo as some customers liked it sort of, but it was broken, but not valuable enough to finish… the result is many rough edges and missed opportunities to delight customers.

The MVP Provides Validation or Valuable, Intentional Learning

One of the most disappointing results to hear from a failed MVP is, “we learned it didn’t work”. Aside from the obvious desire for projects to be successful and delight customers, this result represents a failure to intentionally learn. A great indicator this is happening is a product manager presenting data harvested after the fact, hand picking metrics that were not identified before the product was built, creating learning theater.

The MVP should reduce uncertainty, either by validating previous decisions or providing information necessary to make specific future decisions.

When building the MVP, there should be a clear hypothesis, identification of the metrics that will be used to gauge progress, the ability to capture those metrics, and an understanding of the critical decisions that will be influenced by the results. In addition to creating a discipline around honest assessment of progress, these requirements guide the team’s product development decisions.

 

Have you learned something valuable from building a MVP? I’d love to hear your story! Please leave a reply in the comment section.

Posted on

Leadership Requires Taking a Stand

For reasons I’ll cover in the future, I took a break from blogging. I did not intend to resume this week, and I did not expect that this would be my returning topic, but recent events have been a catalyst for me, and silence wasn’t really an option.

“We must take sides. Neutrality helps the oppressor, never the victim. Silence encourages the tormentor, never the tormented” – Elie Wiesel

I had no intention of covering politics on this blog. In speaking to the importance of leaders taking a stand, the public and obvious failure of Donald Trump was an example that could not be avoided. Further, it would be hypocritical for me to cover this topic without taking a stand myself.

A Leadership Softball

History’s losing flags on display in Charlottesville

Last weekend Nazis rallied in Charlottesville, spouting words of hatred and eventually murdering Heather Heyer. As much as the “Unite the Right” mob wants to claim that they are non-hateful and simply defending white heritage, chants like “Jews will not replace us”, “Fuck you, faggots”, and “Blood and soil” (which comes from Nazi roots), combined with marching under the flags and other imagery of Nazi Germany, clearly reveals the true intent. Describing these people as Nazis is not hyperbole – they are literally marching under the flag that many of our grandfathers gave their lives to defeat.

Denouncing the actions of Nazis is a leadership softball. In my home town of Berkeley, which is frequently (and sometimes fairly) considered socialist and crazy, Top Dog, an establishment that is staunchly libertarian and pro-free-market, fired an employee participating in the Nazi rally. The owner of Top Dog is not a delicate snowflake with hurt feelings, he took a stand against what was morally wrong and backed it up with actions.

In contrast, Donald Trump failed to even take a swing at this leadership softball. His initial comments appeared sympathetic or even supportive of the Nazis, receiving praise from former KKK leader David Duke. In an uncharacteristic two day delay to ensure, “what I said was correct, not make a quick statement”, Trump denounced the Nazi groups, in what appeared to be a forced reading of a prepared statement, days after most leaders (and bipartisan elected officials) took a firm stance against the Nazis. After what would have simply been considered a disastrous display of failed leadership,  yesterday Donald Trump destroyed what little credibility he may have garnered, when he effectively backtracked on his condemnation of Nazis, and seemed to equate our founding fathers to people that committed acts of treason waging war against the United States.

The Nazis in Charlottesville were largely Trump supporters, so Donald Trump making a clear and decisive statement against these hate groups may have come at a cost of losing some of their support. If you assume Donald Trump was simply attempting to remain neutral, the lack of a commitment against something so obviously anti-American (Nazis), was largely interpreted as support for the hate groups – this interpretation was echoed by politicians from both sides of the isle and from the hate groups themselves. After Trump’s impromptu shit show on Tuesday in which he doubled-down on his “many sides” to blame, it left little doubt where he really stands, although he still hasn’t displayed the leadership to clearly define his position.

Taking a Clear Stand

It’s necessary for leaders to take a clear stand on issues that impact their organizations, both to act as a beacon for what is expected for the organization, and to enable people to leave the organization if it is inconsistent with their own values.

A good way to test whether a company is committed to its cultural values is looking at how the company acts when holding to those values comes at a real cost. Similarly, leaders should be judged by their actions as they face adversity… are they willing to make personal sacrifices to maintain their integrity and live by their values.

In 2015, as CEO of IMVU, I made the decision to not allow the confederate flag in IMVU’s products. Some customers reacted unfavorably, some directed hostile remarks at me, and customer service received complaints. There was also some impact resulting from customers that had purchased or sold the products. I had expected all of that. And as much as I value freedom of speech, ultimately the value of IMVU being an inclusive community for millions of customers outweighed the impact of eliminating the emblem representing a war waged on the United States to defend the right to own humans. My actions were not big and bold, they were simply doing what I thought was the right thing given the values of the company and its community.

Regarding Donald Trump’s failed leadership, six business leaders have stepped down from presidential advisory councils, citing their own values as the primary motivation for distancing themselves from Trump. These leaders have clearly taken actions consistent with their personal values, and did so at a cost, as Trump quickly attacked and belittled these leaders the moment they stepped down. Those remaining on the presidential advisory councils may not explicitly support Trump’s defense of hate groups, but their continued support of him as a leader acts as an enabler, and casts doubts on their values or the ability to act consistently with their values. Trump’s top economic adviser Gary Cohn is reportedly ‘disgusted’ and ‘appalled’ by Trump’s responses this week, yet plans to remain in the administration, implicitly supporting Trumps behavior. Gary Cohn, who was born into an Eastern European Jewish family, continues to support a man that can’t denounce Nazis – as a citizen (a member of the US organization) I draw the conclusion that Cohn values tax reform and deregulation above what I would consider a non-starter, supporting somebody that can’t condemn hate groups.

Live Your Values

An organization’s culture and values are just pleasant little phrases in the employee handbook unless the organization reinforces the values in all actions, especially in tough times.

As a leader, if you are unwilling to state a position consistent with your values or sacrifice to take actions supporting those values, you don’t actually hold those values, or you are not a leader.

 

Posted on

Fairness in Employee Intellectual Property Rights

Silicon Valley is still in the Jurassic age when it comes to employee intellectual property rights.  It’s not that Silicon Valley has lagged behind others in this regard, but there has been no innovative leadership while there is ample opportunity to set an example for fair employee policies.

Before I was the CEO of IMVU, I was SVP Engineering, and in 2011 I drove an initiative to change the company’s policy regarding the ownership of employee side projects. At the time my basic argument was we were actively looking to hire employees that are builders, creators, tinkerers and then had a policy (like every other company) that oppresses the same qualities we actively sought.  The new policy created a path for employees to have guaranteed ownership of their side projects and be protected against any future claims from the company.  I detailed the outcome in my article IMVU’s Employee-Friendly Policy on Side Projects. (sadly no longer posted, but accessible via Wayback Machine). My hope was other companies would embrace and improve on this first step.

6 Years of Progress!

In the 6 years that followed,  there has been a massive wave of companies acknowledging that some of the best employees they can recruit are passionate builders that actively contribute to open source and hack on pet projects to feed their creativity and passion for learning new skills.  These same companies have changed their culture and employment agreements to support these employees by recognizing that traditional intellectual property assignment agreements are over-reaching.  Actually, none of that happened.

For the most part, the state of employment agreements and employee intellectual property rights hasn’t changed.  Many companies still have policies with far-reaching claims on anything the employee creates, at any time, even if not directly related to the business and whether or not company resources were utilized.  It doesn’t matter that some of these claims are not enforceable (in particular, California has much more employee-friendly laws), many employees would simply give up rather than incur the legal costs to defend their rights.

The result of the continued inconsistency between company policies and employee behavior is an awkward cultural and legal situation, where employees have side projects and sometimes kind of keep them secret and the company sort of doesn’t acknowledge the side work when it knows about it… a wink wink, nudge nudge arrangement until it isn’t, and the company decides it owns the employee’s thoughts.

I’ll take a moment to call out (and praise) a recent exception… GitHub recently introduced a policy to let employees keep their intellectual property.  GitHub’s policy is called Balanced Employee IP Agreement (BEIPA) and recognizes that the employee has rights to projects that are not related to the company business, and also that “free time” and “company time” is fuzzy (the policy doesn’t explicitly state that employees can use company resources, but it also doesn’t claim rights either).

The Challenge of Change

As I went through the process of changing an industry-standard policy, I gained a much better understanding of the challenges. Ultimately the challenge of innovation in these policies comes down to no perceived upside for the company with fear of embarrassing failures from the innovation

Standard Employee Agreements (which include assignment of intellectual property) are heavily weighted in favor of the employer and, since they are pretty much the same at every company, there is no competitive market and little reason to change. The company’s fear of losing out on an amazing invention can also come into play, with concerns that the company will forfeit rights to what could have been a game-changing development (who wants to be the idiot that let go of the billion dollar idea?). And finally, lawyers… corporate counsel provides tried-and-true boilerplate Employee Agreements, and the same corporate counsel that reviews the policy change is typically risk-averse, seeing rights-releasing changes as mostly downside with unknown benefits.

I found that most of the challenges in changing this policy were key stakeholders taking a “why we can’t” approach instead of a “how can we” attitude.  Now having 6 years of experience with the policy, I can unequivocally state that it resulted in no downside for the company and only goodwill for the employees.

Getting to Fair Employee IP Rights

I believe the first critical step in getting to fair employee intellectual property rights is bringing awareness that change is desired and possible.  Without a push from employees, it’s too easy for employers to just keep doing things the way they’ve always been done.

If you are an employee that would value a more equitable arrangement around intellectual property rights, let your employer know!  As a starting point for what is possible, point them to the improvements made at IMVU or GitHub.  Make an offer to your employer to promote the company’s leadership in this area and use it as a recruiting tool for creative talent.  If you are interviewing with a company, ask about employee IP rights – if this becomes a common topic from candidates, HR (recruiting) will see the value in making a fair policy be a benefit.

We’re seeing progress in other areas that have similar challenges around change… I am excited that some Silicon Valley companies are establishing or updating their policies to consider employee fairness around stock option plans that actually help employees keep the rewards from their contributions.  As these companies intentionally make the choice to not just do the same thing every company has done before, I encourage them to use that same open-minded process to examine their employment agreements and create policies that are fair to the employees they strive to attract.

This guy wrote your boilerplate IP Agreement

As a leader in a company, consider whether the policy you have today was intentional, reflecting the culture and values of what you are trying to build, or if the policy is just a generic hand-me-down from the corporate dinosaurs of the past. If you experience too many challenges around making sweeping changes, at least make incremental changes and try to use them as a differentiator for your company (really, go on Quora or Hacker News – potential employees looking for companies with fair IP policies are left with almost no good examples… your company could stand out).

As more companies show that employee fairness is a differentiator that attracts and retains great talent, it will push others to do improve their policies to be competitive.

Know of other companies that have great Employee IP rights?  Think Brett is crazy and giving away all of a company’s value?  Leave a comment!

Posted on

Interviewed on #ModernAgileShow

I recently had the pleasure of being interviewed by Joshua Kerievsky on the #ModernAgileShow, where we talked about a lot of my experience working at IMVU, ranging from the early days of Continuous Deployment (without all of those fancy automated tests or cluster immune systems) to changes in experiment systems and challenges of building a culture where people feel safe.  I also provide some insights into the sausage making of The Lean Startup.

In the interest of accuracy, my title in the video should be “former CEO of IMVU“.

For more information about Josh’s work to setup agile processes and cultures independent of a specific framework, check out the Modern Agile website.

On a semi-related note, Josh mentioned that the original video of Timothy Fitz presenting on Continuous Deployment at IMVU: Doing the impossible fifty times a day was lost as the result of server corruption…. if anybody happens to have a local copy please let me know – it would be great to restore this historic presentation for the Interwebs!