How to Respond After Leaking Your Customer’s Data

The most recent consumer-hostile disclosure of an account breach was Uber’s leaking of 57 million accounts almost a year ago. I’d like to say this is an extraordinary event, but much like a favorite character getting killed in Game of Thrones, companies leaking customer data is just another regular occurrence we’ve come to expect. What continues to surprise me is how badly so many companies screw-up their response to a breach. The one principle that should guide companies following a breach is, “make the decisions you would want a company to make if it was your account that was compromised.”

And sure, it’s easy to point fingers when it’s not you in the hot seat, so I’ll use the breach I managed as an example… The breach I was responsible for was in September 2015, when I was CEO of a company that had over 100 million registered accounts.

Initial Response

The breach was caught around 11:00 PM at night… within a couple of hours we had a fire-team of employees in the office. The priority was confirming that the breach was indeed fully contained, and then validating we understood the full extent of the breach. We wanted to communicate to customers as quickly as possible, and we wanted to be able to accurately convey the amount of exposure. Every other project was de-prioritized and employees were working 24/7 on projects related to the breach.

Thanks to some security precautions we had in place, we were able to detect the breach in real-time, limit the data that was accessed, and understand exactly what data was exposed. Also, due to the nature of the data that was accessed, the actual customer exposure was minimal (e.g. no credit cards, social security, addresses)… assuming the attacker had planned to use the data for malicious purposes, the actual value of that data was extremely low.

As we reached morning, we contacted law enforcement and legal counsel, both of which informed us that the data exposed was insignificant in terms of risk. We were also told that, because of the type of data accessed, there was no requirement to disclose the breach.

While we had a pretty solid understanding of what happened as part of the breach, we didn’t want to be overly confident, so we continued the process of going through hundreds of servers and employee computers to look for anything that might have been missed, a process that took a little over two full days.

The Ransom

Within 24 hours of the breach I started receiving emails that threatened to release the customer data and publicly announce the breach if we didn’t pay a sum of money. My response to the blackmail was letting them know I would consider their proposal, but ultimately the damage they would do is to customers that didn’t deserve to be exploited, and to employees, good people that already feel a ton of weight from the responsibility. They gave me a few days to make a decision.

Talking to Our Customers

After we had confidence that we had contained the breach, removed any attack vectors, and fully understood the data accessed, we were ready to talk to our customers. Less than 72 hours had passed, but it felt like an eternity getting to this moment.

We posted to our forums and messaged our customers individually with the details of the breach, specific data accessed, how that data can be used, and what steps to take (on our service and others) to protect against any further attack. We also disclosed that the hacker had tried to extort money in exchange for silence.

While I can’t say that any customer was pleased that the exploit occurred, many responded very positively to our handling of it. Earlier that year credit card and health care breaches of highly-sensitive data took many months to be announced, so many of our customers appreciated how quickly we moved to keep them informed.

Evidently the hacker didn’t read our forum post, as the next day they gave me the final warning that they were about to announce the breach to our customers and the media. I informed the hacker that we would not be paying the ransom, reminded them that the people they will hurt don’t deserve it, and pointed them to the forum posting fully disclosing the breach, accessible to all of our customers and the media.

Post Breach

Through a process of many, many postmortems and follow-up action items, the company continued to improve security in several areas, projects that extended many months. We understood exactly how the breach occurred, and the human component that enabled the breach. What we explicitly didn’t do is punish or threaten anybody – throughout the whole process we made all employees feel safe, which enabled people to be fully transparent and quickly disclose their mistakes, a critical aspect of quickly understanding how the breach occurred.

The moment that sticks out in my mind the most was an email I received from an employee in response to a detailed summary of the events I sent to the company. That employee expressed that they had never been so proud to be at a company, in the integrity we demonstrated to our customers, and the unwavering support for the employees. It was one of those emails that CEOs move to their “save forever” folder.

Key Takeaways

While there are a lot of opportunities for companies to make customer data more secure, the unfortunate reality is even the companies with the best security practices experience breaches – this is going to happen. However, a few steps can provide better outcomes for all parties:

Treat your customers as you would want to be treated.
Make your employees feel safe. Fearful employees will conceal critical information that is necessary to fully understand the problem.
Don’t negotiate with criminals. It’s bad for your customers, there is no way to enforce the criminal’s end of the agreement, and the deception is likely to be revealed at some point. Perhaps one acceptable variation on this takeaway is, if you do negotiate with criminals in the interest of your customers (e.g. to get details about how the leak occurred), still be transparent with your customers and disclose that a transaction occurred.
Do the follow-up work. After an exhausting amount of effort getting past the initial breach it’s easy to feel like your work is done… make sure all of the known exploit vectors are eliminated.

Have you been impacted by a company’s data breach? I’d like to hear about your experience – please leave a comment!

August 16, 2017August 17, 2017

Leadership Requires Taking a Stand

For reasons I’ll cover in the future, I took a break from blogging. I did not intend to resume this week, and I did not expect that this would be my returning topic, but recent events have been a catalyst for me, and silence wasn’t really an option.

“We must take sides. Neutrality helps the oppressor, never the victim. Silence encourages the tormentor, never the tormented” – Elie Wiesel

I had no intention of covering politics on this blog. In speaking to the importance of leaders taking a stand, the public and obvious failure of Donald Trump was an example that could not be avoided. Further, it would be hypocritical for me to cover this topic without taking a stand myself.

A Leadership Softball

History’s losing flags on display in Charlottesville

Last weekend Nazis rallied in Charlottesville, spouting words of hatred and eventually murdering Heather Heyer. As much as the “Unite the Right” mob wants to claim that they are non-hateful and simply defending white heritage, chants like “Jews will not replace us”, “Fuck you, faggots”, and “Blood and soil” (which comes from Nazi roots), combined with marching under the flags and other imagery of Nazi Germany, clearly reveals the true intent. Describing these people as Nazis is not hyperbole – they are literally marching under the flag that many of our grandfathers gave their lives to defeat.

Denouncing the actions of Nazis is a leadership softball. In my home town of Berkeley, which is frequently (and sometimes fairly) considered socialist and crazy, Top Dog, an establishment that is staunchly libertarian and pro-free-market, fired an employee participating in the Nazi rally. The owner of Top Dog is not a delicate snowflake with hurt feelings, he took a stand against what was morally wrong and backed it up with actions.

In contrast, Donald Trump failed to even take a swing at this leadership softball. His initial comments appeared sympathetic or even supportive of the Nazis, receiving praise from former KKK leader David Duke. In an uncharacteristic two day delay to ensure, “what I said was correct, not make a quick statement”, Trump denounced the Nazi groups, in what appeared to be a forced reading of a prepared statement, days after most leaders (and bipartisan elected officials) took a firm stance against the Nazis. After what would have simply been considered a disastrous display of failed leadership, yesterday Donald Trump destroyed what little credibility he may have garnered, when he effectively backtracked on his condemnation of Nazis, and seemed to equate our founding fathers to people that committed acts of treason waging war against the United States.

The Nazis in Charlottesville were largely Trump supporters, so Donald Trump making a clear and decisive statement against these hate groups may have come at a cost of losing some of their support. If you assume Donald Trump was simply attempting to remain neutral, the lack of a commitment against something so obviously anti-American (Nazis), was largely interpreted as support for the hate groups – this interpretation was echoed by politicians from both sides of the isle and from the hate groups themselves. After Trump’s impromptu shit show on Tuesday in which he doubled-down on his “many sides” to blame, it left little doubt where he really stands, although he still hasn’t displayed the leadership to clearly define his position.

Taking a Clear Stand

It’s necessary for leaders to take a clear stand on issues that impact their organizations, both to act as a beacon for what is expected for the organization, and to enable people to leave the organization if it is inconsistent with their own values.

A good way to test whether a company is committed to its cultural values is looking at how the company acts when holding to those values comes at a real cost. Similarly, leaders should be judged by their actions as they face adversity… are they willing to make personal sacrifices to maintain their integrity and live by their values.

In 2015, as CEO of IMVU, I made the decision to not allow the confederate flag in IMVU’s products. Some customers reacted unfavorably, some directed hostile remarks at me, and customer service received complaints. There was also some impact resulting from customers that had purchased or sold the products. I had expected all of that. And as much as I value freedom of speech, ultimately the value of IMVU being an inclusive community for millions of customers outweighed the impact of eliminating the emblem representing a war waged on the United States to defend the right to own humans. My actions were not big and bold, they were simply doing what I thought was the right thing given the values of the company and its community.

Regarding Donald Trump’s failed leadership, six business leaders have stepped down from presidential advisory councils, citing their own values as the primary motivation for distancing themselves from Trump. These leaders have clearly taken actions consistent with their personal values, and did so at a cost, as Trump quickly attacked and belittled these leaders the moment they stepped down. Those remaining on the presidential advisory councils may not explicitly support Trump’s defense of hate groups, but their continued support of him as a leader acts as an enabler, and casts doubts on their values or the ability to act consistently with their values. Trump’s top economic adviser Gary Cohn is reportedly ‘disgusted’ and ‘appalled’ by Trump’s responses this week, yet plans to remain in the administration, implicitly supporting Trumps behavior. Gary Cohn, who was born into an Eastern European Jewish family, continues to support a man that can’t denounce Nazis – as a citizen (a member of the US organization) I draw the conclusion that Cohn values tax reform and deregulation above what I would consider a non-starter, supporting somebody that can’t condemn hate groups.

Live Your Values

An organization’s culture and values are just pleasant little phrases in the employee handbook unless the organization reinforces the values in all actions, especially in tough times.

As a leader, if you are unwilling to state a position consistent with your values or sacrifice to take actions supporting those values, you don’t actually hold those values, or you are not a leader.

April 19, 2017January 25, 2018

Firing People Respectably

Firing people is perhaps the most unpleasant responsibility that comes with being a manager. I’ve read many articles on “the right way” to handle firing, but my experience has taught me every case is different, and even following the best advice can result in a challenging interaction.

I’ve created guidelines for myself that feel fair (this is how I want to be fired), and I accepted that firing is unpleasant for everybody involved, so it’s ultimately about making the best out of a shitty situation.

My guidelines come from the perspective of a culture I want to see in a company, not the legal perspective (which tends to err on the side of corporate protection over recognizing the human components).

Guidelines for a Firing Manager

My guiding principle, be respectful, helping the employee retain their dignity, drives these guidelines:

Always remember you’re firing a person, not a resource. In almost every case being fired is an emotionally painful situation, and being mindful that you are firing a person, with feelings, fears, and personal responsibilities that will be compromised as a result of job loss. People react unpredictably in emotion-filled situations. As the firing manager it is important to be respectful through the whole process and be balanced in responses to the other person’s (re)actions.
Don’t get into a detailed discussion. A common pattern is the person being fired will want to get into the details about the decision to fire. The firing discussion should be efficient (there is nuance in balancing not being insensitively fast vs. dragging out the pain). The manager should absolutely provide a high-level explanation, and the next steps (ideally the company has a standard document that explains the issues that will be important to the employee), but the person being fired is very unlikely to actually hear a detailed discussion – they are too emotional to process it. If a person being fired wants to get into details, I suggest scheduling coffee the following week, giving them enough time to figure out what questions are really important and getting past the initial shock so they can be receptive to the answers.
Never discuss individual details with others. When a person is fired, other employees frequently want to understand more details. It can be tempting to want to bring others into the loop or calm an underlying “am I next?” fear they may have by sharing the details, but it is disrespectful to the person being fired (it’s also probably a liability for the company). Instead, have a culture that is transparent about the process (why and how) people are fired, while never discussing an individual’s specific situation.

Reasons for Firing

The reasons for firing an employee generally fall into three categories: performance, role eliminated, and violating the company relationship. Each impact the person being fired, other employees, and possible outcomes differently.

Performance Problems

When an employee is under-performing it is their manager’s responsibility to make that employee successful and, if that fails, fire the employee. An employee’s performance should be a regular discussion with their manager, and missing expectations should be made explicitly clear, along with clarity around the exact expectations and a plan to improve. If the improvement doesn’t happen, the firing discussion should be more of a final conclusion to the mutual recognition of the problem, with both parties aligned on the shared data. My rule is, “if the employee was surprised they were fired for performance reasons, this is a failure of their manager”.

Role Change

The role change scenario is one where the company’s requirements or constraints have changed and an employee is no longer appropriate for the role. I’m including layoffs / downsizing in this category (not being able to pay people is a constraint). A commonality in these firings is it includes qualified, successful employees. This is the one firing scenario where additional insights into the decision can be shared with other employees, as the decision is not about an individual (but be sure that the role change is the real reason for the firing, otherwise it will eventually result in distrust from employees).

A role change specific to an individual feels the most personal for the person being fired and can be hardest for other employees to understand. The message of “great for previous role, wrong skills for what the company needs going forward” is easy to say, harder for employees to process, often because a good employee will be leaving, and many employees won’t have the insights into the need for the change (or may simply disagree). The best analogy I’ve been able to come up with is sports teams, where a great player may be traded to make room for a player that has different skills that make the team better as a whole (as in Moneyball, where trading stars for players that just got on base resulted in a better team).

When a role change is impacting many people (typically driven by financial situations or discontinuing a product / service), explaining to the people impacted can be more comforting than when it is a single role, since the reasons don’t feel as personal (make no mistake, for the people being fired the impact will feel very personal, it just won’t feel like they were individually targeted).

Violating the Company Relationship

Every company has it’s own unique culture, principles, rules, and expectations in the relationship with each employee, and between employees. I’ll use “don’t steal” as an example, since I this is probably a common deal-breaker even in the most toxic environments.

When there is a violation of the relationship, the employee needs to be fired, otherwise the company is signaling that it isn’t an actual expectation of the relationship, or perhaps worse, that enforcement is selectively applied. In this firing the employee should not be surprised, however an employee willing to violate the relationship in one dimension is likely willing to double down and deny their responsibility in the situation. Unfortunately, this is one of those nobody wins outcomes that, as a manager, you simply need to get thorough it, look for the learning opportunity, and move-on.

A particular challenge in this type of situation is the inability to offer an explanation to other employees, especially if the violation was concealed. Using the stealing example, the company could have liability is disclosing the violation to others, so employees just see somebody fired for no apparent reason. As recommended in my guidelines above, if your company has a (trusted) transparent culture around how and why people get fired, many may infer that it was either a performance problem or violation, which a better outcome than the firing feeling random.

Management Failures

Employment is a relationship, and the manager and company have to acknowledge their responsibility in the failed relationship, both in why it failed and the importance of properly handling the failure.

Passing the Buck

If there are other existing opportunities where the employee could be successful at the company, that can provide a solution that is both a win for the employee and the company. However, since firing is so unpleasant, managers should be challenged to understand if they are diverting the problem to somebody else or do they really feel the employee is best for the opportunity. Ask the question, “if the employee didn’t work here but was applying for the new opportunity, would you hire them?” If the answer isn’t a confident, “yes”, the manager is likely passing the problem to somebody else. Another red flag is the creation of a new role for an employee that would otherwise be fired… in almost every case I’ve experienced, this is a manager avoiding a tough (and necessary) decision.

Performance Improvement Plans

Performance Improvement Plans (known as “PIPs” in HR speak) are formal documentation explaining the employee’s performance problem, the expectations, a process to improve and a success evaluation date. On the surface this is all great – issues that should have been discussed in 1:1 meetings. When used as a tool with the intention of making the employee successful, PIPs can be really helpful in providing clear expectations.

The dark side of PIPs is when they are used as an HR cover your ass maneuver, in which the employee’s fate has already been decided but, because of risk or liability, there is a desire fore the company to have ample documentation around the termination. Don’t do this. When a firing outcome has been determined, fire the employee. Dragging-out a process or giving false hope is disrespectful, and arguably cruel.

Learning from Failure

A firing may not reflect a failure, it might actually be the best decision for the company and perhaps even for the person being fired. However, all firings can be an opportunity for the company to learn and improve its processes. If it was a new employee, try to understand how the interview / hiring process could have identified the issue. With longer-term employees, look for training opportunities (for the employee or management) that could have resulted in a more successful outcome. Understand when the firing should have happened and what should be done next time. Since firing has such a big impact to both the employee and the company, there is value in continually improving the process to reduce or avoid any firings that could have been saves.

Have you been on either end of the firing process and have suggestions for improving how it gets handled? Please leave a comment!

April 5, 2017January 5, 2022

Fairness in Employee Intellectual Property Rights

Silicon Valley is still in the Jurassic age when it comes to employee intellectual property rights. It’s not that Silicon Valley has lagged behind others in this regard, but there has been no innovative leadership while there is ample opportunity to set an example for fair employee policies.

Before I was the CEO of IMVU, I was SVP Engineering, and in 2011 I drove an initiative to change the company’s policy regarding the ownership of employee side projects. At the time my basic argument was we were actively looking to hire employees that are builders, creators, tinkerers and then had a policy (like every other company) that oppresses the same qualities we actively sought. The new policy created a path for employees to have guaranteed ownership of their side projects and be protected against any future claims from the company. I detailed the outcome in my article IMVU’s Employee-Friendly Policy on Side Projects. (sadly no longer posted, but accessible via Wayback Machine). My hope was other companies would embrace and improve on this first step.

6 Years of Progress!

In the 6 years that followed, there has been a massive wave of companies acknowledging that some of the best employees they can recruit are passionate builders that actively contribute to open source and hack on pet projects to feed their creativity and passion for learning new skills. These same companies have changed their culture and employment agreements to support these employees by recognizing that traditional intellectual property assignment agreements are over-reaching. Actually, none of that happened.

For the most part, the state of employment agreements and employee intellectual property rights hasn’t changed. Many companies still have policies with far-reaching claims on anything the employee creates, at any time, even if not directly related to the business and whether or not company resources were utilized. It doesn’t matter that some of these claims are not enforceable (in particular, California has much more employee-friendly laws), many employees would simply give up rather than incur the legal costs to defend their rights.

The result of the continued inconsistency between company policies and employee behavior is an awkward cultural and legal situation, where employees have side projects and sometimes kind of keep them secret and the company sort of doesn’t acknowledge the side work when it knows about it… a wink wink, nudge nudge arrangement until it isn’t, and the company decides it owns the employee’s thoughts.

I’ll take a moment to call out (and praise) a recent exception… GitHub recently introduced a policy to let employees keep their intellectual property. GitHub’s policy is called Balanced Employee IP Agreement (BEIPA) and recognizes that the employee has rights to projects that are not related to the company business, and also that “free time” and “company time” is fuzzy (the policy doesn’t explicitly state that employees can use company resources, but it also doesn’t claim rights either).

The Challenge of Change

As I went through the process of changing an industry-standard policy, I gained a much better understanding of the challenges. Ultimately the challenge of innovation in these policies comes down to no perceived upside for the company with fear of embarrassing failures from the innovation

Standard Employee Agreements (which include assignment of intellectual property) are heavily weighted in favor of the employer and, since they are pretty much the same at every company, there is no competitive market and little reason to change. The company’s fear of losing out on an amazing invention can also come into play, with concerns that the company will forfeit rights to what could have been a game-changing development (who wants to be the idiot that let go of the billion dollar idea?). And finally, lawyers… corporate counsel provides tried-and-true boilerplate Employee Agreements, and the same corporate counsel that reviews the policy change is typically risk-averse, seeing rights-releasing changes as mostly downside with unknown benefits.

I found that most of the challenges in changing this policy were key stakeholders taking a “why we can’t” approach instead of a “how can we” attitude. Now having 6 years of experience with the policy, I can unequivocally state that it resulted in no downside for the company and only goodwill for the employees.

Getting to Fair Employee IP Rights

I believe the first critical step in getting to fair employee intellectual property rights is bringing awareness that change is desired and possible. Without a push from employees, it’s too easy for employers to just keep doing things the way they’ve always been done.

If you are an employee that would value a more equitable arrangement around intellectual property rights, let your employer know! As a starting point for what is possible, point them to the improvements made at IMVU or GitHub. Make an offer to your employer to promote the company’s leadership in this area and use it as a recruiting tool for creative talent. If you are interviewing with a company, ask about employee IP rights – if this becomes a common topic from candidates, HR (recruiting) will see the value in making a fair policy be a benefit.

We’re seeing progress in other areas that have similar challenges around change… I am excited that some Silicon Valley companies are establishing or updating their policies to consider employee fairness around stock option plans that actually help employees keep the rewards from their contributions. As these companies intentionally make the choice to not just do the same thing every company has done before, I encourage them to use that same open-minded process to examine their employment agreements and create policies that are fair to the employees they strive to attract.

This guy wrote your boilerplate IP Agreement

As a leader in a company, consider whether the policy you have today was intentional, reflecting the culture and values of what you are trying to build, or if the policy is just a generic hand-me-down from the corporate dinosaurs of the past. If you experience too many challenges around making sweeping changes, at least make incremental changes and try to use them as a differentiator for your company (really, go on Quora or Hacker News – potential employees looking for companies with fair IP policies are left with almost no good examples… your company could stand out).

As more companies show that employee fairness is a differentiator that attracts and retains great talent, it will push others to do improve their policies to be competitive.

Know of other companies that have great Employee IP rights? Think Brett is crazy and giving away all of a company’s value? Leave a comment!