We started anew, like a seedling in the sand (source)

A team reborn after the fiery departure of its misanthropic guru

We stared at the flickering embers of our previous life. We wouldn’t survive another journey through the flames. We had one choice: grow or perish. We chose to grow.

Time is the school in which we learn, time is the fire in which we burn —Delmore Schwarz (image source)

Some years ago I was witness to — and part of — the death and rebirth of an organization. It had previously subsisted on an exclusive diet of hero worship.

Our hero lived long enough to see himself become the villain.

We failed to return him to the fold. Without our hero we now faced a do-or-die deadline that meant life or death for the company. We had two immediate priorities.

First, we had months to finish a project that had struggled in limbo for years. If we didn’t do this we’d all be out of work very soon. This would mean the dreaded “crunch”: eighty, ninety, one hundred hour work weeks for each team member to get us over the finish line. That’s a story for another day.

This story is about our other immediate priority:

We had to ensure this couldn’t happen again.

This essay describes the lessons we took away from our hero’s fall from grace. It’s about how reinvented our organization from the top down, and what we learned along the way.

Our first failure was a failure of management. Certainly our ex-hero’s manager had failed him in myriad ways. Firing his manager was a start, but the problem didn’t emerge in isolation.

We reviewed our management structure. How did we as an organization allow such a catastrophe?

We had structured the group along product lines. Each product line had its own team led by a product manager. The team members were technical resources, each responsible for a different part of the product.

For software products that meant supplying a team member for each technical layer. A database administrator, back-end developer, front-end developer, and user a experience designer.

Together the team was responsible for the complete product.

Assembly lines: good for factory production. Bad for creative work. (source)

We had grown this way organically from our first product. When we added new products, we ramped up new teams.

This was the first ingredient in our demise.

Organizing ourselves this way created silos. Across silos each staff member had taken different approaches when solving the same problem.

The most salient issue to the loss of our hero was isolation. Other team members couldn’t come and help our hero without significant ramp-up time.

Colleagues from other silos didn’t know the product. Colleagues within his silo didn’t know the tech.

Non-technical product managers were not effective at supervising technical work. Staff weren’t effective at collaborating within silos because they each worked on a different part of the product.

Enter the matrix.

No, not this matrix. (source)

We realigned ourselves around component lines into a “matrix organization” structure.

For software, that meant a database team, a back-end team, a front-end team, and a user experience team.

Each team now had a technical lead who could understand and provide meaningful feedback on their work. The product managers coordinated their projects across the technical teams.

This prevented individuals from creating bottlenecks on their products. We could shift more resources onto delayed components much more easily.

It also fostered much stronger collaboration. The team members could bounce ideas off each other and get useful feedback. Other people on their team understood their work.

Visibility was another key benefit.

Where there is smoke, there is probably fire. But what if you can’t see the smoke? (source)

A hard lesson we had learned from our fallen hero was that he was never really the hero we’d thought him to be. He established a good reputation with solid work early on.

Later he abused the silo effect to keep other people out of his sandbox. He blustered his way around any inspection of his work. His manager didn’t have the technical knowledge to second-guess him and took him at his word.

Once we pulled back the curtain we realized what he’d really been producing. He had taken a lot of shortcuts and ultimately produced a worthless mess.

Who will guard the guards themselves?


The new matrix structure kept our technical staff from spinning out of control. Their new technical lead held them accountable for producing good work. They also learned to hold each other accountable, since they were now on the hook for supporting each other’s product.

It also provided at least two management reviewers for every product and component: the technical lead and the product manager. The product managers still didn’t have technical know-how but were useful for spotting bottlenecks. They also provided a check on the tech leads.

Our product lifecycle process was not sustainable. We had used a “waterfall” approach.

It’s called waterfall because everything’s going great until you sail off the edge. (source)

We spent weeks or months with stakeholders iterating through mock-up products. This generated dozens or hundreds of pages of documents describing all the “required” features.

Every feature had equal weight, and the process was all-or-nothing: the product was not ready to release until every feature was done.

The problem we encountered with this approach — the problem many encounter — is that the rest of the world didn’t stand still while we were building the product.

Our customers wanted more. They saw what other people had done, they got new ideas for new features, and some old features became irrelevant.

The pejorative label for this is “scope creep.”

I prefer the term “reality.”

There must be many former Blockbuster execs who cursed the “scope creep” of the movie rental business. Surely many Kodak product managers damned “scope creep” in the imaging industry as they shuttered their factories.

Our product management process did not align with the reality our customers’ needs. So we switched it up.

This was our entry in to the world of Lean process management. For our software products we went Agile.

We focused our releases on creating the greatest differentiation from existing products. We started buying a lot more than we built.

It was possible over time that the greatest value to add could be from customizing a commodity component. But this was many, many cycles after minimum viable product release.

Our customers didn’t need or want completely bespoke products, so most time spent re-inventing commodity components was time wasted.

Scope “creep” became an opportunity to further differentiate ourselves from existing products. We baked everything into two-week cycles. Innovation became low-risk and changing direction became easy.

But innovation was only possible in an environment where everyone was willing to work together.

I was married by a judge. I should have asked for a jury.

Groucho Marx

We learned to nip problem behavior in the bud. Our experience with our hero-turned-villain burned this lesson into my psyche.

At a more recent job, one of my leads had a hotshot young employee who had a lot to offer but also a lot to learn. The lead provided thorough and considerate feedback on his first performance evaluation.

The employee rejected the feedback. He escalated to me, and to my boss. He felt that his performance outweighed his behavior. We responded with a kind but firm rejection of this thesis. The employee quit the next day.

We replaced him with a new junior resource who was willing to learn. Everyone was happier.

What do you mean, nobody’s perfect?! I’m perfect! (source)

The feedback process is not unidirectional. I reach out individually to the direct reports of each of my leads to see if they have feedback. I encourage my leads and their reports to provide feedback to my boss on my performance.

This is a continuing process, but it’s also included in formal performance evaluations.

The world of work is changing.

A lot of people are just waking up to the fact that they are accountable for their behavior.

Some companies tolerate the insufferable genius because his or her performance is so good that it excuses anything.

Let’s be honest: it’s usually “his” performance.

This begs several questions. What behavior isn’t tolerated? What is the acceptable work product to bad behavior ratio? Who must be bothered by the behavior for us to take notice?

If you tolerate obvious and public displays of toxicity, you should expect that your team will not feel empowered to report more subtle misbehavior. Bad behavior happening under the surface is even more damaging.

Ultimately, a very destructive work culture — such as one that accepts harassment — can result.

I always establish clear boundaries on acceptable behavior. When a team member crosses the boundary I make clear that a repeat performance will result in a specific consequence.

Sometimes they don’t listen.

I follow through.

This is usually all it takes. The person gets one of two wake up calls. In the best case, they learn that their behavior was unacceptable and reform themselves.

Sometimes they learn instead that we will not tolerate their behavior, and they leave. Either way, we’re both better off.

Defects are not free. Somebody makes them, and gets paid for making them.

W. Edwards Deming

People don’t scale, but teams do. We had allowed our hero to become a bottleneck. It turned out that he was a paper tiger, but even if he’d been all he claimed it would have been wrong to depend only on him.

We had lacked all ground rules for how to contribute to our product. Everyone built their own pieces in isolation without any guidelines or guard rails. We had no formal processes.

It was nearly impossible to bring new people into the team silos. New team members needed months of one-on-one training before they could contribute.

Instead of tackling this problem head-on we’d taken the easy way out. We simply added new teams to build support new products.

We chalked it up to growing pains and assumed we’d figure it out later.

This was bad for us and it was bad for our staff. People couldn’t jump into new, exciting projects. They had to spend all their time supporting their existing work. This contributed to the crash-and-burn that we experienced.

Paradoxically, our ex-hero had fomented this approach. He and others in our organization saw silos as job security: if they alone could support a product, they’d be set for life.

Job security. (source)

The problem is that security has two sides. A gate can keep you out or keep you in — it depends on who holds the keys. We had allowed our people to build gates to which nobody had a key.

After we realigned our teams we re-invented our team processes. We collaborated across teams to define company-wide boundaries for common tasks.

Each team was also delegated responsibility to create and document their own internal processes. Managers reviewed these processes to ensure that no team became too idiosyncratic.

Standardizing and documenting our processes and products allowed us to easily on-board new staff. It also meant people could move between teams which opened up a new avenue for career growth.

In fact, we encouraged staff to move between teams from time to time. This helped to spread good ideas across the organization.

It also allowed our products to get “fresh eyes” as people who had only looked at the product from one perspective moved to support another. Staff gained an appreciation for what made life easier for each other.

Processes and documentation aren’t a panacea, but they provide a baseline. This baseline enables iterating on and improving how you work, not just what you produce.

Hopefully you don’t have to improve as much as we did, but everybody starts somewhere.

The sun also rises. (image by the author)

People are the most expensive resource in a knowledge organization. The mistakes they make are very costly. And, as Deming pointed out, they get paid while making them.

Mistakes, then, are a kind of investment.

It’s up to us all to ensure we get a return on that investment. If we do nothing the return will be a legacy of failure. It will make customers unhappy. It will burn out staff. It can end a company.

Alternatively, a mistake can provide the insight that saves a company.

Most mistakes will fall somewhere between these extremes. We can put our finger on the scale if we act in time. We can refuse to allow history to repeat itself.

To succeed requires honest introspection: was the mistake inevitable? Were we aware of the risks we ran? Where did we as individuals fail? Where did our team dynamic fail?

Where did I fail?

We may have been able to avoid many of the consequences we suffered, had we done things differently. It is unknowable. We did reform ourselves just in time to avoid total conflagration.

We learned a great many more lessons than those shared here. We anonymously surveyed staff satisfaction. We enforced more vacations. We created specific quality assurance processes. We re-worked our performance evaluation process. We changed our reward structure to discourage hero syndrome.

All this, and more.

We knew we had to learn from our mistakes or be doomed to repeat them. So we learned. We’re not mistake-free, and we’re still learning today.

I hope that you, too, can learn from our mistakes. Most of all you should learn from your own mistakes. After all, the phoenix only arises every 500 years, and we already used it up.

Engineering Manager @Netflix

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store