Hexawise started a book club in 2018. The book club evolved from a Slack channel for sharing interesting articles to help people improve their technical knowledge into a series of thought-provoking readings and discussions on a regular basis. The discussions have been more enjoyable and impactful than any of us anticipated.

Since our book club started, we’ve been able to build context from The Mythical Man Month as to what software development was like before the Internet, and how companies developed software using the waterfall method. We discussed how modern DevOps practices were developed and made major improvements to our planning/prioritization process by implementing some of the lessons learned from The Phoenix Project, primarily by being more explicit about the types of work we’re doing (planned vs. unplanned). We also try to reduce the amount of unplanned work (and/or re-work) we have to do in the future by following the practice of dogfooding as described in Showstopper!. More recently, we dove deep into big data with Designing Data-Intensive Applications and marketing with Traction.

These books have done a better job preparing me for my day-to-day responsibilities of a software engineer than any of the lectures/readings I had in college.

The Birth of the Hexawise Book Club: The Mythical Man Month

As testing continues to trend away from manual testing to test automation, some folks here at Hexawise realized they have some gaps in their knowledge of highly-technical topics. They reached out to our CTO, Sean Johnson, and asked him to provide some recommendations for material that is technical, but written in “layman’s terms.” Sean had the thought that these readings would not only be beneficial to the business team, but would also be hugely beneficial to the engineers here at Hexawise.

The book chosen to kick-off the Hexawise Book Club was a venerable one. At the time it was written, many of its contents were what kids these days would call a “hot take.” However controversial these themes may have been at the time, the ideas of Dr. Fred Brooks in The Mythical Man-Month: Essays on Software Engineering turned into gospel and have withstood the test of time.

As we read through The Mythical Man-Month, we were presented with a central theme that seems counter-intuitive. This central theme is known as Brook’s Law. Brooks Law states that “adding manpower to a late software project makes it later.” This wasn’t the first time I’ve heard of this concept. In fact, I was lucky enough to have Dr. Brooks as a lecturer while I was an undergraduate at UNC Chapel Hill. His teachings did not resonate with me though until I had some experience working as a full-time engineer. Even then, it was hard to believe that companies operated so inefficiently by planning projects large projects that span multiple years involving, at times, hundreds of people often leaving themselves with little or no room for delays or setbacks.

Book Two: The Phoenix Project

I am fortunate enough to begin my software engineering career in a time period where the waterfall approach has more or less fallen by the wayside and the Agile or Lean approaches have become the gold standard. That has a downside though, it’s difficult to put myself in the shoes of someone who works in an IT department still using waterfall methods.

Waterfall is much different than my experience working at Hexawise. We are very nimble so we are able to pivot quickly as our priorities or the marketplaces change. To understand what it’s like to work in a much larger company, Sean suggested that The Phoenix Project should be the next book we read as a group.

If you told me that I would thoroughly enjoy reading a book about working in IT, I would have called you a liar, but it turns out that The Phoenix Project is a page turner! I was invested in the storylines of each character. The authors are able to paint a portrait of what it’s like to work at a dysfunctional corporation whose stock price has been plummeting year after year with no rebound in sight. By the time we got to the second or third chapter, I felt like I was an employee of the fictional Parts Unlimited.

In an effort to get a mission critical business initiative (the Phoenix Project of the title) developed, tested, and out the door, Parts Unlimited worked to identify their bottleneck and find creative ways to eliminate it. They identified their most talented engineer as their bottleneck. All work seemed to flow through him, yet a single person can only get so much done at one time. Naturally, one might try to surround this person with an army of people with similar experience. If you think back to Brooks’ Law, “adding more manpower to a late software project makes it later”, you quickly realize that this approach is doomed. Instead, Parts Unlimited incorporate the philosophies behind Toyota’s Lean manufacturing into their own IT processes.

The Parts Unlimited team eventually learns that any singular task can be classified as one of four types of work: business projects, IT projects, changes, and unplanned work.

A Business project is any work initiated with the goal of delivering additional value to the customer. IT projects are work initiated with the goal of improving processes and operations used to deliver this new value to the customers, often times to reduce costs. Changes are simply routine tasks to accommodate the current state of the world. Provisioning a laptop for a new employee is an example of a change. Lastly, we have unplanned work. Unplanned work is work to fix any problems that originated from the previous three types of work.

Once they started placing their work into these four buckets, they realized that unplanned work took up an overwhelming majority of their total time, thus reducing the amount of time available for planned work. No wonder the important Phoenix Project kept getting delayed with no end in sight!

They re-oriented their thinking around two major objectives. Their first objective, and the reason they are employed, is to support the business, so they did everything they could to maximize the time they spend on planned business projects. Their second goal is at the heart of Lean processes, and is the most important takeaway from the book. It is to minimize the amount of work in progress. By minimizing the amount of work in progress, Parts Unlimited was able to control the release of work from the backlog into the software manufacturing pipeline and focus all of their energy on quickly delivering increments of value to the business. All of a sudden, Parts Unlimited was able to climb out of the hole they dug themselves over the years.

Resulting Changes to Our Process

Obviously this book is fiction, but after feeling like Parts Unlimited employees for a month or two, the Hexawise engineering team decided that we too should rethink how we prioritize and plan development work. We’ve always utilized a streamlined, small-team, agile process, but after reading The Phoenix Project we decided to adopt a more deliberately Lean software development process focused around reducing work-in-progress. Yes, this means we don’t do sprints! Heresy!

Sean spent some time going through every single card on our Kanban board and labeled each item as one of the four types of work. Next he broke down the monolithic Hexawise R&D Trello board down into two separate boards: Hexawise Backlog and Hexawise R&D Kanban.

The Hexawise Backlog has swimlanes for each major category of work. As bug reports come in or new feature requests are dreamed up, they get added to our backlog based on type of work. This allows us to ensure we have a healthy balance between business projects, engineering projects, and unplanned work.

The Hexawise R&D Kanban has three major sections: Ready for Work, Work in Progress, and Deployed. Items in the Ready for Work section have often lived on the Hexawise Backlog for some time, but are now ready to be pulled for work.

Work-in-Progress encompasses multiple lanes on our board, but can be described as all the work that has been started, but has not yet been completed and deployed. Our ideal is to keep these in-progress work items to a minimum, at most one or two tasks per person, and it’s everyone’s responsibility to do whatever they can to get each in-progress card to the finish line ahead of other priorities, especially ahead of pulling new work for themselves from the backlog.

By focusing on getting committed work out the door quickly, we’re able to deliver value to our customers on a more frequent basis. At the same time, we’re able to avoid unnecessary headaches and issues due to merge conflicts between two or more branches that have been hanging around for a while.

Changes to Our Engineering Standups

Aside from restructuring our Kanban board, we also changed how we conduct our standups. Previously our standups were oriented in a typical Scrum fashion around people by having everyone go around and quickly state what they’ve been working on and provide details into what they have going on for the next day or two.

Instead, we now orient our standups around work. Specifically work that is in-progress, and our focus starts with the items on the right-most side of our R&D Kanban: work that is almost ready to be shipped, but for some reason or another has not been deployed yet. These items become everyone’s top priority. After discussing this work in QA/Review, we move onto tasks that have been worked on, but are blocked and cannot be moved forward for one reason or another. We decide what can be done to unblock them, and that’s everyone’s next priority. Only then do we start to discuss what we’re currently working on or new items to pull from the backlog. This reinforces the importance of getting work-in-progress through the entire process and out the door as quickly and efficiently as possible, and helps us structure our days around achieving this goal.

The whole engineering team has gotten behind this mindset shift, and we’re excited to be reaping the benefits. We’re happy to see a > 80% reduction in the number of work-in-progress items at any given time. Once we pull an item from the backlog it’s getting into customer’s hands where it can provide value much more quickly than before.

Book Three: Showstopper!

After living and breathing this new process for a bit, it was reassuring to see our list of tasks done growing at a more rapid pace than ever before. *pats self on back* What was unsettling was that more than half of the cards belonged to the same category:

We had the same holy crap moment that Parts Unlimited had when they realized that the reason they have been unable to deliver the Phoenix Project: they were swimming in unplanned work. Understanding the impact of the unwanted side-effects caused by unplanned work, we started to get innovative about ways to reduce the amount of self-inflicted unplanned work. After all, unplanned work is well… unplanned and there’s only so much you can do to stop it.

The first step in this direction was to get serious about our automated test suite. Each defect that lands in our lap is unplanned work. If there were an automated test for that, our CI builds would fail. Test-driven development (TDD) has become my sidekick for bug squashing. With TDD, I am able to watch tests go from red (failing) to green (passing) providing reassurance that the bug(s) in question have been eradicated. This approach also minimizes the amount of potential rework since builds that fail don’t make it out into production.

The second step in this direction was to get more serious about dogfooding. Dogfooding is when an organization uses its own product to test real-world usage. Dogfooding is a theme in the  book: Showstopper!. QA at Hexawise has always been combinatorially inspired, but it wasn’t until recently that we started to build up a repository of testing ideas in the form of Hexawise test plans.

After we isolate the underlying cause of unplanned work, we find a way to incorporate it as a testing idea that can be reused at a later date. Parameter and parameter value names including special characters (e.g. “&’<>^) have proven to be problematic in the past when it came to rendering and exporting plans. Recently we ran into a defect related to importing a plan that included a requirement with a forced value that was an integer. That’s a 4-way defect for anyone counting. This defect only surfaced itself when attempting to edit the parameter with an integer value via the parameter dialog rather than bulk edit or the mind map. That’s a 5-way defect!

While working on the necessary changes to address this wild defect, I went ahead and created a new test plan that captures these problematic characters and then some.

As we go through our extensive code review and QA process we are not simply testing that features work with the happy path. We practice what we preach and vary as much as possible from test to test in order to maximize the number of defects we uncover before the code gets put out into production.

Book Four: Designing Data-Intensive Applications

Do you know the tale of Icarus and his waxen wings? The quick version is that Icarus made some wings and learned to fly, but he got a bit full of himself and flew too high. Too close to the sun, his wing attachments melted and he plunged to his death. That’s a bit of what happened to us with our fourth book, Designing Data-Intensive Applications.

Everything started off well, if a bit more challenging for the participants that aren’t software engineers, but after a very rewarding first few chapters, Designing Data-Intensive Applications started to dive a bit too far into the weeds.

Ultimately we decided to cut this one short and not finish the book. We did this with no regrets as Part 1 of the book — the first four chapters — was both unique and amazingly helpful to everyone. The history and defining forces that have led us to build data applications the way we do today has never been told like this as far as I know. It’s a story everyone in our industry should know well, and it continues to play out all around us every day.

Designing Data-Intensive Applications is the first book in our book club that no one had ever read before, so it was a riskier choice since no one could vouch that it was a great book, worth everyone’s time, and with the right content for everyone. It turns out Designing Data-Intensive Applications is an incredible book, better than expected, but it wasn’t right for our book club’s mixed audience.

Book Five: Traction

Possibly as a reaction to the deeply technical Designing Data-Intensive Applications, we decided our next book would be our first non-technical book. Sean and I had previously read Traction and thought it was an excellent choice to support a company-wide effort that was just getting underway to invest in new marketing initiatives for Hexawise.

Traction turned out to be the right book at the right time, and we had a series of book club discussions about how each of the 19 marketing channels might be used for Hexawise. We’re now using the process distilled in the book and initiating experiments to validate our hypothesis that a couple of these channels will work well for us.

Over to You!

I encourage you to start your own technical book club at your company. I promise you will be surprised at how much you can learn from the readings and discussions, and you’ll be surprised at how much you enjoy getting to know your colleagues better in a book club!

 

Blog Cover Photo by Aleksi Tappura