When it Comes to Software Development – Size Matters

size matters

Sorry for the tongue-in-check title, but when it comes to software development the size of the teams does matter. When I talk to people about good Agile teams, especially good scrum teams, I talk about five attributes – size, co-location, dedication, stability and cross functionality. I also tell them that if we have to make compromises my preference is to make them in reverse order (ie compromise on cross functionality first, then stability, etc. – not that we want compromise). This means that optimal size (7 +- 2) is the most important attribute. Why is this so important?

One reason is that studies have shown that small teams are much more productive than large teams. One such study was conducted by QSM on 564 projects (from their database of over 4000) titled Top Performing Projects Use Small Teams. I urge folks to check out the entire study, but what is most telling is their conclusion:

To complete projects of 100,000 equivalent source lines of code… they found the large teams took 8.92 months, and the small teams took 9.12 months. In other words, the large teams just barely (by a week or so) beat the small teams in finishing the project!

In this study small teams were 5 or fewer and large teams were 20 or more so basically you can pay for 4X the number of people, but you won’t get 4X the productivity.

Tug of WarOther studies have shown that as the size of teams increase that the effort for each individual decreases, even when the people are engaged in physical tasks such as tug of war. This curious phenomenon has a name, the Ringelmann Effect. In addition to other things:

Ringelmann discovered that as more and more people are added to a group, the group often becomes increasingly inefficient, ultimately violating the notion that group effort and team participation reliably leads to increased effort on behalf of the members.

According to Ringelmann, there are two main reasons for this effect – Loss of Motivation and Coordination Problems.

Two Pizza TeamsThe antiquated notion, borrowed from a physical and mechanistic view of software development, that adding people to knowledge work will increase productivity and result in faster software development is still prevalent in many companies, but there are a few software development companies who understand the importance of team size. Amazon, for example, has the concept of two pizza teams. If you cannot feed a team on two pizzas then it is time to split into smaller teams.

The issue of communication (Ringelmann’s Coordination Problems) is alleviated by the two pizza team rule because the number of communication channels increases exponentially as more people are added to a team. This is often described by the equation n(n-1)/2 where n is he number of people on a team. So for teams of 5 (like in the QSM Study) there are 10 communication channels (which is easily manageable) while for the large teams of 20 there are 190! Obviously 190 communication channels is very difficult to manage.

Communication channels would not be a problem is software development was merely a mechanistic activity (as it is all too often mistaken to be), but because software development is a knowledge activity that relies heavily on communication and collaboration, large teams are no a good idea since all the productivity that each new member would bring is easily cancelled out by the additional communication channels (not to mention the attendant loss of motivation).

In my own consulting I have suggested that the simplest way to double productivity of large teams is merely to break it into two smaller teams. My own experience in employing this technique has resulted in a large increase in overall productivity and quality.

As one begins to understand that software development is a complex, knowledge worker activity, one begins to understand that the most important aspect of producing high quality software is communication and collaboration. Communication (and motivation) is optimal when teams are small – so in the end, size does matter.

We Must Inject (not just inspect) Quality

quality inspection for blog

Not sure where I first heard the phrase “you can’t inspect quality into a product” but I have certainly used the phrase myself all too often in my consulting gigs. After a quick Google search, I found the originator of the quote was Harold Dodge and it was first used in a manufacturing context. While I generally eschew appropriating manufacturing analogies for comparison to software development, in this case, it is certainly apt.

In many of the companies I consult with, the prevalent view of “quality” is nothing more than inspection after the fact, a misplaced notion that quality can be insured by finding all of the defects built into the software. The great majority of these inspections are conducted by a separate team, a half a world away by manually banging on keyboards and mice. If one were to write an article on how NOT to ensure software quality it would look frighteningly similar to this (which is what I experience all too frequently when I work with companies).

I find that often this goes hand in hand with the mistaken notion that we are performing projects as opposed to creating long term stable quality products. In some cases our quality leaders are merely naive or simply unquestioningly following the misguided policies and practices of those who came before. For those who truly wish to understand true software quality, who desire to build quality into a product instead of merely trying to inspect it out, what practices should we use?

The first thing is to recognize that building quality into our software code must begin with those who are building the software, the developers. We might have some nice motivational posters that state quality is everyone’s business, but until we tear down the arbitrary and dangerous walls we have erected (incorrectly taking waterfall to its bizarre conclusion) between development and “testers” we will certainly fail to put quality into the code. Development can ensure the quality comes first by writing tests first and performing true TDD (test driven development).

Sometimes, as Dan North found, developers may be convinced that TDD is a great idea yet find it difficult to implement. This is a great opportunity to inject BDD (behavior driven development) as a means of helping developers with TDD. In fact, I often recommend that BDD style scenarios can be used to express acceptance criteria even if developers are not yet ready for TDD. The tremendous increase in understanding and early inclusion of test cases as expressed through scenarios is the biggest benefit of BDD and goes a long way to injecting quality into our code.

Another technique for injecting quality is the concept of code review. There is nothing like another set of eyes to make sure that our code is high quality before we push it out of the development environment. You may want to go one better and get a more immediate feedback through pair programming. Like many things in software development pair programming is counter intuitive but the quality of code produced by judicious use of pairing far outweighs the perceived “double” expense.

Akin to code reviews conducted by humans is the ability of code to be “reviewed” by software through static code analysis tools. This software can “read” code and determine “quality” through a set of programmed rules. It will, for example, let you know which classes are too long or complex. These rules become instant feedback for code in progress and help us to easily avoid technical debt. For existing code these tools can help us to understand existing code quality and provide us with candidates for refactoring.

Whatever the technique you choose, the key is to understand that quality is something that is built into the code not something that we can inspect out.

How I Know You Are Flailing at Agile

As a consultant I try my best to keep abreast of industry trends in technology and software development. Often one of the best ways to observe trends is to check out job boards to see what positions are being posted. For example, job postings can show trends in programming languages. You can see which companies are trying to make Agile transformations (How many big financial companies can hire multiple coaches in Foster City?). You can see which companies have caught the “DevOps” bug and you can tell which companies are flailing and failing their Agile transformations.

How can I tell which companies are failing? When you see multiple posts for the same position over time is a good indicator. For example, you see a post for an Agile Coach that goes unfilled for months. While good ones can be hard to find and take some time, a position open for months can be telling that good coaches are avoiding the company. Most good coaches can very quickly size up a company’s commitment to Agile and avoid those situations where they are likely to fail no matter how good a coach they are. They can sense the fundamental dysfunction. Sometimes you will see a posting one day and the same posting a few short months later. Chances are whoever was chosen didn’t last and that certainly raises red flags.

Sometimes the title that is advertised will show a company’s inherent failure to grasp Agile. You will see a ton of advertisements for Scrum Master or Project Manager. I personally love these as they serve the same purpose as someone wearing a “Stupid” sign in that we know to avoid both. Advertising for a Scrum Master or Project Manager indicates that the company searching understands neither position. The good thing is that if one is only interested in doing Agile then one can save time by avoiding these.

Once I drove nearly two hours for an interview with a company that does only “Agile” projects. During one of the many lengthy interviews, I was asked, “What would happen if we gave you a waterfall project?” “Well for one thing you would not be 100 percent Agile anymore.”

The interview went downhill from there – a waste of a lot of people’s time. Advertising for a Scrum Project Manager would have saved everyone!

Lately I have been seeing another title that gives me pause – Technical Project Manager, especially in the context of Agile companies. I have been a part of and witnessed many abject failures of software development projects (maybe why Project Management should be replaced by Product Management). On none of these have I ever thought that maybe, just maybe, if we had a Project Manager that had more technical knowledge we would have snatched victory tron the jaws of defeat. And the colleagues I talk to agree.

Project Managers have the least amount of authority in making projects successful. A Technical Project Manager is kind of like blaming the weatherman for the rain. It is a silly tactic by management to deflect blame from them. Agile transformation is difficult because it needs change from management and change of culture. Hiring Technical Project Managers is a red herring that distracts an organization from the real work to be done, pushing the transformation finish line even farther away. And finding the kind of person who can successfully straddle tech and management is a tall order, especially when people who know what Agile transformations are all about and who will see through you and realize you are merely flailing at Agile.

Larry Apke

Technical Debt in the Real World

technical debt, larry apke, agile

technical debt, larry apke, agile

I came across a video the other day that compares the relative devastation caused by two similar earthquakes – a recent 6.0 magnitude earthquake in August 2014 in Northern California and a 6.1 magnitude one in that same month in Yunnan Province, China. The differences are astounding. I created a simple table below to show the particulars.

California

China

Deaths

0 619

Homes Destroyed

4

25,800

To what do we attribute the differences in outcomes in response to similar events? The difference is one of quality. Homes in China are built quickly with only cursory inspections while those in the United States, especially those in earthquake prone areas, are built to exacting standards.

Being a huge proponent of software quality, I could not help but think of the connection between this and code quality and technical debt. In my mind, this is an example of the dangers of technical debt personified. Yes, they build habitable homes in China, but they are not built with the requisite quality necessary to withstand an earthquake. In this case, the corresponding loss is horrific in loss of life and loss of quality of life.

We can see the same things in software development. In our rush to complete our features, we take shortcuts in quality, building, as it were, houses that will not be able to withstand earthquakes.  In the case of software it is rare that the technical debt created has cost of human life, but it certainly can manifest itself in a cost to the quality of life. And much like the house that cannot withstand an earthquake, poor quality code may look fine from the outside, hiding the cost that will be paid at a later date.

Not a day goes by where we do not see some of the “earthquakes” in the world of software – the recent hacks into Apple, Target, JP Morgan Chase et al ad nauseum should provide some adequate examples. And these are just the flashy incidences that we hear about in the media.

Please don’t think I am making light of the devastation in China. My heart goes out those affected by this terrible tragedy. I take this very seriously, as I take the concept of technical debt very seriously. My hope is that us folks who make software for a living can see the similarity and leverage it to continue the fight for high quality software development. Let’s not continue making brittle houses – figuratively and literally.

Larry Apke

Why Agile is the Answer for Healthcare.gov

larry apke agile doctor

Healthcare.gov - www.agile-doctor.com

I have purposely tried to not comment about the Healthcare.gov website and it’s issues, but the landslide of wrong-headed information written on the subject has prodded me into adding my small voice on the upside that someone reading my this post will learn from this software development fiasco that we all have paid so dearly for.

Most unfortunate for those trying to learn from this is that the issue is so politically charged that to discuss it objectively is nearly impossible. The desire to place blame is too deeply ingrained in the human psyche to not point fingers – and this, for anyone capable of laying politics aside is the second lesson – the lesson that assigning blame is a human instinct that is not always productive in solving problems. Root cause analysis is essential to fixing systemic issues and may tangentially point to individual failings, but placing blame should not be primary.

And this leads to the first problem – the fact that this issue can rarely be discussed without politics. Once we have chosen sides, not only do we move from objective root cause analysis to finger pointing, but we begin to be tainted with confirmation bias in that everything we hear, read, experience is pulled through this lens. Therefore, this means that anything contrary to our preconceived political leanings are completely ignored or logically forced into their respective procrustean bed.

In my agile world the cautionary tale is about how office politics can certainly derail our best efforts to be agile. It is only in an apolitical atmosphere where true organizational change can occur. Where facts are not valued and the truth depends on the individual posting it, then the frank, brutally honest conversations (borrowing from Jim Collins’s saying in his book Good to Great)  that are necessary to effect real change are never had. In some cases the only people capable of such brutal honesty are consultants with no vested interest or employees so fed up with the politics that they no longer fear losing their job.

One other political struggle that this instance has exposed is the continual struggle between the agile and waterfall camps. Each side has cherry picked their arguments in attempt to bolster their claims and for me to weigh in on one side or the other does little to further understanding. Of course, I come down on the agile side and believe that this project’s failures can be tracked fundamentally to the fact that the project was in no discernible way agile. This argument is not even vaguely interesting to me.

The most interesting point I have heard (and agree wholeheartedly with) is that there are a number of individuals, myself included, who feel, given the proper non-political circumstances could accomplish this work for a small fraction of the cost with a much smaller team. The reason for our confidence is that most of us already have. interestingly not a single soul who has accomplished something of this magnitude has done so using anything other than agile methodologies.

While the political right claim it is big government’s fault, this argument is somewhat disingenuous because the entities that failed are precisely the private sector companies that are supposedly our economic white knights. The real problem, when all the red herrings are swept away, is the procurement process itself that only allows for the largest, bureaucratic and bloated from the private sector to win contracts. Those companies who, by the very necessity of being small, are nimble will never get these contracts so failure is to actually be expected at the same rate (or worse) identified over the years by The Chaos Report.

A long time ago, government misinterpreted Winston Royce’s “waterfall paper” and all contracts are written to favor waterfall development which has been shown to be a poor methodology in delivering complex software. Again and again we run into the core issue that the nature of software development is fundamentally misunderstood as being merely complicated so that throwing money and bodies at a problem (and these bodies can reside anywhere, with any company like interchangeable widgets) will lead to success. Because software development is complex it works best under agile which emphasizes feedback and communication. Simply the fact that so many vendors were working this project shows the complete lack of knowledge and gross incompetence of people who are paid well to supposedly understand software development. In fact, if people in charge actually understood software development then they would have known the job would have been done much better with a small band of government workers and not the disaffected armies of “private” companies.

The reason I and my agile friends know we could have done the work for much less and with infinitely better quality is because we know what software development is and we would have used smaller teams, fewer teams and co-located teams. We would have increased feedback and communication. We would have completed the most critical scope first and phased in improvements. We would have used test driven development and continuous integration and testing to ensure performance all along the way. In other words, we would have operated in a completely egalitarian, non-political manner and achieved success.

What are your thoughts?

Larry Apke

Why Project Focused Mentality is Killing Software Development

california damian morys larry apke agile doctor
California by Damian Morys
California by Damian Morys

So, the other day I was listening to NPR in the car (like most people when they listen to talk radio). And the talk was about peace in the Middle East. One of the experts mentioned that, in his opinion, unless both sides owned the process, it was never going to come to fruition. This reminded me of Mark Fritz, international speaker on leadership in today’s organizations, and a compelling blog post he wrote about ownership. I thought this was interesting because there was an interwoven thought he drew upon throughout the post that I believe is applicable to Agile and software development:

“You never wash a rental car.”

I have used a similar phrase many times before when describing the project-centric mentality that pervades most large software development shops. I tell people that, in order to have good quality code, in order to do the right things by the code (BDD/TDD, continuous integration, etc), you must own the code. Therefore, we must move from project to product management, from renting our code to owning the code. So now you know why my brain went from the Middle East to Mark Fritz to rental cars, and now here we are!

Also, I distinctly remember a recent talk when I proposed that in order for better quality products we must eliminate the PMO and replace it with a PPMO (Product and Project Management Office). Interestingly enough, I remember this talk was given to a roomful of project managers who conveniently heard the first part of the phrase and not the second part.

The bottom line is that Agile is about creating high quality software quickly. This cannot be done long term without doing the things that come with owning the code. No project, which merely rents the code, is going to spend their budget making sure that the next group to rent the code has an easier and better time. There is no incentive in project focused organizations to do the right thing! The pressure is on chasing the next bright, shiny object as quickly as possible (e.g: feature chasing) with zero regard for the long term stability of the product.

Before you know it all you have is a big pile of mud that no one can safely enhance or refactor without high chance of regression defects. The time it takes to bring the new, shiny objects to market becomes longer and longer because no one bothered to create the necessary infrastructure. Why not? You might as well ask yourself why no one washes a rental car.

Larry Apke