The High Cost of “Low Cost” Software Development

money

I recently spoke with some software development professionals about the economics of software development and how Agile Values and Principles, when properly applied through a framework like Scrum, could improve a company’s bottom line. One thing we discussed was that so many companies who develop software are ignorant of the economics of software development and the Total Cost of Ownership (TCO) of software development. Companies often try to save money by choosing the lowest cost software development option. My friends referred to this as sourcing software development on price alone. At the new division I manage, 10XP Solutions, we call this the high cost of “low cost” software development.

crowdWhat is the high cost of “low cost” software development? This is the tendency for people involved with financial decisions regarding software development to put too great an emphasis on the cost of software developers. I have recently coined the Law of “Low Cost” Software Development that states, “In the absence of additional information and a lack of understanding of the economics of software development, the choice of software developers will be based on cost alone.” Unfortunately, in practice this law leads to an overall higher cost of software development.

One example I use frequently involves the concept of technical debt. Once I was asked to write an article on technical debt. When my article was published, it was stripped of a definition of technical debt because the editor believes everyone in software development knows what technical debt is. I protested, but to no avail. Nevertheless, I still maintain this assumption is not only incorrect, but it is fundamentally dangerous. I have found the number of people who make decisions affecting the creation of software who do not understand, or may have not even heard of, technical debt remains shockingly high.

For those uninitiated with the concept, technical debt was invented by Ward Cunningham as a metaphor to explain the real cost associated with short-term decision-making and shortcuts taken in software development. A classic example would be the first time code is shipped and market forces dictate that speed to market trumps all other concerns. Cunningham’s conception would be that technical debt would be a conscious decision made with the trade-offs known. Over time it seems that technical debt has morphed because these days a great deal of debt accrued by organizations is unconscious. In other words, they are creating technical debt with little or no awareness. In these cases technical debt is like high blood pressure – a silent killer.

moneyThere are now ways to quantify technical debt. The CRASH Report calculated the cost to remediate technical debt concluded that the current levels of technical debt average $3.61 per line of code (the amount is higher for Java code at $5.42). As technical debt increases so does the complexity of the code and the difficulty in making changes to existing code. This means that new features added to existing code will take much longer to develop. How much longer? A study by Dan Sturtevant at MIT, entitled “Technical Debt in Large Systems: Understanding the cost of software complexity” found that complex (technical debt-laden) code resulted in:

  • Up to a two-fold increase in the amount of time to enhance software (50% decrease in developer productivity)
  • Up to a 310% increase in defect density
  • Developers working on poor quality code had up to a 10x increase in employee turnover

How these translate into hard dollars may be difficult to determine, but we can certainly infer that defect laden code increases the maintenance cost, QA cost, tracking cost, defect reporting cost, and costs related to poor customer satisfaction. The most surprising finding was developers working on poor quality code had a greatly increased amount of turnover. There is as obvious hard cost for replacing already difficult to find developers, but also untold morale cost for those who remain.

Total cost of ownership (TCO) addresses the total cost of software development from inception to sun setting. In 2011, the CRASH report stated the total cost of ownership for software code was $18/Line of Code (LOC). Of this, it is generally accepted that the majority of this cost is related to the maintenance of the software after its initial creation, with estimates ranging from 60-90%.

Because the majority of our cost involves maintenance it doesn’t make a great deal of sense for us to spend our effort trying to pare down the initial cost of development by employing lower cost developers. If we use less expensive resources, we could expect technical debt to increase. As technical debt increases, so too does the cost of maintenance. If we assume a slight increase in technical debt (50%) which results in maintenance negatively impacted by only 33%, our “low cost” resources have now actually cost us $4.92/LOC more. In contract, an approach that focuses on higher quality, while a little more expensive in initial cost, results in overall savings.

Strategy Average “Low Cost” High Quality
Initial Cost 4.50 2.25 6.75
Technical Debt 5.42 8.13 2.71
Maintenance 13.50 18.00 7.50
Total Cost 23.42 28.38 16.96

There are other factors to consider in addition to just initial cost, technical debt and maintenance. Many people employing the “low cost” software development model are rarely paying attention to another hidden cost – the cost of delay. Frequently a trade off is made between cost of developers and productivity with lower cost developers being less productive. This results in software products that take longer to produce and deploy.

crowdOf course, because these developers are lower cost, one could always just throw more people at the problem, which is often done. However, adding more people to solving software development problems does not result in a corresponding increase in productivity and obviously eats into cost “savings”. There are many who believe doubling the number of people results in a doubling of productivity. This is an example of applying mechanistic thinking to knowledge problems. There are numerous studies indicating increasing the size of teams results in productivity increases much lower than expected. Therefore, “low cost” development leads to longer cycle times and a higher cost of delay.

While it may be seductive to think that you can save money on software development by using “low cost” developers, it rarely results in overall cost savings when considering TCO and cost of delay. The cost of delay and technical debt are generally hidden costs (at least on the balance sheet). Over the years I have had numerous discussions with software development professionals (CIO, CTO, Development Managers, Product Owners, etc.) regarding the “low cost” software development models and there is nearly a universal befuddlement over why the model continues to flourish. Unfortunately, many people making financial decisions regarding software development resourcing simply do not understand the nature of software development and TCO. If they did, my guess is that they would make drastically different decisions.

 

 

Real World Agile Q&A

question

One of the joys of being an Agile Coach is that it is a passion and a calling that doesn’t end at 5pm. Frequently I have the pleasure of staying in contact with individuals who I have worked with via phone and email. With the permission of a good friend, and changing a few details to render our exchange more generic, I present our latest exchange in hopes that people will find it helpful or at least interesting.

 Larry,

I keep reading your articles and really enjoy the insight you continue to bring to the Agile world. I’m particularly stuck with a situation here at my current employer.

questionWe’ve organized a scrum team that has done a fantastic job pushing out working, quality software since we started 10 weeks ago. However, this team primarily has worked on our website, but as I see it, we are bringing all of our development work to the team. Consequently, we are about to kick off work for redesigning our Intranet. This too falls into the team’s backlog, but now my business contact is pushing back, saying her backlog for Internet is separate and she doesn’t want to have to prioritize her work including the Intranet. I don’t have the luxury of just adding more developers to an additional team.

What tactics would you use to approach solving this? It’s a fragile situation. Your thoughts?

 

My response:

Thanks for the kind words. My first thought is a question – how many people are on the team? If you have enough, I would float the idea of splitting the team. The rule of thumb is one or more teams per backlog. What your proxy PO is doing is to (perhaps arbitrarily) split one backlog into two. One team cannot really service multiple POs or multiple backlogs. This way you can give the PO a choice – work with me to understand resources are limited, work with me to value all of the work for the team (not just your work) and then sequence the work so that the entire company is getting the benefit of a stable, focused agile scrum team.

question marksThat said, the real root of your problem might be that you really don’t have a product owner in the classic sense. Perhaps you would be well served to find someone who could represent a full backlog, especially if your team is too small to split as outlined above. I would suggest that all stories on your backlog should have both effort (in points) and value. Perhaps the best way to achieve your story value is to have your people compute the cost of delay (COD) for each item. You can then show all the work of the team with respect to value/effort to get an idea of which items to sequence first. If you find stories too granular to perform COD, group them (call them Epics, Themes, etc). Though I don’t recommend wholesale use of SAFe in your case, I would recommend that you have your folks look into concepts like COD and WSJF (Weighted Shortest Job First) that SAFe describes fairly well. Donald G. Reinertsen further elucidates these ideas in the book The Principles of Product Development Flow.

If you cannot get a different PO who can oversee more than just their parochial concerns and you cannot split the team in a way to service the two different products you are working on then your options become even more limited. My next question would be to ask who is in charge of your Agile effort? One of the reasons I have proposed the concept of CAO (Chief Agile Officer) is for situations such as these where someone at the top level of the organization can arbitrate using the entire organizational as a lens to determine best courses of action. It sounds like you are both on the same organizational level so it would help to appeal to a higher management level in instances where two of the same level cannot come to a satisfactory conclusion. Obviously this is a last resort in that people involved tend to react negatively to people “going over their heads.”

I wish you all the best, hope that you are able to resolve this situation and that my advice will prove valuable, and if not so, will at least not result in a negative outcome.

In the end, my friend decided to split the team into two smaller teams. Maybe in a later blog I can follow up to see how well this tactic worked.

Individuals, Teams, Systems – The Evolution of Agile at Scale

building for scaling agile blog 2

If you have any interest at all in agile, if you read agile blogs, if you follow agile trends, you most certainly have been (over) exposed to one phrase “scaled agile.” I, myself, recently received a certification for one of the new scaled agile programs. While I enjoyed the training I find myself not 100% convinced of its effectiveness.

Perhaps it is because of its very novelty that I remain skeptical.  Perhaps it is because I know front line people in companies referenced as successfully adopting who do not agree to the efficacy.  Perhaps it is because “scaling ” has simply become the latest excuse from management why agile cannot be adopted at their company (as in why should we bother when agile has not been Provence to scale).  In the end though I think the reason that I am lukewarm on the idea is because companies have been latching on to scaling before doing the initial groundwork.  in other words, scaling has become a red herring that induces companies to put the cart before the horse or the system before the team.

Once upon a time I coached one of the most amazing agile scrum teams. They were able to deliver things that their management found quite unbelievable so much so that they conveyed a meeting to find answers to why this particular group of people were able to so greatly outperform others in the organization. As an aside I was not originally invited to the meeting but the team lobbies for me to be there as they considered me one of the team and, as a team member, partially responsible for their success.

I remember very well one manager in particular gushing about the team performance and wondering aloud how such a miraculous feat could be “scaled.” “It is very much like the legends of people working together out of their garages,” the manager said, “but how can such a thing” he paused briefly to allow the group to understand what came next would be important, “be scaled?”

Before I realized that the question was merely a rhetorical one to introduce us to the new management buzzword “scaling”, I opened my mouth to answer. “Why we just create more garages,” I said. The answer, which I still stand by to this day, was roundly ignored by aforementioned manager and his manager reports and was about as welcome as a tart in a crowded elevator.  What I realized too late was that “scaling” was management’s way of avoiding any action, especially action that might involve hard work and opposition.

So, you might say, scaled agile to the rescue. Not so fast my friend. My amusing anecdote above was all about creating good teams (and creating more good teams). Agile at scale is about creating a good overall system. There is no need to scale agile until you have enough teams doing agile.

Here is the way I like to explain the evolution to scaled agile. Waterfall is inefficient because it tries to optimize individuals at the expense of teams. Good teams have a higher productivity, shorter time to market, higher quality, etc than individuals.  Enter Scrum. Scrum, when done correctly, optimizes teams. This is a huge advance and if your agile evolution ended here, you would have come a very long way.  Nevertheless it frequently happens that we cam optimize teams at the expense of the system. Once we reach this point it is appropriate to look at ” scaling” and how we can optimize the entire system.

To recap – waterfall optimizes individuals at the expense of teams. Scrum optimizes teams at the expense of the system. Scaling is the attempt to optimize the system. My experience (and it may be biased because like Freud I work mainly with the “sick”) is that companies abuse scaling because they either use it as a means to avoid agile or they rush to scaling without first making the tremendously difficult transition to creating effective scrum teams. My advice remains “build more garages.”

The Gang of Four – How Optimization Perseveres

room, lights, black and white

room, lights, black and white

It must be the new year that has me waxing nostalgic again. My last post was of rockstars past and while I was working on that on that one I was also reminded of some others I had a great affection for, a group that I was a part of we called the “Gang of Four”. This was a tongue in cheek nod to the Gang of Four who were authors of a book on design patterns.

The impetus for the Gang of Four was the desire of four agilists to try to do the right things with regards to implementing agile and scrum – to do things that were generally acknowledged as good agile practices but things that were not necessarily politically palatable. In other words, we were something of a clandestine organization. In fact, our meetings, while not in a secret location, were held in a location well off the beaten path – so much so that some of us had to literally walk 15 minutes to get to our meeting room. Over time we added another like-minded individual but kept the original name. Hell, we even had a ironic salute we would give to each other when we would pass in the hall. It was in the company of these wonderfully dedicated people that I spent the most enjoyable time I had on this gig.

What is important and pertinent to this blog is not the nostalgia or shout out to my compadres (though it pleases me to think they might read this and reflect with happiness), but that regardless of your office politics, your cronyism, your brown-nosing, your ladder climbing or your sheer incompetence and general ineptness, there will always (or rather you should hope there to always) be those who will have the uncompromising desire to succeed in spite of your inanity.

Like a tree whose roots will break rocks in order to drink of the water that sustains it, there will be individuals and teams in your company that will persevere against the all odds to do what is right, to do what is best for the company. Seek these out and when you find them give them the environment that they need because if they survive on rocky soil just imagine what they can do on a proper plot of earth.

Long live the Gang of Four.

Managers from Hell and Agile Transformation

larry apke, whiteboard, agile

larry apke, whiteboard, agile

“Here’s something they’ll probably never teach you in business school: The single biggest decision you make in your job — bigger than all of the rest — is who you name manager. When you name the wrong person manager, nothing fixes that bad decision. Not compensation, not benefits — nothing.”

Jim Clifton, Chairman and CEO, Gallup – State of the American Workplace

A few months back I stumbled onto some research done by Gallup on employee engagement quoted from above. I would like to say that their findings were shocking, but having been an Agile Coach at many large organizations, I find the statistics (and conclusions based on the statistics) to track quite close to my own experiences.

For example, Gallup found that only 30% of all employees in the United States are “engaged and inspired at work” and about 20% are what have been defined as “actively disengaged” employees who “aren’t just unhappy at work; they’re busy acting out their unhappiness. Every day, these workers undermine what their engaged coworkers accomplish.” The remaining 1 out 2 workers are defined as “not engaged” and “essentially ‘checked out.’ They’re sleepwalking through their workday, putting time — but not energy or passion — into their work.”

And take a wild guess what the number one indicator of employee engagement is. You got it – their manager. In another article, Gallup estimates that “companies fail to choose the candidate with the right talent for the job 82% of the time” (Why Great Managers Are So Rare). In other words, next time you are in a meeting with five managers, it is safe to assume that only one (maybe two) were the result of good hiring decisions. If you are a good manager meeting with other managers chances are good that everyone you are speaking with was the result of a poor hiring decision!

The fact that management hiring decisions are so poor should be of some comfort to you the next time you are passed over for the management job you know you can do well, but is little comfort to active agile coaches like myself who are actively trying to help a company get agile and stay agile. On the other hand, knowing this goes a long way in explaining why most companies have so much trouble with the transformation.

Without the support of managers, lasting agile transformation will not occur. The fact that the bulk of your managers are poor leaders just makes it that much harder. Gallup estimates that only 1 in 5 current managers have the abilities to “naturally engage team members and customers, retain top performers, and sustain a culture of high productivity.” Agile is all about engagement. It demands leaders who can engage developers.

So you say you want to be agile? Why don’t you start with hiring better managers? Stop relying on third-party vendors to find your talent. Grow your talent. Recruit your own talent. Find CIOs, VPs, Directors, Managers, Coaches who understand agile, can truly lead agile teams and engage developers.

Go ahead. I’ll be waiting for your call. Of course, it may be a long wait.

Predictability and The Gold Standard

I am often asked as an Agile Coach when I know that I have been effective at my job. The answer is simple- my work as a coach is done when the team in question is capable of being predictable.

And what, you may ask is predictable? For me it is a team that is capable of consistently delivering 90% or greater of points that have been planned for an iteration. I have given this capability a name. I call it the Gold Standard.

Continue reading “Predictability and The Gold Standard”