General Agile - 5/10 - Agile Doctor

14 October, 2015

Technical Debt on the Balance Sheet

My recent article on the high cost of “low cost” software seems to have struck a nerve with lots of views, comments, likes, shares, etc. A lot of software development professionals feel the weight daily of the tendency to chase the lowest price of software development and have added to the conversation I started last week. Below are some of the comments:

There is always a price for “free.”

In the end, it’s all about understanding the nature of software development in order to make wise decisions.

As the saying goes: “if you believe a senior developer is expensive, wait till you hire a Junior”.

Another aspect of poorly written code is the inefficient use of hardware which it runs on … leading to negative user experience .. Resulting in losing $$ in terms of user productivity .. Adding to the higher cost !!

My unscientific guess is corporate America is wasting billions each year on “low cost software “.

It’s funny how there never seems to be enough time to do it right, but always enough time to do it twice.

The technical debt gets worse when you add an army of developers to it.

The old saying “you get what you pay for” rings true.

Perhaps the most interesting comment came from Jason Ross who stated:

It’s often the case that people get “value” and “cost” confused. Technical debt is a very debt that must eventually get serviced. In the same manner that debt service kills cash flow and becomes a drag on the balance sheet, so does technical debt deprive an the ability to deliver new features and effectively update the code base as new technologies arrive that could be leveraged. In the long run, the maintenance and upkeep costs becomes the dominant factors in many things, and especially in software, they are often entirely neglected as factors.

When I read his comment and saw his mention of “a drag on the balance sheet” it reminded me of an article I had read a few years back from Israel Gat for his blog The Agile Executive titled Technical Debt on Your Balance Sheet

Israel’s assertion is now that there are ways to rather easily assess technical debt (like SonarQube with SQALE plugin), technical debt could (and should) be added to a company’s balance sheet. I applaud and support his efforts.

Unfortunately, it appears that any efforts to include technical debt on a company balance sheet might run into some accounting industry standards that would make this not currently possible. In his post, Clear Costs and Technical Debt, Trent opines

Accounting standards don’t allow provisions or liabilities to be shown for “not performing enough maintenance” or similar intangibles. There needs to be a present obligation before a liability is incurred and a present obligation is only usually formed through a contract.

In his scenario, financial wizards will chalk up the lower cost of software development as a savings.

The accountants will note a(n)… asset on the books, congratulate themselves for saving … capex and chalk up the increased opex of maintenance as the cost of doing business.

My guess is this is precisely what happens now. If Trent is correct, then the issue facing companies developing software is that current accounting practices are inadequate and need to be modified. Technical Debt can be measured and tracked and is a very real liability.

The question remains: can there be a new GAAP around technical debt or can an existing GAAP be instructive? While I am not a CPA, I have done work in accounting in the past so I know “enough to be dangerous.” My line of thought is that technical debt probably resembles something that currently exists in GAAP. After doing some research I think that maybe GAAP around warranties may prove helpful. In reading about warranties in an article GAAP Accounting for Product Recalls. I noticed some similarities:

Product warranties present manufacturers with a bit of a conundrum. …The manufacturer is actually responsible for the product for the length of the warranty. In accrual accounting, the finality of the transaction does not occur until the period of responsibility ends. However, warranties … present a bit of an exception to the rule. (Emphasis mine)

Notice how, like technical debt and the Total Cost of Ownership (TOC) of software, the complete transaction is not final until the period of responsibility ends. I can see that the life cycle of software (and liability of technical debt – present or future) is like the life cycle of a product with a warranty attached. Even I can see how warranties could cause accounting headaches. However,

Rather than being on the hook for the cost of the product for the entire life of the product or throughout the product’s warranty period, manufacturers can instead follow the generally accepted accounting principles regarding products under warranty. … Under GAAP, product warranties and recalls can be reasonably accounted for with the initial sale of the product by estimating the potential warranty expense of a return or recall.

I can reasonable see how it might be possible to substitute an estimate of Technical Debt liability for the similar liability of estimated warranty expense. For example, with each release we classify software as a capital expense (CapEx). At this point we could also include on the balance sheet an estimation of technical debt as a long-term liability.

In the case of warranties, the actual accounting mechanism under GAAP is done by

…making two separate entries, one a debit and the other a credit for the same amount. These are entered into the ledger as a warranty expense and a separate entry as an allowance for warranty costs.

What is to stop a company for entering two separate entries for technical debt – one a debit as technical debt expense and the other a credit under allowance for technical debt? I certainly see this as in the realm of possibility. This way GAAP can be followed AND technical debt can be raised to true financial visibility. It is this visibility that will allow companies to begin seeing the true total cost and guard against the high cost of “low cost” software development.

6 October, 20156 January, 2016

The High Cost of “Low Cost” Software Development

I recently spoke with some software development professionals about the economics of software development and how Agile Values and Principles, when properly applied through a framework like Scrum, could improve a company’s bottom line. One thing we discussed was that so many companies who develop software are ignorant of the economics of software development and the Total Cost of Ownership (TCO) of software development. Companies often try to save money by choosing the lowest cost software development option. My friends referred to this as sourcing software development on price alone. At the new division I manage, 10XP Solutions, we call this the high cost of “low cost” software development.

What is the high cost of “low cost” software development? This is the tendency for people involved with financial decisions regarding software development to put too great an emphasis on the cost of software developers. I have recently coined the Law of “Low Cost” Software Development that states, “In the absence of additional information and a lack of understanding of the economics of software development, the choice of software developers will be based on cost alone.” Unfortunately, in practice this law leads to an overall higher cost of software development.

One example I use frequently involves the concept of technical debt. Once I was asked to write an article on technical debt. When my article was published, it was stripped of a definition of technical debt because the editor believes everyone in software development knows what technical debt is. I protested, but to no avail. Nevertheless, I still maintain this assumption is not only incorrect, but it is fundamentally dangerous. I have found the number of people who make decisions affecting the creation of software who do not understand, or may have not even heard of, technical debt remains shockingly high.

For those uninitiated with the concept, technical debt was invented by Ward Cunningham as a metaphor to explain the real cost associated with short-term decision-making and shortcuts taken in software development. A classic example would be the first time code is shipped and market forces dictate that speed to market trumps all other concerns. Cunningham’s conception would be that technical debt would be a conscious decision made with the trade-offs known. Over time it seems that technical debt has morphed because these days a great deal of debt accrued by organizations is unconscious. In other words, they are creating technical debt with little or no awareness. In these cases technical debt is like high blood pressure – a silent killer.

There are now ways to quantify technical debt. The CRASH Report calculated the cost to remediate technical debt concluded that the current levels of technical debt average $3.61 per line of code (the amount is higher for Java code at $5.42). As technical debt increases so does the complexity of the code and the difficulty in making changes to existing code. This means that new features added to existing code will take much longer to develop. How much longer? A study by Dan Sturtevant at MIT, entitled “Technical Debt in Large Systems: Understanding the cost of software complexity” found that complex (technical debt-laden) code resulted in:

Up to a two-fold increase in the amount of time to enhance software (50% decrease in developer productivity)
Up to a 310% increase in defect density
Developers working on poor quality code had up to a 10x increase in employee turnover

How these translate into hard dollars may be difficult to determine, but we can certainly infer that defect laden code increases the maintenance cost, QA cost, tracking cost, defect reporting cost, and costs related to poor customer satisfaction. The most surprising finding was developers working on poor quality code had a greatly increased amount of turnover. There is as obvious hard cost for replacing already difficult to find developers, but also untold morale cost for those who remain.

Total cost of ownership (TCO) addresses the total cost of software development from inception to sun setting. In 2011, the CRASH report stated the total cost of ownership for software code was $18/Line of Code (LOC). Of this, it is generally accepted that the majority of this cost is related to the maintenance of the software after its initial creation, with estimates ranging from 60-90%.

Because the majority of our cost involves maintenance it doesn’t make a great deal of sense for us to spend our effort trying to pare down the initial cost of development by employing lower cost developers. If we use less expensive resources, we could expect technical debt to increase. As technical debt increases, so too does the cost of maintenance. If we assume a slight increase in technical debt (50%) which results in maintenance negatively impacted by only 33%, our “low cost” resources have now actually cost us $4.92/LOC more. In contract, an approach that focuses on higher quality, while a little more expensive in initial cost, results in overall savings.

Strategy	Average	“Low Cost”	High Quality
Initial Cost	4.50	2.25	6.75
Technical Debt	5.42	8.13	2.71
Maintenance	13.50	18.00	7.50
Total Cost	23.42	28.38	16.96

There are other factors to consider in addition to just initial cost, technical debt and maintenance. Many people employing the “low cost” software development model are rarely paying attention to another hidden cost – the cost of delay. Frequently a trade off is made between cost of developers and productivity with lower cost developers being less productive. This results in software products that take longer to produce and deploy.

Of course, because these developers are lower cost, one could always just throw more people at the problem, which is often done. However, adding more people to solving software development problems does not result in a corresponding increase in productivity and obviously eats into cost “savings”. There are many who believe doubling the number of people results in a doubling of productivity. This is an example of applying mechanistic thinking to knowledge problems. There are numerous studies indicating increasing the size of teams results in productivity increases much lower than expected. Therefore, “low cost” development leads to longer cycle times and a higher cost of delay.

While it may be seductive to think that you can save money on software development by using “low cost” developers, it rarely results in overall cost savings when considering TCO and cost of delay. The cost of delay and technical debt are generally hidden costs (at least on the balance sheet). Over the years I have had numerous discussions with software development professionals (CIO, CTO, Development Managers, Product Owners, etc.) regarding the “low cost” software development models and there is nearly a universal befuddlement over why the model continues to flourish. Unfortunately, many people making financial decisions regarding software development resourcing simply do not understand the nature of software development and TCO. If they did, my guess is that they would make drastically different decisions.

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Android | RSS

28 September, 20154 January, 2016

Accenture Ends Annual Review (and Admits Earth Orbits the Sun)

Recently I read an article from the Washington Post titled “In big move, Accenture will get rid of annual performance reviews and rankings” which was in a section called “On Leadership”. I like my headline better (though it has probably blown any shot I had to work at Accenture).

To classify this as somehow a bold or “big move”, place it in a section titled “On Leadership” only highlights the complete lack of scientific understanding of what actually works and does not work. This is not leadership, but a damning indictment of the lack of leadership in the workplace. The real issue is this is “big news” and it takes companies so long to not only admit the obvious, but to act on overwhelming scientific evidence suggests some of their unquestioned practices are utterly wrongheaded.

In the case of annual reviews we have such a wealth of evidence they do not work it is amazing so few companies have actually done away with them. Daniel Pink, in his book Drive, states

…performance reviews are rarely authentic conversations. More often, they are the West’s form of kabuki theatre – highly stylised rituals in which people recite predictable lines in a formulaic way and hope the experience ends very quickly.

Pink’s book has been out for six years now, but it is based on scores of science that was conducted years and decades prior to its publication. We don’t need to look farther than W. Edward Deming, who wrote way back in 1986, in his seminal book “Out of the Crisis”, that annual reviews are “Deadly Disease #3” and

The performance appraisal nourishes short-term performance, annihilates long-term planning, builds fear, demolishes teamwork, nourishes rivalry and politics… it leaves people bitter, crushed, bruised, battered, desolate, despondent, dejected, feeling inferior, some even depressed, unfit for work for weeks after receipt of rating, unable to comprehend why they are inferior. It is unfair, as it ascribes to the people in a group differences that may be caused totally by the system that they work in.

What amazes me most about this annual “kabuki theatre” is everyone intimately involved with the system (managers and front-line employees) literally hates this process. Even if the people at the top of the company bury their heads in the sand when it comes to real, scientific evidence, how can they fail to acknowledge the tidal wave of anecdotal evidence from their own people?

Steve Jobs was once asked how he learned to run a company in his early 20s since he had no formal business training. His answer sheds a great deal of light on why only 6% of fortune 500 companies have gotten rid of annual reviews and rankings though the evidence is overwhelming that they do not work.

You know, throughout the years in business I found something, which was I’d always ask why you do things. And the answers you invariably get are, “Oh, that’s just the way it’s done.” Nobody knows why they do what they do. Nobody thinks about things very deeply in business. – Steve Jobs – The Lost Interview

So, if Jobs was right, as I believe him to be, then a plausible explanation would be that even highly compensated CEOs are unable to properly ask the “why” of things or perhaps top-down control has led to a culture where why is not asked out of fear or complacency. I would guess the CEO of Accenture, since he is quoted in the article, was the decision maker of the “big move.” While he might think himself progressive since he is on the vanguard with respect to his peers, the more appropriate “why” to ask would be “why did it take so long?” and “why haven’t others made the change?” or maybe “which CEO will be next to admit that the world is indeed round?”

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Android | RSS

14 September, 2015

One Effective Interview Question for a Scrum Master or Agile Coach

I have had the good fortune to be involved with Agile software development for close to a decade now. During that time I went from a somewhat skeptical Director of Software Development to someone who now makes his living helping others create great software by applying the Agile philosophy as an Agile Practice Director. One of the things I have noticed during this time is, as Agile (and especially Scrum) have become more mainstream, the quality of individuals calling themselves scrum masters and agile coaches has become more variable.

The problem is certainly not new or particular to Agile Scrum Masters and Coaches, However, I feel that that, given their centrality to software development teams, the fact that Agile is new to many organizations and in many cases these individuals are expected to aid in transformation, having an ineffective Scrum Master or Coach can lead to disproportionally negative effects. For example, a single inadequate developer can be a real drag to a team, an ineffective Scrum Master can be detrimental to multiple individuals and multiple teams and the same inadequacy at the Coach level can lead to wholesale bedlam for large parts (if not the entire) organization.

The problem is further exacerbated by, as I (and others) have pointed out in the past, by a weak and inadequate system of certification. It is good business to mint as many scrum masters as possible from a two-day training, but it does the outside world no service. This begs the question – in a world where every former project manager with a CSM calls themselves a scrum master and every scrum master with a few teams under their belt calls themselves a coach, how can I really assess those who have talent from those who do not?

Of course, one could use someone’s resume to assess the breadth of work. How many different teams has one worked with? How many years has one been involved with scrum? How many different organizations has one coached at? Unfortunately, individuals looking for work have a tendency to prevaricate at times so how is one to know if that time was really spent doing Agile or did I conveniently reclassify this work because it was not “entirely waterfall”?

A good deal of my current work is trying to make these kinds of judgments. If you too need to assess the ability of a scrum master or coach, let me share the question that I have found to be most effective:

What book have you read recently that has changed the way you think about Agile and the way you go about your job as a scrum master (or coach) and why?

If you are merely a paper candidate who has relied on the minimum set of qualifications (ie you have a good number of certifications) then this question is very difficult to answer. Being a scrum master or coach requires a level of passion that transcends merely implementing a small set of learned behaviors. It requires an active mind that is constantly seeking new ways of thinking,, new experiments to try, new behaviors to model – any and everything that one can do to truly be a servant leader. Passionate people always want to improve. There is no better indication than reading as a means of improving one’s ability.

The question not only shows ones ability to think critically, but it also indicates a higher level of maturity. To read something that changes ones view requires the ability to realize ones knowledge is never perfect or complete. I have to acknowledge that something I hold to be true today may not be true or that I have a true ignorance of certain things.

Agile and scrum are about continuous improvement. We should not expect our scrum masters and coaches to have all the answers any more than we can expect our teams to function perfectly. There is always room for improvement and it is the good scrum master and coach who is not only able to admit this, but also has taken the steps necessary to improve. One of the most accessible ways for this improvement is to read books and to apply this knowledge. So the next time you are asking yourself “Is this scrum master or coach any good?” perhaps you might want to ask them this simple question:

What book have you read recently that has changed the way you think about Agile and the way you go about your job as a scrum master (or coach) and why?

9 September, 20159 September, 2015

Type Three and Four Errors – Solving the Wrong Problems Flawlessly

For some reason, I find myself attracted to statistics and probability. This is one of the reasons I love the books of Nassim Taleb, anything that Stephen J. Dubner and Steven D. Levitt of Freakonomics fame write, the social economics work of Dan Ariely, etc. Most recently I was doing some research and ran into the work of Ian I. Mitroff and Abraham Silvers and “The Error of the Third Kind (E3)”.

Because I had read so many books that have statistics and probability as themes, I was familiar with Type One and Type Two errors. Type One errors can be defined as incorrectly detecting an effect that is not present. Type Two errors, on the other hand, occur when one fails to detect an effect that was present. However, Type Three and Type Four errors were new to me.

An Error of the Third Kind (E3), as presented by Mitroff (and others), are those where the wrong problem has been not only proposed, but it is solved flawlessly so that it has the perception of being correct. Mitroff states:

The vast majority of books on management contribute to E3 because they imply that managers already know what their important problems are, for example: how to downsize in the most efficient way, how to improve chances for success in the global economy, how to install the correct Total Quality Management program, how to design the right reward system and so on. In each case, the unstated assumptions are that the essential problem the organization is facing is downsizing, global competitiveness or whatever it may be. While the assumptions may be correct, they are so crucial to formulating a problem correctly that they deserve to be challenged in the strongest possible way by asking tough questions.

This type of error reminds me of the argument I often hear from clients that “we have always delivered” or “waterfall has worked for us for years.” The real question that needs to be answered is not if one has “always delivered” it is “what has one always delivered?” If, in software development, the code is difficult to maintain, if it has a great deal of technical debt (for example, only concentrating on on-time delivery), can it really be said that one has delivered successfully?

When someone mentions “waterfall has always worked for us”, I believe this is an example of Type Three error. The real question – has waterfall been optimal? The example I always give is that the covered wagon was successful for transportation, but when I look out the window of a plane, I don’t see any crossing the prairie. In the case of those applying the values and principles of agile properly there is little doubt as to which is the airplane and which is the covered wagon.

Type Four errors may be even more insidious. These are sometimes better known as HiPPO errors (Highest Paid Person’s Opinion). These are seen in companies where top down hierarchical control is the norm. In their book, Dirty Rotten Strategies: How We Trick Ourselves and Others Into Solving the Wrong Problems Precisely, Mitroff and Silvers state:

The Type Three Error is the unintentional error of solving the wrong problem precisely. In sharp contrast, the Type Four Error is the intentional error of solving the wrong problems…

The Type Three Error is primarily the result of ignorance, a narrow and faulty education, and unreflective practice. In contrast, the Type Four is the result of deliberate malice, narrow ideology, overzealousness, a sense of self-righteousness, and wrongdoing.

By allowing others to improperly frame the question, a troublesome groupthink is created that allows many people to solve the wrong problems precisely.

As companies go down the road to Agile transformation, it is helpful to understand the Type Three and Four errors so that we don’t fall into the trap of solving the wrong problems flawlessly.

21 August, 201528 August, 2015

A New Chapter Begins

I recently accepted the position of Agile Practice Director for 10XP Solutions, Inc. (an affiliate of Systems Integration Solutions (SIS), Inc.) – a new and very exciting chapter in my life.

As always, the beginning of a new chapter is bittersweet. I really enjoyed working with the people at Apple and wish all of them continued personal and professional success. Fortunately I was able to handpick my replacement so I know they are in good hands and that they will continue on their Agile journey.

I am very excited about my new position because I will have the opportunity to create a brand new practice whose primary focus is creating great agile teams. While it is widely known that good agile teams are small, co-located, dedicated, stable and cross functional, there are very few folks in software development that actually get to work in such an environment. The possibility of actually creating such an environment is an honor and a dream come true. Additionally, we have the chance to operate like a startup but still have the backing of a strong, staffing company with over 25 years experience. The fact that I get to work with some really incredible people at SIS is amazing. I really appreciate their warm welcome.

The world is always changing. The world of staffing needs to change as well. As more and more organizations try to become Agile, they are finding it difficult to create optimal agile teams. We can provide intact agile teams to these and also to startups who need to ramp up quickly but may find it difficult to ramp up quality teams quickly.

We call our new effort 10XP Solutions. “10X” comes from the well known adage that the best programmers are ten times more productive than the least. The “XP” alludes to Extreme Programming whose emphasis has always been on high quality code, something that is central to the value that we bring to our clients.

I have always been fortunate and I am grateful to Peter Ling (CEO of SIS) for giving me this opportunity, the people at SIS for their support as we move into new territory and, most of all, to my wife and son for allowing me to pursue my dream.