technical debt - Agile Doctor

14 March, 201628 June, 2016

Will Your Development Practices Shield You From Malpractice?

There are a number of reasons why companies who develop software should use Agile. Agile is superior to phase gate approaches like Waterfall in quicker time to market, the practices associated with Agile produce a much higher quality product with less technical debt, maintenance of software (which is the bulk of the SDLC in terms of time and money) is much easier, the engagement of employees is much higher and results in less employee turnover, frequent feedback results in better risk mitigation (to name a few benefits). As I look to the future of software I believe we can begin to add one more thing –protection from software malpractice litigation.

The thought came to mind recently as I was listening to an NPR segment on autonomous automobiles. According to the story, a large segment of the American public is hesitant to trust their safety to autonomous vehicles, even though there are some estimates that cite a potential 80% reduction in traffic fatalities. As human beings we are comfortable with the notion of human error, but find it difficult to find comfort in computer, especially software, errors. We find it especially difficult to accept those that would result in loss of life, even if the average loss of life were greatly reduced. I think that the potential for litigation (and concurrent drag on profits) of such defects is quite high. Not only do I expect software defect litigation to be increased because of autonomous vehicles, the Internet of Things (IOT) and the corresponding explosion in software in all areas of our lives will also create scenarios for increased litigation.

In some research for this article, I unearthed a paper by Cem Kaner that speaks directly to this issue. While I do not possess the legal mind that he does, I believe I can understand the two potential issues of Negligence and Malpractice he outlines. In regard to negligence, he states, “How do we decide whether a product was developed or tested negligently?…A critical issue to keep in mind is that the plaintiff must prove that the failure to use some ‘best practice’ actually caused the defect in the system.” As software becomes ubiquitous, the methods used to create software will begin to be questioned more often. I believe there to be overwhelming evidence that software developed using Waterfall methods tend to result in greater technical debt, and therefore, the possibility for being on the losing end of a lawsuit is greater. In other words, improper development methods that have been (and continue to be used in many places) not only result in greater technical debt, but also will expose companies to more potential lawsuits. Should I ever be called as an expert witness it would cause me no compunction to reveal poor development processes and practices. We have all heard of lawsuits that have been filed, litigated and judgments made for less.

With regards to malpractice, Kaner states, “Before calling for the professionalization of software quality advocates, please consider the problem that we need a solid basis for distinguishing unacceptable from acceptable practices.” Of course, this paper was written back in 1997 so the amount of evidence available for determining unacceptable from acceptable is growing and the evidence is strong that Waterfall development is no longer the best or accepted practice.

My argument is this – most software development is complex. It is a well-established truth that complexity is best attacked through frequent feedback, the kind supplied by the agile methodologies and frameworks, as well as the commonly applied practices associated with XP like pair programming, BDD/TDD, code reviews, etc. Software that has been written using methodologies other than Agile generally result in greater number of defects and higher degree of technical debt. As these things are generally accepted by software development professionals as better ways to produce quality software, companies that do not use Agile will become exposed. Furthermore, studies indicate hours worked beyond about 30-40 per week can result in increased numbers of defects. Companies employing this already counter-productive measure have yet another reason for adhering to the agile principle of sustainable development in avoiding potential litigation.

Perhaps malpractice litigation will not affect the realm of software development as I anticipate, but that does not mean it is not appropriate. In some cases, people who with authority to make decisions regarding software development show a willful ignorance of the nature of software development. I believe their behavior is not only detrimental to the production of quality software and the satisfaction of customers and employees alike, but certainly borders on the realm of malpractice. Maybe the hint of malpractice will incentivize people making software development decisions to treat development and developers in a manner more congruent with the true nature of software development.

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Android | RSS

14 October, 2015

Technical Debt on the Balance Sheet

My recent article on the high cost of “low cost” software seems to have struck a nerve with lots of views, comments, likes, shares, etc. A lot of software development professionals feel the weight daily of the tendency to chase the lowest price of software development and have added to the conversation I started last week. Below are some of the comments:

There is always a price for “free.”

In the end, it’s all about understanding the nature of software development in order to make wise decisions.

As the saying goes: “if you believe a senior developer is expensive, wait till you hire a Junior”.

Another aspect of poorly written code is the inefficient use of hardware which it runs on … leading to negative user experience .. Resulting in losing $$ in terms of user productivity .. Adding to the higher cost !!

My unscientific guess is corporate America is wasting billions each year on “low cost software “.

It’s funny how there never seems to be enough time to do it right, but always enough time to do it twice.

The technical debt gets worse when you add an army of developers to it.

The old saying “you get what you pay for” rings true.

Perhaps the most interesting comment came from Jason Ross who stated:

It’s often the case that people get “value” and “cost” confused. Technical debt is a very debt that must eventually get serviced. In the same manner that debt service kills cash flow and becomes a drag on the balance sheet, so does technical debt deprive an the ability to deliver new features and effectively update the code base as new technologies arrive that could be leveraged. In the long run, the maintenance and upkeep costs becomes the dominant factors in many things, and especially in software, they are often entirely neglected as factors.

When I read his comment and saw his mention of “a drag on the balance sheet” it reminded me of an article I had read a few years back from Israel Gat for his blog The Agile Executive titled Technical Debt on Your Balance Sheet

Israel’s assertion is now that there are ways to rather easily assess technical debt (like SonarQube with SQALE plugin), technical debt could (and should) be added to a company’s balance sheet. I applaud and support his efforts.

Unfortunately, it appears that any efforts to include technical debt on a company balance sheet might run into some accounting industry standards that would make this not currently possible. In his post, Clear Costs and Technical Debt, Trent opines

Accounting standards don’t allow provisions or liabilities to be shown for “not performing enough maintenance” or similar intangibles. There needs to be a present obligation before a liability is incurred and a present obligation is only usually formed through a contract.

In his scenario, financial wizards will chalk up the lower cost of software development as a savings.

The accountants will note a(n)… asset on the books, congratulate themselves for saving … capex and chalk up the increased opex of maintenance as the cost of doing business.

My guess is this is precisely what happens now. If Trent is correct, then the issue facing companies developing software is that current accounting practices are inadequate and need to be modified. Technical Debt can be measured and tracked and is a very real liability.

The question remains: can there be a new GAAP around technical debt or can an existing GAAP be instructive? While I am not a CPA, I have done work in accounting in the past so I know “enough to be dangerous.” My line of thought is that technical debt probably resembles something that currently exists in GAAP. After doing some research I think that maybe GAAP around warranties may prove helpful. In reading about warranties in an article GAAP Accounting for Product Recalls. I noticed some similarities:

Product warranties present manufacturers with a bit of a conundrum. …The manufacturer is actually responsible for the product for the length of the warranty. In accrual accounting, the finality of the transaction does not occur until the period of responsibility ends. However, warranties … present a bit of an exception to the rule. (Emphasis mine)

Notice how, like technical debt and the Total Cost of Ownership (TOC) of software, the complete transaction is not final until the period of responsibility ends. I can see that the life cycle of software (and liability of technical debt – present or future) is like the life cycle of a product with a warranty attached. Even I can see how warranties could cause accounting headaches. However,

Rather than being on the hook for the cost of the product for the entire life of the product or throughout the product’s warranty period, manufacturers can instead follow the generally accepted accounting principles regarding products under warranty. … Under GAAP, product warranties and recalls can be reasonably accounted for with the initial sale of the product by estimating the potential warranty expense of a return or recall.

I can reasonable see how it might be possible to substitute an estimate of Technical Debt liability for the similar liability of estimated warranty expense. For example, with each release we classify software as a capital expense (CapEx). At this point we could also include on the balance sheet an estimation of technical debt as a long-term liability.

In the case of warranties, the actual accounting mechanism under GAAP is done by

…making two separate entries, one a debit and the other a credit for the same amount. These are entered into the ledger as a warranty expense and a separate entry as an allowance for warranty costs.

What is to stop a company for entering two separate entries for technical debt – one a debit as technical debt expense and the other a credit under allowance for technical debt? I certainly see this as in the realm of possibility. This way GAAP can be followed AND technical debt can be raised to true financial visibility. It is this visibility that will allow companies to begin seeing the true total cost and guard against the high cost of “low cost” software development.

6 October, 20156 January, 2016

The High Cost of “Low Cost” Software Development

I recently spoke with some software development professionals about the economics of software development and how Agile Values and Principles, when properly applied through a framework like Scrum, could improve a company’s bottom line. One thing we discussed was that so many companies who develop software are ignorant of the economics of software development and the Total Cost of Ownership (TCO) of software development. Companies often try to save money by choosing the lowest cost software development option. My friends referred to this as sourcing software development on price alone. At the new division I manage, 10XP Solutions, we call this the high cost of “low cost” software development.

What is the high cost of “low cost” software development? This is the tendency for people involved with financial decisions regarding software development to put too great an emphasis on the cost of software developers. I have recently coined the Law of “Low Cost” Software Development that states, “In the absence of additional information and a lack of understanding of the economics of software development, the choice of software developers will be based on cost alone.” Unfortunately, in practice this law leads to an overall higher cost of software development.

One example I use frequently involves the concept of technical debt. Once I was asked to write an article on technical debt. When my article was published, it was stripped of a definition of technical debt because the editor believes everyone in software development knows what technical debt is. I protested, but to no avail. Nevertheless, I still maintain this assumption is not only incorrect, but it is fundamentally dangerous. I have found the number of people who make decisions affecting the creation of software who do not understand, or may have not even heard of, technical debt remains shockingly high.

For those uninitiated with the concept, technical debt was invented by Ward Cunningham as a metaphor to explain the real cost associated with short-term decision-making and shortcuts taken in software development. A classic example would be the first time code is shipped and market forces dictate that speed to market trumps all other concerns. Cunningham’s conception would be that technical debt would be a conscious decision made with the trade-offs known. Over time it seems that technical debt has morphed because these days a great deal of debt accrued by organizations is unconscious. In other words, they are creating technical debt with little or no awareness. In these cases technical debt is like high blood pressure – a silent killer.

There are now ways to quantify technical debt. The CRASH Report calculated the cost to remediate technical debt concluded that the current levels of technical debt average $3.61 per line of code (the amount is higher for Java code at $5.42). As technical debt increases so does the complexity of the code and the difficulty in making changes to existing code. This means that new features added to existing code will take much longer to develop. How much longer? A study by Dan Sturtevant at MIT, entitled “Technical Debt in Large Systems: Understanding the cost of software complexity” found that complex (technical debt-laden) code resulted in:

Up to a two-fold increase in the amount of time to enhance software (50% decrease in developer productivity)
Up to a 310% increase in defect density
Developers working on poor quality code had up to a 10x increase in employee turnover

How these translate into hard dollars may be difficult to determine, but we can certainly infer that defect laden code increases the maintenance cost, QA cost, tracking cost, defect reporting cost, and costs related to poor customer satisfaction. The most surprising finding was developers working on poor quality code had a greatly increased amount of turnover. There is as obvious hard cost for replacing already difficult to find developers, but also untold morale cost for those who remain.

Total cost of ownership (TCO) addresses the total cost of software development from inception to sun setting. In 2011, the CRASH report stated the total cost of ownership for software code was $18/Line of Code (LOC). Of this, it is generally accepted that the majority of this cost is related to the maintenance of the software after its initial creation, with estimates ranging from 60-90%.

Because the majority of our cost involves maintenance it doesn’t make a great deal of sense for us to spend our effort trying to pare down the initial cost of development by employing lower cost developers. If we use less expensive resources, we could expect technical debt to increase. As technical debt increases, so too does the cost of maintenance. If we assume a slight increase in technical debt (50%) which results in maintenance negatively impacted by only 33%, our “low cost” resources have now actually cost us $4.92/LOC more. In contract, an approach that focuses on higher quality, while a little more expensive in initial cost, results in overall savings.

Strategy	Average	“Low Cost”	High Quality
Initial Cost	4.50	2.25	6.75
Technical Debt	5.42	8.13	2.71
Maintenance	13.50	18.00	7.50
Total Cost	23.42	28.38	16.96

There are other factors to consider in addition to just initial cost, technical debt and maintenance. Many people employing the “low cost” software development model are rarely paying attention to another hidden cost – the cost of delay. Frequently a trade off is made between cost of developers and productivity with lower cost developers being less productive. This results in software products that take longer to produce and deploy.

Of course, because these developers are lower cost, one could always just throw more people at the problem, which is often done. However, adding more people to solving software development problems does not result in a corresponding increase in productivity and obviously eats into cost “savings”. There are many who believe doubling the number of people results in a doubling of productivity. This is an example of applying mechanistic thinking to knowledge problems. There are numerous studies indicating increasing the size of teams results in productivity increases much lower than expected. Therefore, “low cost” development leads to longer cycle times and a higher cost of delay.

While it may be seductive to think that you can save money on software development by using “low cost” developers, it rarely results in overall cost savings when considering TCO and cost of delay. The cost of delay and technical debt are generally hidden costs (at least on the balance sheet). Over the years I have had numerous discussions with software development professionals (CIO, CTO, Development Managers, Product Owners, etc.) regarding the “low cost” software development models and there is nearly a universal befuddlement over why the model continues to flourish. Unfortunately, many people making financial decisions regarding software development resourcing simply do not understand the nature of software development and TCO. If they did, my guess is that they would make drastically different decisions.

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Android | RSS