Agile Values: Why Contracts And Software Development Don’t Mix

larry apke agile doctor

customer collaboration

“Customer collaboration over contract negotiation”

We all have customers. If we didn’t there would be no reason to do what we do. If we didn’t their would be no one to pay our invoices. And when someone agrees to pay you for work, they generally want to have some kind of agreement on the nature of the work for the money that is being paid. This agreement is usually put in writing and voila, we have a contract. This is an important part of the process and as everyone knows, contracts are valuable documents for both the customer and yourself. But as the Manifesto states, it’s important to not get caught up in negotiation fever.

While it is a good thing to have an agreement before work begins, there are a number of unfortunate aspects to even having such an agreement.

The main problem is complexity.

Software development, by its very nature, is a complex endeavor, dependent on communication and creativity to succeed. While it would be wonderful if all our customers were omniscient, they are often far from it. If we are building a house, we could easily choose things like materials and the architecture of the project and expect that the final building will resemble what was originally agreed upon. Such a thing is rarely true in something as complex as software development, but it hasn’t stopped people from trying (even today).

There is a field of study called complexity theory that applies well to software development in my opinion. In his article, ”On Understanding Software Agility—A Social Complexity Point Of View”, Joseph Pelrine writes:

Many people still regard building software as a complicated undertaking, but in fact it is a defining example of a complex or a ‘wicked’ problem. The concept of wicked problems was originally proposed by Horst Rittel and Marcus Webber. Wicked problems have incomplete, contradictory, and changing requirements; and solutions to them are often difficult to recognize as such, because of complex interdependencies. Rittel and Webber stated that while attempting to solve a wicked problem, the solution of one of its aspects may reveal or create other, even more complex problems. Rittel expounded on the nature of ill—defined design and planning problems, which he termed ‘wicked’ (that is, messy, circular, aggressive) to contrast against the relatively ‘tame’ problems of mathematics, chess, or puzzle solving.

If the nature of software development is indeed ”wicked” then trying to agree on all the requirements at the outset in contractual form is not only wasteful but counterproductive. A much better way is to form an understanding with broad brush strokes and work together to attack the problems, hence the need for collaboration over contract negotiation.

The same is true when the customer is internal. This is why Scrum has someone called the product owner who is available daily to the team to work on the software. Instead of assuming we know everything at the beginning and then have mind—numbing change reviews and contract addenda, is it not better to forge a lasting partnership? Or as Deming calls it, “a long-term relationship of loyalty and trust”.

Larry Apke

Be sure to follow me on Twitter and Quora! 

Agile Values: Working Software Over Documentation

Papers everywhere

“Working software over comprehensive documentation”

Of the four agile values, this is probably the least understood and most often misinterpreted. It certainly does not say that there should be no documentation as some (the less ambitious developers and teams) propose. It says that there is more value to actual software than comprehensive documentation.

While I have a great deal of affection for the Agile Manifesto (I believe that dedicating an entire series of blogs to just this particular topic demonstrates this abundantly), I think that the original writers could have been more specific to remove some of the confusion and misinformation that has sprung up around this value. It might have been more appropriate if the original writers would have said, ”Working software over comprehensive requirements and design documentation,” because I think this is more what they meant.

This level of specificity would keep teams from using the excuse that there should be no documentation at all. We have to remember that the writers of the manifesto were doing so as a reaction to what they perceived to be short comings in the prevalent way software was delivered. In that phase—gated approach, copious documentation of requirements and design were produced before any substantive software was developed. Prior to the manifesto, comprehensive documentation could be delivered but working software was either not delivered or poorly delivered because so much emphasis was placed on requirements and design documentation.

In other words, documentation is important but it should be less important than actually building the software. What is the use of great documentation of a system that is poorly or not fully built? Also, documentation tends to be easily outdated so I suspect that the writers of the manifesto were also alluding to this fact. Why create copious amounts of documentation that does not match the final product? Or why spend all our time updating copious documentation as the product changes?

There are usually two types of documentation — internally facing documentation, like design and requirements documents, and externally facing, like user manuals. So far my argument has been for these internally facing documents. Agile addresses these with user stories and acceptance criteria. I like to attack internal documentation using Behavior Driven Development (BDD) since then the code always matches the internal system documentation (see Specification by Example).

What about external documentation? It might be that they would argue that good design would preclude the need for end-user documentation. I suspect the writers of the manifesto would agree since they are user-focused and that they would be more supportive of more of this type of documentation versus internal documentation.

Larry Apke

Also be sure to check out the other post in this series:

Agile Values: The Importance of Individuality

Agile Values: The Importance of Individuality

hands together

agile manifesto

Recently, I have witnessed a disturbing trend.

I’ve witnessed some companies who are trying to transform themselves into ”agile” organizations without ever referring to the original philosophical underpinnings of the Agile Manifesto and its principles. And in some cases, I have heard of companies or divisions within a company rewriting the entire Agile Manifesto, sometimes removing verbiage that would either challenge the existing company culture or would be too difficult to realize in their current environment. I have also worked for or know of consulting companies wanting to make money by creating their own principles and methodologies.

As I said before, I find this troubling.

Hollow ceremonies are ways to “do” agile, but in order to achieve true agility we must be Agile.

In an effort to emphasize the values and principles of the Agile Manifesto, I have decided to create a series of blogs with my thoughts on each value and principle.

With Agile we are now uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value certain things, one of which is:

Individuals and interactions over processes and tools

That is, while there is value in the items on the right, we value the items on the left more.

The Agile Manifesto is best seen as a reaction to the environment at the time. Many authors are saying that software development values have gotten out of whack because the majority of companies now value processes and tools more than individuals and interactions. Obviously in order to create better software this needs to be corrected!

Those of us challenged with transforming organizations to Agile see the attractiveness of putting processes and tools over individuals and interactions. There are a great number of individuals in positions of power (and influence) who believe that only if they can find the right processes and get people to slavishly follow then we can deliver. In this mindset, people are “resources”; chess pieces to be moved around. The chance to be master of the universe is intoxicating and plays well into our mistaken notion that we can be in control. However, let me be clear that it is a fool’s errand because it doesn’t work well in the real world and doesn’t work well with software development.

In some respects it reminds me of socialism in that if we only move the right levers and offer the right incentives, we can get complex social constructs to behave in predefined ways. And the ideas around process management can be traced back to the theory of a mechanistic world that can be known and tamed. We would be better to look at theories of complex systems for inspiration.

Some things look great on paper (the Dallas Cowboys would always win if they didn’t actually have to take the field), but what looks good in theory doesn’t always work well in practice. The Soviet Union fell because grand theories and plans do not always work well with real human beings because humans are not cogs or machines and will not behave predictably. For instance, real humans do not like being managed (especially micro-managed!) and workers are motivated more by intrinsic factors than extrinsic rewards.

Humans are messy. They are individuals. They work well with some people and not well with others. They cannot be treated as if they are all the same. I have three children of different ages, therefore I treat the seven year-old much differently than the twenty-one year-old. And though they share genetics and a similar environment (parents, socio—economic, etc) they each have their own personality and how we interact and react to each other is vastly different. There is not one playbook that will necessarily work with each child so why would we expect it to work with every individual at work? Of course, we all strive for fairness and there are certain ways we should all be treated (this is where HR comes in), but one process fits all? I think not.

Because teams are made up of individuals, every team has its own individual personality. Even if I found a process that works well with one team, it might not work with another. That is not to say that there isn’t universal processes that will work with almost every team, but the more prescriptive and detailed the less likely it is that it will work with everyone (think waterfall). This is why agile methodologies, like scrum, rely on fewer controls. The philosophy of Agile should be shared by all teams, but even the basic methodology (scrum vs kanban, etc) need not be shared by all teams because the needs of the teams and the individuals comprising the teams is not shared.

Just as one process will not work with all individuals or teams, one tool (or set of tools) cannot work for all teams. The manifesto does not say that tools are not important, but their use should be subservient to the individuals doing work and should not interfere with the interactions among individuals.

Software development is communication and creativity

It only works when people are treated like individuals so their inherent creativity can be evidenced. Furthermore, individuals on a team must constantly interact but processes and tools are often in the way of this collaboration. The writers of the manifesto knew that processes and tools were necessary to software development, but our very human need to have an illusion of control can lead us to value processes and tools over individuals and interactions.

Larry Apke

Understanding The Agile Manifesto: A Brief & Bold Guide to Agile – Podcast Now Available

Understanding The Agile Manifesto: A Brief & Bold Guide to Agile by [Apke, Larry]Those of you who may follow this blog may have noticed it has been some time since the last post. It was an eventful summer and it was good to spend some time to catch my breath. Now that the kids are heading back to school, it is time for me to renew my posting. In addition to posting the podcasts, please check back for new Agile posts.

I am pleased to announce that my first book, Understanding the Agile Manifesto: A Brief & Bold Guide to Agile is now available as a podcast. Over the next few days / weeks I will be releasing the book, chapter by chapter, on this website and through the iTunes store under my “Agile Doctor” podcast. Since the book was a series of blogs, the podcasts will also be made available under the original posts.

This project is very dear to my heart as the podcasts were recorded by my son, Vadim Kim, who assisted me over the summer as my intern. I think he did a fantastic job of recording my “little” Agile book and I am sure you will enjoy.

The most recent version of the book includes an Introduction which was never part of any individual post. I have included the podcast of the Introduction below.

 

The Five Attributes of a Good Scrum Team

One of the best experiences I can have as an Agile Coach is presenting Agile and Scrum to people with little or no experience. It grounds me and after many years of living Agile, keeps me from falling into the Curse of Knowledge cognitive bias. Team01smallBetween these presentations and my daily coaching practice, I am reminded yet again at the importance of forming a good scrum team. This, along with proper backlog compilation and maintenance, can make the difference between success and endless frustration. In my experience, proper scrum team formation is the area where companies who are unsuccessful in Agile transformations fail most often.

Over the years, with the help of my teams and the many knowledgeable colleagues that I have worked with, I have distilled everything I have learned and come up with five attributes of a good scrum team. I use it much like a mantra. Those attributes are: Small, Co-Located, Dedicated, Stable and Cross-Functional. That is not to say that a good scrum team will not have other elements or that you can’t witness improvement without all five, but like the sculptor who is only done when there is nothing left to take away, these five represent an ideal essence of a good team. None of these five things can be removed without real consequences to the success of the team.

Small

I write about size at length (and mention the five attributes) in a previous post, When it Comes to Software Development – Size Matters, so I encourage you to read it for some eye-opening account of a major study that confirms this fact. Suffice it to say that Software Development is about communication and collaboration. It is complex knowledge work. In a physical, manufacturing model, adding a person to the effort would result in the expected proportional increase in productivity. In software development, however, adding an additional person merely exponentially increases the number of communication channels whereby any expected increase in productivity is quickly overtaken by the number of communication channels created. There is a reason Agilists recommend small teams. Basically, you can save money because your cost per unit of productive (and quality) work decreases.

Co-Located

Co-Location seems to be an issue with all of the companies that I consult for. We know that the best software developers can be 10 times or more productive than the least. We know that face-to-face communication is multiple times better than any of the substitutes we employ when that mode is not available. We all laugh at the Youco-location01smallTube video “A Conference Call in Real Life”. We know that the unacknowledged creation of technical debt will cause us financial havoc for years to come. Not convinced, I encourage you to check out my blog on the High Cost of Low Cost Software Development. And yet we still continue to source our software development to low bidders around the globe. Here’s the really interesting thing. For many years I have asked a simple question – does it work? I get two answers. The majority shake their head, laugh, and provide a solid “No”. The optimists of the group say that “We have found a way to make it work.” I also see a great deal of blogs that are titled “How We Found a Way to Make Offshore Work”, but it takes little research skills to uncover the author as someone employed by a company that works offshore. Never is there an enthusiastic “Yes.” Most would bring the work back if they could, but it appears that there is a mysterious formation in many software development organizations that continues to think they are doing the right thing by piece-mealing work to various locations. I can only guess it is accounting, but I have yet to meet anyone whom I respect who actually understands software development who would recommend it.

Dedicated

I often rail against the project-centric focus of many software development organizations. For more information, I encourage you to check out my blog “Why Project Focused Mentality is Killing Software Development”dedication. The problem is that in projects we build teams around projects so that one, especially a key employee, will be spread over multiple concurrent projects. There are a number of very serious negative effects to doing this. First, context shifting means that every time a developer must change their concentrated area of focus they will lose 15 minutes minimum. If I am working on two different projects concurrently, there is not a huge loss, assuming that I might work on one in the morning and one after lunch, but many developers are bounced from thing to thing like so many pinballs with context shifting throughout the day. Hours each day are wasted and no project gets the attention it deserves. A better way is to create a small, co-located scrum team around a product backlog and bring the work of these multiple projects to the team via their backlog. The ability for individuals on the team to not fall prey to context shifting will allow them to focus on their work, even at times leading to a state called Flow in which developers (and athletes, artists, etc) are at their peak performance. For those interested, I refer you to the work of Mihaly Csikszentmihalyi.

Stable

 Project-centric thinking also results in teams that are short-lived. We put the team together for a particular period (sometimes relatively short) of time. When the project is done, the team is disbanded. This causes a number of issues. First, every team goes through a period of “storming, norming, and performing.” This is easy to see visually when you plot the velocitiScaleStability01es of multiple teams on a timeline. You can visually see the storming period as teams learn to work together as a team instead of a collection of individuals, through very low (if non-existent) velocity. In my experience, I have come to the conclusion that I expect nearly nothing from a brand new team for the first couple of iterations. It is not until they reach iterations 3, 4 or even 5 that they will reach what is their performing velocity. Therefore, every time I break up a team, I have a very high overhead associated with the team’s “storming and norming.” Another benefit of keeping the same people on teams for a longer period of time is by simply learning how to better communicate with each other over time. Since software development is about communication and collaboration, the more time we spend together, the more effective we become in communicating. I remember reading about the notion of “hyper-productive” teams from Scott Downey and Jeff Sutherland.  The interesting thing is that their “hyper-performance” did not manifest until after the team had been together for some time.

Cross Functional

 In some respects, cross functionality of people is a by-product of the fact that team size should be kept small. If there are 15 things I need to do to get my code into production, I obviously will not be able to do so if there crossfunctional01needs to be an expert for each piece. The creation of “silos” is great if your goal is to create a MDD (mortgage driven development) environment. It is bad for the flow of the product. Silos create queues. Queues, if you want to deliver something quicker, are bad. I highly recommend the book Principles of Product Development Flow by Don Reinersten for a deeper understanding of queuing theory and the damage that out-of-control queues can have. This book talks about the concept of generalizing specialists. In my experience, developers are more motivated by status and mastery of new technology and ideas than any other group of people I have worked with. Creating an environment where developers are expected to know more than a single specialized domain is not only good for moving the software through the system quicker, but it also leads to greater developer engagement. A team should be responsible for all aspects of the software product and the development cycle. This cannot be accomplished without having team members who are cross functional.

Final Thoughts

 An old African proverb states, “If you want to go fast, go alone. If you want to go far, go together.” In order to be successful in software development we need to understand teams and team dynamics. My many years of experience working with what I would estimate around a hundred development teams has led me to conclude that there are five basic attributes for an optimal development team – small, co-located, dedicated, stable and cross functional. I hope that these five attributes help you as you continue your quest for agility.

 

Will Your Development Practices Shield You From Malpractice?

Gavel

GavelThere are a number of reasons why companies who develop software should use Agile. Agile is superior to phase gate approaches like Waterfall in quicker time to market, the practices associated with Agile produce a much higher quality product with less technical debt, maintenance of software (which is the bulk of the SDLC in terms of time and money) is much easier, the engagement of employees is much higher and results in less employee turnover, frequent feedback results in better risk mitigation (to name a few benefits). As I look to the future of software I believe we can begin to add one more thing –protection from software malpractice litigation.

The thought came to mind recently as I was listening to an NPR segment on autonomous automobiles. According to the story, a large segment of the American public is hesitant to trust their safety to autonomous vehicles, even though there are some estimates that cite a potential 80% reduction in traffic fatalities. As human beings we are comfortable with the notion of human error, but find it difficult to find comfort in computer, especially software, errors. We find it especially difficult to accept those that would result in loss of life, even if the average loss of life were greatly reduced. I think that the potential for litigation (and concurrent drag on profits) of such defects is quite high. Not only do I expect software defect litigation to be increased because of autonomous vehicles, the Internet of Things (IOT) and the corresponding explosion in software in all areas of our lives will also create scenarios for increased litigation.

Law BuildingIn some research for this article, I unearthed a paper by Cem Kaner that speaks directly to this issue. While I do not possess the legal mind that he does, I believe I can understand the two potential issues of Negligence and Malpractice he outlines. In regard to negligence, he states, “How do we decide whether a product was developed or tested negligently?…A critical issue to keep in mind is that the plaintiff must prove that the failure to use some ‘best practice’ actually caused the defect in the system.” As software becomes ubiquitous, the methods used to create software will begin to be questioned more often. I believe there to be overwhelming evidence that software developed using Waterfall methods tend to result in greater technical debt, and therefore, the possibility for being on the losing end of a lawsuit is greater. In other words, improper development methods that have been (and continue to be used in many places) not only result in greater technical debt, but also will expose companies to more potential lawsuits. Should I ever be called as an expert witness it would cause me no compunction to reveal poor development processes and practices. We have all heard of lawsuits that have been filed, litigated and judgments made for less.

With regards to malpractice, Kaner states, “Before calling for the professionalization of software quality advocates, please consider the problem that we need a solid basis for distinguishing unacceptable from acceptable practices.” Of course, this paper was written back in 1997 so the amount of evidence available for determining unacceptable from acceptable is growing and the evidence is strong that Waterfall development is no longer the best or accepted practice.

Judge StatueMy argument is this – most software development is complex. It is a well-established truth that complexity is best attacked through frequent feedback, the kind supplied by the agile methodologies and frameworks, as well as the commonly applied practices associated with XP like pair programming, BDD/TDD, code reviews, etc. Software that has been written using methodologies other than Agile generally result in greater number of defects and higher degree of technical debt. As these things are generally accepted by software development professionals as better ways to produce quality software, companies that do not use Agile will become exposed. Furthermore, studies indicate hours worked beyond about 30-40 per week can result in increased numbers of defects. Companies employing this already counter-productive measure have yet another reason for adhering to the agile principle of sustainable development in avoiding potential litigation.

Perhaps malpractice litigation will not affect the realm of software development as I anticipate, but that does not mean it is not appropriate. In some cases, people who with authority to make decisions regarding software development show a willful ignorance of the nature of software development. I believe their behavior is not only detrimental to the production of quality software and the satisfaction of customers and employees alike, but certainly borders on the realm of malpractice. Maybe the hint of malpractice will incentivize people making software development decisions to treat development and developers in a manner more congruent with the true nature of software development.