agile development - Agile Doctor

9 February, 201618 February, 2016

Hit Rock Bottom? Maybe Now You’re Ready for Agile

One of the things I enjoy most about my work is meeting other Agile practitioners and coaches and sharing war stories. Recently I was at an Agile Coffee with people who had various experience with Agile, ranging from complete neophytes to folks with many years of experience. The topic chosen was “What are the preconditions for a large company to be successful implementing Agile?” – a very pointed, yet valid, question to be sure. All of the answers were ones that one might expect; support from upper management, learning culture, etc. except for one. The lone unobvious answer was provided by someone whose opinion I respect. I was somewhat surprised at the time and it has reverberated in my mind ever since, “one precondition for a company to be successful in an Agile transformation is they have failed miserably and have hit rock bottom.”

The term “rock bottom” is most commonly used by folks in NA and AA (Narcotics Anonymous and Alcoholics Anonymous). I found the following definition of rock bottom on a website for alcohol addiction:

… often used to describe a point … when they are finally willing to seek help. Things are now so bad for them that it is impossible to deny their problem anymore. Hitting rock bottom may result due to a particular event, or it can be a slow decline over time. This is a subjective term because some … will be willing to suffer a lot more … than others.

I first came in contact with Anonymous groups through my work in healthcare (nearly a lifetime ago when I was a licensed Health Care Administrator) and have more than one commented on how Agile transformations remind me of some of the twelve steps – with the first step being to admit one has a problem.

So when my friend mentioned that a good indicator for agile transformation success was a company had hit rock bottom I knew exactly what he was referring to. In this particular case he used the examples of the FBI Sentinel Project and Healthcare.gov website debacle. In both cases, it wasn’t until each was a total disaster that Agile was actually tried with any seriousness and rigor and in both cases the results were amazing.

The total spend of these [two] failed [Waterfall] attempts to replace the ACS system was $597m and wasted 10 years. The Agile project, which is now delivering a solution, will only cost $114m for a three-year long project.

In another recent talk I went to on Agile a gentleman explained how Agile helped with an ad agency. He was only brought in AFTER the agency blew millions on a website that never made it to production. After losing millions I would guess this agency hit “rock bottom”. The good news was that Agile was able to help them to change and allowed them to produce high quality software with a much better time to market (than never).

In my agile coaching practice I have had more than one conversation with enlightened management that has acknowledged our Agile transformation wasn’t working as good as hoped and, barring a complete meltdown, most folks were happy to continue along a non-optimal path. In fact, relative success is a particularly sticky barrier to change as people often confuse the ability to get something done with the ability to get something done optimally. Bill Gates has often remarked, “Success is a lousy teacher. It seduces smart people into thinking they can’t lose.” Sometimes what we mistakenly call “failure” is a much better teacher. Sometimes the lesson is more important than the perception of relative success or failure.

Just this morning I was listening to my local NPR station as they were having a fund drive. Like many others, I couldn’t wait for the drive to be over so I could no longer had to feel the shame and guilt of being a “free rider”. Fortunately, it was the last day of the pledge drive and my mind was only half listening to the chatter of the announcers when they introduced Shankar Vedantam who does segments on the Hidden Brain (which I enjoy immensely). His topic was why people donate (or don’t donate) to pledge drives. He talked about humans’ tendency to pay more attention to emergencies and crisis than what is important. Being the last day of the pledge he hoped what was merely important (donating) would now also be given the status of an emergency because time was running out.

My mind went immediately back to the concept of “rock bottom” and why this is one good indicator of where Agile can be successful. Sometimes it is only when we hit rock bottom that the importance of being Agile begins to align with the immediate crisis of having to be Agile. It is then, and unfortunately for some, only then when Agile gets the attention and focus it deserves. It is my hope companies realize becoming an Agile organization is essential to their long term survival (at least in organizations that rely heavily on software development), seek the help they need to transition and actually carry out the transition before they hit rock bottom. In the meantime, if you have hit rock bottom (are you listening Yahoo!?), please seek out Agile help.

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Android | RSS

7 December, 201517 December, 2015

Brainstorming – Effective Technique or Sacred Cow?

I have spent a great deal of time studying and reading about human cognitive biases and their effect on business, especially the business of software development. This past weekend I finished the groundbreaking book by Stuart Sutherland, appropriately title “Irrationality: The Enemy Within”.

Since I have made quite a bit of study on the topic previously, some of the material was either referenced by other materials or has lost its shock value since I have become thoroughly convinced of humankind’s built in propensity not only for irrational behavior, but their inability to recognize that these biases are a problem. In fact, my experience is that a large segment of our population is not only ignorant of biases but seems to revel in a willful ignorance of scientific evidence. Certainly there appears to be a great deal of cognitive bias (mostly the confirmation bias) in the debate on climate change.

My previous understanding of human cognitive bias withstanding, while the book was published in 1992, the information is still relevant, interesting and cogent. I would suppose that there are a number of things that are worthy of note, but since there is such a wealth of information in the book, I decided to choose a single instance to write about here and encourage those interested in more examples to actually get a copy of the original material.

The one thing that caught my attention and has stuck in my mind is the example of using a technique called “brainstorming” to improve creativity and productivity. For those who have lived on another planet, brainstorming is the process of getting as many ideas out as possible without judging or filtering of the ideas. It has been used for decades since its introduction by Alex Olsen in the book Applied Imagination. Olsen claimed that in his experience using brainstorming in advertising agencies resulted in 44% more worthwhile ideas than individuals thinking up ideas without the benefit of group discussion.

Ever since that time, brainstorming has been widely used to improve creativity and productivity of groups. However, here’s the kicker, since as long ago as 1958, Osborn’s claims has been subject to numerous studies which almost universally cast doubt upon the effectiveness of brainstorming. Keith Sawyer, a psychologist at Washington University in St. Louis, states: “Decades of research have consistently shown that brainstorming groups think of far fewer ideas than the same number of people who work alone and later pool their ideas.” In other words, brainstorming doesn’t work quite as well as we think it does (or should).

With scientific evidence questioning the effectiveness of brainstorming vast, the real question is why does the use of brainstorming persist? The question is at the heart of much of my agile practice in that the prime issue is not whether one is merely effective, but that one is optimal. It is obvious to me that several cognitive biases are in play in keeping brainstorming around.

There is something of the availability cascade to brainstorming “which is a self-reinforcing process in which a collective belief gains more and more plausibility through its increasing repetition in public discourse (or ‘repeat something long enough and it will become true’)” (Wikipedia). Furthermore, a whole host of cognitive biases around groupthink, herd behavior and the bandwagon effect certainly have their influence on the popularity of brainstorming. Since brainstorming “seems” to make sense it is also subject to the belief bias, which is seen when the believability of the conclusion leads us to misunderstand the true effectiveness of the process. Frankly, I would suppose that I could find literally dozens of cognitive biases, which allow brainstorming to proliferate as the “go to” technique for group creativity and productivity.

Given that brainstorming may very well not be optimal, what are the alternatives that have actually been scientifically proven to be more effective? In a 2012 article for Psychology Today, Ray Williams proposes a few modifications to the brainstorming approach:

Have groups collaborate frequently by having them in close physical proximity to each other;
Pay attention to creating physical spaces that enable good collaboration, which facilitates people frequently “running into each other” while at work;
Revise the “no criticism” script of brainstorming to encourage debate about ideas;
Use appreciative inquiry techniques, where group participants build on ideas suggested by each individual in the group.

Most interesting to me about these suggestions is how closely they align to the things that Agile (and I) speak to, namely close attention to co-location of people within an Agile team to increase good collaboration, allowing an environment where there is embracing of feedback as opposed to “failure” and using iterative feedback to improve ideas (and software) incrementally.

There are a great number of cognitive biases inherent in human beings. The first step is to be aware that these irrationalities exist. We must also acknowledge that we, as individuals, are subject to these irrationalities. Furthermore, we need to create an environment of safety that gives us the freedom and encouragement to continually explore and seek the underlying scientific truths, the “why” of what we do – the freedom to gore the sacred cows.

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Android | RSS

1 December, 20159 December, 2015

Agile – It’s All About Making Better Decisions

I’ve been spending a lot of time recently doing research, reading and presenting on human cognitive biases. To the initiated, cognitive biases are defined as

“…a systematic pattern of deviation from norm or rationality in judgment, whereby inferences about other people and situations may be drawn in an illogical fashion. Individuals create their own ‘subjective social reality’ from their perception of the input.” (Wikipedia Definition)

In other words, cognitive biases exist when there is a gap between our perception of reality and objective reality. For example, there is the “confirmation bias” which is our human tendency to seek out or interpret information that confirms one’s existing opinions.

While the term “cognitive bias” is relatively new (it was coined in 1972 by Amos Tversky and Daniel Kahneman), researchers have already uncovered literally over a hundred cognitive biases, some which are relatively tame like the “google effect” (or digital amnesia), where there is a tendency to forget information that can be easily researched, to ones that can lead to more disastrous consequences like the Sunk Cost Fallacy where people justify increased investment in a decision based on prior investment instead of looking only at future efficacy. The Sunk Cost Effect, along with the Overconfidence Effect and Receny Effect, played a role in the May 1996 mountain climbing tragedy, made famous in the movie Everest, that resulted in the death of five experienced climbers.

A great number of cognitive biases have been found through the work of behavioral economics researchers like Dan Ariely who wrote the wonderful books Predictably Irrational and The Upside of Irrationality. Underlying all of classic economics is the concept of homo economicus, or economic man who behaves in rational ways to maximize individual returns and acts in his own self-interest. Unfortunately, this is not the case and humans often act irrationally (and predictably so) because of their inherent cognitive biases. Humans all have biases for loss aversion and would choose to avoid loss over a larger corresponding potential gain and thus act as “homo irrationalis” as discovered by behavioral economics instead of “homo economicus” as predicted by classic economics.

It is our cognitive biases that cause us to make irrational decisions. Since behavioral economists found many of these cognitive biases, it was not a great leap to see how cognitive biases would be a paramount concern for the economics of software development. In my coaching practice, a great deal of my time and effort is used in helping organizations make better decisions about software development. Many times the optimal decisions are counter intuitive to people’s inherent biases so my job (and my passion) is helping companies see the world of software development differently so that, when it comes down to making a decision, they have all the knowledge necessary to make the optimal economic decision.

One of the most prevalent biases in software development is to see the world in a mechanistic / Tayloristic manner. Taylor’s viewpoint was fine for the old world of physical work, but does not hold up in the complex knowledge work being done by software development professionals today. Unfortunately, most of the people making software development decisions are predominantly influenced by this old, less optimal way of viewing the world, and, as a result, make sub-optimal decisions. For example, in the mechanistic worldview, adding more people to an effort results in a corresponding increase in output. If there is an existing team of seven people and we add seven more then we would (if we hold this mechanistic bias) expect the work to be approximately twice as fast. However, like the behavioral economists that found the real world to be counter intuitive to homo economicus, actual studies have found that the need for increased communication of knowledge work nearly outpaces any incremental increase in individual productivity (see Top Performing Projects Use Small Teams). I have always said that if you want to double productivity of a fourteen person team all that is necessary is to create two teams of seven.

The mechanistic bias can also be seen in many of the ways that the Agile philosophy is implemented. For example, the scrum framework is often trained as a series of ceremonies and actions with little or no understanding of the reason such mechanistic actions are successful. “Scrum Masters” are “certified” with only two days of training and a simple test. The training deals with ideal situations, but when the scrum master actually has to implement scrum, he or she is woefully unprepared. In the real world compromises and decisions must be made. Without understanding the underlying “why” of agile and the basic nature of software development, the decisions and compromises that are made are not optimal. In my experience, this is why project managers are tougher to train than people with no project management experience. When faced with ambiguous information and the need to make optimal decisions, project managers tend to fall back on existing mechanistic knowledge and the decisions made range from mildly irritating to completely disastrous. As I have often pointed out, to say that one was successful with waterfall reeks of confirmation bias because it begs the question of whether or not one would have been more successful using another methodology or framework like Lean or Scrum.

In addition to the mechanistic bias, software development suffers from another bias, the project-centric bias, which is the tendency to see all work done in terms of projects. Unfortunately, the project-centric bias is so ingrained in companies that there needs to be some radical changes to the way we view software development across all areas, including accounting. Viewing work as a project when we are actually working on software products results in a whole raft of poor software economic decisions like concentrating on features more than quality and security. Remember that no one washes a rental car.

As I think back on my coaching work in agile, the blogs I have written, the many discussions I have had and the presentations I have made, I think that all of these boil down into one very simple thing – my work is all about helping people understand the true nature of the software development business process and, thereby helping them to make better decisions. Understanding our cognitive biases, therefore, is extremely important for my clients and myself because, in the end, Agile is all about making better decisions.

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Android | RSS

4 May, 20154 May, 2015

We Must Inject (not just inspect) Quality

Not sure where I first heard the phrase “you can’t inspect quality into a product” but I have certainly used the phrase myself all too often in my consulting gigs. After a quick Google search, I found the originator of the quote was Harold Dodge and it was first used in a manufacturing context. While I generally eschew appropriating manufacturing analogies for comparison to software development, in this case, it is certainly apt.

In many of the companies I consult with, the prevalent view of “quality” is nothing more than inspection after the fact, a misplaced notion that quality can be insured by finding all of the defects built into the software. The great majority of these inspections are conducted by a separate team, a half a world away by manually banging on keyboards and mice. If one were to write an article on how NOT to ensure software quality it would look frighteningly similar to this (which is what I experience all too frequently when I work with companies).

I find that often this goes hand in hand with the mistaken notion that we are performing projects as opposed to creating long term stable quality products. In some cases our quality leaders are merely naive or simply unquestioningly following the misguided policies and practices of those who came before. For those who truly wish to understand true software quality, who desire to build quality into a product instead of merely trying to inspect it out, what practices should we use?

The first thing is to recognize that building quality into our software code must begin with those who are building the software, the developers. We might have some nice motivational posters that state quality is everyone’s business, but until we tear down the arbitrary and dangerous walls we have erected (incorrectly taking waterfall to its bizarre conclusion) between development and “testers” we will certainly fail to put quality into the code. Development can ensure the quality comes first by writing tests first and performing true TDD (test driven development).

Sometimes, as Dan North found, developers may be convinced that TDD is a great idea yet find it difficult to implement. This is a great opportunity to inject BDD (behavior driven development) as a means of helping developers with TDD. In fact, I often recommend that BDD style scenarios can be used to express acceptance criteria even if developers are not yet ready for TDD. The tremendous increase in understanding and early inclusion of test cases as expressed through scenarios is the biggest benefit of BDD and goes a long way to injecting quality into our code.

Another technique for injecting quality is the concept of code review. There is nothing like another set of eyes to make sure that our code is high quality before we push it out of the development environment. You may want to go one better and get a more immediate feedback through pair programming. Like many things in software development pair programming is counter intuitive but the quality of code produced by judicious use of pairing far outweighs the perceived “double” expense.

Akin to code reviews conducted by humans is the ability of code to be “reviewed” by software through static code analysis tools. This software can “read” code and determine “quality” through a set of programmed rules. It will, for example, let you know which classes are too long or complex. These rules become instant feedback for code in progress and help us to easily avoid technical debt. For existing code these tools can help us to understand existing code quality and provide us with candidates for refactoring.

Whatever the technique you choose, the key is to understand that quality is something that is built into the code not something that we can inspect out.

3 February, 20158 February, 2015

Timely Feedback – Why Agile Works

Most folks who know me well know I like to talk.

I especially like the times when I get to speak in front of groups, whether a face-to-face presentation, a small user’s group or 160 on a conference call all over the world (If you have a group and want a passionate speaker – feel free to call on me).

Sometimes my passion provokes long stories, explanations or asides. Lately though when I speak to others about why agile works, I eschew my sometimes verbose ways and reply with two words – “timely feedback”.

These two words refer back to my argument that software development is, as described by Snowden in his Cynefin Model, complex as opposed to complicated. And complexity begs for timely feedback.

Take a look at the most popular Agile processes and practices and you can see why they are effective. They all are merely ways to produce timely feedback.

Daily standup = daily feedback.

Review, retrospective, grooming, planning – all are timely feedback.

What is automated testing on code check in or continuous integration and deployment? Feedback. Pair programming is nothing more than continuous immediate feedback.

Conversely, what is waterfall than a dearth of feedback? Compared to the continuous timely feedback of agile, waterfall is a vast wasteland of opacity.

So next time you are asked why agile works, or why a shorter iteration is better, or why we should do pair programming, or why continuous integration, take the quick route and prove that brevity is indeed the soul of wit and simply say, “Timely feedback”.

– Larry Apke

17 November, 20141 December, 2014

How I Know You Are Flailing at Agile

As a consultant I try my best to keep abreast of industry trends in technology and software development. Often one of the best ways to observe trends is to check out job boards to see what positions are being posted. For example, job postings can show trends in programming languages. You can see which companies are trying to make Agile transformations (How many big financial companies can hire multiple coaches in Foster City?). You can see which companies have caught the “DevOps” bug and you can tell which companies are flailing and failing their Agile transformations.

How can I tell which companies are failing? When you see multiple posts for the same position over time is a good indicator. For example, you see a post for an Agile Coach that goes unfilled for months. While good ones can be hard to find and take some time, a position open for months can be telling that good coaches are avoiding the company. Most good coaches can very quickly size up a company’s commitment to Agile and avoid those situations where they are likely to fail no matter how good a coach they are. They can sense the fundamental dysfunction. Sometimes you will see a posting one day and the same posting a few short months later. Chances are whoever was chosen didn’t last and that certainly raises red flags.

Sometimes the title that is advertised will show a company’s inherent failure to grasp Agile. You will see a ton of advertisements for Scrum Master or Project Manager. I personally love these as they serve the same purpose as someone wearing a “Stupid” sign in that we know to avoid both. Advertising for a Scrum Master or Project Manager indicates that the company searching understands neither position. The good thing is that if one is only interested in doing Agile then one can save time by avoiding these.

Once I drove nearly two hours for an interview with a company that does only “Agile” projects. During one of the many lengthy interviews, I was asked, “What would happen if we gave you a waterfall project?” “Well for one thing you would not be 100 percent Agile anymore.”

The interview went downhill from there – a waste of a lot of people’s time. Advertising for a Scrum Project Manager would have saved everyone!

Lately I have been seeing another title that gives me pause – Technical Project Manager, especially in the context of Agile companies. I have been a part of and witnessed many abject failures of software development projects (maybe why Project Management should be replaced by Product Management). On none of these have I ever thought that maybe, just maybe, if we had a Project Manager that had more technical knowledge we would have snatched victory tron the jaws of defeat. And the colleagues I talk to agree.

Project Managers have the least amount of authority in making projects successful. A Technical Project Manager is kind of like blaming the weatherman for the rain. It is a silly tactic by management to deflect blame from them. Agile transformation is difficult because it needs change from management and change of culture. Hiring Technical Project Managers is a red herring that distracts an organization from the real work to be done, pushing the transformation finish line even farther away. And finding the kind of person who can successfully straddle tech and management is a tall order, especially when people who know what Agile transformations are all about and who will see through you and realize you are merely flailing at Agile.

– Larry Apke