Agile - 5/13 - Agile Doctor

7 December, 201517 December, 2015

Brainstorming – Effective Technique or Sacred Cow?

I have spent a great deal of time studying and reading about human cognitive biases and their effect on business, especially the business of software development. This past weekend I finished the groundbreaking book by Stuart Sutherland, appropriately title “Irrationality: The Enemy Within”.

Since I have made quite a bit of study on the topic previously, some of the material was either referenced by other materials or has lost its shock value since I have become thoroughly convinced of humankind’s built in propensity not only for irrational behavior, but their inability to recognize that these biases are a problem. In fact, my experience is that a large segment of our population is not only ignorant of biases but seems to revel in a willful ignorance of scientific evidence. Certainly there appears to be a great deal of cognitive bias (mostly the confirmation bias) in the debate on climate change.

My previous understanding of human cognitive bias withstanding, while the book was published in 1992, the information is still relevant, interesting and cogent. I would suppose that there are a number of things that are worthy of note, but since there is such a wealth of information in the book, I decided to choose a single instance to write about here and encourage those interested in more examples to actually get a copy of the original material.

The one thing that caught my attention and has stuck in my mind is the example of using a technique called “brainstorming” to improve creativity and productivity. For those who have lived on another planet, brainstorming is the process of getting as many ideas out as possible without judging or filtering of the ideas. It has been used for decades since its introduction by Alex Olsen in the book Applied Imagination. Olsen claimed that in his experience using brainstorming in advertising agencies resulted in 44% more worthwhile ideas than individuals thinking up ideas without the benefit of group discussion.

Ever since that time, brainstorming has been widely used to improve creativity and productivity of groups. However, here’s the kicker, since as long ago as 1958, Osborn’s claims has been subject to numerous studies which almost universally cast doubt upon the effectiveness of brainstorming. Keith Sawyer, a psychologist at Washington University in St. Louis, states: “Decades of research have consistently shown that brainstorming groups think of far fewer ideas than the same number of people who work alone and later pool their ideas.” In other words, brainstorming doesn’t work quite as well as we think it does (or should).

With scientific evidence questioning the effectiveness of brainstorming vast, the real question is why does the use of brainstorming persist? The question is at the heart of much of my agile practice in that the prime issue is not whether one is merely effective, but that one is optimal. It is obvious to me that several cognitive biases are in play in keeping brainstorming around.

There is something of the availability cascade to brainstorming “which is a self-reinforcing process in which a collective belief gains more and more plausibility through its increasing repetition in public discourse (or ‘repeat something long enough and it will become true’)” (Wikipedia). Furthermore, a whole host of cognitive biases around groupthink, herd behavior and the bandwagon effect certainly have their influence on the popularity of brainstorming. Since brainstorming “seems” to make sense it is also subject to the belief bias, which is seen when the believability of the conclusion leads us to misunderstand the true effectiveness of the process. Frankly, I would suppose that I could find literally dozens of cognitive biases, which allow brainstorming to proliferate as the “go to” technique for group creativity and productivity.

Given that brainstorming may very well not be optimal, what are the alternatives that have actually been scientifically proven to be more effective? In a 2012 article for Psychology Today, Ray Williams proposes a few modifications to the brainstorming approach:

Have groups collaborate frequently by having them in close physical proximity to each other;
Pay attention to creating physical spaces that enable good collaboration, which facilitates people frequently “running into each other” while at work;
Revise the “no criticism” script of brainstorming to encourage debate about ideas;
Use appreciative inquiry techniques, where group participants build on ideas suggested by each individual in the group.

Most interesting to me about these suggestions is how closely they align to the things that Agile (and I) speak to, namely close attention to co-location of people within an Agile team to increase good collaboration, allowing an environment where there is embracing of feedback as opposed to “failure” and using iterative feedback to improve ideas (and software) incrementally.

There are a great number of cognitive biases inherent in human beings. The first step is to be aware that these irrationalities exist. We must also acknowledge that we, as individuals, are subject to these irrationalities. Furthermore, we need to create an environment of safety that gives us the freedom and encouragement to continually explore and seek the underlying scientific truths, the “why” of what we do – the freedom to gore the sacred cows.

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Android | RSS

1 December, 20159 December, 2015

Agile – It’s All About Making Better Decisions

I’ve been spending a lot of time recently doing research, reading and presenting on human cognitive biases. To the initiated, cognitive biases are defined as

“…a systematic pattern of deviation from norm or rationality in judgment, whereby inferences about other people and situations may be drawn in an illogical fashion. Individuals create their own ‘subjective social reality’ from their perception of the input.” (Wikipedia Definition)

In other words, cognitive biases exist when there is a gap between our perception of reality and objective reality. For example, there is the “confirmation bias” which is our human tendency to seek out or interpret information that confirms one’s existing opinions.

While the term “cognitive bias” is relatively new (it was coined in 1972 by Amos Tversky and Daniel Kahneman), researchers have already uncovered literally over a hundred cognitive biases, some which are relatively tame like the “google effect” (or digital amnesia), where there is a tendency to forget information that can be easily researched, to ones that can lead to more disastrous consequences like the Sunk Cost Fallacy where people justify increased investment in a decision based on prior investment instead of looking only at future efficacy. The Sunk Cost Effect, along with the Overconfidence Effect and Receny Effect, played a role in the May 1996 mountain climbing tragedy, made famous in the movie Everest, that resulted in the death of five experienced climbers.

A great number of cognitive biases have been found through the work of behavioral economics researchers like Dan Ariely who wrote the wonderful books Predictably Irrational and The Upside of Irrationality. Underlying all of classic economics is the concept of homo economicus, or economic man who behaves in rational ways to maximize individual returns and acts in his own self-interest. Unfortunately, this is not the case and humans often act irrationally (and predictably so) because of their inherent cognitive biases. Humans all have biases for loss aversion and would choose to avoid loss over a larger corresponding potential gain and thus act as “homo irrationalis” as discovered by behavioral economics instead of “homo economicus” as predicted by classic economics.

It is our cognitive biases that cause us to make irrational decisions. Since behavioral economists found many of these cognitive biases, it was not a great leap to see how cognitive biases would be a paramount concern for the economics of software development. In my coaching practice, a great deal of my time and effort is used in helping organizations make better decisions about software development. Many times the optimal decisions are counter intuitive to people’s inherent biases so my job (and my passion) is helping companies see the world of software development differently so that, when it comes down to making a decision, they have all the knowledge necessary to make the optimal economic decision.

One of the most prevalent biases in software development is to see the world in a mechanistic / Tayloristic manner. Taylor’s viewpoint was fine for the old world of physical work, but does not hold up in the complex knowledge work being done by software development professionals today. Unfortunately, most of the people making software development decisions are predominantly influenced by this old, less optimal way of viewing the world, and, as a result, make sub-optimal decisions. For example, in the mechanistic worldview, adding more people to an effort results in a corresponding increase in output. If there is an existing team of seven people and we add seven more then we would (if we hold this mechanistic bias) expect the work to be approximately twice as fast. However, like the behavioral economists that found the real world to be counter intuitive to homo economicus, actual studies have found that the need for increased communication of knowledge work nearly outpaces any incremental increase in individual productivity (see Top Performing Projects Use Small Teams). I have always said that if you want to double productivity of a fourteen person team all that is necessary is to create two teams of seven.

The mechanistic bias can also be seen in many of the ways that the Agile philosophy is implemented. For example, the scrum framework is often trained as a series of ceremonies and actions with little or no understanding of the reason such mechanistic actions are successful. “Scrum Masters” are “certified” with only two days of training and a simple test. The training deals with ideal situations, but when the scrum master actually has to implement scrum, he or she is woefully unprepared. In the real world compromises and decisions must be made. Without understanding the underlying “why” of agile and the basic nature of software development, the decisions and compromises that are made are not optimal. In my experience, this is why project managers are tougher to train than people with no project management experience. When faced with ambiguous information and the need to make optimal decisions, project managers tend to fall back on existing mechanistic knowledge and the decisions made range from mildly irritating to completely disastrous. As I have often pointed out, to say that one was successful with waterfall reeks of confirmation bias because it begs the question of whether or not one would have been more successful using another methodology or framework like Lean or Scrum.

In addition to the mechanistic bias, software development suffers from another bias, the project-centric bias, which is the tendency to see all work done in terms of projects. Unfortunately, the project-centric bias is so ingrained in companies that there needs to be some radical changes to the way we view software development across all areas, including accounting. Viewing work as a project when we are actually working on software products results in a whole raft of poor software economic decisions like concentrating on features more than quality and security. Remember that no one washes a rental car.

As I think back on my coaching work in agile, the blogs I have written, the many discussions I have had and the presentations I have made, I think that all of these boil down into one very simple thing – my work is all about helping people understand the true nature of the software development business process and, thereby helping them to make better decisions. Understanding our cognitive biases, therefore, is extremely important for my clients and myself because, in the end, Agile is all about making better decisions.

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Android | RSS

16 November, 2015

On Death and Dying and Agile Transformation

I was recently involved with a large scale Agile transformation and noticed what I thought was an interesting correlation, jotted down a note to blog about it and then promptly did nothing for a very long time. Usually these blinding flashes of light quickly lose their luster and find themselves relegated to the bottom of the blog backlog, never seeing the light of day, but this particular one reignited my attention as I sat down to write my newest blog.

My earth-shattering insight was that any organizational transformation, which obviously includes an Agile transformation, involves the very same stages that were first identified by Swiss psychiatrist Elisabeth Kübler-Ross in her 1969 book, On Death and Dying. For those who were sleeping through Psych 101, Kübler-Ross proposed that there are a series of stages that are experienced by survivors when faced with the death of a close friend or relative. These stages could be experienced linearly but also in no particular order, but that everyone would go through the five stages she recognized through her work with terminally ill patients.

The five stages are: Denial, Anger, Bargaining, Depression, and Acceptance. Though the model was originally created to explain the stages of grief following the death of a loved one, it was later expanded to encompass the grief stages associated with any major loss like the loss of a job or income or divorce / end of a relationship. It is my opinion that these stages can also be applied to the loss of a treasured idea. In fact, I think these stages are better explained by the death (loss) of a cherished idea since love, attachment, etc. are all associated with mental constructs (ideas). Our world is merely the sum of our mental perception so the loss of a loved one, loss of a job, or loss of a relationship are nothing more than the loss of an idea.

Once we understand that the grief stages are in response to the loss of an idea, it is not a great leap to apply this to any company transformation. It is well known that there are some who will readily embrace change, but there are a great number that see any change as a threat. What is the nature of the threat? I think that either consciously, or more often subconsciously, the threat is to an idea that one has grown to “love” and that there is a very real fear that if this idea were replaced that its death would cause grief. In my experience with a great number of agile transformations I do tend to see the five stages that Kübler-Ross outlined.

There is certainly a large share of denial when I have tried to help companies become more agile. There is never a shortage of people who will defend the status quo and insist that the current way of creating software (nearly always waterfall) is already successful and that there is no need to bring agile. The minds of the people in denial are closed to any external threat to their enshrined beliefs.

I also see a great deal of anger during transformations. People have loved their ideas for so long that they are like a member of the family. How dare you agile folks try to kill off my favorite processes? I will do everything in my power to try to stop you, railing at the purveyors of such dangerous ideas.

I see my share of bargaining too. If we cannot outright defeat the new ways, we can at least try to keep as many of the old ways intact. Maybe we don’t have to kill off all the waterfall phases. Maybe we can keep the phases, but just do them in shorter time frames. Maybe we can just do this “agile” thing for development and leave the rest of the sacred cows not slaughtered. I don’t have to give up my old way of thinking or deal with the death of my ideas, is there not room for both?

As new ideas begin to take hold, I have also seen my share of depression. People have viewed the world in one way for so long that once their ideas are shown to be outdated or not optimal, they begin to look forlorn and some even begin to despair. With waterfall gone, how am I to complete a software development project?

And finally, if the company has the intestinal fortitude to stick it out through the first four stages, you will finally get to acceptance. No with your new idea having taken hold, it probably won’t be long before you will have to go through it all again with another new idea. I think the more that we realize that changes our ideas result in the death of old ideas and that the death of old ideas will result in some recognizable stages the more we will be able to quickly move through those stages and adopt new ideas more readily.

Postscript: Interestingly enough, when I had just finished the above blog, I did some research to see if I had written on this subject before because it had seemed eerily familiar. I could find no blog that I had published on the subject, but I may have written about it before and not published. What my research did find was that I am not alone in my link between the Kübler-Ross model and Agile. I have found other references to this on Mindstorm, and Agile Helpline. While I do not recall reading these blogs prior to my current blog and it is possible that we have all come to the same insight independently, I reference these here just in case I did read them at some point and perhaps forgot. Regardless, the fact there are others who have written about the very same topic leads me to believe in the concept’s applicability.

Post Postscript: This blog represents my 100th blog post under the agile-doctor.com website. While this is mostly symbolic and my 100th blog post will not guarantee me syndication (like a sitcom), it is still a moment to celebrate. Many thanks to everyone who has given support over the years! Stay agile my friends!

26 October, 2015

You Can Be Right or You Can Be Successful

I recently had lunch with one of my friends and we spoke about some issues that one of his teams was having. As a new team, they were still in the storming and norming stages and they were having some fairly heated discussions around some philosophical issues of software development. For those not familiar with software development you might think there is not really anything to get all worked up about, in which case you would be surprised at the level of intensity some folks have. The discussions the team was having bordered on “a holy war” and there appeared to be little hope for compromise.

I asked my friend a simple question, “would you rather be right or be successful?” This question is one I like to use and is a variation of the quote “you can be right or you can be happy” which I have, until I did the research for this blog, incorrectly attributed to Dr. Phil McGraw (for those who care, the quote is from Gerald Jampolsky). Basically, there are times when we can be right and we can be happy but often we have to make a choice. The same applies to success at work. There are times when we can have it all, but sometimes we need to choose whether we wish to be right or effective.

Recently there has been an outbreak of measles in the United States due to the fact that some parents have not been vaccinating their children. The reason these parents were not vaccinating was due to false science that has since been refuted. In other words, the truth is unequivocal; vaccines do not cause the reactions that anti-vaccination parents fear. These parents are 100% wrong.

Since these parents are 100% wrong and not vaccinating children can lead to adverse health issues in the community, it makes sense that public health officials would try to change the minds of the “anti-vaxxer” parents. They did try. Their first approach was to try and convince the parents by presenting the scientific facts. They theorized that by knowing the true facts, people would make the correct decisions. That effort failed. It seems that merely presenting facts did not change peoples’ minds.

If presenting facts does not work, public health officials concluded the reason for the resistance was emotional and that emotional arguments would be more effective in convincing parents to vaccinate their children. That effort failed as well. (For a more details analysis see http://web.missouri.edu/~segerti/3830/Pediatrics-2014-Nyhan-e835-42.pdf).

This was surprising. Furthermore, not only are people not convinced by facts or emotion, but that when presented with factual or emotional arguments, the anti-vaxxer parents in the study were LESS likely to vaccinate their children. Evidence suggests that once someone has a strong belief in something, anything (or anyone) contradictory to that belief is seen as a threat. People who are threatened will cease to listen, much like a child will put a finger in their ears. People will erect walls to keep you out.

Knowing this, should we just give in to despair? Of course not. We need to stop trying to be right, whether presenting factual or emotional arguments, and concentrate more on being successful. The question is – what works in convincing people with strong beliefs to change their position? There is something that has been shown to work, something that lowers peoples’ walls and defenses and that thing is listening.

Instead of telling or implying people are wrong, we need to take the time to ask them about their position. We need to stop threatening them. Given the space to explain their position, people will not only feel less threatened, but also they will have to defend their own position and when tasked with defending their own position, people will slowly be able to see that they might not be the experts they think they are. This is where the walls and defenses are lowered so that other arguments might be entertained and listened too. Furthermore, showing respect to individuals and spending time with them will also make it more difficult to not listen. We listen to our friends not our antagonists.

A study of midwives in India found that arguments did not work with them either. What did work was “the rule of seven touches” which marketers know well. You must have meaningful contact and get to know someone (seven touches) before they are willing to trust and listen to you.

This applies to my work of transforming companies from waterfall to agile approach. While I may be 100% correct and it may make me feel good to be right, by presenting my viewpoint as THE way, I will not meet my objective. If I choose to be successful then I must take a different approach. I need to get to know the people involved, understand their concerns, not threaten their ideas, but allow their defenses to be lowered by listening to their ideas and then, and only then, if my ideas are truly right will I have the chance to convince them of or, better yet, guide them to the truth.

In the end you can be right or you can be successful. You can’t always be both.

31 August, 2015

Patterns, Anti-Patterns and Pilots – Guidance versus Governance

In the past, I have proposed that an agile transformation should follow “three pillars of success – Structure, Practices and Governance.” This was a concept I decided to borrow from something I read in the past. It made sense at the time that to successfully transform we need to change structure (build good teams, etc.) and adopt new practices both at the framework/process level (ie Scrum) and for software development (ie TDD, Continuous Integration). It was the concept of governance that, while it seemed to make sense, gave me some pause.

It didn’t take me long before I began to understand why I was uncomfortable with “governance” – primarily because governance seemed too top-down and heavy-handed. It seemed a holdover from the very school of thinking our transformation was trying to change. Perhaps it bothered more so because governance was something easier to implement via the PMO (and something that people were too comfortable with).

It wasn’t that governance isn’t needed – even the games we play need rules. It would be chaos indeed if the length of a football field, number of downs or number of players changed from game to game. However, elevating governance seemed to embolden attempts by people in the organization to dictate even the smallest minutia. It was as if governance gave implicit authority to make every team run a west coast offense instead of concentrating on eleven players on each side.

Once governance lost its cache, the question remained as what to replace it with. It made sense to pick a word with a more appropriate servant-leader feel. “Guidance” became a much more palatable alternative. This was better, but it didn’t provide everything needed because there were some rules that would need to be followed. After a brief flirtation with the concept “Governance and Guidance”, in the end, we settled on “Guidance and Tools” as a replacement to the original concept of “Government.” The reason for this was that most of the “rules” revolved around reporting and the tools support necessary to some basic reporting.

Once we settled on the concept of guidance, a definition of guidance was needed. Over time, a working definition emerged that was referred to as:

Patterns, Anti-Patterns and Pilots

Not only does this option have alliterative qualities, but it also allows for a much greater environment for teams to self-organize and experiment. The process is simple. There are certain things that we know to generally work well for teams. One example is Behavior Driven Development (BDD). This would be considered a pattern. It would be a practice that is not only highly recommended, but would be fully supported by the organization from a training perspective for teams that wanted to try it. The key is that this practice would not be mandated. Each team has the option to choose and would be supported in their decision, but not every team would choose to (or have the maturity to successfully) implement. I often tell my teams, while I think driving is great there is a reason I don’t hand my car keys to my nine year old.

The second category would be anti-patterns, which are defined as those things that teams have tried but there is consensus that these have not worked. In other words, something becomes an anti-pattern when multiple teams have tried something unsuccessfully. In addition, there are things our experience as agile coaches and scrum masters have shown to not be effective which are added to the list. One example would be splitting stories between offshore and onshore for the same team. In practice, the anti-patterns begin to look a lot like a “scrum but” list.

The final category, pilots, is where the power resides. This category that allows teams to freely experiment with different ways of doing business without a great deal of pressure. The approach is empirical like the Agile Manifesto. Try something and see what works. There is no failure – only data.

The team identifies these experiments during retrospectives and is free to experiment on anything that is neither a pattern nor anti-pattern. These experiments follow the scientific method in that they have an identifiable hypothesis. The outcome from this experiment (usually lasting a single iteration) is either that the experiment has proven successful in which case it can be elevated to a pattern, the pilot did not achieve the desired outcome or the experiment needs to gather more data.

In the end, after changing from “Governance” to “Guidance and Tools” and outlining an approach to using Patterns, Anti-Patterns and Pilots, I found myself much more comfortable with the underlying “three pillars” approach of our agile transformation. I believe the new approach was better aligned to the underlying Values and Principles outlined in the Agile Manifesto. And, more importantly, I noticed that organization itself seemed more comfortable with the agile transformation and the teams were much more empowered to self-organize.

26 August, 201528 August, 2015

Fear, Slack and Agile Transformation

This past week I got the chance to finally check out the Silicon Valley Agile Trends and Leadership meetup at Yahoo in Sunnyvale. I really enjoyed the presentation that was given by Dan Kimble on The Leadership Crisis of the Digital Age.

Dan did a great job of framing the problems endemic in today’s high tech world – working too many hours, allowing too many interruptions, superfluous meetings, incessant checking of emails, the inability to unplug, etc. It was interesting to see that nearly everyone in the audience not only recognized the behavior, but agreed that it was irrational.

After presenting the group with the problem, Dan allowed discussion as to the possible root causes for such irrational behavior. There were a number of comments that seemed to dance around the true cause like one gentleman who offered that the problem was “culture”. Since this was my maiden voyage, my goal was to keep a low profile, but to those who know me, this is not always possible when the topic is Agile or software development (though on most other subjects I am likely to remain silent). I was compelled to speak up. The real root cause – FEAR.

Most interesting to me was that once that comment was made, it seemed to me that light bulbs starting going off. Fear. Ah yes. I check my emails at all hours so that my boss and co-workers can see how hard I work, how committed I am to the cause. I stay late so that people can see me and will know that I am not replaceable. If I have anything that looks like slack time, I may get sent back home (in some cases home can be very far away). We all seemed to know it, but it was almost like, even in that room of agile aficionados, we were afraid to give it a name.

Fear is a very good motivator. Like sugar it is great for short-term gain, but one cannot live on junk food alone. The problem is that our so-called leaders these days rely almost exclusively on this motivational junk food. The facts are that working over 40 hours is not productive, that not having down time leads to stress and burn out, that a culture of fear leads to employee disengagement and turnover, but facts be damned with our new breed of leaders.

The fear is so bad that all slack has been driven out of day. We do not have the time to think. We do not have the time to plan. We do not have the time to learn. We do not have the time to create long-term strategy. We do not take the time to train. We do not have time to form personal bonds with our co-workers. We do not have the time to build healthy relationships with our clients, let alone our own families. We do not have time to change. We do not have time to transform.

It is the lack of slack that is the single biggest factor that will derail your agile transformation (or any other significant workplace initiative), even though everyone will benefit from a proper adoption of agile. We are so busy implementing on the newest bright shiny object that we cannot take the time necessary to allow for a proper agile implementation.

So, if there are any “leaders” out there looking to make significant changes in your organization, like an agile transformation, please pay attention. If you want change, you must first remove fear. In its place, install some slack. Give back some time. Allow people to learn. Allow people to reflect. Allow people to change. Allow people to fail gracefully. You want to lead? Give them a reason to follow, something that aligns with their own intrinsic motivation. Stop using motivational junk food. Stop using fear.