From the Agile Transformation Trenches

Putting People First in the Transformation

Hi! I am Stephen R. Palmer, a principal consultant at Borland (the middle initial helps separate me from the multitude of other Stephen Palmers on Google and Yahoo, etc.). My work at Borland revolves around two main themes, model-driven software development and agile process coaching. My interest in these areas really started when I worked as the development lead on a major project in Singapore in the late 1990’s.  The project was the first to use Jeff De Luca’s Feature-Driven Development (FDD) model-centric agile process and Peter Coad’s ‘modeling in color’object modeling technique. In addition to the privilege of working closely with Jeff and Peter, the project team also included other agile practitioners like Paul Szego and David Anderson.

More recently, I have written a book about FDD, worked with enterprises developing model-driven service-oriented architectures, and coached teams adopting agile processes based on Scrum (I am a Certified Scrum Practitioner) and Smart Agile.

One team I coached this year was a large defense client. The client was keen to see if an agile approach could improve their productivity and give management a better handle on the status and progress of projects.

The agile manifesto states that it values “Individuals and interactions over processes and tools”. It puts people and the ways they work together at the center of things. Processes and tools support these collaborations. For organizations undergoing an agile transformation, this ‘people first’ idea is proving useful in introducing and establishing agile practices.

In nearly twenty years of working as a developer or consultant, one thing I have learned is that simply deploying a software product or mandating a particular process rarely produce the desired transformation in the way people work. It is all too easy for people to continue applying the same thought processes and working practices while using the terminology of the new tool and process. ‘Play it again, Sam! … but this time call it something different.’

This is not something particular to agile approaches. I have seen the same thing happen when introducing UML modeling tools as a replacement for generic diagramming tools, requirements management tools for monolithic requirements documents, and configuration management tools for basic version control of source code. A shift in paradigm needs more than a switch in technology. Those old enough to remember the switch from structured programming languages like Pascal and C to object-oriented programming languages like C++, Java and C# will remember that for a long time, developers simply used these new languages in much the same way has they had the old ones. What we needed to realize the benefits of these new languages was a change in the way developers thought. The programming languages supported this new way of thinking but they could not cause the change. The same is true of the shift from traditional software development management to agile approaches.

For my defense client, the ‘people first’ principle has worked well. We decided to concentrate initially on establishing the values, principles, and practices of a Scrum-based approach within the team, postponing any discussion about the use of software tools to manage the project in which they would be piloting the agile approach. Instead, we started with index-cards and a large corkboard.

The idea was for individuals to get a feel for the new ways of working without having to worry about operating a particular software product. With this approach,the team was able to concentrate on establishing the new ways of communicating and interacting, instead of learning the idiosyncrasies of a particular user interface.

Once the team had two or three iterations (sprints) under their belt we started to discuss how software tools might help improve management of the project. For example, one member of the team was having difficulty because he worked from home - in another country - part of the time. He had to attend standup meetings by phone, call or e-mail his daily updates for the corkboard to the scrum master, and so on. Another problem this team had was the growing size of the backlog. The product owner was struggling to manage the stories coming in from multiple different sources. Because we had established the new way of thinking, the selection of tooling focuses on addressing real problems experienced by the team.

From a vendor’s perspective, it might seem counter-intuitive not to push your product hard from the start but it actually makes considerable sense. It is all too easy and common to blame software tools for frustration caused by lack of understanding, bad communication, and poor process definition. Agile transformation programs are no different. As a vendor looking to establish a lasting relationship with a client, you do not want to see a process-improvement program struggle or fail. You certainly do not want your products cast wrongly as one of the causes of that struggling or failure. And yet, ‘Play it again, Sam!... but this time call it something different,’ is likely to happen if a tool is viewed as the key to transforming traditionally managed teams into agile teams.

It is not a coincidence, therefore, that all consultants in Borland UK are Certified Scrum Masters or Practitioners. We have recognized that for us to be successful we cannot simply throw our products at people and expect them to transform the ways they work. We have to help change the way people think if they are to realize the larger benefits from using our products.

The Mature Agile Organization

Here's a quick note about a recently published technical note from the SEI: CMMi or Agile: Why Not Embrace Both? There's talk about it from some of the authors: David Anderson, Hillel Glazer and Jeff Dalton.  I saw them and Mike Konrad discuss the draft at an impromptu session at last year's SEPG Conference and am heartened to see the progress being made in the communities working together. 

I'll have some more thoughts on the technical note itself, but a bigger question in your mind may be "CMMi and Agile?  Those can't go together."  I heard that comment from a colleague the other night at dinner.  I disagreed and my answer was "A poorly implemented CMMi based process improvement- one that mistakes the model for a process description and that doesn't keep a firm eye on the goal of delivering business value- won't be very Agile.  And, a poorly implemented Agile based improvement effort- one that doesn't recognize the amount of discipline required, for example, or doesn't address what's necessary to scale up to an enterprise level- won't look good from a CMMi view.  But, I don't want either one of those things!"

Do you agree that there's value to an Agile organization in looking at the CMMi?

Bootstrapping a Transition to Agility: Looking for a place to start?

We work with many companies at Borland who are looking to transition to agile but haven’t quite pulled the trigger.  One of the first questions that comes up in our initial conversations is “how do you get started?

When it comes to deciding on where or why to make any major change in an organization, it helps to look at examples of effective change – even if (and especially if) they are in other domains.  Learning from and citing these examples can help shift the perspective of both individuals or teams on where they can drive change in their organizations

I find the success stories at the Lean Enterprise Institute very useful for helping me shift my perspective.

Take the story of the HighGrove Partners for example.  HighGrove Partners is one of the largest privately owned diversified landscape design, installation, and maintenance companies in Atlanta

One of the changes HighGrove Partners introduced was creating maps of their trucks, which detailed where the equipment was to be stored in the trucks, and attaching these maps on to the trucks.  This simple change was meant to improve the process that the maintenance crews carried out each day.  Each maintenance job requires a crew to arrive with the truck on a jobsite, unload equipment, carry out the job, reload the equipment and continue to the next jobsite.

The storage and retrieval of equipment in these trucks does not add value to what HighGrove does. Customers of the company are not paying for delivery and retrieval of this equipment, but rather the result of using that equipment effectively on their grounds. The transit, load and unload activities are necessary, but a form of waste because they are adding no value.  The article doesn’t describe what the value-stream map looks like for this part of HighGrove, but by minimizing this waste through a simple enhancement, the company now sees these benefits:

• Minimizes the time to search for particular equipment because workers know where the equipment is, so they can start the job sooner after arriving on the jobsite.
• Minimizes the time to pack up because workers know where the equipment goes, so they can depart the jobsite quicker when they’re done.
• Reduces the risk of injury or damage because workers can consistently store equipment securely during transit, load and unload.
• Reduces the amount of time for a new worker to learn these things because truck configurations are standardized, so they can fully contribute their efforts more quickly.

Looking at this example, what parallels do you see in your organization?

From my point of view, it reminds me of the ongoing need to introduce and improve continuous integration. Teams need consistent and automated procedures for checking in and checking out of artifacts, and also for launching builds, test sets, analysis, documentation generation, and so on.

At a different level, the example also reminds me to think about repetitive tasks as I encounter them, and to contemplate how I would standardize them.  What value add or waste reduction would that standardization bring?

Lastly, it reminds me to examine whether I understand the value stream of the organizations I’m involved in.

Cheers, for now!

More on QA specialists vs. testers

In my previous post, QA specialists vs testers, I mused that agile is good for the careers of QA engineers:

Nobody is relegated to the role of tester--a subset of QA work that's generally considered lower status. Instead, every QA engineer on an agile team must function more like a lead QA engineer: planning testing strategies, advising the team on what testing is necessary, and making sure that the testing work is accomplished by various team members.

The post garnered several comments. In this post, I'd like to address some of the thoughts and questions in the comments.

First, Andrew and Jorge question my use of terminology. Andrew writes:

To me, the role of Quality Assurance on a project is to make sure deliverables (test scripts, code, functional designs) live up to a certain standard, which you inact at the beginning of a project. The QA is seen as a subject matter expert for this area. To me, a tester is someone who actually physically tests the application.

I agree 100% with that definition. But Jorge asks:

Does the author try to say QA Specialist = QA Engineer? If that's the case, there is no reason to use different naming for the same meaning (QA best practice should avoid that)

In general, I would say, yes, the two terms are interchangeable. The point I was trying (unsuccessfully, I see) to make in using the term "QA Specialist" was specific to agile: a scrum team consists of members with various specializations, which makes the QA engineer the team's QA specialist. But I agree with Jorge, it's probably an unnecessary distinction to make. "QA Engineer" has all the same connotations.

Mark Johnson points out that this change of focus for QA engineers has also been good for the company and products overall:

As the manager of the QA team, I had been pushing for them to do less testing and more QA for several years.

Agile legitimised this and made it clear that the developers role is to develop working features, not throw half-working software over the wall to testing.

I have changed the role of the senior QA staff to include the negotiation with the stakeholders of the acceptance criteria before design and coding are started; this has given them higher profile within our organisation and our clients and raised their status.

I don't really have any reply to that except to congratulate Mark. That's the outcome that I would expect from a successful agile implementation.

Finally,Sean Wilson asks about how I undertook the mentoring that I described in my earlier post.

The QA engineer whom I described was originally assigned with me to the same scrum team. Our mentoring relationship started with some credibility issues with one of the developers. To over-simplify the situation, one of the developers on the team felt that the QA engineer was not sufficiently trying to solve problems or find information herself before involving him and he felt his time was being wasted.

The other QA engineer and I discussed the problem, and we agreed that I would become her 'safety.' If she couldn't figure out a problem or find information, she should come to me first, and I would help her find other methods to address the issue at hand before resorting to the developer in question. We also explicitly agreed that she could ask me anything; no question was dumb with me. And finally, if she did have to take an issue to the developer, I mentored her on how to present it, e.g., describe the issue as methodically as possible, list the steps she'd already taken to resolve it herself, etc.

Due to this agreement, we formed a relationship of trust, and it eventually led to broader mentoring. I have to say, though, that the success of the relationship lay squarely with the other QA engineer: she recognized her weaknesses, sought out a mentor and actively worked to better herself professionally.

I eventually left that team and this QA engineer assumed a more active role in leading quality efforts. Over the course of a few months, I was pleased to see that she was flourishing in this role.

What's missing from the Agile manifesto?

Coincidental to my fellow blogger Dale's attendance and coverage of the Agile Development Practices conference, iTunes downloaded an interesting podcast into my StickyMinds  conferences subscription. It was for a short Q&A with Brian Marick about his keynote address at the conference with the intriguing title "Seven Years Later: What the Agile Manifesto Left Out." 

Questioning the Manifesto? Isn't this, like tugging on Superman's cape, one of the things you just don't do?  Well, since Brian 's one of the original thirteen signatories, he's in a good position to offer
commentary on it.  The full text of his keynote is found on his blog, but the thing I took from
the interview was a main weakness in the Manifesto is not describing the internal values necessary for Agile to be successful, as opposed to the external values to make the business and dev team interaction successful.  In the text, he lists the internal values as Courage, Working Software, Ease, Being Reactive, Fast Feedback, Naivete', Visibility to the point of Exhibitionism, and Joy.

One line from the written text that jumps out at me is "There are always tradeoffs between the values".  To me, that means not just between the items on the left and right on each line of the original Manifesto, but between the lines.  While this is not missing from the Manifesto, more people need an appreciation for this and to think about when it makes sense to trade off one against another.  And that it is ok to do this.  Everything can't be priority #1 at all times.

Two things I think are missing from the Manifesto are a recognition that it should change over time and laying out the means for such change.  It seems remarkable that a document putting "Responding to change" as one of the top four values for software development would not see value in acknowledging that it, too, might need to change along the line.  Brian touches on this in the interview and sees it as remarkable enough that the thirteen signatories could come together and come to agreement once.  The odds of getting this to happen more than once are just too low, in his opinion, to believe it could happen.  Still, that disappoints me.

Interestingly, a difference between the interview and the written text is that in the interview he mentions Skill and Discipline, not just as values but as ones that are lacking (in the case of skill) and more necessary (in the case of discipline) to make Agile work.  I think these are very good points and are often overlooked.  Do some of the values, such as fast feedback, visibility and working software combined with courage (to be honest about them and not fudge them for appearance's sake),
require discipline?  A different set, like working software + ease, might require strong skills? 

I'd be interested in hearing Brian's take on the relationships between skill and discipline and his other values.  I was slightly disappointed that there wasn't more discussion of these two in the text of his talk since I'd say those are two big gaps in the internal workings of teams. 

I leave readers with a different set of questions:

• Are we doing enough to recognize what skills gaps there are in our teams? 
• How are we addressing those rather than making the team relearn all of the ideas that really experienced folks know?
• Are we sending enough developers out to conferences or other events with peers? 
• Are teams putting up information radiators to reflect to themselves how disciplined they are? 
• What do you think is missing from the Manifesto?

Key Questions for a Customer Proxy

Over a year ago, Roman Pichler wrote an important post on Being an Effective Product Owner. In it he writes:

Proactive stakeholder management is particularly important in larger organizations, where the stakeholders include not only customers but also internal functions, such as production support, service or sales, as well. Taking into account and prioritizing the various interests early on and involving stakeholders regularly, for instance in form of user story writing workshops and sprint reviews, is crucial to ensuring that the software can successfully work in its target environment.

While frequent, iterative, and incremental deliveries help course correct based on working code, this is not an excuse to "just code it and see". Getting good at Agile means getting good at managing stakeholder expectations. As a Product Manager, when I deal with a changing requirement, there are 4 essential questions I need to answer over its lifecycle:

• Is everyone on-board? - I need to ensure the business itself is setting common goals.
• Should we make this change? - I need to enable changing goals. Ironically, this means saying no to some changes.
• Is everything in place? - I need to know that code, tests, documentation, or any other part is done.
• What did we get? - I need to express the value of what was delivered to all stakeholders.

It is not that asking the questions requires much effort. Rather, it is the delay between asking the question and getting the answer that determines how much change I can accommodate. If it takes 2 weeks to get stakeholders to agree to the meaning of a change and another 2 weeks to get estimates from the development team, then I will be unable to take lessons learned in one 4-week iteration into the plan of the next. Even if the effort is measured in man-minutes, the latency of the process impedes my capacity to change requirements. In order to become change adept with regard to requirements, a customer proxy needs to become very efficient in answering these questions.

I would be very eager to know how other people are answering these questions. What practices help make the answers readily available? What tools make it easier to get the answers on demand?

Agile Development Practices conference take-aways

I've just returned from the superb Agile Development Practices conference in Orlando, Florida.  One of the most inspiring sessions I experienced was Brian Marick's opening keynote.   It set a pretty high bar for the rest of the conference.  I'll try to give you my impressions and take-aways from Brian's talk, but I would strongly encourage you to read the original, which Brian has posted on his blog.

Looking back on the movement christened by the publication of the Agile Manifesto, Brian shared with us his perspective on how that mission statement should be enhanced and extended.  Brian focused on agile values as a way to resist the temptation to chase short-term gains at the expense of long-term benefit.  The eight values he discussed were:

• Courage
• Working Software
• Ease
• Reactive Behavior
• Fast Feedback
• Naiveté
• Visibility
• Joy

Courage is needed to overcome the many kinds of Fear that can distract us from our goal.  It allows us to question the rules, and break or change them when appropriate.  Working Software is our goal.  This means software that fulfills the needs of the people who use the software.  This implies both quality and fitness for purpose.  Brian emphasized that small increments of this goal should be produced very frequently, citing Kent Beck's advice to have no more than 15 minutes of discussion before someone makes the ideas concrete by writing code.  In all the work we do, we must strive for Ease.  Taken in an appropriate context of long-term value, this means making many small investments along the way so that our work becomes more streamlined rather than bogging down under the drag caused by allowing barnacles to accumulate.  Ultimately, this allows us to avoid Yak Shaving, but you'll have to see Brian's transcript for more on that.

A pervasive thread through the presentation was the idea of keeping software Soft.  This was especially emphasized by valuing Reactive (rather than the traditionally valued Proactive) Behavior.  This does not mean that we don't have a plan, but that we retain the Softness required to bend our plans while constantly moving forward.  Fast Feedback helps us become aware of important information that we should integrate into our plans as we react to what we learn.  The faster the feedback, the better we will do at staying on target, so we should strive for feedback on the order of minutes, such as that provided by Test Driven Development and Pair Programming. 

The value of Naiveté really amounts to what Eastern martial artists refer to as Beginner's Mind.  This is the transcendence of what we "know" about what is possible, so that we are open to succeeding with the seemingly impossible.  Visibility is the art of simply exposing the problem and letting our human nature generate a solution.  In this way we find that people can be soft like software.  When the consequences of our choices are visible, we become motivated to change, where we would have resisted exactly the same change if it was imposed on us.  Taken together, these values lead to a sense of Joy.  As David Anderson said in his closing keynote, we can create a place where we want to work, rather than have to work.  From a motivational perspective, this kind of environment captures Discretionary Effort, which is the hallmark of high performance.

At the end of his talk, Brian invited a panel of agilists to offer their thoughts and values.  This list included:

• Simplicity (J.B. Rainsberger)
• Integrity (Linda Rising)
• Questioning (Rachel Davies)
• Angst & Purpose (Jeff Patton)
• Experience (David Hussman)
• Community (Anthony Marcano)

J.B. began with a story about Simplicity that encouraged us to ask ourselves what it would take to start making money sooner.  Linda followed with admonitions to avoid Delusion—a lacking of Visibility, Courage and Ease that leads us to overlook challenging, yet often obvious, opportunities for improvement.  I interpret this as the value of Integrity—doing what we know is right even when it's not easy in the short term.  Rachel asked us to have the Courage to say "No" to taking on too much work, illustrating the value of Questioning. 

Jeff contributed the value of Angst—the motivating sensitivity to things that could be better, and the value of Purpose—to make sure we all understand the strategic vision that guides our tactical decisions.  David offered potential manifesto additions like "Success over Dogma" and "Description over Prescription", ultimately focusing on the value of Experience communicated through sharing our stories.  This really resonated with me as there always seems to be some nuanced understanding to be gained through listening to the struggles and triumphs of others.

Anthony concluded with the value of Community—a kind a Family Spirit that keeps us looking out for each others' best interests.  For me, this relates to the value of Collaboration, which is mentioned in the original manifesto, and seems central to the culture change required for successful agile adoption.  Focusing on Collaboration means replacing win-lose relationships with win-win relationships.  In asking how we all can benefit, we find the path to sustainable success.

What values do you think should be added, removed, clarified or modified?

[Special thanks to Joey McAllister of SQE for allowing me to review the tape of Brian's session and capture the comments of the panelists]

QA specialists vs testers

Back in July, I wrote on my agile testing blog:

I'm becoming increasingly concerned about the suitability of conventional QA engineers in agile environments. Since agile methodologies stress that the entire team is responsible for testing and that as much testing as possible be automated, the QA specialist has limited usefulness on an agile team.

When I re-read that entry today, it felt like I wrote it years ago, not a few months. I still hold the same general belief--that distributing testing among scrum team members reduces the role traditionally played by the QA engineer--but my take on the situation is much more positive now.

While agile adoption could indeed lead to a total reduction in the number of QA engineers needed in the industry, it can be a good opportunity for individual QA engineers. Nobody is relegated to the role of tester--a subset of QA work that's generally considered lower status. Instead, every QA engineer on an agile team must function more like a lead QA engineer: planning testing strategies, advising the team on what testing is necessary, and making sure that the testing work is accomplished by various team members.

I have seen first-hand the positive effects of scrum on the professional lives of individual engineers. I mentored a QA engineer on one of our teams here who first struggled with the challenges of her role on her scrum team. But I'm happy to report that she has since flourished and really come into her own as a 'QA specialist' on the team. She has gained the respect of the entire team and raised the overall quality of the software that her team develops.

I'm interested to know if others have witnessed the same types of effects.

The benefits of continuous software delivery

In my previous two posts (part one and part two), I discussed one of the benefits that Borland derives from delivering working software to customers at the end of each sprint. Allowing customers to try out and provide feedback on sprint builds is probably the most obvious goal of the first agile principle: Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

But Borland also derives other, perhaps less obvious, benefits from demonstrating working software at the end of each sprint. The first such benefit is that Borland's management is able to concretely monitor the progress of a product release. Here's how (somewhat oversimplified).

At the beginning of a release, stakeholders in the application hold an agile roadmapping session. The product of this session is a prioritized list of desired functionality for that release. The product owner further decomposes this functionality into prioritized and roughly scoped user stories. Based on the scoping of the stories and the development team's historic velocity, the product owner is able to draw a line in the product backlog and tell management what amount of functionality the team should be able to complete in the release.

At the end of each sprint, management can see the stories that are checked off the top of the list of user stories targeted for the release and therefore see whether the development team is roughly on track. Furthermore, management gets feedback from customers that the completed stories actually meet their needs.

How does Borland's management benefit from this type of progress monitoring?

For one, they don't panic if a team gets behind on their planned work. Change is a given, not an exception. If a conventional waterfall project gets behind schedule, the entire release is at jeopardy and the only real option is to extend the release date. In the agile process described above, however, only the functionality at the bottom of the prioritized user story list  is at risk. Furthermore, management has two options for dealing with this: if they feel that all targeted functionality is necessary for the release, then the choice is to move the release date. If, however, they decide that they can release without the incomplete functionality, then the release date is firm and content is the variable. Since the unfinished functionality was prioritized lowest to begin with, and since so many other company activities depend on the release date, at Borland we tend to assume from the beginning that the release date is firm and functionality is not. This acknowledgment, in turn, affects user story prioritization from the beginning. In fact, for some products, I've seen the product owner draw a 'must-have' line that is well below the team's total velocity for the release, followed by a 'nice-to-have' list of stories. That makes it even clearer that functionality is the chosen variable.

Management's ability to feel confident in the accuracy of their monitoring of product development has been a huge benefit at Borland. In fact, our SVP of R&D, Pete Morowski, has been giving presentations all over the place about the benefits he derives from agile within his organization. In fact, you can listen to a podcast of Pete's thoughts on Borland's Agile Transformation site. (When I've heard Pete's presentations, I have agreed that he's accurately portraying our situation, not sugar-coating it for marketing reasons. That says a lot coming from a cynical guy like me).

How do members of Borland's agile product development teams benefit from all this?

I think the most obvious benefit is the relationship of trust that has developed between management and the teams. Management is trusting the teams to take care of their business and not 'managing' their work so much as monitoring it and changing plans based on their progress. That makes for a pleasant, low-stress workplace.

The first commandment, continued

In my previous post, Thou shalt deliver software early and continuously, I replied to a blog post by Kishore Kumar about the agile principle: Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Kishore subsequently left a comment on my blog post and posted a response on his blog.

In his comment and follow-up post, Kishore compares scrum sprint builds to alpha releases in a waterfall project. I agree with Kishore to a point: one purpose of both sprint builds and alpha releases is to solicit customer feedback before the end of the release.

But the similarities stop there. In my opinion, with alpha releases of waterfall projects, the developing company is demonstrating that it is set to deliver what it committed to months before. With sprint builds of agile projects, the developing company is showing the customer that it is delivering what the customer wants.

With an alpha release near the end of a months-long waterfall release, the customer is seeing completed functionality for the first time near the end of the release. The customer has a lot to digest in a short amount of time, and there is little time to incorporate customer feedback or changes before the final release. In my experience with alpha releases, the level of customer feedback that's actually acted upon before the final release is typically limited to fixing major bugs that the customer uncovers.

With sprint builds, however, the customer gets to try out the new functionality in small chunks throughout the release, and, especially with the earlier sprints in the release cycle, there is ample opportunity for the development team to incorporate customer feedback into the final product.

I promised Kishore that I would discuss some specific benefits that Borland derives from sharing sprint builds with customers, but since this post has already gotten pretty long, I think I'll save that for my next post.