I enjoyed reading Planning Extreme Programming by Kent Beck and Martin Fowler. I found the book to be an easy read and finished it in just a few days.  Beck and Fowler are generous in their on-line writings and you can find a lot of information about agile process in various articles; however, I found it to be very useful gathered together in a narrative that clearly spelled out basic principles and best practices. One point that they made toward the end of the book was that you can borrow any of the practices individually, but that XP works best when the different practices are combined in a process that reinforces itself. They weren’t speaking of development practices, like pair programming, but planning processes, such as:

  • A clearly identified product owner who communicates directly and regularly with the development team
  • Iterations, where the date never slips (only the scope if needed)
  • Planning with yesterday’s weather technique: you only promise to deliver as much as you did in the last iteration
  • The product owner writes stories (e.g. short sentences on index cards)
  • The people who will write the code make the estimates
  • The product owner meets with the dev team at the beginning of each iteration and communicates the stories, answers questions, clarifies details
  • The development team breaks stories into tasks (detailed estimates are then given to the tasks without reference to previous estimates)
  • If you aren’t going to get everything done in the time available, then you reduce scope by talking to the product owner and letting him or her prioritize, including breaking larger stories into smaller ones that will fit.  (This should happen rarely)

They made the point early on that if you aren’t working in an environment or with a client who is willing to have a product owner regularly spend time communicating with the engineers and involved in the process, don’t bother with XP at all.  I admired this strong stance, and agree that the whole thing falls apart without an individual willing to make business decisions and spend some time refining requirements and prioritizing.

I’ve been practicing many agile processes since before the terms XP or agile were coined; however, I appreciated the focused perspective and am planning to try the process as described (which is not very far from my typical process).   I do have a few questions not addressed by the book:

  • Is there a way to use “yesterday’s weather” as a planning tool if you are working with elastic resources? (e.g. volunteers who work different number of hours per week or in a bootstrap startup environment where people are working variable hours)  if you have a more laissez-faire approach to milestones, then having simple priorities rather than well defined scope for each iteration seems to make sense to me. Would Beck and Fowler consider this an abandonment of XP or a fair variation?
  • What are effective ways to handle the planning process when this week’s stories have unexpected complexities? (e.g. the product owner describes the story, the developer asks some key questions which require a decision, the developer wants a decision on the requirements, but the product owner needs to understand the consequences of the decision before deciding)  My approach to this is defer the story and allocate some time during the iteration for design/brainstorming sessions on the challenging story.
  • When does the UI design happen? I have a number of answers to this one, but was surprised that I don’t hear this more frequently addressed in XP.

I used to pair program now and then, but only for specific challenges. When I needed to implement a new feature in an unfamiliar area of the code, I might pair with the original author. I might collaborate on a particularly tricky algorithm or when we’re up against a tight deadline and it must absolutely be correct. And, of course, you put a team of engineers on it, when there’s that critical bug that happening to real people in production right now. A few months ago, I took a different approach. I started to pair program routinely. The experience changed my perspective entirely.

Background

I was working on a project where I wore two hats: product manager and engineer. In the engineering role, I had a small task to develop a Flash video client app and a Rails API that it would call. The other job was to write down requirements and work with the business owners to come up with a well-defined scope and project plan for the next iteration of the software. At the outset of the project, they gave me a demo. The site looked good and seemed to work well, it was just missing a few features and the application needed to be brought up to production readiness. After digging in a bit, I realized that the project needed more than one engineer to reach the desired launch date and ended up hiring Wolfram Arnold to work with me in building the Ruby on Rails web application and web services APIs. He asked me how I felt about pair programming and proposed that we pair about 2/3 of the time. Since I was relatively new to Rails and he was expert, plus he was new to the project and I knew the requirements inside and out, I felt this would strike a good balance. The business owners were unsure, but I argued that the easiest bug to fix is one that you don’t write. It was a relatively short 6 week contract, so we decided to give it a try. But first, we had to make sure that we had the right candidate for the job.

The Interview

I’m used to having a team of engineers at my disposal to vet a candidate, but in this case there was just me and a remote engineer to do a phone interview. I’ve always relied on seeing a candidate write code on the whiteboard to get a feel for how they think about code. However, I have always felt the whiteboard coding exercise to be a bit awkward, so we asked Wolf if he had any ideas about what we could do to affirm that he was the sharp engineer he seemed to be in conversation. Wolf suggested that we just pair on a real coding problem to complete the interview process. So we got together one afternoon and I picked a specific bug that I thought we could fix in about an hour. I was then introduced to a common pair programming setup that was nonetheless new to me: one computer, two keyboards, two mice. While unexpected at first, this was a very effective way to work. I found the switching to be naturally mediated by social convention. We discussed the problem, then dove in, taking turns, with me navigating to the right area of the code and us taking turns typing. We ended up spending a few hours together and fixed the bug (and wrote some tests and refactored code along the way), happily confirming that Wolf was as good as he seemed on paper and that we could easily work together.

Getting Started

The task at hand was a fairly tricky one. We were migrating a codebase from Rails 2.0.2, that relied heavily on the Globalize plugin for translation, to Rails 2.3 which includes built-in localization support. Additionally the plugin had been modified and the database had quite a bit of data which needed to be migrated forward. I immediately felt the speed of pair programming: I understood the database schema, product requirements and current feature set; Wolf could readily explain common Rails patterns (and the occasional anti-pattern) in the code and recommended good solutions. Together we were easily twice as fast as each of us individually. We found that there were a number of requirements that were unclear when we dug in and we spent about three weeks documenting these detailed requirements by writing tests for them. This was a critical step since we didn’t want to lose or break functionality when upgrading the code.  My favorite time savings in this phase was that we never had to call a meeting, whenever we would need to meet, there we were.

Escape Velocity

When we felt we understood the detailed requirements and has a core set of the features working well with tests to prove it, we dove into the upgrade preparation which involved removing the modified Globalize plugin, a few huge database migrations and some other important details. I had never coded anything so risky, so fast. If anyone were to code such a change solo, it would require careful focus and iteration ensuring that every detail were taken care of. You would naturally go slowly as your mind searches for each dependency and where you miss aspects and see test failures, you go back and fix your code and run again. As a pair of programmers, who were now familiar with the codebase, schema, and requirements, we flew. We each had our own checklists in our heads, as one of us typed, one kept a list on paper. When making any change, before we even ran the tests once, the one not coding would remember dependencies, notice typos or test cases not written that needed to be handled. The iterations were fewer. Each of us had time to think when the other typed. While waiting for tests to run, we discussed what else could go wrong or what we might do to double-check that it had all gone right. We were now coding at three or four times the pace that either would have going solo. And more importantly, we caught bugs in our brainstorming that would otherwise have been caught in a longer deploy-test-report-fix-deploy-verify cycle that is several order of magnitudes slower than when a developer notices an issue in the middle of a coding session.

Routine Speed

After we got through the most challenging and risky parts of the project, we continued to pair. Beyond my initial expectations that pairing would only accelerate the learning process, I found that even routine coding tasks and feature development were executed with breathtaking speed when we paired. While there were logistical reasons that we occasionally worked separately, the project moved forward most efficiently when we coded together. If anyone is starting a Ruby on Rails project, wants to adopt TDD or simply accelerate their development, I would recommend hiring Wolf. When we took the site live and moved on to work on separate projects, I found that I had grown used to the routine speed of pairing and missed the camaraderie as well. I wanted to achieve a similar acceleration on all of my projects, so I hired an intern, Lee Lundrigan, to pair with me. Lee didn’t have as much domain expertise, but he’s a very sharp engineer and does have some experience that complements mine. It’s not as fast as when I pair with Wolf — a dozen years or so of experience makes a difference. I’m guessing that it only doubles my speed.  I find the collaboration sharpens my attention and focus, in addition to bringing his perspective to the problem at hand.  The extra pair of eyes catch bugs before the first test run reducing debugging time.

Conclusion

If you haven’t tried pairing and you work in a team, just do it.  Give yourself a project for a couple of months to get a feel for it before you decide what you think.  If you work solo, consider hiring someone who is experienced with pairing and agile development to work with you to give it a try.  Take an opportunity when you are learning something new or have a big project you are taking on.  I’d be interested in hearing other people’s pairing experiences.

Last night Wolfram Arnold and I gave a talk at the SF Ruby Meetup about a recent collaboration where we created specs, documentation and tests using cucumber to streamline workflow and communication between client and server engineers.

It was Wolf’s idea to start the talk with a role play of the real-life situation that we were faced with at the outset of the project.  He convinced me to do it as improv, which was a lot of fun, but for those who missed here’s the script that I had written out earlier:

Biz Guy (ring):  Hey Pat, this is Charlie. I’ve got an investor pitch in the morning and I just tried the app on my Nokia phone and I’m no seeing search results any more.  Can you look into it?
Client Engineer: I haven’t touched the code in  months.  It must be the server code.

Biz Guy (ring): Hi Sarah, this is Charlie.  The mobile app isn’t working on my Nokia phone and Tony says he hasn’t changed the code in months.  You said that server engineer you hired was really good, maybe he could fix it by tomorrow.
Manager: hmmm… I’ll check.

Manager (ring): Hey Wolf, this is Sarah.  Did you make any changes lately that could have affected the mobile client?
Server Engineer: No… none of the APIs have changed and all of the tests pass. In fact, the mobile app demo on the website works and that uses the same APIs, so it must be a client problem.

Manager (ring): Hey Pat, this is Sarah. Glen told me the mobile app isn’t working, but the demo on the web does work.  I understand they use the same APIs, so I’m guess the issue is in the mobile client code.
Client Engineer: Actually, the mobile app uses a slightly different authentication than the app on the web
Manager: Oh really, is that documented?
Client Engineer: well, there is detailed documentation of the platform SDK that the mobile phone doesn’t support http header responses so the auth token needs to be sent back in the post body
Manager: I see.  Unfortunately, our server engineer isn’t actually familiar with the mobile platform SDK.

This is a critical problem in my experience and a very common one. Whenever you have API documentation, it is usually incomplete, inaccurate or both, often despite the best intentions of everyone involved. In this case, we solved the problem of documenting and testing XML APIs by writing cucumber scenarios which serve as both the documentation and tests. By making the documentation executable with published results in a continuous integration system, we always have up to date documentation.

The slides from the talk are posted on slideshare: