Paul Julius > Blog

Cup of Coffee Metric for Continuous Integration

September 14th, 2009

How much can you learn in the time it takes to get a cup of coffee?

How much can you learn in the time it takes to get a cup of coffee?

What happens in the time it takes to get a cup of coffee can tell you a lot about the quality of your continuous integration system. Try something out for me. Commit some code. Leave your desk to get a cup of coffee, or some other tasty beverage. When you get back to your desk, do you know whether your changes were any good? No? Yes? I hope so, and so should you. It’s called the cup of coffee metric.

The idea is simple. In the about the amount of time it takes to get a cup of coffee you should be able to get information about the quality of your changes. Look at it this way. You just committed some work. You need a short break before you move on to your next task. You don’t want to wait too long, or you might find yourself surfing the web instead of writing code. So, leave your desk. Get a coffee. Take a bathroom break. See what’s hot around the water cooler. Once refreshed, return to your desk. You don’t want to move on until you make sure your last commit didn’t cause any immediate problems. A good CI system will support this behavior by giving you some feedback in about that much time.

The level of information may not be complete. Enterprise build systems are notorious for the length of their builds. Enterprise systems are also notorious for waiting to provide any feedback until the end of the build cycle. That sucks. A little information would be better than none. It might only be an initial piece of feedback, like “Your changes compiled!” If that’s all your CI system can get done in cup of coffee time, that’s better than nothing. Your CI system might tell you everything is going smoothly, only to return later to tell you something broke. No worries, sometimes that will be the case. Hopefully not always.

As a build engineer, I am going to try my darnedest to provide you with as much information as possible while you are getting your refreshment. That’s how I keep my team humming along. I might even use a “build pipeline” to help facilitate this sort of thing, but that’s an implementation detail. I am always asking myself, “What’s the most I can possibly discover about a commit within about 5 or 10 minutes from the time a developer presses the button?”

This idea came from Jeffrey Fredrick‘s real life experiences. He and I co-present on various topics. We talk about CI a lot at CITCON, of course. Invariably we discuss the cup of coffee metric when talking about evaluating CI systems. I stole the idea from him for use in my consulting. (Most of my good ideas come from our community. Thank you internet!) With his permission, I figured it would make sense to write it up for the world. So, here you go.

9 Responses to “Cup of Coffee Metric for Continuous Integration”

  1. Curits says:

    I think it was Glover and Duvall and guys at his old firm (Stelligent?) that started the 10-minute rule (Didn’t you contribute a foreward to this book too?). Build need to finish and report in <10minutes, your cup-of-coffee but with real numbers. Through modularization of our build process we have consistently been hitting this metric. f course we are only compiling, packaging and running the Unit Test but that should be enough. Immediately after that,, after the buld success or failure notification went out, we go on to run our longer running, more comprehensive build complete with code scans, Site and Sonar reports.

    So, like 'technical debt' while cute and fun to talk about I feel Coffee Metric is a little too fuzzy for people to apply. I'm sticking with 10 minutes and let them decide what to do with their time ūüėČ

  2. Hi Curtis,

    the book you’re talking about came out in 2007. I started talking about the “cup-of-coffee” rule back in 2004 (though oddly not blogging about it, so I’m happy Paul did).

    I think the advantage of the fuzzy coffee rule is that when you give it with a story it helps people understand why it is important ‚ÄĒ avoiding the cost of context switching ‚ÄĒ in a way that “5 minutes” or “10 minutes” don’t convey. Sure, a fixed time is more actionable, and I’m all in favor of that too, but I think the story around coffee/tea is more memorable.

    (Example of the story in this article.)

    Jtf

  3. Pinoy Sikat says:

    Our community needs this kind of advices. Keep it up!.

  4. […] I made some changes locally and committed them. A few minutes later, I was back at my desk after¬†refreshing my coffee¬†and immediately was concerned. My instant message client was not blinking as it should be. Where was […]

  5. […] facilitate continuous integration. And by “fast” we mean the time it takes to get a¬†cup of coffee¬†or just at least¬†¬†under 10 […]

  6. […] to the CFO of a company. One can use “Cup of Coffee Metric for Continuous Integration” (pauljulius.com): hours wasted per developer * number of developers * average hourly rate * 20 working days a month […]

  7. […] build script that takes 10+ minutes to compile and run unit tests and I lament that it fails the cup of coffee test, but should I look further. ¬†The faster the feedback, the faster you find bugs, the faster you […]

  8. […] In North America, though, the last few CITCON events have been focusing more on advanced Continuous Delivery topics. That does not mean that practitioners new to the field should shy away. On the contrary, they are the most important attendees in many ways. They challenge the thinking of old timers like myself. They make me re-rationalize things I take for granted, like the importance of the cup-of-coffee metric. […]

  9. […] Next I took part in a discussion about making tests effective: we studied how and why the test pyramid often turns upside down, how the tests near the top get more and more slow and unstable, and how to fix them. These are so called second generation problems, since they imply that, at least, there is a bunch of automated tests for the code. There’s nothing wrong about running series of end-to-end or through-the-UI tests during whole nights frequently, but you need to care about the lower levels of your testing pyramid: the much faster unit tests (close to or above a thousand tests per second!) and integration tests. Have you heard about the Cup of Coffee Metric? […]

Leave a Reply