November 29, 2010

Evidence-Based Scheduling: Seth Coster Style

After stumbling across a post about evidence-based scheduling at Joel on Software, I decided to apply this to my own game design. I skimmed a few key points from this, which I used to develop my own modified version of evidence-based scheduling. Since my game design is fairly complex (and involves around 700 steps), I'll simplify the method by demonstrating its use in planning a novel about goats.

Again, note that the method described in this post is my own take on evidence-based scheduling. I have simplified it and made it easily usable in a spreadsheet. If you want the original version, head to the link above.

When planning a project, we try to set deadlines and benchmarks. However, our original schedule often gets thrown into the trash because it is so ludicrously unreliable and inaccurate and tells us nothing. Once we fall two weeks behind schedule, the original plan is useless.

Why is this? We tend to plan in broad strokes, such as "write chapter 1." This method is fairly useless because it is nebulous. When we plan in this way, we don't think about the individual steps involved in writing chapter 1, so any time we allocate to that overall task is arbitrary. And in the end, when we estimate a date at which the project will be finished, the date is meaningless because the time allocated to each task is meaningless. You won't know how long it will take to write chapter 1. Writing a chapter requires research, revisions, proofreading, note organization, etc... If you are a writer, you probably have some idea as to how long it takes you to do each of these small tasks. So break it down into those smaller components, and plan around those. Joel on Software says that you should never plan a task that takes more than 16 hours. This might be too large. I'd go for 10. If a task is this big, it should be broken down into smaller tasks. The bigger it gets, the less reliable your estimates will be.

Okay, so let's say we want to write a novel with three chapters. It's a children's novel. About goats. To plan it out, we start by breaking it down into small tasks, along with our estimates for how many hours each task will take (I'd advise doing this in a spreadsheet, as you'll see below):

Goat Novel Chapter 1:
  • Goat research: 8 hours
  • Note organization: 1 hour
  • Write chapter: 3 hours
  • Revision 1: 3 hours
  • Revision 2: 3 hours
  • Proofreading/Editing: 2 hours
  • Final draft: 2 hours
And rinse and repeat for chapters 2 and 3. Now that we have the tasks broken up into small chunks, we can take advantage of the "evidence" part. As each task is completed, we compare our estimates to the actual time each task took. We can then take "Estimated time" divided by "Actual time" to get a variable called "Velocity." So for this hypothetical situation, let's say that you finished your research, you organized your notes, and you wrote the chapter, which gave you the following actual times and velocities:

As you can see, some of our estimates are closer than others, which means we have different velocities. But those velocities give us good information, which is how good you are at estimating and whether you tend to overestimate or underestimate the time it takes you to complete a task. The higher the velocity, the faster you were relative to your estimate. Now, we can use our velocities to adjust our future estimate. To do this, we now set up another section of the spreadsheet that does the following:
  1. Calculates how many hours we have left to complete the task
  2. Uses our average velocity to adjust our estimated remaining hours to a more realistic level
  3. Reports back how many hours per day we must spend in order to reach the deadline
To do this, we first set up a separate column off to the side, called "Time Remaining." If our estimated time is in column C and actual time is in column D, here is the formula we put in the Time Remaining column:

=if(D2>0, "", C2)

This formula checks to see if we have completed the task. If so, it ignores it. If we haven't completed the task, it dumps the remaining hours into our "time remaining" column. This will give us the following:

Now, we can use our average velocity and our time remaining to calculate a more reliable estimate of how much time we have remaining. Down below what you see here, we can set up the following cluster of cells:

The "time remaining" cell (C26) is nothing more than a sum of column F. Cell C27, the "adjusted time remaining," is where the magic happens. It takes Time Remaining divided by Average Velocity. This adjusts your total time remaining based on how accurate your previous estimates were.

Now that we have this information, we can do all kinds of things with it, such as the following:

This is meaningful information. As a final demonstration, I'll add a few random completion times into the original data so you can see what this project would look like as it progressed.

As you can see, we are much slower than our estimates on average. So even though, at this point, we had estimated that the whole project would take us 44 more hours, our evidence-based scheduling kicks in and tells us that realistically, it's probably more like 57 hours. The nice thing about this scheduling method is that the more work you do, the more reliable the final estimate becomes. Remember, your goal isn't to "beat" your estimated times; it's to have an average velocity close to 1. Reliability and consistency are best.

I've put together a monster spreadsheet for Skybrawler using the above method. It employs about 700 tasks and could probably kill a goat if I let it loose. My spreadsheet currently tells me that I have about 3,500 hours of work left, and if I want to finish in time for the Indie Games Festival in 2012, I have to put in about 4.5 hours per day on average for the next two years. Hopefully I've severely overestimated my time requirements. We'll find out!

If you have questions about what I've written here, please drop a comment below!


  1. I have one concern: WEIGHTING!

    If you are a half hour off on a 1hr estimate, the effect is relatively large. If you are half hour off on a 10hr estimate, the effect is relatively small. What if your average velocity was based on the sum of each column instead of the average of the velocities?

    And I just thought of something else. What if you were to adjust your estimates based on your previous ones, since the work you put in at the beginning of a large project may totally change the downstream work required. But your re-estimates would be better informed and so your velocity-scaling based on the previous work would be wrong.

  2. Weighting shouldn't make much of a difference in a large project. If you have a project which is comprised of 100 tasks (which is certainly a reasonable number), totaling 500 hours, and one of your tasks is 10 hours and one is 1 hour, by the time you get 20-30 of those tasks under your belt, the weights would have a negligible effect. The purpose of the velocity variable is to show how accurate you are in proportion to the size of the task, which it does. For example, if you are off by 30 minutes on a 10 hour estimate, you were pretty close, and your velocity reflects that. If you were off by 30 minutes on a 1 hour estimate, you were way off, and your velocity reflects that as well.

    I'm not quite sure what you mean by basing it on the sum of each column.

    And as far as adjusting future estimates, if you feel that there are past velocities which are outliers, you can just delete them and they won't be used in computing the average.

  3. This is goddamn awesome. And I read the joel post. GOING TO BUILD ONE OF THESE TOMORROW WHEN MY AWAKE-VELOCITY IS HIGHER.