It is sad to see so many projects get shocking news that their schedule is not according to the one planned and estimated. I work as a consultant at on site in the client locations. At times I study the feasibility of automating an application and at the end normally the manager who interacts with the client from my side urges me to estimate the project accurately so that after we get the project it won’t lead to any issues after the project starts. There are a lot of things fundamentally wrong in that opinion. The first is software development project is a dynamic system and with so many unknown factors in a project it is never possible to predict anything remotely accurate. The second is the only accurate thing we know about a plan for a project is that it is bound to change. And finally and most important of all is estimation is a prediction into future. As far as I know there isn’t anybody who can predict into future accurately unless you have extremely high levels of ESP.
Are there ways in which we can make estimations accurate? What are the mistakes we are doing currently? I write this in relation to testing projects as I work mainly on them but they are general principles of estimation. As I can get my analogy easily from the environment I am currently in I am using it. But feel free to switch it to any other scenario with similar circumstances.
When we look at the current projects, the estimation is done together for all the requirements as one big task and never seem to updated at the later stage of the project. Typically they come to months of effort or worse (to the delight of the manager) to years. At a later stage of the project changing estimation is considered as a sin in a lot of companies as it will reduce your reputation in the eyes of customer. This is quiet understandable but it has to be taken into account that estimations are vision into future. They are bound to be inaccurate if we want to do it once and follow the plan till the end. As I have been working on these for sometime I try my experiments and gather my thoughts on what work and what won't. I write my observations here. These are in no way exhaustive but a small set of practices I found useful.
1) Do estimations in short iterations. if you try to estimate for longer time the mistakes we make add up together to make our estimate more inaccurate. When working in iterations, make concrete for current iteration which is starting, plan for next iteration, anything after that is a vision. It is not a problem to have a vision of what should happen but don't expect it to work directly as planned.
2) It is okay to create a release plan by taking all the functionalities into consideration but it should be noted that release plan is bound to change. After the project starts, across the iterations the project current status should be used to change the release plan for the better rather than keeping it constant and forcing people to work by it.
3) It is better to estimate for small amount of work at a time. The more work you take in hand the longer it will take to complete, longer time period will lead to inaccurate estimations. We will be back to square one where we started with a bulky estimation.
4) Estimations are typically done by some consultant and then handed on to the project manager who will use a team of developers to complete the work. Unfortunately this won’t work most of the time for the reason that there are differences in the way each and every developer works. It will be better to get the developer together and ask them to estimate as they know their work ability. Forcing developers to choose what you think as best won’t be a good choice. Discuss with them and if you don’t feel comfortable you can ask for a justification.
5) Estimating done in iterations will also help you to understand the speed of the project as well the pain issues which are hindering it. Using the previous iteration’s experience can help you to estimate better for the next iteration. You can also use this to update the release plan to make it more realistic for the moment.
6) Don't strive for using more accurate algorithm, estimation is always a guess. Putting a lot of effort in making it a bit more accurate won't be very useful in real time for the effort put in the process of calculating it.
A lot of these points correspond to some of the practices in agile development. But I tried these methods when I faced problems with estimation when I was working at onsite with a client and incidentally understood that they are also used in Agile. The woes I faced when we were fighting over an estimation because it was too unrealistic made me to rethink the way in which estimation has to be approached and the above points are the result of it. I am not advocating you to follow any agile methodology even though it will be better and I am big fan and follower of it. But these methods can be used without implementing agile and still work together to make your estimates accurate. Finally I urge you to understand and approach the estimation in a humane way rather than thinking that it as a mere part of a big process.