Fork me on GitHub

Thursday, June 18, 2009

How to measure QA velocity?

Answer - Trick question. There is nothing called QA velocity. (Applause for those who answered correctly :). To make the point strong I have seen having a separate QA burn down charts and story walls as well.

I always wondered what drives people to measure such numbers? I found that the main argument is QA backlog in a project is increasing, so we need to measure the velocity to effectively manage and track testing resources. I don't think measuring QA velocity will lead to any meaningful answers because testing is not a separate process from development. Unless a feature is developed, tested and accepted by the customer it doesn't make any sense. I have seen people measuring the number of stories completed by the development team and having a dev complete phase. To reiterate my point I think doing this also doesn't make any sense. In Agile development, testing and working with the customer happen collaboratively and are not handoffs from one group to another. It is always the team owning the quality and not a group of people who are given the position QA or tester.

So how do we help teams understand that there is a problem in flow and the constraint lies in a specific part of development process. I used the term flow specifically because I believe software is developed as a flow of value (throughput). It doesn't happens in phases (staggered waterfall model) and handoffs doesn't happen between groups. When working with teams with such problem, one tool which really helps me with this problem is the kanban boards. Not that I am also joining kanban bandwagon but I think it is really a good add on to the normal story boards we have when we want to identify the constraint which is obstructing the flow.

Illustration of a kanban board is shown in the figure.


One of my favourite features of the kanban board is the WIP limits in each swim lane. I will not go into describing each phase in the board but the numbers in each phase is the maximum number of working items in the lane. If the number of items becomes too low then we need to pull work from the upstream. But if the number of items in a lane is high, it clearly indicates the constraint. Say if the number of items in testing lane becomes high for some reason, then may be developers in team needs to pitch in to see if things can be automated or help QA's with some of the tasks like writing fixtures or Watir scripts (I love Watir :). If the number of items in development lanes becomes high, QA can pitch in the help writing fixtures of acceptance tests or setting up of the environment and build and sometimes write some production code as well (It is really a good way to learn the system internals). I normally help the team fix the maximum number of items in each lane by discussing with them to understand their capacity.

The example I use for software development is flow of water through a pipe. It is not a staged process. If the water flows out of the pipe it is good and adds value. If there is some problem in the flow it is good to know where the problem is and correct it to bring the flow back to normal. It is good to concentrate on flow and throughput than individual stages.

Let me know what you think and share your experiences through comments. :)