Fork me on GitHub

Thursday, June 18, 2009

How to measure QA velocity?

Answer - Trick question. There is nothing called QA velocity. (Applause for those who answered correctly :). To make the point strong I have seen having a separate QA burn down charts and story walls as well.

I always wondered what drives people to measure such numbers? I found that the main argument is QA backlog in a project is increasing, so we need to measure the velocity to effectively manage and track testing resources. I don't think measuring QA velocity will lead to any meaningful answers because testing is not a separate process from development. Unless a feature is developed, tested and accepted by the customer it doesn't make any sense. I have seen people measuring the number of stories completed by the development team and having a dev complete phase. To reiterate my point I think doing this also doesn't make any sense. In Agile development, testing and working with the customer happen collaboratively and are not handoffs from one group to another. It is always the team owning the quality and not a group of people who are given the position QA or tester.

So how do we help teams understand that there is a problem in flow and the constraint lies in a specific part of development process. I used the term flow specifically because I believe software is developed as a flow of value (throughput). It doesn't happens in phases (staggered waterfall model) and handoffs doesn't happen between groups. When working with teams with such problem, one tool which really helps me with this problem is the kanban boards. Not that I am also joining kanban bandwagon but I think it is really a good add on to the normal story boards we have when we want to identify the constraint which is obstructing the flow.

Illustration of a kanban board is shown in the figure.


One of my favourite features of the kanban board is the WIP limits in each swim lane. I will not go into describing each phase in the board but the numbers in each phase is the maximum number of working items in the lane. If the number of items becomes too low then we need to pull work from the upstream. But if the number of items in a lane is high, it clearly indicates the constraint. Say if the number of items in testing lane becomes high for some reason, then may be developers in team needs to pitch in to see if things can be automated or help QA's with some of the tasks like writing fixtures or Watir scripts (I love Watir :). If the number of items in development lanes becomes high, QA can pitch in the help writing fixtures of acceptance tests or setting up of the environment and build and sometimes write some production code as well (It is really a good way to learn the system internals). I normally help the team fix the maximum number of items in each lane by discussing with them to understand their capacity.

The example I use for software development is flow of water through a pipe. It is not a staged process. If the water flows out of the pipe it is good and adds value. If there is some problem in the flow it is good to know where the problem is and correct it to bring the flow back to normal. It is good to concentrate on flow and throughput than individual stages.

Let me know what you think and share your experiences through comments. :)

4 comments:

Anand Vishwanath said...

Yes ! There is no point measuring velocity in isolation, velocity of any kind. Because improving just velocity numbers is never going to make the plant more efficient, since you will end up building more inventory.

One problem I see is that people on the ground should be able to visualize the flow through stages in real time. I had a vanilla one running once http://blog.anandvishwanath.in/2008/10/scope-monitor.html .

The problem with Kanban boards and story walls is that they tend to get out of date unless you have someone playing card monkey.

Like we have radiators for builds we should ideally have radiators for flow.

Sai Venkatakrishnan said...

Radiators for flow is a nice concept. For me I like using boards because it is big and visible to everyone and people can play around with it. I don't believe in having a card monkey. It works as much as the concept of build monkey or test monkey :). For me team needs to be responsible for moving the right cards. Not just for the sake of self organization but also because it is the only effective way out of the people problem.

Nigel Fernandes said...
This comment has been removed by the author.
Nigel Fernandes said...

The definition of velocity is the crucial bit.

Most agile projects measure velocity based on when a story can be called "Done".

"Done" should mean a status beyond the QA stage, and usually after the customer has signed off on it.

By that practice, any attempts at a QA velocity is a near meaningless subset of your team velocity.

Long story cycles are one reason why velocity maybe calculated before QA completion. My belief is that such approaches are flawed.

Thoughts?