Fork me on GitHub

Tuesday, April 14, 2009

Invest in tests wisely for a secure future for your application

Tests are there for different reasons like design drivers, safety net, executable specification. The reason I am taking here is safety net where your tests will help you to understand that if something is broken in the application after a change has been made. For an application which is in flux this is a really important usage of test.

It is possible and many would be doing this, to write tests at different layers. We can have tests in Domain layer, at service layer and at UI layer. The metaphor I see tests across these layer is "The different tests are your application's investment portfolio". (WARNING - I am not really expert on finance terms.). Invest wisely the application will live happily or the investment is going to drag the application down.
As with the real investment portfolio tests are risk limiting strategy. And nobody with their brains working will invest on one kind of investment too much and ignore the rest. That kind of increases the risk. So it is important to distribute the investment across the options we have. Again the percentage we are going to use for distribution is really important.

I am not going to say here that investing on UI tests is a bad thing as it all depends on the context . On general UI tests are fragile, slow and have a high probability to drag the application down. But they are the easiest to create (with all the cool drivers like Watir and Selenium, recorders and everything). On the flip side the domain layer tests are kind of stable, run fast but they tests units in isolation (This has both +ve and -ve aspect).

The biggest problem I have is when a functional test fails because the id of an button has changed. From a user's perspective, I don't care as long as functionality works. So this raises a question should I be testing functionality through UI? We have to have those tests because that is how the app is going to be used. So I tend to keep at dozen of end to end UI tests. In my case anything more than that will potentially turn to pain at the end of the day.

So before you go ahead and create a recorded test using the new shiny tool you got there, ask yourself is it the right investment you are making for you application. Secure the future of your app with the right investment :)

Thanks @lisacrispin - Look at your ROI when thinking about investing in a kind of test. (Warning - ROI doesn't mean Excel sheet which your managers use :)

(I kind of tend to use this as my investment guideline - http://c2.com/cgi/wiki?TestFoodPyramid)
(Picture from Flickr. Thanks to lastminute.)

2 comments:

Lisa said...

So true and well said! The automation pyramid is a great guideline. We definitely try to push testing down to the lowest level possible. Most functional testing is "behind the GUI". However, you can also choose your GUI tools wisely. We have gotten super ROI from Canoo WebTest for smoke regression tests. Easy to specify the tests, little logic in the tests, integrates easily with build process, quick feedback, and catches almost 100% of regressions that slip past unit tests. Consider ROI when evaluating tools is my advice.

Sai Venkatakrishnan said...

Aha.. ROI was the term I was looking for. How can I talk about investment and miss about ROI ^_^.

I hope when we say ROI people don't think it as a excel or a web application where they fill numbers before testing and get another number out of it and say that is going to be my ROI.