Fork me on GitHub

Monday, September 8, 2008

Why GUI tests fail a lot? (From tools perspective)

I was in a discussion today which involved brittle automation tests done through GUI as usual. The problem statement -- GUI changes continuously more specifically the properties of the object changes continuously. The reason behind this is the pages are generated dynamically at build time and so the properties for identifying the objects are not constant. Their whole test suite is GUI based and they don't want to throw away this as well don't want to rework every time a new build comes. My only choice to give advice was to work with the developers to find a way to fix one of the properties most probably html id so that it will give them a stable interface to work with automation.

But I see this pattern of problem repeatedly. Sometimes it is with the generated html or at times with manually created ones where developers change if they wanted to name something better or change some object's property (May be refactor the html). I know that automating and concentrating a lot of testing through GUI is not a good choice although it is needed at some point. I know the reason for this that the GUI changes continuously. But I needed a better way of expressing this. So I thought of the good reason why we still go for GUI automation and why we fail on this?

Why do we go for GUI automation?

Evil lies in the availability of easy to use tools which entice us to create tests quickly by record and playback and create brittle test code. They also guarantee us to test the application end to end which is needed to test the complete integrated application.

But why do the code we write dependent on the GUI fails? (GUI Testing from the tools perspective)

One of the compelling answer I found was GUI is an human interface which people like you and me use and any other  interface is programmatic one to use your application as a service. Now programmatic interfaces, when published change less because any change to its contract will break the external programs which consume them. But a GUI interface being used by users change for two reasons.

1) If the html internal properties like id or class change then that doesn't affect the user directly.

2) If the appearance, location, text or description of the object changes then the user may not be able to immediately identify it but human beings have the ability to adapt to change, understand and act accordingly.

The problem comes when a GUI automation tool tries to use the application's GUI as a programmatic interface. They try to expect a constant interface contract (i.e. the properties of the objects and objects wouldn't change time after time) which can be used to consume your application GUI as a service. But this is not possible as GUI is not meant to be consumed by programs in the first place and when the GUI change, the contract between the automation tool and the application is broken. GUI testing tools also don't have the ability to understand and adapt unless they have a built in AI in them :).

So the better solution would be to concentrate your testing through a programmatic interface which can work with domain objects directly. This will help you to create a cleaner design of the application by making our GUI layer lean with no complex domain logic. As well the testing will be faster when we bypass the GUI layer. The test code will be more domain oriented as testing will concentrate more on the actual functionality of the application rather than on the GUI layer. Using tools like FitNesse, Concordion are suitable in this context.

But is testing through GUI not necessary?

The answer is we need. We need a set of tests which can run on GUI to tests it as well gives us feedback at end to end level. As well GUI needs to be tested when we have client side components like Ajax, Applets or others. The plus side of the contract enforcement of GUI testing tools needing a constant interface is a better naming of html objects like <HtmlInput id='txtUserName'> instead of absurd one like <HtmlInput id = 'txt0012345'>.

So this again brings up the concept to have a small set of tests which work on GUI using tools like Watir or selenium but concentrate most of the tests on the programmatic interface to test the domain layer directly.

No comments: