Fork me on GitHub

Friday, September 26, 2008

Things I learned in the past 3 years

I started this blog more as my personal notebook to write my learnings and till now it has served its purpose as well helped me share my views to the world.

I have this habit of revisiting where ever I started and see if the things I learned has changed or evolved as a result of my work and increased understanding. For the past three years I have been in the field of testing and here are some of the things whose definitions or understandings have evolved for me as a result of my work, the things I read, people I talk with and my most precious thoughts :). This is not the complete list but something which came up in my mind as I was thinking about this. I will try to keep this list updated whenever I can find something changed.

1) Testing is an experiment conducted in controlled environment and conditions to gather information about the application.
2) Acceptance tests are expectations of the customer.
3) Regression tests or tests created by recorders are tests for invariance.
4) Agile is more about people and their communication than about techniques and tools. (Something which dawned upon me after years of seeing agile in terms of techniques and process)
5) Domain is the core of any project. The project is there to solve a problem in the domain.
6) Speak the language of the customer. Using technical language of the customer will lead to complications in communication. (Personal experience)
7) Don't try to solve people based problems by putting up a process. (Everyday experience by working with the managers in my place)
8) Testers working closely with developers and customers is the most effective way (Understood this even before I started agile)
9) Every tool or language has a legacy. The usage of the tools depend on their legacy (Interesting point I understood. Will write about this more in detail later)
10) Automation can never replace manual testing. (Exploratory testing is invaluable)
11) GUI testing is unreliable and difficult. (Learned this as a bitter lesson)

Well thats what I can think of now. There are lots more things I will remember especially when I speak with people. Will keep a note of them and update this as and when I can

Wednesday, September 24, 2008

Are table structured tests good for regression testing?

I have been seeing cases lately where people try to use Fitnesse or Fit like infrastructures or a custom excel based fixture where in the tests are written in the form of tables. The reason behind this may be they have seen the success of using Fit or Fitnesse and want to extrapolate it everywhere. The problem I have with this approach is whether this table structure is enough for meeting the conditions needed for regression testing?

I am big fan of Fit and Fitnesse and advocate, use it as a way to write executable specifications. They are really good when you work with the customer to etch concrete examples about the business domain conditions. But using them as a framework for regression testing is something I am skeptical about. I prefer to use a language like Ruby or Python or Java to write my tests using drivers like watir or webdriver or schnell. The reason behind this is more than acceptance tests, regression tests cover much more conditions in form of data driven tests as well need constructs for iterating, conditional constructs, variable storage and passing and reusing code by modules. Having these constructs will help us simplify the code we write. As well regression tests don't have very close customer involvement as acceptance tests do and in this case for a tester or an automation engineer it is easier to use a language than a table structure.

This doesn't mean that we can't do or have these things in a table driven test. But for this we need to put an extra infrastructure in place which we have to maintain as well these constructs will not look elegant or natural in a table.

Implementing the regression tests in a developer language also doesn't mean that they will be just raw code which have the user actions flow driven using watir or webdriver or schnell. These tests are also written in a way that they reflect the domain language by building proper abstractions may be as an internal domain specific language. But these abstractions need not be structured as a table as explained in the above case.

This is from my opinion and experience. Different people will have varied experiences and thoughts. I would be happy if you can share your thoughts on this as it will help to understand this problem more or may be in a different angle...

Thursday, September 18, 2008

Controllability, Observability and Testability

I was an electrical engineering major in college and control systems was one of my most favorite subjects. The fundamental and important principle I understood in it was to control any system you need controllability (to effectively direct it to a particular state) and observability (to find out the application behavior or state at any point of time).

Now working in the field of testing, automation comes into play most of the time. To automate any application testability is a important. So what does this testability mean? what does the test interface for an application represent? Testability is the ability to control and observe the application's behavior to validate it and test interface facilitates this by helping in making calls to the application and in observing the behavior or state of the application at any point of time.

Based on my understanding of control systems, testability of an application is actually the combination of controllability and observability. Test Interface is a facade to the application which helps us to control it and observe its behavior and state.

But does controllability and observability assure us testability? This is something I need to think about. For an application there can be multiple interfaces through which you can control and observe its behavior. But not all interfaces are appropriate for testing for example user interface is a bad candidate for a test interface.

So testable interface has controllability and observability but not every controllable and observable is testable. For an interface to be useful for testing it is necessary to be easy to work with it programatically as well the interface to be stable to a certain extent.

Testability and test interfaces are an important aspect for automating an application. They are also an interesting topic and I hope to explore it more in future.

Saturday, September 13, 2008

Wordle's Perspective

Wordle is in rage now and it wants to tell everyone what it thinks about your blog and web pages. I wanted to give it a try and here is my word list according to wordle that my blog concentrates one. I expected bigger testing and examples and other words but let me try it after a few more months to see what it finds out ;-) .

Monday, September 8, 2008

Why GUI tests fail a lot? (From tools perspective)

I was in a discussion today which involved brittle automation tests done through GUI as usual. The problem statement -- GUI changes continuously more specifically the properties of the object changes continuously. The reason behind this is the pages are generated dynamically at build time and so the properties for identifying the objects are not constant. Their whole test suite is GUI based and they don't want to throw away this as well don't want to rework every time a new build comes. My only choice to give advice was to work with the developers to find a way to fix one of the properties most probably html id so that it will give them a stable interface to work with automation.

But I see this pattern of problem repeatedly. Sometimes it is with the generated html or at times with manually created ones where developers change if they wanted to name something better or change some object's property (May be refactor the html). I know that automating and concentrating a lot of testing through GUI is not a good choice although it is needed at some point. I know the reason for this that the GUI changes continuously. But I needed a better way of expressing this. So I thought of the good reason why we still go for GUI automation and why we fail on this?

Why do we go for GUI automation?

Evil lies in the availability of easy to use tools which entice us to create tests quickly by record and playback and create brittle test code. They also guarantee us to test the application end to end which is needed to test the complete integrated application.

But why do the code we write dependent on the GUI fails? (GUI Testing from the tools perspective)

One of the compelling answer I found was GUI is an human interface which people like you and me use and any other  interface is programmatic one to use your application as a service. Now programmatic interfaces, when published change less because any change to its contract will break the external programs which consume them. But a GUI interface being used by users change for two reasons.

1) If the html internal properties like id or class change then that doesn't affect the user directly.

2) If the appearance, location, text or description of the object changes then the user may not be able to immediately identify it but human beings have the ability to adapt to change, understand and act accordingly.

The problem comes when a GUI automation tool tries to use the application's GUI as a programmatic interface. They try to expect a constant interface contract (i.e. the properties of the objects and objects wouldn't change time after time) which can be used to consume your application GUI as a service. But this is not possible as GUI is not meant to be consumed by programs in the first place and when the GUI change, the contract between the automation tool and the application is broken. GUI testing tools also don't have the ability to understand and adapt unless they have a built in AI in them :).

So the better solution would be to concentrate your testing through a programmatic interface which can work with domain objects directly. This will help you to create a cleaner design of the application by making our GUI layer lean with no complex domain logic. As well the testing will be faster when we bypass the GUI layer. The test code will be more domain oriented as testing will concentrate more on the actual functionality of the application rather than on the GUI layer. Using tools like FitNesse, Concordion are suitable in this context.

But is testing through GUI not necessary?

The answer is we need. We need a set of tests which can run on GUI to tests it as well gives us feedback at end to end level. As well GUI needs to be tested when we have client side components like Ajax, Applets or others. The plus side of the contract enforcement of GUI testing tools needing a constant interface is a better naming of html objects like <HtmlInput id='txtUserName'> instead of absurd one like <HtmlInput id = 'txt0012345'>.

So this again brings up the concept to have a small set of tests which work on GUI using tools like Watir or selenium but concentrate most of the tests on the programmatic interface to test the domain layer directly.