Fork me on GitHub

Friday, May 21, 2010

Continuous Integration - Stop lying to me

Here is the point. If your Continuous Integration build isn't validating the system as it will be deployed in production from end user perspective, it is not telling you the truth.

If the continuous integration build is green but your packaging or deployment is broken, it is still lying.
If the continuous integration build is green but your production configuration has broken, it is still lying.
If the continuous integration build is green but you find 20 days later after deployment that the assumption you made about session has started creating problems with load balance, it has lied big time to you.
If the continuous integration build is green but if the functional tests run against a deployment optimized to make it work and different from production, it is still lying.

The above points illustrate that a lot of teams consider running unit tests and functional tests on a utopian or development environment mirrored build environment. This is equivalent to running the tests before you check in from your pairing station. The result is not completely useless. CI is still telling you the valuable information that the code you wrote has not broken anything.

But if you don't have anything in the CI build which validates the code and assumptions you have made, against the production mirroring environment from the point of view of end user we will be missing very important piece of information. It basically means even though the CI build is green the team only has a false confidence that the system they are building will work as intended from the end users point of view (Both from functional point as well as deployment point).

CI build should be end to end. They should run unit tests as a developer does. Should deploy to a production mirroring environment which validates packaging, deployment scripts and configuration and run functional and load tests against that so that we get a realistic understanding.

I also understand this is a challenging activity. But unfortunately on the other hand the price we pay for not having this infrastructure in place is actually heavy as we won't have our assumptions validated in time and need to resort to hacky methods when we come to understand that the decisions we made don't work in the real world as well as we expected.

oh.. And important point. When you check in code on the build server side don't update but always recreate the code from scratch. That way it is easy to find if we have tests passing because of stale assemblies lying around even if we have accidentally deleted them :)

What is the ultimate nirvana of this approach? Continuous Deployment. The best way to understand that things work as we have expected to put deploy them to production. And if we make it a non event it is the best kind of test we can ever have. Continuous deployment is a much bigger subject and may need another blog post for discussion.

Let me know your thoughts and experience about this.

Wednesday, May 12, 2010

Dang!! I should have looked before I lept

Old saying but mostly true (Mostly not always.. :). Let me be first clear up my opinion about Selenium. It may have been successful in a lot of project, it is now backbone for companies like Saucelab, it may be one of the first successful open source project. But still in my opinion Selenium is one big hack. I mean both its api and internals... (No flame wars.. I mean nothing wrong. Just saying it is not soundly structured). And when ever I see problems with selenium (as in my current project) my thought process goes like this... How can I make this better and the first thing come to mind is let's replace with Webdriver. Unfortunately my current project is already running for long time before I joined so no chance of Watir in it :(.

The reason I am writing this is to share my experience of this journey I undertook to replace a test code base which relied on Selenium with Webdriver. Webdriver in my opinion is one of the sound code bases I have seen equivalent to Watir. They have nice design, good abstractions, solid internals and it is fun to work with the API. Coming to my current code base... It follows the pattern of lots of test code base I have seen before.. Abstraction from application using page objects but none to the underlying driver. This may be because most of the time it is not natural to change the test driver in a project. But here we are with the need.. To make functional tests stabler and faster. Without abstraction with underlying driver the driver code would have leaked through out the test framework. Lesson - Remember test or production, properly abstract from third party stuff.

I started with building a adapter layer (Webdriver backed Selenium) between test cdeand Webdriver as we wanted to keep the test code constant. Things went pretty smooth for simple stuff. Lots of strategies, webdriver functions are simpler. But then I started hitting some road blocks. Most of this is not because Webdriver is lacking but because Webdriver and Selenium are two different things. Sometimes there is no direct equivalent. Or we have to bend a lot to find an equivalent. Also the reference implementation I was using was in Java which again posed some problems as I had to build a similar functionality in C#. Although we can jokingly say that change all the starting lower case in java to upper case will make it C# in a lot of cases there is not direct equivalent again... Lots of reading, lots of research and making mistakes and correcting them are the way to solution. (Lesson - It will not be just an adapter as we say when we start....)

Also another problem and a major one I hit was CSS selector. There is no css selector implementation for InternetExplorer in Webdriver. I probably may have missed out the documentation somewhere but looking at the code base we had css selectors everywhere (Lesson - Research your current landscape before jumping into a solution). I don't have a direct solution in had but as we needed this I used Ian Bicking's Css_to_Xpath python code called from C# using DLR (Because I had used this css_to_xpath before and found it to be reliable. Didn't want to pull some sample C# code from internet and patch things up). Not a great solution but it seems tow work fine till now except for really complex css selectors (I will rant on css and xpath later). So had to hunt down and convert some css selectors to xpath.

I had learnt lot more lessons doing this conversion than I am putting here. Good news is I have convinced people to open source this WebDriverBacked *adapter* so other's can use it directly :)

Also it is important to remember that at times we should just say "screw it.. I am gonna do this". Just take the risk and work with it. I am in the final phase of this work and things seem to go relatively smooth. Now that I am back on the testing side for sometime I will write a lot about tests. Next one coming will probably be about Css selector (A spicy one :)..

I will update this post with the repo url once the adapter has been open sourced. Please let me know your thoughts with comments.