Fork me on GitHub

Friday, May 21, 2010

Continuous Integration - Stop lying to me

Here is the point. If your Continuous Integration build isn't validating the system as it will be deployed in production from end user perspective, it is not telling you the truth.

If the continuous integration build is green but your packaging or deployment is broken, it is still lying.
If the continuous integration build is green but your production configuration has broken, it is still lying.
If the continuous integration build is green but you find 20 days later after deployment that the assumption you made about session has started creating problems with load balance, it has lied big time to you.
If the continuous integration build is green but if the functional tests run against a deployment optimized to make it work and different from production, it is still lying.

The above points illustrate that a lot of teams consider running unit tests and functional tests on a utopian or development environment mirrored build environment. This is equivalent to running the tests before you check in from your pairing station. The result is not completely useless. CI is still telling you the valuable information that the code you wrote has not broken anything.

But if you don't have anything in the CI build which validates the code and assumptions you have made, against the production mirroring environment from the point of view of end user we will be missing very important piece of information. It basically means even though the CI build is green the team only has a false confidence that the system they are building will work as intended from the end users point of view (Both from functional point as well as deployment point).

CI build should be end to end. They should run unit tests as a developer does. Should deploy to a production mirroring environment which validates packaging, deployment scripts and configuration and run functional and load tests against that so that we get a realistic understanding.

I also understand this is a challenging activity. But unfortunately on the other hand the price we pay for not having this infrastructure in place is actually heavy as we won't have our assumptions validated in time and need to resort to hacky methods when we come to understand that the decisions we made don't work in the real world as well as we expected.

oh.. And important point. When you check in code on the build server side don't update but always recreate the code from scratch. That way it is easy to find if we have tests passing because of stale assemblies lying around even if we have accidentally deleted them :)

What is the ultimate nirvana of this approach? Continuous Deployment. The best way to understand that things work as we have expected to put deploy them to production. And if we make it a non event it is the best kind of test we can ever have. Continuous deployment is a much bigger subject and may need another blog post for discussion.

Let me know your thoughts and experience about this.

1 comment:

Lewis said...

I couldn't agree more on your lead in to continuous deployments - something I've been blogging on about for a while. I think that the work of DevOps should be shared beyond a single CI build or company. Any thoughts on Soft-service distro's?