Fork me on GitHub

Sunday, June 27, 2010

Introducing Multiverse Software Transactional Memory for JRuby

I haven't written for quite sometime. I have been busy working on one of areas I really love, Concurrency. People who have followed me for sometime in twitter or heard my talk in Rubyconf about Concurrency Paradigms with Hari would have understood this. Recently I have been exploring alternate forms of concurrency other than the normal locks, mutex and conditional variable ones.

I love the message passing concurrency for its simplicity in removing the concept of shared memory (Still doesn't mean we will not have state dependence :). But there are at times when we need shared state and safe access to it. This problem of strategy for accessing a shared state among multiple threads of execution has been already solved in database world. Transactions gives a really good answer for this. Currently I am working on using Multiverse STM in JRuby (As earlier I had used Clojure). This time it is a complete Software Transactional Memory with support for atomic (transactional), retry (blocking) and orElse (choice) primitives. Multiverse is the transactional Memory on JVM and is the one behind Akka's STM. It uses TL2 algorithms (which has global version clock and commit time locking) as well as flexible to accommodate other algorithms.

Check out the code in multiverse git repo git://git.codehaus.org/multiverse.git. There is a nice implementation of Dijkstra's Dining Philosopher's problem in it using atomic and retry...

Let me know your thoughts. Look out for more posts on the internals of the STM :)

Friday, May 21, 2010

Continuous Integration - Stop lying to me

Here is the point. If your Continuous Integration build isn't validating the system as it will be deployed in production from end user perspective, it is not telling you the truth.

If the continuous integration build is green but your packaging or deployment is broken, it is still lying.
If the continuous integration build is green but your production configuration has broken, it is still lying.
If the continuous integration build is green but you find 20 days later after deployment that the assumption you made about session has started creating problems with load balance, it has lied big time to you.
If the continuous integration build is green but if the functional tests run against a deployment optimized to make it work and different from production, it is still lying.

The above points illustrate that a lot of teams consider running unit tests and functional tests on a utopian or development environment mirrored build environment. This is equivalent to running the tests before you check in from your pairing station. The result is not completely useless. CI is still telling you the valuable information that the code you wrote has not broken anything.

But if you don't have anything in the CI build which validates the code and assumptions you have made, against the production mirroring environment from the point of view of end user we will be missing very important piece of information. It basically means even though the CI build is green the team only has a false confidence that the system they are building will work as intended from the end users point of view (Both from functional point as well as deployment point).

CI build should be end to end. They should run unit tests as a developer does. Should deploy to a production mirroring environment which validates packaging, deployment scripts and configuration and run functional and load tests against that so that we get a realistic understanding.

I also understand this is a challenging activity. But unfortunately on the other hand the price we pay for not having this infrastructure in place is actually heavy as we won't have our assumptions validated in time and need to resort to hacky methods when we come to understand that the decisions we made don't work in the real world as well as we expected.

oh.. And important point. When you check in code on the build server side don't update but always recreate the code from scratch. That way it is easy to find if we have tests passing because of stale assemblies lying around even if we have accidentally deleted them :)

What is the ultimate nirvana of this approach? Continuous Deployment. The best way to understand that things work as we have expected to put deploy them to production. And if we make it a non event it is the best kind of test we can ever have. Continuous deployment is a much bigger subject and may need another blog post for discussion.

Let me know your thoughts and experience about this.

Wednesday, May 12, 2010

Dang!! I should have looked before I lept

Old saying but mostly true (Mostly not always.. :). Let me be first clear up my opinion about Selenium. It may have been successful in a lot of project, it is now backbone for companies like Saucelab, it may be one of the first successful open source project. But still in my opinion Selenium is one big hack. I mean both its api and internals... (No flame wars.. I mean nothing wrong. Just saying it is not soundly structured). And when ever I see problems with selenium (as in my current project) my thought process goes like this... How can I make this better and the first thing come to mind is let's replace with Webdriver. Unfortunately my current project is already running for long time before I joined so no chance of Watir in it :(.

The reason I am writing this is to share my experience of this journey I undertook to replace a test code base which relied on Selenium with Webdriver. Webdriver in my opinion is one of the sound code bases I have seen equivalent to Watir. They have nice design, good abstractions, solid internals and it is fun to work with the API. Coming to my current code base... It follows the pattern of lots of test code base I have seen before.. Abstraction from application using page objects but none to the underlying driver. This may be because most of the time it is not natural to change the test driver in a project. But here we are with the need.. To make functional tests stabler and faster. Without abstraction with underlying driver the driver code would have leaked through out the test framework. Lesson - Remember test or production, properly abstract from third party stuff.

I started with building a adapter layer (Webdriver backed Selenium) between test cdeand Webdriver as we wanted to keep the test code constant. Things went pretty smooth for simple stuff. Lots of strategies, webdriver functions are simpler. But then I started hitting some road blocks. Most of this is not because Webdriver is lacking but because Webdriver and Selenium are two different things. Sometimes there is no direct equivalent. Or we have to bend a lot to find an equivalent. Also the reference implementation I was using was in Java which again posed some problems as I had to build a similar functionality in C#. Although we can jokingly say that change all the starting lower case in java to upper case will make it C# in a lot of cases there is not direct equivalent again... Lots of reading, lots of research and making mistakes and correcting them are the way to solution. (Lesson - It will not be just an adapter as we say when we start....)

Also another problem and a major one I hit was CSS selector. There is no css selector implementation for InternetExplorer in Webdriver. I probably may have missed out the documentation somewhere but looking at the code base we had css selectors everywhere (Lesson - Research your current landscape before jumping into a solution). I don't have a direct solution in had but as we needed this I used Ian Bicking's Css_to_Xpath python code called from C# using DLR (Because I had used this css_to_xpath before and found it to be reliable. Didn't want to pull some sample C# code from internet and patch things up). Not a great solution but it seems tow work fine till now except for really complex css selectors (I will rant on css and xpath later). So had to hunt down and convert some css selectors to xpath.

I had learnt lot more lessons doing this conversion than I am putting here. Good news is I have convinced people to open source this WebDriverBacked *adapter* so other's can use it directly :)

Also it is important to remember that at times we should just say "screw it.. I am gonna do this". Just take the risk and work with it. I am in the final phase of this work and things seem to go relatively smooth. Now that I am back on the testing side for sometime I will write a lot about tests. Next one coming will probably be about Css selector (A spicy one :)..

I will update this post with the repo url once the adapter has been open sourced. Please let me know your thoughts with comments.

Tuesday, February 23, 2010

Simple Reactive Programming in Ruby

Reactive Programming is an interesting programming paradigm based on propogation of changes, data flows. A programming model based on side effects, I think this is especially useful when we need to control the state of one entity based on what goes on in outside itself. For example take the equation


x = 1; y = 3
z = x + y
puts z #<= z = 4


Since z depends on x and y, what will happen when we update value of x or y. z remains same as 4 because z is already computed and stored using the values of x and y. Now if the values of x and y changes z will still remain the same.

But what if you want z to get adjusted depending upon x and y. This is more common problem in everyday programming than we think. Imagine you have to update the GUI based on what is changing in underlying model (MVC?) or take an example of spreadsheet where changing a value in a cell recalculates some other cell based on a formula.

Here is a very simple example of Reactive Programming in Ruby using EventTarget.
Here in this example value of z is x + 2. We evaluate z by setting an initial value for x and then later we change value of x and observe the value of z changed as well.


require 'EventTarget'

module Rx
class Change
attr_accessor :old, :new
end
class ObservableValue
include EventTarget
def initialize(value, on_change)
add_event_listener(:change, &on_change)
self.value = value
end
def value=( v )
change = Change.new
change.old = @value
change.new = v
@value = v
evt = Event.new( :change, change )
dispatch_event( evt )
end
end
end

if $0 == __FILE__
z = 0
on_change = Proc.new {|e| z = e.change.new + 2}
x = Rx::ObservableValue.new(3, on_change)
puts "Initial value of z is #{z}" #<= z=5
x.value = 5
puts "Current value of z is #{z}" # <= z=7
end


This is a very simple and contrived example of Reactive programming but I hope this illustrates the point. The concept is way more powerful and is (Functional Reactive Programming) available in a lot of functional languages like Haskell. Take a look at Reactive Framework in .Net and Javascript, Traits, Trellis in Python or Cells in Common Lisp for more understanding of how Reactive Programming works.

Let me know your thoughts and happy hacking :)

Friday, February 19, 2010

Software Transactional Memory in JRuby - Redux with Cloby

After I wrote about using Clojure's Software Transactional Memory infrastructure in JRuby in my previous post. @headius has written an awesome wrapper Cloby in Java to use Clojure STM in JRuby and writing Ruby objects which can be used as Clojure Refs.

Here is an example code of using a Ruby object in STM.


require 'clojure'

class MyClojureObj < Clojure::Object
def initialize
dosync { @foo = 'foo' }
end

attr_accessor :foo
end

obj = MyClojureObj.new
puts "obj.foo = " + obj.foo

begin
puts "Setting obj.foo to 'bar'"
obj.foo = 'bar'
rescue ConcurrencyError
puts "Oops, need a transaction"
end

puts "Trying again with a transaction"
dosync { obj.foo = 'bar' }
puts "Success"

puts "obj.foo = " + obj.foo


Some pretty good abstractions of using dosync and Refs in STM. So how do we use it?
Clone the code from http://github.com/headius/cloby/blob/. Build ClojureLibrary.jar with clojure and jruby.jar. Keep ClojureLibrary.jar and clojure.jar in classpath.
And here is the bridge I wrote between your awesome code using STM and ClojureLibrary.jar


#!/usr/bin/env jruby

require "java"
require "ClojureLibrary.jar"
require "clojure-1.0.0.jar"
include_class "org.jruby.clojure.ClojureLibrary"

clj_lib = ClojureLibrary.new
clj_lib.load(JRuby.runtime, true)


Now its pretty easy to use Clojure's STM. No raw usage of LockingTransactions or Refs. Enjoy Cloby... :)

Thursday, February 18, 2010

#Concurrency - Software Transactional Memory in JRuby with Clojure

Still playing with JRuby and concurrency... For me JRuby is one of the best implementations of Ruby VM. It has this amazing property of taking the most boring Java projects and turning them into something exciting. Not that clojure is boring but on the other hand it is one of the most wonderful languages I have worked with. Being a big fan of Lisp Clojure is a dream come true.

This post is about my exploration with JRuby and Clojure more in terms of how to use the Clojure's STM infrastructure. Whenever I find an interesting Java library the first thing I do is to import it into JRuby and play with it for sometime. After working with Clojure I wanted to use its STM in Ruby and finally found time to do it. The code is really crude as it explores very basic functionality of STM. May be as I keep working I will try to build something around this and who knows it can end up as a library for JRuby :)...

Instead of defining STMs myself - Here it is from Wikipedia..
"In computer science, software transactional memory (STM) is a concurrency control mechanism analogous to database transactions for controlling access to shared memory in concurrent computing. It functions as an alternative to lock-based synchronization. A transaction in this context is a piece of code that executes a series of reads and writes to shared memory. These reads and writes logically occur at a single instant in time; intermediate states are not visible to other (successful) transactions."

As most of you know the problem is with sharing the state.. In this example my_account balance. Imagine when two different threads of execution trying to update the value at the same time. Disastrous results may ensue. So the typical approach is locking of the shared resource. Problem with locking is it effectively limits concurrency as well is too hard to get it right. So we have started looking for other models of concurrency. Actors, Dataflow and STMs are some of the popular form. As you see in the example let's say there is a shared resource. If we need to update it we can't update it in a separate context. We need to run this as a part of transaction and as database transactions it is all or nothing.

In Clojure's term a ref can be updated only inside the context of a transaction. Else it will throw IllegalStateException. As well the ref getting updated will not be visible to outside work till the end of the transaction. The transactions are composable which is difficult in cases of threads and locks. Also as the example shows STM is a form of optimistic concurrency. Everyone can update the shared resource but whoever finishes first and commits wins. Others fail because the state now is different from what they started with and they can try.





The code is almost self explanatory and simple. Using clojure STM is pretty simple and straightforward. And this will help us to effectively manage concurrent access of shared resource by multiple threads.

Let me know your thoughts...

Tuesday, February 16, 2010

JRuby + Jetlang => Awesome Message Passing Concurrency

It has been quiet sometime since I played around with JRuby.. The last I seriously worked on JRuby was during schnell-jruby age :). And now that celerity has taken the main stage I was back in MRI land for a long time. I recently got a chance to return to JRuby land and play with it for sometime.

Interested in concurrency I tried to work on Software Transactional Memory in JRuby with Akka... But as things didn't go great I was searching for something else interesting to work on and I found Jetlang. Jetlang has been in my interest list of sometime but I never got a chance to work with Java.

Jetlang is based on message passing concurrency. It has the concept of fibers which are independent threads of execution and channels which are conduits attached to fibers. We can pass messages to a fiber through channels and when a message is passed into a channel to a fiber the on receive callback is called with the corresponding parameters..

Following are two examples illustrating the use of fibers and channels for message passing concurrency. The last one is the popular ping pong example used for illustrating message passing across fibers :).








Writing code in Erlang and using Jetlang based message passing concurrency has made me think that... Can Actors (and Message passing fibers) be considered as ideal representations of objects? Isn't this how we want object oriented systems to behave?
Let me know your thoughts...

Monday, January 18, 2010

Are we stuck in a Bratty Programming Model?


This is more of an essay or more of my thoughts than a blog post. So brace yourself to read some lengthy writing :).

Warning - Thoughts are evolving and so may not be complete as you think. If you can help it to move in the right direction, I welcome you.

Ever had your day spoiled by a kid who demands what he needs and doesn't budge until you give it to him (cries, pleads, rolls on the floor and on extreme cases punches and kicks :). The problem with him is he won't let you do anything else till he gets what he wants. But how does it relate to programming? I think most of our programs behave in the same way as the bratty kid.

Let's take a problem - Say we are working on some ticket reservation app. The need is - Given a ticket id, get the reservation info, do some processing on that and then go on to do some other important stuff.

Expressing the above spec in ruby:


reservation = remote_service.get_reservation_info(ticket_id)
process_ticket(reservation)
#Program wishes to continue to do some other important stuff.


The above code is simple and we have all written this code at some point in time. But there seems to be a problem with the piece of code. It demands for some information from a remote service and doesn't move from there until it gets what it demands. Also it doesn't let rest of the code to execute as well until its demands are met. It is not just for service calls.. Think of using the same mode for network calls, database access, file system operations, long running computations. All of them can become bratty and stop your program from moving forward.

Are programs supposed to behave like this? Is this a better method of writing code? For simple code of not much serious use, the above model is acceptable. But when we start moving to systems which need to grow in scale this programming model is not sustainable.

Programs are supposed to be magnanimous in nature. A line of code should behave in a way that if it is not able to get something for moving forward, should move out of the line to let others execute and wait for its input. How would that actually work out in code?

Now lets jump into code which can give a hint of how to solve this. Below is the same ruby code written in a different way.


remote_service.get_reservation_info(ticket_id).addCallback {|reservation| process_ticket(reservation)} # I will get the input when it is available and process it
# other lines of code executing when above code waits for reservation info...


So the above lines of code does the same as above semantically. But the difference between the two is in this piece of code the retrieving of reservation information and processing it doesn't seem to block the program from moving forward. I know just adding a callback doesn't solve the problem. There must be an underlying infrastructure to handle this. And this is where event loops come in. But I think you get the idea of what I am trying to say.

Event loops are a bigger subject of discussion. But we have been using them most of the times. Take GUI programming as example.

The I/O or more importantly programing models we have been using till now have predominantly been focusing being imperative (Do this step, get the result, then do that and continue...) but I think it's time we understand that this model can't help us long. If we need our programs to scale or even be reasonable we need to move from bratty to a magnanimous programming model. Blocking other's from moving because we still need something is not a very scalable model both in real life and programming :).

This is my rant of current programming models. There may be better ways to do things and if you know any please let me know. A lot of this essay is inspired by node.js work done by Ryan Dhal and EventMachine which made non bratty programming so much easier for me to handle.

P.S. Using event loops has a different set of constraints like new way of thinking (inverted control), different way of exception handling as well using different frameworks which we may not be familiar with.

Saturday, January 16, 2010

DNS lookup Experiment in Ruby - Blocking and NonBlocking Modes

Normal DNS lookup in Ruby (Blocking Mode)

require "resolv"

hosts = %w[www.yahoo.com www.google.com twitter.com github.com]

start = Time.now

hosts.each do |host|
p "Getting address for #{host}"
p Resolv.getaddress(host)
end

finish = Time.now

p "Time take to finish #{finish - start} seconds"


Execution Result -

"Getting address for www.yahoo.com"
"69.147.76.15"
"Getting address for www.google.com"
"209.85.231.104"
"Getting address for twitter.com"
"168.143.171.84"
"Getting address for github.com"
"207.97.227.239"
"Time take to finish 40.313232 seconds"

Non Blocking DNS Lookup in Ruby (Using EventMachine Deferrable)

require "rubygems"
require "resolv"
require "eventmachine"

class Dns
include EM::Deferrable
def resolve_hostname(hostname)
begin
ip = Resolv.getaddress(hostname)
set_deferred_status :succeeded, ip
rescue Exception => ex
set_deferred_status :failed, ex.to_s
end
end
end

start = Time.now
EventMachine.run {
p "Requesting DNS info for yahoo"
dns0 = Dns.new
dns0.callback {|response| p "For yahoo #{response}"}
dns0.errback {|response| p "For yahoo #{response}"}
Thread.new { dns0.resolve_hostname "www.yahoo.com"}

p "Requesting DNS info for google"
dns1 = Dns.new
dns1.callback {|response| p "For google #{response}"}
dns1.errback {|response| p "For google #{response}"}
Thread.new { dns1.resolve_hostname "www.google.com"}

p "Requesting DNS info for twitter"
dns2 = Dns.new
dns2.callback {|response| p "For twitter #{response}"}
dns2.errback {|response| p "For twitter #{response}"}
Thread.new { dns2.resolve_hostname "twitter.com"}

p "Requesting DNS info for github"
dns3 = Dns.new
dns3.callback {|response| p "For github #{response}"}
dns3.errback {|response| p "For github #{response}"}
Thread.new {dns3.resolve_hostname "github.com"; EM.stop }
}
finish = Time.now

p "Time take for querying #{finish - start} seconds"



Execution Result -

"Requesting DNS info for yahoo"
"Requesting DNS info for google"
"Requesting DNS info for twitter"
"Requesting DNS info for github"
"For google 209.85.231.99"
"For yahoo 69.147.76.15"
"For github 207.97.227.239"
"For twitter 128.121.146.100"
"Time take for querying 20.311271 seconds"

The results are obvious :). You can see the difference between the blocking and non blocking modes and the effect they may have on your systems. I am also planning to write a DNS library based on NeverBlock which uses Fibers as a way to do non blocking concurrency.

EventMachine is an excellent piece of software which gives a lot of functionality to do things asynchronously. Especially the Deferrable pattern is an awesome way to do long running jobs or calculations and still not block the whole world from running. I have been exploring EventMachine for sometime since I started writing em-couchdb.

The code is written in a simplified way and may not be of direct use to production but hope it reflects the principles of value of doing things in non blocking way.

Wednesday, January 13, 2010

ChromeWatir + Watir-WebDriver Update [Important]

Interesting development in Watir world but not a completely unanticipated one. I started playing around with using WebDriver as platform for Watir in the form of FireDriver and ChromeWatir.

Yesterday Jarib has released a Watir abstraction on top of WebDriver. I am planning to move the effort of working on ChromeDriver to Watir-WebDriver which has ChromeDriver in it.

Installation of Watir-WebDriver

> sudo gem install selenium-webdriver
> git clone git://github.com/saivenkat/watir-webdriver.git

Example code in watir-webdriver (ChromeDriver)

require "watir-webdriver"
browser = Watir::Browser.new(:chrome)
browser.goto "http://www.google.com"
browser.text_field(:name, "q").set "Watir"
browser.button(:name, "btnG").click

The code is same as Watir except the driver changes internally.
The repo is in http://github.com/saivenkat/watir-webdriver. Feel free to fork and enjoy...

Tuesday, January 12, 2010

What is your favorite code metric?

You don't need to run any build or external tool to get this. It is always there when you code and changes actively as your code evolves. What would it be?

My most favorite is obviously "Line Numbers".

Line numbers are available in all IDEs but mostly not actively used as a code metric during coding. Switch them on and notice them actively when you code, they tend to have an interesting effect. Probably the classes and functions will be shorter, less bloated and concise. Based on personal experience :)

What is your simple metric to improve code?