Fork me on GitHub

Thursday, December 3, 2009

Eventlet based CouchDB connector

This is an attempt to write a nonblocking CouchDB API based on Eventlet. Eventlet is a wonderful python library written by Second Life people based on coroutine based nonblocking I/O. I also tried using AsyncHTTPClient provided by Tornado framework released by FriendFeed people but liked the eventlet version more. So here goes... This API may grow to be a full fledged one in future.

Also look out a Ruby non blocking one based on NeverBlock :)


Denis said...

How what you wrote is different from simple:

print urllib2.urlopen(resource_url).read()


(it isn't)

Sai Venkatakrishnan said...

yeah.. if you wanna evolve that into an API then what I stated is simple. If you just wanna make a blocking call then what you wrote is simple. My aim is to do a non blocking call...

Denis said...

if you imported urllib2 from (or monkey patched socket) then it's already a non-blocking call.

It behaves like blocking but it does not block other greenlets, so you can spawn as many as you like and they all be concurrent:

def job(url):
print urllib2.urlopen(url).read()

spawn(job, url1)
spawn(job, url2)
spawn(job, url3)

If what you want is async call, that is your callback called when request is complete, do smth like:

def job(url, callback):
result = urllib2.urlopen(url).read()
callback(url, result)

def on_job_done(url, callback):

spawn(job, url1, on_job_done)
spawn(job, url2, on_job_done)
spawn(job, url3, on_job_done)

If you used gevent, you could use its Pool class which implements apply and apply_async functions that do what want.

from gevent.pool import Pool

def job(url):
return urllib2.urlopen(url).read()

def on_job_done(result)

p = Pool() # you can pass optional concurrency limit here

p.apply_async(job, (url1, ), callback=on_job_done)
p.apply_async(job, (url2, ), callback=on_job_done)
p.apply_async(job, (url3, ), callback=on_job_done)

Denis said...

Check out gevent examples.

Sai Venkat said...

Cool.. Thanks for the examples. This may simplify (as in really simplify :) a lot of stuff.