saush

Processing Rails task in the background

Posted in Rails, Ruby by sausheong on October 22, 2006

Rails as a web application development framework is wonderful but running real-time processing with Rails (or any web application else for that matter) is not something really smart. For example, you need to send some email messages to a large number of recipients, you wouldn’t want to trigger the send and then wait for Rails to finish processing. Despite the nice Ajaxy things you can do, it’s still irritating to wait for 10 minutes Rails to finish its job.

What I used is probably one of most common application design patterns around — create a ‘job’ from the user interface and have a backend job scheduling mechanism to run it separately. In Java the most natural thing to do was to open up the magic open source hat and pull out Quartz(http://www.opensymphony.com/quartz/), the J2EE job scheduling framework. I will create a Quartz job and let Quartz take care of everything else.

But what can I use in Ruby on Rails? Searching through trusty ole Google found me a couple of possible solutions. One was rails_cron, managed by Kyle Maxwell which is a cron equivalent controlled by Rails. Another is BackgrounDRb, by Ezra Zygmuntowicz, a Rails plug-in that removes a processing task away from a normal HTTP request/response cycle. A third is to use Starfish by Lucas Carlson. Starfish, together with MapReduce (really, a ‘reduced’ functionality version of it) allows me to efficiently process large chunks of data using ActiveRecord but separately from Rails. So plenty of possibilities.

Of the few solutions, BackgrounDRb seems to be the most appropriate solution. Rails_cron is no longer maintained, Kyle decided to defer to BackgrounDRb, which is a more complete solution. I couldn’t get Starfish to work properly under Windows, and under Linux I faced problems with DRb and Rinda (Starfish is a wrapper around DRb and Rinda). In the end I didn’t go with BackgroundDRb either as it depends on DRb as well and I thought there isn’t such a need to make things so complex.

My alternative solution is to create a short Ruby script that retrieves all records through ActiveRecord and process them through a number of threads, then loop it over and over again. This can be created as a Windows Service using win32-service or as a *nix daemon using daemons.

This is the code snippet for the processing to be done:

require 'rubygems'
require 'active_record'
require 'app/models/message'

ActiveRecord::Base.establish_connection({
:adapter  => "mysql",
:host     => "localhost",
:username => "xxx",
:password => "xxx",
:database => "mydatabase"
})

# loop over and over again
loop do
messages = Message.find :all,
:conditions => ['sent = (?)', 0] # check if the message has been sent
:limit => 20 # retrieve and process 20 at a time

# array of threads
threads = []

# iterate through each message
for message in messages do
# start a new thread to process the message
threads <  e
# do some rescuing
puts "Failed: #{e.message}"
end
end
end

end

Creating a Windows service

For this I used Daniel Berger’s win32-service package. This is a great package that does literally everything for you to enable you to create a nice Windows service. Just install the gem like this:

gem install win32-service

I used examples/daemon_test.rb as the base template to make my Windows Service. This piece of code contains all that is needed to run the code snippet above as a service. In this file you will find a class called Daemon that has a number of methods necessary for running a Windows service. Put in the necessary requires and place your ActiveRecord initialization code at the beginning of the code. Then under service_main, while the status == RUNNING, put in the main processing code, minus the loop. Your code could possibly look something like this (don’t cut and paste this code, just use it as a reference):

require 'rubygems'
require 'logger'
require 'active_record'
require 'c:/myrailsapp/app/models/message' # remember to put in the absolute path here
require "win32/service"
include Win32

ActiveRecord::Base.establish_connection({
:adapter  => "mysql",
:host     => "localhost",
:username => "xxx",
:password => "xxx",
:database => "mydatabase"
})

# I start the service name with an 'A' so that it appears at the top
SERVICE_NAME = "MyProcess Service"
SERVICE_DISPLAYNAME = "MyProcess"

if ARGV[0] == "install"
svc = Service.new
svc.create_service{ |s|
s.service_name = SERVICE_NAME
s.display_name = SERVICE_DISPLAYNAME
s.binary_path_name = 'ruby ' + File.expand_path($0)
s.dependencies = []
}
svc.close
puts "installed"
elsif ARGV[0] == "start"
Service.start(SERVICE_NAME)
# do stuff before starting
puts "Ok, started"
elsif ARGV[0] == "stop"
Service.stop(SERVICE_NAME)
# do stuff before stopping
puts "Ok, stopped"
elsif ARGV[0] == "uninstall" || ARGV[0] == "delete"
begin
Service.stop(SERVICE_NAME)
rescue
end
Service.delete(SERVICE_NAME)
# do stuff before deleting
puts "deleted"
elsif ARGV[0] == "pause"
Service.pause(SERVICE_NAME)
# do stuff before pausing
puts "Ok, paused"
elsif ARGV[0] == "resume"
Service.resume(SERVICE_NAME)
# do stuff before resuming
puts "Ok, resumed"
else

if ENV["HOMEDRIVE"]!=nil
puts "No option provided.  You must provide an option.  Exiting..."
exit
end

## SERVICE BODY START
class Daemon
logger = Logger.new("c:/myprocess.log")

def service_stop
logger.info "Service stopped"
end

def service_pause
logger.info "Service paused"
end

def service_resume
logger.info "Service resumed"
end

def service_init
logger.info "Service initializing"
# some initialization code for your process
end

## worker function
def service_main
begin
while state == RUNNING || state == PAUSED
while state == RUNNING

# --- start processing code
messages = Message.find :all,
:conditions => ['sent = (?)', 0], # check if the message has been sent
:limit => 20 # retrieve and process 20 at a time

# array of threads
threads = []

# iterate through each message
for message in messages do
# start a new thread to process the message
threads <  e
# do some rescuing
puts "Failed: #{e.message}"
end
end
end

# --- end processing code

end
if state == PAUSED
# if you want do something when the process is paused
end
end
rescue StandardError, Interrupt => e
logger.error "Service error : #{e}"
end
end
end

d = Daemon.new
d.mainloop

end #if

Important to note that you need to put in the absolute path in the require as the service wouldn’t be starting at the Rails app.

Now you can install and start the Windows service (assuming the code is written in a file called ‘message_service.rb’:

c:/>ruby message_service.rb install
c:/>ruby messag_service.rb start

You can also control it from your Windows Services MMC console. What you have now is a Windows service that loops around until there is a message record in your database that is not sent (sent = 0). If there are, it will retrieve up to 20 messages at a go and process them with a thread each (parallelizing the processing to make it faster). Once it is processed, it will indicate the meesage has been sent (sent = 1) and loop again. Now you can happily create messages from your Rails app and stuff them into the database, while your message processor will process them separately.

About these ads

7 Responses

Subscribe to comments with RSS.

  1. 56838 Blog Verification said, on October 23, 2006 at 10:00 am

    56838 Blog Verification…

    56838…

  2. dan said, on December 8, 2006 at 2:47 pm

    just an fyi for all ya out there.. this DOES NOT work with ruby 1.8.5-21. It will work on ruby 1.84-19

  3. [...] http://blog.saush.com/?p=142 http://raa.ruby-lang.org/project/win32-service/ Ooh a native win 32 service. This timer service is quite spiffy where you actually install a win 32 service as a gem in ruby, and create a process which sleeps a certain amount of time and does some work. With some glue code, it’s able to access the active record objects. It works fine, but my problems are that I can’t access the ‘logger’ objects from active record. When I have a call into the container and I have ‘logger.debug( “id” ) it bombs because the logger isn’t in scope of the timer service. Weird weird weird. I e-mailed the guy in the blog and he said it can access a constant declared as a logger which doesn’t help me. BackgroundDRB [...]

  4. Daffy said, on June 7, 2007 at 9:05 pm

    Can I ask why this doesn’t work, and if there’s any fixes for this? I’m trying to accomplish a similar task to this using Ruby 1.8.6 and it doesn’t work in that either.

  5. sausheong said, on June 23, 2007 at 12:41 am

    I’m sorry, I don’t have the time to help in this, perhaps some other reader of this blog can help?

  6. [...] DrbServer Win32 service, but not my lucky day, the closest I found was an article by Sausheong on how to create a Win32 service on Rails. Picking up from there, I created some scripts that allows you to install/remove Ferret DrbServer [...]

  7. Nora Tafer said, on February 4, 2010 at 12:07 pm

    Good post – As an rough gemstone cutting service provider this is interesting.I’m happy:glad I found this


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 446 other followers

%d bloggers like this: