saush

After trying out Sinatra as the interface to my search engine, I got hooked to it. I liked the no-frills approach to developing a web application. I liked it so much that I decided to write a fuller web app using Sinatra. Of course I had no idea what to write, so I decided to clone a random popular application. That was how I ended up writing a Twitter clone.

It was inevitable that halfway writing the Chirp-Chirp, my Sinatra-based Twitter clone, I found out that there are already quite a few Twitter clones out there, 262 to be specific, at least as of August 2008.

The design goals for Chirp-Chirp are very simple. I want to build a reasonably feature complete Twitter clone in the simplest way possible. The code should be minimal and easily understood. The user interface should be usable without additional extensive instructions. Also, the deployment should be straightforward.

To begin with, let’s look at a Twitter clone. Fundamentally it’s all about telling people what you’re doing with minimal text entry, in real-time. The people are friends who ‘follow’ you (naturally, called ‘followers’). The text that you type is  short, which is why such apps are also known as micro-blogs.

Let’s look at the various components of Chirp-Chirp:

• User interface
• Login and user management
• Data modeling
• Home page
• Friends and followers
• Public and direct messages

User interface
Simply put I just copied Twitter‘s user interface and slapped on a self-drawn logo.

Login and user management
I don’t really want to write a login module, nor manage users who log in to Chirp-Chirp but I certainly need one. So my solution is to ‘borrow’ someone else’s login module and kept minimal user management. The first thing I looked into is OpenID. Using OpenID is relatively simple but it still involved some code integration using an OpenID library.

Finally I settled on using RPX to handle the login, and doing minimal integration to single sign-on with Yahoo!, Google and Windows Live ID, 3 of the biggest players on the Internet scene. I figured almost everyone would have at least 1 of these 3 accounts. The advantage in using RPX is that there is really little integration needed, and if the user is already logged into either one of the 3 acounts, he doesn’t need to log in again. There is no user registration, choosing password, taking care of security and all that stuff. Just a simple click on the account the user wants to use, and he’s in.

User management is inevitable since I need to keep track on who write what messages. As the unique user ID, I use the email that is returned by the provider. OpenID (which all 3 of the providers support) returns me a unique identifier but that is comparatively difficult to type and remember. Email is a good compromise as it is something easily remembered and is unique.

Data modeling
I used DataMapper again for data modeling. The data model consists of 3 classes — User, Relationship and Chirp. Simply put — a User has Relationships with other Users, and a User has many Chirps (corny but, hey :)). A Chirp with a specific recipient User is a direct message.

Home page
All chirps sent out by the user and his friends are listed in his home page in a reverse timeline order (i.e. the latest chirp is listed at the top). Direct messages received are only shown in the direct message inbox and the direct messages sent are only shown in the sent box.

Friends and followers
The concept of friends and followers is a one way relationship. A friend is someone you follow, while a follower is someone following you. A user can be a friend or a follower or both. Following does not require the permission of the user. For example, you can follow anyone in the system without the explicit approval of that user.

Public messages (chirps) and direct messages
A chirp is a public message that is sent by a user that is viewable by the user and all his followers. Chirps belong to a user. Chirps containing the user ID (email) starting with ‘@’ will be hyperlinked to the user’s home page. 2 other features involving the message is direct messaging and following.

You can send direct message to specific users, who will receive them in their direct message inbox. To send a direct message start the chirp with ‘dm’, followed by the user ID. To follow a user, start the chirp with ‘follow’, followed by the user ID.

With this in mind let’s jump into the code. Here’s the stuff that I used:

• Sinatra (web framework)
• DataMapper (data modeling)
• RackFlash (flash for Rack-based frameworks)
• ERB (view template)

Without looking the at view templates, the total number of lines of code is around 200 or so. I have only 2 non-view template files — chirp.rb is the web application while models.rb contains the 3 DataMapper data models. As we go along I’ll be extracting code fragments from these 2 files to explain in details.

Let’s talk about chirp.rb first. The first line after the requires tells Sinatra that I want to use sessions. This is followed by telling Sinatra to use the Rack::Flash library to use session-based flash. What you see next are a bunch of get and post blocks. Let’s look at the first one.

['/', '/home'].each do |path|
get path do
if session[:userid].nil? then
erb :login
else
redirect "/#{User.get(session[:userid]).email}"
end
end
end

You will notice that I wrap the get block with an array block, and the array contains 2 strings. These 2 strings are the routes that are associated with the subsequent get block. In the code above, we associate the ‘/’ and ‘/home’ routes to the get block. The get block here checks if the current session contains a user ID. If it doesn’t, it will redirect the user to the login view template. Otherwise it will get the user with this ID and redirect it to the route that contains the email. For example, if the user’s email is user@yahoo.com, then the route is ‘/user@yahoo.com’.

We’re going to look at the login module next. As mentioned, I used RPX (to be specific, I use RPX Basic, which is free!) to do single sign-on so to begin, go to RPXNow and register a web site. Once you do that, you will get an API key and that is really the only thing you need. For Windows Live ID, you’ll need to do a few more steps to register a Windows Live project but you can just follow the steps provided.

Now let’s look at the login template to understand more about the RPX single-sign on mechanism. You will notice that this is entirely HTML, and it contains 3 forms. The URLs for the single sign-on for Google and Yahoo! are OpenID based (which is not surprising as RPX is run by JanRain, one of the main supporters for OpenID). Each form has only 1 input, which is an image button. The user clicks on the account he wants to log in with, and following the given instructions, will be authenticated by the provider and redirected back to our web app. If the login is correct, our web app will receive some data on the user (the amount of data we get is different with different providers, and also configurable from RPX). I won’t go into the actual mechanism, the RPX documentation is pretty detailed.

Once RPX authenticates the user, it will redirect him to our web app along with a token. For Chirp-Chirp this goes back to ‘/login’. The ‘/login’ block then retrieves this token and uses it to retrieve the user information from RPX.

get '/login' do
openid_user = get_user(params[:token])
user = User.find(openid_user[:identifier])
user.update_attributes({:nickname => openid_user[:nickname], :email => openid_user[:email], :photo_url => "http://www.gravatar.com/avatar/#{Digest::MD5.hexdigest(openid_user[:email])}"}) if user.new_record?
session[:userid] = user.id # keep what is stored small
redirect "/#{user.email}"
end

For Chirp-Chirp, I store the unique identifier from RPX, the user’s nickname and his email. For his avatar picture, I use Gravatar which neatly also uses email address as a unique identifier although we need to use MD5 to hash it first. If this is the first time the user is logging in, I save the user information in the database. Finally I store the user’s serial ID (not the unique identifier) in the session and redirect the user to the user home page.

This is the code fragment that retrieves the user information from RPX.

def get_user(token)
u = URI.parse('https://rpxnow.com/api/v2/auth_info')
req = Net::HTTP::Post.new(u.path)
req.set_form_data({'token' => token, 'apiKey' => '<insert RPX API KEY HERE>', 'format' => 'json', 'extended' => 'true'})
http = Net::HTTP.new(u.host,u.port)
http.use_ssl = true if u.scheme == 'https'
json = JSON.parse(http.request(req).body)

if json['stat'] == 'ok'
identifier = json['profile']['identifier']
nickname = json['profile']['preferredUsername']
nickname = json['profile']['displayName'] if nickname.nil?
email = json['profile']['email']
{:identifier => identifier, :nickname => nickname, :email => email}
else
raise LoginFailedError, 'Cannot log in. Try another account!'
end
end

Let’s look at the home page next. I take the example of a user with the email user@yahoo.com, so the home page for him would be ‘/user@yahoo.com’. This is the code fragment with the get block for the home page:

get '/:email' do
@myself = User.get(session[:userid])
@user = @myself.email == params[:email] ? @myself : User.first(:email => params[:email])
@dm_count = dm_count
erb :home
end

The code is pretty self explanatory. The home page retrieves the currently logged in user (@myself) and if the requested user (@user) is not the currently logged in user, it will retrieve that use from the database as well. It also retrieves a count of the direct messages that this user has, and this is for display purposes.

def dm_count
Chirp.count(:recipient_id => session[:userid]) + Chirp.count(:user_id => session[:userid], :recipient_id.not => nil)
end

The number of direct messages is the number of direct messages sent and received.

Let’s look at the friending and following features.

get '/follow/:email' do
Relationship.create(:user => User.first(:email => params[:email]), :follower => User.get(session[:userid]))
redirect '/home'
end

This get block is called when you want to follow a particular user. Again, I just create a relationship between the person whom you want to follow and you. Deleting that relationship is also very simple.

delete '/follows/:user_id/:follows_id' do
Relationship.first(:follower_id => params[:user_id], :user_id => params[:follows_id]).destroy
redirect '/follows'
end

Note that this is a delete block and not get block any more. Finally to display the friends and followers, I have 2 separate routes, but I want to re-use the same code and view template:

['/follows', '/followers'].each do |path|
get path do
@myself = User.get(session[:userid])
@dm_count = dm_count
erb :follows
end
end

Next let’s look at the chirps and direct messages. To chirp, I use a post block:

post '/chirp' do
user = User.get(session[:userid])
Chirp.create(:text => params[:chirp], :user => user)
redirect "/#{user.email}"
end

However, to implement the various features like URL shortening and command level following and direct messaging, I placed the processing logic in the Chirp class itself.

before :save do
case
when starts_with?('dm ')
process_dm
when starts_with?('follow ')
process_follow
else
process
end
end

Before the chirp is save, I check if the text starts with ‘dm’ or ‘follow’ and I process the text accordingly. Let’s look at these various processing blocks of code.

def process_follow
Relationship.create(:user => User.first(:email => self.text.split[1]), :follower => self.user)
throw :halt # don't save
end

Implementing the ‘follow’ command in the chirp box is probably the easier. I just create the relationship and then throw halt to stop saving the chirp.

def process_dm
self.recipient = User.first(:email => self.text.split[1])
self.text = self.text.split[2..self.text.split.size].join(' ') # remove the first 2 words
process
end

Implementing direct message is also relatively simple. I just set the recipient of the message to be the person that is indicated in the chirp box and when saving the chirp, I remove the command. Finally I pass it on to the common processing block.

def process
# process url
urls = self.text.scan(URL_REGEXP)
urls.each { |url|
tiny_url = open("http://tinyurl.com/api-create.php?url=#{url[0]}") {|s| s.read}
self.text.sub!(url[0], "<a href='#{tiny_url}'>#{tiny_url}</a>")
}
# process @
ats = self.text.scan(AT_REGEXP)
ats.each { |at| self.text.sub!(at, "<a href='/#{at[2,at.length]}'>#{at}</a>") }
end

URL_REGEXP = Regexp.new('\b ((https?|telnet|gopher|file|wais|ftp) : [\w/#~:.?+=&%@!\-] +?) (?=[.:?\-] * (?: [^\w/#~:.?+=&%@!\-]| $ ))', Regexp::EXTENDED)
AT_REGEXP = Regexp.new('\s@[\w.@_-]+', Regexp::EXTENDED)

The common processing block scans the chirp text and does a few things:

• I find the URLs in the code and replace it with a shortened URL from TinyURL.
• I find all emails that has ‘@’ prefixed and I replace it with a link to that user email.

As for viewing direct messages, again I reuse the same view template, but for the sake of variety instead of having 2 routes, I used 1 route with a parameter:

get '/direct_messages/:dir' do
@myself = User.get(session[:userid])
case params[:dir]
when 'received' then @chirps = Chirp.all(:recipient_id => @myself.id)
when 'sent'     then @chirps = Chirp.all(:user_id => @myself.id, :recipient_id.not => nil)
end
@dm_count = dm_count
erb :direct_messages
end

In the case above, when I’m looking at the received direct messages (i.e. I’m looking at the inbox) I just want to find all chirps which recipient is myself. For the direct messages I sent, I find all chirps that belong to me, which recipient is not null.

That’s it! Of course there’s the view templates but that’s mainly HTML and some ERB snippets embedded in them. You can check out the entire repository at git://github.com/sausheong/chirp.git.

To run the app, get the code. Then go to the models.rb file and change the database URL accordingly. After that, go to irb, require the models file, and run this:

> DataMapper.auto_migrate!

This will create the necessary database. Then in the directory, run this

> ruby chirp.rb

Go to http://localhost:4567 and you’re up and running locally! Of course, you need to register your RPX account and then change login.erb accordingly.

To view this in a live site, go to http://chirp-chirp.heroku.com. This is deployed to Heroku, which is another amazing service, but that’s another story for another post.