Upgrading MySQL from 5.0.27 to 5.1.34
Posted by: Bob Ngu on: May 28, 2009
- In: Database| MySQL
- Leave a Comment
Upgrading MySQL from 5.0.x to 5.1.x is considered a major upgrade and there are some fundamental internal changes. For example, in MySQL 5.1.23 and earlier, the default location for the data files used by MySQL is located within the corresponding MySQL Server installation directory. For MySQL 5.1.24 and later, the default location of the data directory is the AppData directory configured for the user that installed the MySQL application. This means that it won’t pick up the current databases since it sits in a different location.
I followed the steps outlined in this article. In my first attempt, I thought it might be possible to have side-by-side 5.0.x and 5.1.x installations just in case the upgrade process really messed up my system. That didn’t pan out so I decided to follow the above steps exactly and first uninstall 5.0.27 and then install 5.1.34.
Here are the steps I took for a successful upgrade.
- First do a backup of all the current databases. “mysqldump –all-databases > alldatabases.sql”
- Set a new system restore point just in case something went horribly wrong.
- Uninstall MySQL 5.0.27 and remove the service.
- Download and run mysql-essential.5.1.34.msi.
- Run the following command to restore the database file from above
“mysql –u root < alldatabases.sql”
And I’m up and running.
Update: As it turns out, I spoke too soon, I did get an error
The fix is simple enough, simply run “gem install mysql” as instructed.
Note: I read some articles saying that MySQL 5.1 upgrade had some issues with Rails 2.2 and requires an older MySQL client library, here’s an article on how to fix it. Personally, I didn’t encounter the problem.
Update: Again I spoke too soon, “rake db:create” got this error
ruby 1.8.6 (2008-08-11) [i386-mswin32]
Applying the solution in the above article fixed it.
Upgrading Rails 2.1 to 2.3.2
Posted by: Bob Ngu on: May 22, 2009
- In: Gem| rails| ruby rails
- Leave a Comment
Right before upgrading Rails 2.1 to 2.3.2, I downloaded a trial version of E text editor (the power of Textmate on Windows), and in the process, it upgraded my cygwin installation.
This caused a couple of errors during my Rails upgrade.
- When I ran “gem install rails -y”, I got the error “ruby no such file to load ubygems loaderror”. The problem is caused by E text editor installation and the trick is to go to My Computer properties and unset RUBYOPT variable. It’s set to –rubygems, delete the variable.
- With that error gone, there is a new error
c:/ruby/bin/gem:8:in `require’: no such file to load — rubygems (LoadError) from c:/ruby/bin/gem:8
Error was also caused by E text editor installation which upgraded cygwin and installed its own version of Ruby. Since cygwin was in environment path before c:\ruby, the gem command picked up the wrong copy of ruby (cygwin has its own Ruby exe). Moving the environment cygwin variable to after c:\ruby fixed the problem.
With those 2 errors fixed, I successfully upgraded Rails to 2.3.2. Next it’s time to upgrade the gems, “gem update –system” generated this error
Instead, run these 2 commands to fix it
- gem install rubygems-update
- update_rubygems
As usual, there are leftover gems, so do the cleanup, run with –d (dryrun) option “gem cleanup -d” first to see what the cleanup would do, then run the cleanup “gem cleanup”. This completes the gem upgrade to 1.3.2.
Successful upgrade of Ruby 1.8.5 to 1.8.6
Posted by: Bob Ngu on: May 21, 2009
- In: rails| ruby rails
- 1 Comment
In an earlier post, I explained why I didn’t upgrade Ruby to 1.8.6 several months ago, now I bit the bullet and completed the upgrade process, remember this is on the Windows platform which is my development machine.
First, I should mention that I did a fair amount of research as to whether I should upgrade Ruby to 1.8.7 instead and there are concerns on multiple Ruby and Rails forums that there may be backward compatibility issues with 1.8.7, so I decided to stick with 1.8.6. The concern with 1.8.7 is that it has backport of several Ruby 1.9 features causing potential compatibility issues, however there are also developers who don’t think there are any 1.8.7 compatibility issues, so do your own research and decide for yourself the version you wish to upgrade to. My plan is to stick 1.8.6 until 1.9 is certified to be compatible with Rails.
First thing I did was download ruby-1.8.6-p287-i386-mswin32.zip from a pseudo secret Ruby ftp location. This may or may not be obvious but unless someone compiles Ruby for you on Windows, you are out of luck since you don’t have the binaries. Note that it isn’t the latest patch level 368 (as of May 2009) but since I don’t have the Windows binaries for that version, I decided to stick with p287 and also not use one-click installer since it includes an older version of gems which by the way is easily remedied by issuing gem update after the installation. I also heard from Luis Lavena, the creator of the Windows Ruby one click installer, about why he decided to stick with 1.8.6 and not 1.8.7. Here’s his post explaining why he stuck with 1.8.6. BTW, he hasn’t gotten much love from developers in helping him maintain the one click installer, so let’s show him some love and appreciation, he is truly a nice guy and has been very helpful in my past interaction with him.
After downloading the zip file, unzipped the file and copied all folders on top of c:\ruby replacing existing files and folders. And it’s as simple as that to upgrade Ruby except I encountered a problem running my app after the upgrade. Note: unless you use REXML in your app, you won’t get this error and don’t need to apply patch.
I got this error
c:/ruby/lib/ruby/1.8/rexml/text.rb:292:in `normalize’
c:/ruby/lib/ruby/1.8/rexml/element.rb:1085:in `[]=’
/app/controllers/application.rb:57:in `resize_videos’
c:/ruby/lib/ruby/1.8/rexml/element.rb:891:in `each’
c:/ruby/lib/ruby/1.8/rexml/xpath.rb:53:in `each’
c:/ruby/lib/ruby/1.8/rexml/element.rb:891:in `each’
c:/ruby/lib/ruby/1.8/rexml/element.rb:393:in `each_element’
/app/controllers/application.rb:56:in `resize_videos’
I posted the question on ruby-forum and got the Ruby author, Yukihiro “matz” Matsumoto, to respond with the following REXML patch
index b5bd805..1edb627 100644
— a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,8 @@
+Thu May 21 08:50:58 2009 Yukihiro Matsumoto
+
+ * lib/rexml/text.rb (REXML::Text.normalize): call to_s for input.
+ [ruby-talk:337069]
+
Mon May 18 21:40:11 2009 Tanaka Akira
* lib/pathname.rb (Pathname#sub): suppress a warning.
diff –git a/lib/rexml/text.rb b/lib/rexml/text.rb
index 2bc0042..a4a30b6 100644
— a/lib/rexml/text.rb
+++ b/lib/rexml/text.rb
@@ -286,7 +286,7 @@ module REXML
EREFERENCE = /&(?!#{Entity::NAME};)/
# Escapes all possible entities
def Text::normalize( input, doctype=nil, entity_filter=nil )
- copy = input
+ copy = input.to_s
# Doing it like this rather than in a loop improves the speed
#copy = copy.gsub( EREFERENCE, ‘&’ )
copy = copy.gsub( “&”, “&” )
According to him, this is a REXML 1.9 backport that caused the issue, not a gsub bug, and he has committed the changes to 1.8 HEAD. I applied the 1 line change to rexml/text.rb and it worked, much thanks to Matz.
And that’s the story of my successful upgrade of Ruby 1.8.5 to 1.8.6. BTW I also successfully upgraded Rails from 1.2.1 to 2.1 to 2.3.2 but that’s another post for another time.
Blueprint CSS template
Posted by: Bob Ngu on: July 2, 2008
- In: General
- 3 Comments
Admittedly, this post is slightly off-topic since it doesn’t have anything to do with Ruby or Rails but this is a common task that a do-it-all web developer (like me) has to do while building an app / site. For my current site, JiggyMe (video aggregation site), I stumbled my way through CSS hacking together with what I needed at different times and it left a less than satisfying feeling with me.
And then I came across Blueprint framework, an excerpt from the site
Blueprint is a CSS framework, which aims to cut down on your CSS development time. It gives you a solid CSS foundation to build your project on top of, with an easy-to-use grid, sensible typography, and even a stylesheet for printing.
After spending a day looking through the samples, I decided that it is a good template for my site. Blueprint CSS 101 is a good introduction post on it although it is a bit abstract especially if you aren’t a CSS expert or web designer. These are key features of Blueprint
- It performs a mass reset of browser default styles.
- It sets up sensible defaults for typography, including font families, header sizes, paragraph styles, list styles, a baseline grid, and more. It does all of this with relative sizes, so that it scales well in any browser.
- It gives you a methodology to use for customizable layout grids. Any number of columns and widths you can dream up is easily achievable.
- It provides a sensible default print stylesheet.
- It does all of these things in ways that work elegantly in most browsers your visitors are likely to be using, including Internet Explorer 6 and 7.
The Grid
Blueprint’s crown jewel is its grid-building tools. By default, Blueprint’s grids.css file sets up a 950px wide grid of 24 columns, each 30 pixels wide with 10 pixel gutters. This grid is likely to be flexible enough for most of your needs.
However, the grid is fully customizable. If you need more or less columns or a wider or narrower total width, you’ll want to use the Blueprint Grid CSS Generator tool. It not only provides a replacement grids.css file, but it also creates a grid.png image file for use as a background image during development—very handy for making sure everything lines up properly. The grid generator is a key piece to the framework, even if it’s technically a third-party tool. Without it, Blueprint is limiting and controlling, forcing designers to use a single layout grid. With it, designers have endless freedom.
Let’s take an example that should clarify how you can use it.
In the above code, the CSS classes from Blueprint are container, span-4, span-24, last, span-8, span-16 and it looks like this in the browser (click on the image to see the full layout).
Hope that’s enough to get you started with Blueprint, have fun with it.
UPDATE
Someone asked how to incorporate the CSS templates into Rails, so here are the instructions (for Windows).
First you have to download the “Blueprint 0.7.1.zip” from
http://code.google.com/p/blueprintcss/downloads/list
Unzip the file and copy the blueprint folder to your Rails app folder [yourapp]\public\stylesheets folder.
In any HTML file, for example [yourapp]\public\index.html, you have to include the following lines between the <head> tags
<link rel=”stylesheet” href=”stylesheets/blueprint/print.css” type=”text/css” media=”print”>
<!–[if IE]><link rel=”stylesheet” href=”stylesheets/blueprint/ie.css” type=”text/css” media=”screen, projection”><![endif]–>
After that, you can use the blueprint CSS classes within the HTML code as shown above.
Switching to Aptana Studio
Posted by: Bob Ngu on: June 30, 2008
- In: Subversion| rails| ruby rails
- 2 Comments
My IDE of choice has been RadRails 0.7.2 until now. I just switched over to Aptana Studio 1.1. Aptana has taken over the development of RadRails as Aptana Studio so I highly recommend the switch especially since Aptana Studio will be maintained and enhanced but not RadRails. For those of you with RadRails, you will like the “Check for Aptana Studio updates now…” action, something that wasn’t working for RadRails.
The first thing I did was to install the Subclipse Subversion plugin, enter the authentication credentials and I have access to my project. Next thing I did was set Aptana default workspace to my existing workspace and it imported all the settings.
Note that I am on Windows, hence why I am not using textmate. IMO, Aptana Studio is the best RoR IDE for Windows and free to boot. Enjoy!
New Ruby on Rails Windows group
Posted by: Bob Ngu on: June 29, 2008
At the suggestion of a fellow RoR Windows developer, we thought it is a good idea to create a separate group for RoR Windows developers seeing how Windows issues tend to be rather different than Linux. Here is the new group
http://groups.google.com/group/ruby-on-rails-windows
I have also sent an email to the ruby-forum webmaster to add the group there. Please come and join us.
Upgrading to Rails 2.1 on Windows
Posted by: Bob Ngu on: June 27, 2008
- In: rails| ruby rails
- 5 Comments
Ok, now that I have decided not to upgrade Ruby to 1.8.6, see Upgrading Ruby for details, I then moved onto upgrading Rails from 1.2.1 to 2.1, this is still on my Windows XP development machine which is treated like the bastard redheaded step child that it is 
Note that I have an existing app in Rails 1.2.1, in fact the app is frozen to Rails 1.2.1, and I have no plans of upgrading the app to Rails 2.1. I am upgrading my development environment to create a new app on Rails 2.1.
First off, different articles have slightly different ways of doing the upgrade. According to this article Migrating to Rails 2.0, it suggested the following commands (it’s the one I ended up following):
gem install rails -y
gem update --system
In another Wiki article HowToUpgrade, it recommends the following commands
sudo gem update rails -y
sudo gem update rake -y
sudo gem update activerecord -y
sudo gem update activesupport -y
sudo gem update actionpack -y
sudo gem update actionmailer -y
sudo gem update activeresource -y or possibly sudo gem install activeresource -y
Yet in another different article, Setting Up Rails Development Environment on Windows Vista/XP, it suggests… In future, whenever you want to update Rails to a newer version, simply run the following command:
C:\> gem update rails
Since I am not a Rails internals expert, I am starting to feel like the blind leading the blind here. After doing an eenie, meenie, minie, moe (not quite but close enough) to catch a mongrel by its tail (sorry couldn’t resist 
), I went with the first option.
The first command went without a hitch – this one updates Rails itself
gem install rails –y
The second command seemed to work – this ones updates RubyGems from 0.9.4 to 1.2.0
gem update --system
However, when I tried to start my existing app (it uses Mongrel), it threw an error
c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require’: no such file to load — c:/ruby/lib/ruby/gems/1.8/gems/mongrel_service-0.3.2-x86-mswin32/lib/mongrel_service/init.rb (MissingSourceFile)
If I start the app using webrick, it works fine. Also, I noticed that the old Rails 1.2.1 gems are still there along with the new 2.1 gems – this is in the C:\ruby\lib\ruby\gems\1.8\gems folder. After getting some advice from the Ruby on Rails forum, I decided to do a gem cleanup with the following command
gem cleanup
Running the command gave me this
Cleaning up installed gems...
Attempting to uninstall rails-1.2.1
Successfully uninstalled rails-1.2.1
Attempting to uninstall actionmailer-1.3.1
Successfully uninstalled actionmailer-1.3.1
Attempting to uninstall activerecord-1.15.1
You have requested to uninstall the gem:
activerecord-1.15.1
actionwebservice-1.2.1 depends on [activerecord (= 1.15.1)]
rails-1.2.1 depends on [activerecord (= 1.15.1)]
If you remove this gems, one or more dependencies will not be met.
Continue with Uninstall? [Yn]
This was unexpected so I terminated the cleanup and did some research and found out that Rails 2.0 made a conscious decision to chuck ActionWebService (SOAP) in favor of ActiveResource (REST). Quoting from this article
ActionWebService out, ActiveResource in
It’ll probably come as no surprise that Rails has picked a side in the SOAP vs REST debate. Unless you absolutely have to use SOAP for integration purposes, we strongly discourage you from doing so. As a naturally extension of that, we’ve pulled ActionWebService from the default bundle. It’s only a gem install actionwebservice away, but it sends an important message none the less.
So the warning regarding actionwebservice makes sense since gem cleanup doesn’t know about actionwebservice for Rails 2.x. To get around the problem, I answered “Y” and proceeded. I encountered a similar prompt later and answered “Y” as well.
Attempting to uninstall fxruby-1.2.6-x86-mswin32
You have requested to uninstall the gem:
fxruby-1.2.6-x86-mswin32
fxri-0.3.3 depends on [fxruby (>= 1.2.0, < 1.3)]
If you remove this gems, one or more dependencies will not be met.
Continue with Uninstall? [Yn]
I hope nothing breaks, if you have any details about this, let me know.
Next, onto resolving the mongrel failure problem. After digging around the net, I came across Latest RubyGems and Rails is a deadly combo. As it turns out all I need to do is go into c:\ruby\lib\ruby\gems\1.8\gems and rename mongrel_service-0.3.2-mswin32 to mongrel_service-0.3.2-x86-mswin32 (note the x86 addition). Mongrel runs fine afterward.
All in all, upgrading Rails to 2.1 is a major PITA and not for the faint of heart.
Upgrading Ruby
Posted by: Bob Ngu on: June 23, 2008
- In: ruby rails
- 2 Comments
I have been silent on this blog for the past few months partly because I am busy with a new blog Ungeek DaPo (data portability technologies) and partly because I haven’t been coding. However I am looking to get back into coding again, so I expect to be more active on this blog once again.
In the few months that I haven’t coded, Rails 2.0 and now 2.1 have been released. My first task is to upgrade my current development (and production) platforms from Ruby 1.8.5 to 1.8.6 (the recommended version for Rails 2.1) and from Rails 1.2.3 to Rails 2.1. I quickly found out that there isn’t a clean way to upgrade Ruby on Windows and preserve the current installed gems.
I came across this thread suggesting a workaround, quoting from the thread
There’s no ruby update AFAIK. You can work around this by renaming c:\ruby to something else, installing ruby, then overwrite C:\ruby\lib\ruby\gems with the old version and call ‘gem update’ to fetch the new versions.
I don’t have much of a choice so I decided to give it a try, wish me luck!
UPDATE (later in the day)
This is not as easy as what’s stated above as the workaround. Once I have a chance to sort things out, I will update this post. In the meantime, here are some of Ruby’s directory structure (courtesy of this post Installing Ruby) that might prove useful to some of you in understanding where Ruby puts things
Ruby Programming Environment
Let’s assume that you have Windows operating system and have installed Ruby in the folder c:\ruby, then the installation creates a number of sub-folders. The folders:
c:\ruby\bin is where the Ruby executables (including ruby and irb) have been installed.
c:\ruby\lib\ruby\1.8 you’ll find program files written in Ruby. These files provide standard library facilities, which you can require from your own programs if you need the functionality they provide.
c:\ruby\lib\ruby\1.8\i386-mswin32 contains architecture-specific extensions and libraries. The files in this directory generally have names ending in .so or .dll (depending on your platform). These files are C-language extensions to Ruby; or, more precisely, they are the binary, runtime-loadable files generated from Ruby’s C-language extension code, compiled into binary form as part of the Ruby installation process.
c:\ruby\lib\ruby\site_ruby folder is where you and\or your system administrator store third-party extensions and libraries. Some of these may be code you yourself write; others are tools you download from other people’s sites and archives of Ruby libraries.
c:\ruby\lib\ruby\gems is the Ruby-Gems packaging system.
UPDATE 6/25/2008
After a couple of days of research, I have decided that the above suggested workaround is a NO-GO for me for the following reasons
- There is at least one new gem in Ruby 1.8.6, hpricot, that is not in 1.8.5, so the above workaround would lose hpricot.
- c:\ruby\lib\ruby\gems (for the current Ruby 1.8.5) also include all of Rails 1.2.3 gems. Although it seems like gem update would also update Rails (presumably to 2.1), I am not comfortable that something isn’t missing, could be unjustified paranoia but I am not taking any chances.
- I use HTree which as it turns out is not installed as a gem, the author didn’t bother to package it as a gem and instead uses Ruby 3rd party extension / library mechanism instead meaning that it is installed under c:\ruby\lib\ruby\site_ruby instead of c:\ruby\lib\ruby\gems. The above workaround does not take this into account.
In the end, I deemed the hackish workaround of upgrading Ruby to be too risky. So I have 2 options currently
- Stick with Ruby 1.8.5 and upgrade to Rails 2.1
- Blow away the current Ruby 1.8.5, install Ruby 1.8.6, reinstall all the gems and extensions I need, and upgrade to Rails 2.1.
At this point, it doesn’t seem worth the time and effort to upgrade to Ruby 1.8.6 since Rails 2.1 runs on Ruby 1.8.5, so I am going to give option 1 a try. Details on the next post on how it went.
UPDATE 7/1/2008
Best advice yet I received on this matter from Luis Lavena on the Ruby on Rails Windows group
There are several changes from 1.8.5 that will break your gems, no matter how easy we make that process. Also the update to latest rubygems will break projects like mongrel and such that uses pre-compiled extensions (and thus are platform specific gems).
> Is there any chance of building a one-click installer
> that can preserve the current installed gems?There is no easy way, sorry. First thing first, backup your installation
The new version of the installer we are working on will honor your installed gems and will leave them in place, but right now this installer is not based on that
Jumping from 1.8.5 to 1.8.6 is a difficult step and I suggest you write a list of your gems and manually install them, since there a lot ones that will not work under 1.8.6
UPDATE 5/21/2009
I did finally upgrade ruby 1.8.5 to 1.8.6, see the post here.
Migrating from Apache to Nginx
Posted by: Bob Ngu on: February 26, 2008
- In: web server
- 2 Comments
My RoR efforts have culminated in JiggyMe, check it out!
About a week ago, I decided to install PHP and WordPress on my production server. I immediately noticed that my server started heavy swapping. This has never happened before because the server has been pretty stable so far. PHP pushed the memory usage over the edge and caused the heavy swapping.
After several days of investigation, I found that in general Apache is a memory hog. I came across a few articles talking about how lean and fast Nginx is, so I decided to migrate. Turns out it wasn’t simple because Nginx is not as well-documented as Apache.
Here are my steps to get it installed, maybe it will help you, the following instructions are for FC6.
Prerequisites to installing Nginx
Install zlib library (required by gzip module)
sudo wget http://www.zlib.net/zlib-1.2.3.tar.gz
tar –xvf zlib-1.2.3.tar.gz
cd zlib-1.2.3
sudo ./configure
sudo make
sudo make install
Install pcre library (required by rewrite module)
yum install pcre-devel
sudo ./configure
sudo make
sudo make install
Install nginx
Initially, I used yum to install nginx but it picked up an older version 0.5.33. I wanted the latest stable version 0.5.35, so I compiled it from source.
wget http://sysoev.ru/nginx/nginx-0.5.35.tar.gz
tar –xvf nginx-0.5.35.tar.gz
cd nginx-0.5.35
sudo ./configure
sudo make
sudo make install
This installed nginx in /usr/local/nginx
Configure nginx
Modify /usr/local/nginx/conf/nginx.conf to the following. Note: I use a typical Mongrel cluster of 2 Mongrel processes. Adjust the following variables for your own needs: user, server_name, root.
Updated on 2/27/2008, gzip now handles additional file types, access log turned off (don’t want to deal with growing log file), and have nginx serve up all sorts of static files like images, css, javascript, etc. Thank me later =)
user deploy deploy;
worker_processes 5;
error_log logs/error.log debug;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include conf/mime.types;
default_type application/octet-stream;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
tcp_nodelay on;
gzip on;
gzip_min_length 1100;
gzip_buffers 4 8k;
#2/27/08 – add more file types to use gzip
gzip_types text/plain text/html text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript text/xhtml;
upstream mongrel {
server 127.0.0.1:8000;
server 127.0.0.1:8001;
}
#the following to redirect example.com requests to www.example.com, important for SEO.
server {
server_name example.com;
rewrite ^(.*) http://www.example.com$1 permanent;
}
server {
listen 80;
server_name www.example.com;
root /var/www/apps/example/current/public;
index index.html index.htm;
#2/27/08 – Also serve the static files in rails public directory using nginx
location ~* ^.+.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js|mov)$ {
access_log off;
expires 30d;
root /var/www/apps/example/current/public;
}
location / {
access_log off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect false;
if (-f $request_filename/index.html) {
rewrite (.*) $1/index.html break;
}
if (-f $request_filename.html) {
rewrite (.*) $1.html break;
}
if (!-f $request_filename) {
proxy_pass http://mongrel;
break;
}
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
Start / stop nginx
When I used yum to install nginx, I could start nginx using service
sudo service nginx start
However, when I compiled and installed it from source, I couldn’t. If anyone knows the intricacies of how how to get it to work with service command, let me know.
Anyway, to start the service, type
sudo /usr/local/nginx/sbin/nginx
If you want to stop it, type
kill nginx-master-pid
You can obtain the master process id here
/usr/local/nginx/logs/nginx.pid
Done
Watch the memory usage drops!
Making Twitter 10000 Percent Faster
Posted by: Bob Ngu on: January 28, 2008
Found a couple of other interesting links to this topic. One is Scaling Twitter: Making Twitter 10000 Percent Faster. It is from a site called High Scalability dedicated to describing (in high level) various scalable architectures like MySpace, Plentyoffish, eBay, Google, Flickr, etc. Definitely worth reading if you are interested in exploring the various architecture designs employed by these different high traffic sites.
The most interesting comment on scaling Twitter is this
Twitter started as a side project and blew up fast, going from 0 to millions of page views within a few terrifying months. Early design decisions that worked well in the small melted under the crush of new users chirping tweets to all their friends. Web darling Ruby on Rails was fingered early for the scaling problems, but Blaine Cook, Twitter’s lead architect, held Ruby blameless:
For us, it’s really about scaling horizontally – to that end, Rails and Ruby haven’t been stumbling blocks, compared to any other language or framework. The performance boosts associated with a “faster” language would give us a 10-20% improvement, but thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10000% faster than it was in January.
This is different than what Alex Payne, another developer at Twitter, said when he was interviewed here. Subsequent to Alex’s interview, Rails has been blamed incessantly for not being able to scale. Even open source drama follows mainstream media drama, i.e., take the sound bite and run with it, never mind the details.
It is as I suspected all along, the language / framework are all but bit players in the grand scheme of scalable architecture. IMO this also explains why other scripting languages like Perl, PHP, VBScript, Python, etc are not front and center when it comes to scalability. Don’t get me wrong, I think Ruby performance can be improved but the real challenges lie elsewhere in the backend. I guess all the initial hype about RoR productivity led to exuberant expectations and the inevitable crash. Truth be told, the scaffolding bits that allow you to put up a web page in 5 mins is useless when it comes to real development. Once the hood is lifted, the real work begins. Sites developed with other scripting languages/framework went through the same growing pains but without the glitz and media attention that RoR garnered. I believe J2EE went through the same growing pains early on in their adoption. These are expected growing pains, nothing more, nothing less.
Another interesting article is Could Instant Messaging (XMPP) Power the Future of Online Communication?. It talks about using push instead of pull. This is nothing new, this design has been around for years and is being re-discovered and re-use in a new domain. Something old becomes something new.
