One of my first tasks as a new developer here at GinzaMetrics has been to help migrate our production servers to AWS, not because our current setup is failing us in any kind of egregious way, but because we’re looking to better automate provisioning and scaling of the platform itself up to millions of keywords. It also helps that Amazon has very recently launched a new data center in Tokyo, right in the backyard of many Ginza customers.
If you’ve never worked with AWS before, your first foray will most likely be somewhat confusing. Part of it is that Amazon’s documentation, while thorough, is needlessly verbose and labyrinthine, to the point where it might take three or four hours of ceaseless jumping, scanning, and focused reading before you have even the slightest grip on how to bring up an EC2 instance. And even if you are familiar with the AWS-EC2 ecosystem, you’re probably always looking for better tools to make life easier.
Enter Rubber, a Capistrano/Rails plugin that promises to automate the provisioning of both vertically and horizontally scalable multi-instance EC2 deployment configurations.
The Rubber gem is essentially a set of Rails generators—called “vulcanizers”—that spit out a bunch of really nifty YAML files and Capistrano recipes, which when combined let you create entire “cloud” server configurations by issuing simple commands like cap rubber:create. Different vulcanizers give you different stacks: there’s a mysql vulcanizer, a passenger vulcanizer, an apache vulcanizer, and many more. These can be combined in arbitrary ways to create more complicated setups like the complete_passenger_mysql stack.
Behind the scenes, those YAML files make clever use of the AWS API and features like security groups to automagically link instances so that, say, your two mysql servers (master and slave) only allow connections from your load-balanced app servers. The Capistrano tasks handle the download, installation, and configuration of all the software you need, from Git and Ruby and Rails all the way up to Passenger and Apache and monit. Like vanilla cap recipes, each time you “deploy” they fetch your application’s latest code from your Github repository, put it in the right directory on the right instance(s), run the necessary database migrations, and reboot whatever parts of the stack need rebooting.
What I want to do in the rest of this post is walk you through the process of pushing a Rails app to EC2 using Rubber. We won’t be provisioning multiple instances—just one that handles all three of the “web”, “app”, and “db” roles—but you’ll find that breaking out roles onto their own boxes is almost just as easy.
The first step is to sign up for EC2 access if you don’t already have it. All you’ll need for the purposes of this walkthrough are three key credentials: your AWS Access Key, your AWS Secret Access Key, and an SSH keypair (which you should let Amazon generate for you).
For the most part all I’ll be doing is fleshing out Rubber’s Quick Start guide. I’ll assume that you’re working with Ruby 1.8.7 and Rails 2.3.x, but Rubber will work with lots of different Rubies, including Ruby 1.9 and Rails 3. We’re using RVM and Bundler, two must-haves for RoR developers. (If you don’t use them, that’s fine—instead of steps 3-5 just sudo gem install rubber. You may also have to SSH into the server post deploy to rake gems:install.)
Step 0: Review the steps and pitfalls enumerated in Rubber’s FAQ, especially those relating to SSH keys. You have to make sure that both your private and public key are in your ~/.ec2 folder, and that the private key doesn’t use the “.pem” extension that Amazon gives you by default.
Step 1: Navigate into your Rails project’s root directory and create a new branch, maybe something like “rubber_test”, just so that you can make a big mess without worrying about mucking anything up too seriously.
git checkout -b rubber_test
Step 2: Edit your .rvmrc to use a new gemset. The top line should now read:
rvm 1.8.7@your_app_rubber_test —create
Step 3: cd out and back into the your_app directory to activate the new gemset. (Or just do rvm gemset use your_app_rubber_test). You can confirm that this worked by running rvm gemset list. The arrow should be pointing at your_app_rubber_test.
Step 4: Edit your Gemfile to include “rubber”, and “mysql” if it’s not already there. If you have gem that depends on soap4r, you should remove it, because it’ll interact dangerously with a gem that rubber wants to load called mumboe-soap4r, through these dependencies: rubber → nettica → mumboe-soap4r.
Step 5: Install the bundle:
gem install bundler; bundle install
Incidentally, here is a good resource for understanding the finer points of gems and Bundler.
Step 6: Vulcanize! Recall that this is the name for Rubber’s own generators that create all sorts of config files, like for Apache and Monit and Passenger and MySQL and so on. There are many different vulcanizers, each providing a different system setup. For now we will use one that creates a “Complete Passenger MySQL” installation.
script/generate vulcanize complete_passenger_mysql
If this script appears to hang, don’t worry: just hit the ENTER key. (This is due to a weird bug on some versions of Rails. It was pointed out to me by nirvdrum who hangs out in the #rubberec2 channel on freenode—very responsive.)
You should now see a prompt warning you that some files are going to be overwritten. Accept the changes by typing “Y” (or review them first by selecting the “diff” option) and hit ENTER again. There might be three or four of these prompts, and all are okay to authorize. (That’s why we spun off a new branch!)
Step 8: Edit your_app/config/rubber/rubber.yml, which is one of the many files just generated that defines the setup Rubber will use once you tell it to build your server(s).
You’re going to have to make a few simple changes. Here’s a diff:
-app_name: your_app_name
+app_name: [Your actual app name]
-access_key: XXX
+access_key: [Your actual AWS Access Key]
-secret_access_key: YYY
+secret_access_key: [Your actual AWS Secret Access Key]
-account: ZZZ
-key_name: gsg_keypair
+key_name: [This might still be gsg_keypair, but change the name if necessary]
Step 9: Set “rvm_ruby” to 1.8.7 in your_app/config/rubber/rubber-rvm.yml.
Step 10: mv .rvmrc rvmrc. Otherwise your new instance will try to load the .rvmrc file and halt because it doesn’t trust it. So the server will just end up sitting there waiting for you to press ENTER.
Step 11: If any of your gems have system dependencies, like the way nokogiri depends on libxslt, you will need to add them in config/rubber/rubber-rvm.yml. For example, add libxslt-dev just before libxml2-dev in the “packages:” list.
Step 12: Make sure your migrations can run cleanly from start to finish, because otherwise the cap recipe will crap out and you won’t be able to finish the deployment. Comment out lines for migrations that might fail.
Step 13: Fire up the instance!
cap rubber:create_staging
Accept the defaults and say “yes” to the prompts.
Step 14: Wait about 18 minutes.
Step 15: Navigate to in your browser and bask in the vulcanized glory of EC2. Rubber very cleverly updates your “/etc/hosts” file so that you won’t need to type in your EC2 instance’s public DNS address. Pretty cool!
Steps 16-∞: Make local changes to your code and run cap deploy to push them in 10s or so. Note that Rubber defaults to just grabbing a tarball of your local project directory; for real use you’ll want to set it up to work with your actual repository, which you can do by changing the “:scm” parameters in “your_app/config/deploy.rb”.
Once you get this cooking it’s like a whole new world of pain-free system administration. You can trivially add new instances, roles, or even entire sequestered staging deployments, one for each developer. You can let Rubber’s defaults rule the day or dive right into making all sorts of granular tweaks to individual config files, all of which are available locally right there in your project’s source tree.
I plan to write another post that will show you how to bring up more complex deployments based on the production environment we’re using at GinzaMetrics (with a couple of load-balanced web servers, a handful of headless workers, and two db instances with master-slave replication).
The upshot for now, though, is that with something like Rubber you get the easy scalability of, say, a Heroku, with a whole lot more transparency. It’s an alternative that’s well worth considering.