My Feeds

Hungry Machine - All

MRI Ruby + MySQL + Threads == Stop the world... JRuby doesn't 10:04am

As we have been internally discussing how to scale our databases from 10's of millions of rows to 100's of millions, database sharding came up.

Depending on your data model and your application, sharding data into tables by some natural key is great if any given request uses only one shard. FiveRun's DataFabric seems to help with that. Its obviously best to shard the data in the most used way, but occasionally you'll need to write broadcast queries across shards. It'd be even better if those broadcast queries were executed concurrently. Well, apparently, its not that simple in MRI Ruby.

The mysql (2.7) gem stop's the world while executing a query. A sloppy but reproducible test is here. This is not a mutex lock on the re-use of a single connection object. Eddie also reached out to the Sequel maintainer, who agrees it is likely due "to the fact that the C drivers don't release the interpreter lock while they wait for a response from the server." JRuby, or more accurately JDBC, acts as expected. We even tested DataMapper's DataObjects::MySQL, as it appears they've re-implemented the mysql gem. Unfortunately it suffers from the same stop-the-world issue.

$ ruby mysql_locking_test.rb 
Loading MRI MySQL
Serial: 4.00969815254211s
Multi-threaded: 4.00785183906555s

$ data_mapper=true ruby mysql_locking_test.rb 
Loading DataMapper
Serial: 4.01330804824829s
Multi-threaded: 4.01132893562317s

$ jruby mysql_locking_test.rb 
Loading JDBC-MySQL
Serial: 4.2802369594573975s
Multi-threaded: 1.0499329566955566s

Only in JRuby was the multi-threaded != sequential time, as expected.

Potentially unrelated, no one has touched the mysql gem in 3 years?!?!

$ gem spec -v 2.7 mysql | grep date
date: 2005-10-09 00:00:00 +00:00

So even Ruby ORM frameworks (Sequel, DataMapper) that say they're thread-safe, are not concurrent on MRI... at least for mysql. For folks not using Rails, which already has a mutex lock higher in the stack, this must be a performance issue. For example, Merb + DataMapper + MySQL. If there is a 2s SQL query, all threads in that process stop for 2s.

Can others verify? "select sleep(2) from dual;" is a great way to test for this.

UPDATE: Multiple have asked, so to clarify. The sample code here creates a new connection PER thread. The mysql docs states: "Two threads can't send a query to the MySQL server at the same time on the same connection", but the test is creating a new connection object per thread, so that should not apply.

UPDATE 2: A couple folks have mentioned asynch drivers. One for mysql and one for postgres, but I believe they're based on NeverBlock, which is Ruby 1.9 only. That sounds like awesome progress. What's the realistic ETA for folks running 1.9 in a production environment? At least until Christmas for a 1.9 official release?

UPDATE 3: Looks like lots is happening here. There now is a mysql driver that supports async/threaded operations on 1.8! See the NeverBlock MySQL project. Looking forward to testing this in a production environment.

Using helpers in a controller: with_helpers 8:49am

Although there is supposed to be a clear separation between views and controllers, often when it comes to helper functions, there is a small bit of overlap and there are situations where it'd be nice to simply use a few helpers from inside an action.

ActionController::Base.class_eval do
  def with_helpers(&block)
    template = ActionView::Base.new([],{},self)
    template.extend self.class.master_helper_module
    add_variables_to_assigns
    template.assigns = @assigns
    template.send(:assign_variables_from_controller)
    forget_variables_added_to_assigns
    template.instance_eval(&block)
  end
end

Here is what it looks like in a controller. Although this is a non-sensical example, it shows off how you can use a helper (i.e. link_to) in an action. It also shows how the instance variables (i.e. @person) set in the action are available in the with_helpers block... the same way that instance variables are available in views.

class MyController < ApplicationController
  def my_action
    @person = Person.find(params[:id])
    render :text => with_helpers { link_to(@person.full_name, person_path(@person)) }
  end
end

Breaking the Rails static asset timestamp cache in development mode 1:11pm

Rails automatically adds the File.mtime to static assets when using stylesheet_link_tag and javascript_include_tag. The file's mtime is cached to prevent excessive file system access... even in development mode. This is problematic in a Facebook canvas during development mode because often you won't immediately see the changes you make to your stylesheets and javascripts.

Here is a monkey patch you can throw into a config/initializers/break_asset_cache_in_dev_mode.rb to fix this:

if RAILS_ENV == 'development'
  require 'action_controller/dispatcher'
  ActionController::Dispatcher.before_dispatch do
    ActionView::Base.computed_public_paths.clear
  end
end

Cleaning up old releases 3:12pm

Instead of relaying on running cleanup of old releases via capistrano, we have a cron job to only keep releases for last two days (but at least three latest).

#!/usr/bin/env ruby

require 'fileutils'

KEEP_RELEASES = 3
KEEP_DAYS = 2
EXCLUDE_APPS = %W(uploadr)

cut_time = (Time.now.utc - KEEP_DAYS*24*60*60).strftime("%Y%m%d%H%M%S").to_i

Dir['/u/apps/*'].each do |app|
  next if EXCLUDE_APPS.include?(File.basename(app))
  dirs = Dir["#{ app }/releases/*"]
  fresh = dirs.select { |dir| (dir.split('/').last).to_i > cut_time }
  latest = dirs.sort.last(KEEP_RELEASES )

  (dirs - fresh - latest).each do |dir|
    FileUtils.rm_rf dir
  end
end

Rake task for syntax checking a Ruby on Rails project 11:00am

Here's a quick rake file that will crawl through your Rails project and syntax check ruby, erb, and yml files. You should always run this before doing a "cap deploy" and it even doesn't hurt to run it before a "svn commit".

require 'erb'
require 'open3'
require 'yaml'

task :check_syntax => [:check_ruby, :check_erb, :check_yaml]

task :check_erb do
  (Dir["**/*.erb"] + Dir["**/*.rhtml"]).each do |file|
    next if file.match("vendor/rails")
    Open3.popen3('ruby -c') do |stdin, stdout, stderr|
      stdin.puts(ERB.new(File.read(file), nil, '-').src)
      stdin.close
      if error = ((stderr.readline rescue false))
        puts file + error[1..-1]
      end
      stdout.close rescue false
      stderr.close rescue false
    end
  end
end

task :check_ruby do
  Dir['**/*.rb'].each do |file|
    next if file.match("vendor/rails")
    next if file.match("vendor/plugins/.*/generators/.*/templates")
    Open3.popen3("ruby -c #{file}") do |stdin, stdout, stderr|
      if error = ((stderr.readline rescue false))
        puts error
      end
      stdin.close rescue false
      stdout.close rescue false
      stderr.close rescue false
    end
  end
end

task :check_yaml do
  Dir['**/*.yml'].each do |file|
    next if file.match("vendor/rails")
    begin
      YAML.load_file(file)
    rescue => e
      puts "#{file}:#{(e.message.match(/on line (\d+)/)[1] + ':') rescue nil} #{e.message}"
    end
  end
end

And here's what the output looks like:

warren:tmp_project$ rake check_syntax
(in /Users/warren/tmp_project)
app/controllers/application.rb:37: syntax error, unexpected '='
app/views/people/home.html.erb:60: syntax error, unexpected '<', expecting $end
vendor/plugins/will_paginate/test/fixtures/users.yml:13: syntax error on line 13, col 0: `dev_<%= digit %>:'

This could probably be expanded to support lots of other types of files as well (css, javascripts, etc).

Updates:

• Exclude all files in vendor/rails (thanks szeryf)

ESI & Mongrel-ESI.. Request for Feedback 10:59am

  I'm wondering what is the main set of features preventing folks
today from using mongrel-esi in production?  Is it :

* performance
* documentation
* stability

If it's performance, can someone do some load testing maybe provide
some numbers and even some target numbers?  I am working on an
improved concurrency model, that should help improve page performance
for pages with lots of esi:include tags

if it's documentation, what's the missing bits, and can any of you
help out by filling in the gaps?

if it's stability, can you provide some samples that fail?

-Todd

RailsConf 2008 Presentation - Assembling Pages Last 4:41pm

The End of Slideshows: Animoto 4:29pm

UPDATE: Animoto just raised a round of investment from Amazon! Congrats Guys!

Animoto is a great idea. They take your photos and create a production quality video to the music of your choice. Its the end of those boring slide shows, for good.

We had the pleasure to work with the Animoto guys to launch their Facebook application, "Animoto Videos", which leveraged all of the existing photos on Facebook. The growth was amazing.

(From a recent AllFacebook article here)

Scaling an application from a few hundred users to over a million in just a few days isnt easy, but we had a great team. Their backend rendering farm lived in Amazon's Cloud, and the growth was so impressive, Jeff Bezos even spoke about them at Y Combinator’s Startup School just a few weeks ago. From 50 EC2 instances to over 4k in only a few days. See the video below.

It was a pleasure working with the entire team from Animoto, RightScale, and Amazon. See their blog posts about the application here, here, and here.

I'm sure I'll cross paths with many of you at RailsConf. First round of beers is on me.

Advanced Rails Recipes 11:32am

ActsAsInsertOrUpdate 10:40pm

Problem

With high volume Rails applications, entities with unique constraints are expensive and error prone to create/update. ActsAsInsertOrUpdate helps solve that problem (if you're using MySQL), by leveraging the "INSERT ... ON DUPLICATE KEY UPDATE" functionality.

Scenario

Lets say you have a Person, and Entity, and a Rating. Each user can rate each entity only once, and if they re-rate the entity, it should update the value.

class Entity < ActiveRecord::Base
  has_many :ratings
end

class Person < ActiveRecord::Base
 has_many :ratings
end
  
class Rating < ActiveRecord::Base
 belongs_to :Person
 belongs_to :Entity
end  

Here is the table that back's Rating. Notice the Unique Key constraint on (entity_id, person_id).

CREATE TABLE `ratings` (
  `id` int(11) NOT NULL auto_increment,
  `rating` tinyint(4) default '0',
  `person_id` int(11) default NULL,
  `entity_id` int(11) default NULL,
  `created_at` datetime default NULL,
  `updated_at` datetime default NULL,
  PRIMARY KEY  (`id`),
  UNIQUE KEY `index_ratings_on_entity_id_and_person_id` (`entity_id`,`person_id`),
)

Previously, the logic would be something like:

• 1) Check if a rating exists for the User + Entity
• 2) If so, update
• 3) If not, insert
• 4) rescue the insert in case there is a unqiue constraint error
• 5) retrieve the record (and/or update with the new rating)

If the table is MyISAM, Steps 1-5 aren't transactionally safe. If you're using InnoDB, and experience heavy volumes of traffic, you're prone to Deadlock's. This is even more of a concern is the unique entity is shared across multiple users, as seen with a recent client of ours.

Solution:

class Rating < ActiveRecord::Base
 belongs_to :Person
 belongs_to :Entity
 acts_as_insert_or_update :field_to_update => "rating"
end  

Now Steps 1-5 above become, just one. Rating.create(..)

In the background, ActsAsInsertOrUpdate overwrites the implementation of ActionRecord:Base#create, to leverage an often unsed feature of MySQL called INSERT ... ON DUPLICATE KEY UPDATE. As configured above, if a duplicate record is found for the unique constraint, the rating field will be updated with the new value.

Caution

This is a brute force hack on ActiveRecord::Base#create. Use at your own risk.

Code

Waiting for a rubyforge account. Will post more info soon.