In this tutorial I will describe how to setup gitorious on Ubuntu 11.10. Gitorious – a Ruby on Rails web application – can be used to conclude git projects in an easy to manage user interface. In the README of the gitorious repository I found the evidence “One of the main challenges in Gitorious is its installation process. It is anything but trivial.” – It appears correct During the installation I got some errors by the sphinx search engine and some ruby gems. You will find the workarounds in this tutorial.

This tutorial is based on the gitorious – UbuntuInstallation tutorial.

In this setup I will use the hostname gitorious.linux-adm.de – change this hostname in your installation.

software installation

Required packages

apt-get install build-essential zlib1g-dev tcl-dev libexpat-dev \
    libcurl4-openssl-dev postfix apache2 mysql-server mysql-client \
    apg geoip-bin libgeoip1 libgeoip-dev sqlite3 libsqlite3-dev \
    imagemagick libpcre3 libpcre3-dev zlib1g zlib1g-dev libyaml-dev \
    libmysqlclient15-dev apache2-dev libonig-dev ruby-dev rubygems \
    libopenssl-ruby phpmyadmin libdbd-mysql-ruby libmysql-ruby \
    libmagick++-dev  zip unzip memcached git-core git-svn git-doc \
    git-cvs irb libxslt1.1 libxslt1-dev

Ruby, sphinxsearch, stompserver

apt-get install libchronic-ruby librmagick-ruby libdaemons-ruby \
ruby-hoe ruby-echoe libyadis-ruby ruby-mime-types libopenid-ruby \
libdifflcs-ruby libjson-ruby librack-ruby libhmac-ruby rake stompserver \
libapache2-mod-passenger rails sphinxsearch sphinxbase-utils

fetch gitorious

git clone git://gitorious.org/gitorious/mainline.git /var/www/gitorious

install missing ruby gems

cd /var/www/gitorious/
gem install bundler
bundle install

git user

Create a git user with the homedir in /var/www/gitorious/ and set the correct ownership.

adduser --system --home /var/www/gitorious/ --no-create-home --group --shell /bin/bash git
chown git:git -R /var/www/gitorious/

Create a directory for the repositories and a cache directory. Gitorious will add the SSH public keys of the registered users to the authorized_keys file, therefore we have to create a .ssh file and an empty authorized_keys file with the correct permissions.

su - git
git submodule init && \
git submodule update
mkdir -p tmp/pids
mkdir repositories
mkdir tarballs
mkdir .ssh
touch .ssh/authorized_keys
chmod 700 .ssh/
chmod 600 .ssh/authorized_keys

Put gitorious binary on path:

cat >> /var/www/gitorious/.bash_profile <<EOF
export PATH="$PATH:/var/www/gitorious/script/"
EOF

gitorious configuration

prepare configuration files

cd /var/www/gitorious
cp config/database.sample.yml config/database.yml
cp config/gitorious.sample.yml config/gitorious.yml
cp config/broker.yml.example config/broker.yml

create a secret key

Create a secret key for the session cookie. (we need this key later in the next step.

apg -m 32

setup config/gitorious.yml

Setup gitorious in /var/www/gitorious/config/gitorious.yml:


repository_base_path: "/var/www/gitorious/repositories"

is_gitorious_dot_org: false


Create the logroate configuration and the init scripts.

cp /var/www/gitorious/doc/templates/ubuntu/gitorious-logrotate /etc/logrotate.d/

cp /var/www/gitorious/doc/templates/ubuntu/git-ultrasphinx /etc/init.d/
cp /var/www/gitorious/doc/templates/ubuntu/git-daemon /etc/init.d/
cp /var/www/gitorious/doc/templates/ubuntu/git-poller /etc/init.d/

Set in /etc/init.d/git-daemon and in /etc/init.d/git-poller the RUBY_HOME variable to “/usr”

#RUBY_HOME="/opt/ruby-enterprise"
RUBY_HOME="/usr"


Activate the new gitorious init scripts

update-rc.d git-daemon defaults && \
    update-rc.d git-poller defaults && \
    update-rc.d git-ultrasphinx defaults

fix gem spec files

Try to start /etc/init.d/git-ultrasphinx
If you get the following error message you have to fix the spec files:

root@ubuntu03:/var/www/gitorious# /etc/init.d/git-ultrasphinx start
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
(in /var/www/gitorious)
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
rake aborted!
Bundler couldn't find some gems. Did you run <code>bundle install</code>?
/var/www/gitorious/Rakefile:4
(See full trace by running task with --trace)

Change the “s.date” in the following spec files:

/var/lib/gems/1.8/specifications/json-1.5.4.gemspec
/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec
/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec

from:

s.date = %q{2011-08-31 00:00:00.000000000Z}

in:

s.date = %q{2012-08-31}


fix git-ultrasphinx

You have to fix the /var/www/gitorious/config/boot.rb file,
if you get the following error message after starting /etc/init.d/git-ultrasphinx.

root@ubuntu03:/var/www/gitorious# /etc/init.d/git-ultrasphinx start
(in /var/www/gitorious)
rake aborted!
uninitialized constant ActiveSupport::Dependencies::Mutex
/var/www/gitorious/Rakefile:10:in `require'
(See full trace by running task with --trace)

Add the following line in /var/www/gitorious/config/boot.rb.


# Don't change this file!
# Configure your app in config/environment.rb and config/environments/*.rb

require 'thread'


Apache configuration

Disable the default vhost configuration

a2dissite default
a2dissite default-ssl

create a new vhost configuration

cat >> /etc/apache2/sites-available/gitorious <<EOF
<VirtualHost *:80>
    ServerName gitorious.linux-adm.de         # change the hostname!
    DocumentRoot /var/www/gitorious/public
</VirtualHost>
EOF

create a new SSL vhost configuration

cat >> /etc/apache2/sites-available/gitorious-ssl <<EOF
<IfModule mod_ssl.c>
    <VirtualHost _default_:443>
        ServerName gitorious.linux-adm.de   # change the hostname!
        DocumentRoot /var/www/gitorious/public
        SSLEngine on
        SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
        SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
        BrowserMatch ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
    </VirtualHost>
</IfModule>
EOF

a2ensite gitorious
a2ensite gitorious-ssl
a2enmod ssl   # enable SSL

create MySQL User for gitorious

mysql -u root -p
GRANT ALL PRIVILEGES ON *.* TO 'gitorious'@'localhost' IDENTIFIED BY 'XXXX' WITH GRANT OPTION;
FLUSH PRIVILEGES;

change the password XXXX!

Add the MySQL User to the production section in /var/www/gitorious/config/database.yml


production:
adapter: mysql
database: gitorious_production
username: gitorious
password: XXXX # change the password!
host: localhost
encoding: utf8


Starting gitorious

Run the following commands as user git.

export RAILS_ENV=production && 
bundle exec rake db:create && 
bundle exec rake db:migrate && 
bundle exec rake ultrasphinx:bootstrap

fix ultrasphinx/production.conf

If you get the following error message, you have to fix the /var/www/gitorious/ultrasphinx/production.conf file.

export RAILS_ENV=production && 
bundle exec rake db:create && 
bundle exec rake db:migrate && 
bundle exec rake ultrasphinx:bootstrap

Sphinx 0.9.9-release (r2117)
Copyright (c) 2001-2009, Andrew Aksyonoff

using config file '/var/www/gitorious/config/ultrasphinx/production.conf'...
WARNING: key 'address' is deprecated in /var/www/gitorious/config/ultrasphinx/production.conf line 11; use 'listen' instead.
listening on all interfaces, port=3312
WARNING: index 'main': preload: failed to open /var/www/gitorious/db/sphinx//sphinx_index_main.sph: No such file or directory; NOT SERVING
FATAL: no valid indexes to serve
Failed to start
Done
Please restart your application containers

Change in /var/www/gitorious/config/ultrasphinx/production.conf the word “base_tags” to “tags”.

git-ultrasphinx cronjob

Add the following cronjob for the git user.

crontab -e -u git


* * * * * cd /var/www/gitorious && /usr/local/bin/bundle exec rake ultrasphinx:index RAILS_ENV=production


create an admin user

git@ubuntu03:/var/www/gitorious$ env RAILS_ENV=production ruby1.8 script/create_admin
Type in Administrator's e-mail: 
admin@gitorious.linux-adm.de
Type in Administrator's password: 
test
Admin user created successfully.

After a reboot of your system the gitorious web front-end should be available.

In this tutorial I will describe how to setup gitorious on Ubuntu 11.10. Gitorious – a Ruby on Rails web application – can be used to conclude git projects in an easy to manage user interface. In the README of the gitorious repository I found the evidence “One of the main challenges in Gitorious is its installation process. It is anything but trivial.” – It appears correct During the installation I got some errors by the sphinx search engine and some ruby gems. You will find the workarounds in this tutorial.

This tutorial is based on the gitorious – UbuntuInstallation tutorial.

In this setup I will use the hostname gitorious.linux-adm.de – change this hostname in your installation.

software installation

Required packages

apt-get install build-essential zlib1g-dev tcl-dev libexpat-dev \
    libcurl4-openssl-dev postfix apache2 mysql-server mysql-client \
    apg geoip-bin libgeoip1 libgeoip-dev sqlite3 libsqlite3-dev \
    imagemagick libpcre3 libpcre3-dev zlib1g zlib1g-dev libyaml-dev \
    libmysqlclient15-dev apache2-dev libonig-dev ruby-dev rubygems \
    libopenssl-ruby phpmyadmin libdbd-mysql-ruby libmysql-ruby \
    libmagick++-dev  zip unzip memcached git-core git-svn git-doc \
    git-cvs irb libxslt1.1 libxslt1-dev

Ruby, sphinxsearch, stompserver

apt-get install libchronic-ruby librmagick-ruby libdaemons-ruby \
ruby-hoe ruby-echoe libyadis-ruby ruby-mime-types libopenid-ruby \
libdifflcs-ruby libjson-ruby librack-ruby libhmac-ruby rake stompserver \
libapache2-mod-passenger rails sphinxsearch sphinxbase-utils

fetch gitorious

git clone git://gitorious.org/gitorious/mainline.git /var/www/gitorious

install missing ruby gems

cd /var/www/gitorious/
gem install bundler
bundle install

git user

Create a git user with the homedir in /var/www/gitorious/ and set the correct ownership.

adduser --system --home /var/www/gitorious/ --no-create-home --group --shell /bin/bash git
chown git:git -R /var/www/gitorious/

Create a directory for the repositories and a cache directory. Gitorious will add the SSH public keys of the registered users to the authorized_keys file, therefore we have to create a .ssh file and an empty authorized_keys file with the correct permissions.

su - git
git submodule init && \
git submodule update
mkdir -p tmp/pids
mkdir repositories
mkdir tarballs
mkdir .ssh
touch .ssh/authorized_keys
chmod 700 .ssh/
chmod 600 .ssh/authorized_keys

Put gitorious binary on path:

cat >> /var/www/gitorious/.bash_profile <<EOF
export PATH="$PATH:/var/www/gitorious/script/"
EOF

gitorious configuration

prepare configuration files

cd /var/www/gitorious
cp config/database.sample.yml config/database.yml
cp config/gitorious.sample.yml config/gitorious.yml
cp config/broker.yml.example config/broker.yml

create a secret key

Create a secret key for the session cookie. (we need this key later in the next step.

apg -m 32

setup config/gitorious.yml

Setup gitorious in /var/www/gitorious/config/gitorious.yml:


repository_base_path: "/var/www/gitorious/repositories"

is_gitorious_dot_org: false


Create the logroate configuration and the init scripts.

cp /var/www/gitorious/doc/templates/ubuntu/gitorious-logrotate /etc/logrotate.d/

cp /var/www/gitorious/doc/templates/ubuntu/git-ultrasphinx /etc/init.d/
cp /var/www/gitorious/doc/templates/ubuntu/git-daemon /etc/init.d/
cp /var/www/gitorious/doc/templates/ubuntu/git-poller /etc/init.d/

Set in /etc/init.d/git-daemon and in /etc/init.d/git-poller the RUBY_HOME variable to “/usr”

#RUBY_HOME="/opt/ruby-enterprise"
RUBY_HOME="/usr"


Activate the new gitorious init scripts

update-rc.d git-daemon defaults && \
    update-rc.d git-poller defaults && \
    update-rc.d git-ultrasphinx defaults

fix gem spec files

Try to start /etc/init.d/git-ultrasphinx
If you get the following error message you have to fix the spec files:

root@ubuntu03:/var/www/gitorious# /etc/init.d/git-ultrasphinx start
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
(in /var/www/gitorious)
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
rake aborted!
Bundler couldn't find some gems. Did you run <code>bundle install</code>?
/var/www/gitorious/Rakefile:4
(See full trace by running task with --trace)

Change the “s.date” in the following spec files:

/var/lib/gems/1.8/specifications/json-1.5.4.gemspec
/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec
/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec

from:

s.date = %q{2011-08-31 00:00:00.000000000Z}

in:

s.date = %q{2012-08-31}


fix git-ultrasphinx

You have to fix the /var/www/gitorious/config/boot.rb file,
if you get the following error message after starting /etc/init.d/git-ultrasphinx.

root@ubuntu03:/var/www/gitorious# /etc/init.d/git-ultrasphinx start
(in /var/www/gitorious)
rake aborted!
uninitialized constant ActiveSupport::Dependencies::Mutex
/var/www/gitorious/Rakefile:10:in `require'
(See full trace by running task with --trace)

Add the following line in /var/www/gitorious/config/boot.rb.


# Don't change this file!
# Configure your app in config/environment.rb and config/environments/*.rb

require 'thread'


Apache configuration

Disable the default vhost configuration

a2dissite default
a2dissite default-ssl

create a new vhost configuration

cat >> /etc/apache2/sites-available/gitorious <<EOF
<VirtualHost *:80>
    ServerName gitorious.linux-adm.de         # change the hostname!
    DocumentRoot /var/www/gitorious/public
</VirtualHost>
EOF

create a new SSL vhost configuration

cat >> /etc/apache2/sites-available/gitorious-ssl <<EOF
<IfModule mod_ssl.c>
    <VirtualHost _default_:443>
        ServerName gitorious.linux-adm.de   # change the hostname!
        DocumentRoot /var/www/gitorious/public
        SSLEngine on
        SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
        SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
        BrowserMatch ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
    </VirtualHost>
</IfModule>
EOF

a2ensite gitorious
a2ensite gitorious-ssl
a2enmod ssl   # enable SSL

create MySQL User for gitorious

mysql -u root -p
GRANT ALL PRIVILEGES ON *.* TO 'gitorious'@'localhost' IDENTIFIED BY 'XXXX' WITH GRANT OPTION;
FLUSH PRIVILEGES;

change the password XXXX!

Add the MySQL User to the production section in /var/www/gitorious/config/database.yml


production:
adapter: mysql
database: gitorious_production
username: gitorious
password: XXXX # change the password!
host: localhost
encoding: utf8


Starting gitorious

Run the following commands as user git.

export RAILS_ENV=production && 
bundle exec rake db:create && 
bundle exec rake db:migrate && 
bundle exec rake ultrasphinx:bootstrap

fix ultrasphinx/production.conf

If you get the following error message, you have to fix the /var/www/gitorious/ultrasphinx/production.conf file.

export RAILS_ENV=production && 
bundle exec rake db:create && 
bundle exec rake db:migrate && 
bundle exec rake ultrasphinx:bootstrap

Sphinx 0.9.9-release (r2117)
Copyright (c) 2001-2009, Andrew Aksyonoff

using config file '/var/www/gitorious/config/ultrasphinx/production.conf'...
WARNING: key 'address' is deprecated in /var/www/gitorious/config/ultrasphinx/production.conf line 11; use 'listen' instead.
listening on all interfaces, port=3312
WARNING: index 'main': preload: failed to open /var/www/gitorious/db/sphinx//sphinx_index_main.sph: No such file or directory; NOT SERVING
FATAL: no valid indexes to serve
Failed to start
Done
Please restart your application containers

Change in /var/www/gitorious/config/ultrasphinx/production.conf the word “base_tags” to “tags”.

git-ultrasphinx cronjob

Add the following cronjob for the git user.

crontab -e -u git


* * * * * cd /var/www/gitorious && /usr/local/bin/bundle exec rake ultrasphinx:index RAILS_ENV=production


create an admin user

git@ubuntu03:/var/www/gitorious$ env RAILS_ENV=production ruby1.8 script/create_admin
Type in Administrator's e-mail: 
admin@gitorious.linux-adm.de
Type in Administrator's password: 
test
Admin user created successfully.

After a reboot of your system the gitorious web front-end should be available.

Since there is no patch for Ruby Enterprise Edition 2011.03 to prevent the Hash Collision DoS Attack, I’ve quickly ported ruby 1.8.7 patchlevel 357 patch. Here it is:

You can view it or download it from github.

Disclaimer: This is provided as is, no guarantees are provided, etc.

NoMethodError: undefined method `each' for nil:NilClass

(NoMethodError) "undefined method `each' for nil:NilClass"

/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/mysql2-0.2.4/lib/active_record/connection_adapters/mysql2_adapter.rb:628:in `select'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/connection_adapters/abstract/query_cache.rb:54:in `block in select_all'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `cache_sql'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/connection_adapters/abstract/query_cache.rb:54:in `select_all'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/base.rb:467:in `find_by_sql'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/relation.rb:64:in `to_a'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/relation/finder_methods.rb:143:in `all'

This week, after 3 months in the works, we’ve finally released version 1.7.0 of DbCharmer ruby gem – Rails plugin that significantly extends ActiveRecord’s ability to work with multiple databases and/or database servers by adding features like multiple databases support, master/slave topologies support, sharding, etc.

New features in this release:

• Rails 3.0 support. We’ve worked really hard to bring all the features we supported in Rails 2.X to the new version of Rails and now I’m proud that we’ve implemented them all and the implementation looks much cleaner and more universal (all kinds of relations in rails 3 work in exactly the same way and we do not need to implement connection switching for all kinds of weird corner-cases in ActiveRecord).
• Forced Slave Reads functionality. Now we could have models with slaves that are not used by default, but could be turned on globally (per-controller, per-action or in a block). This is a new feature that brings our master/slave routing capabilities to a really new level – we could now use it for a really mission-critical models on demand and not be afraid of breaking major functionality of our applications by switching them to slave reads.
• Lots of changes were made in the structure of our code and tests to make sure it would be much easier for new developers to understand DbCharmer internals and make changes in its code.

Along with the new release we’ve got a brand new web site. You can find much better, cleaner and, most importantly, correct documentation for the library on the web site. We’ll be adding more examples, will try to add more in-depth explanation of our core functions, etc.

If you have any questions about the release, feel free to ask them in our new mailing list: DbCharmer Users Group.

For more updates in our releases, you can follow @DbCharmer on Twitter.

XKCD (as usual) makes a very good point – this time about password strength, and I reckon it’s something app developers need to consider urgently. Geeks can debate the exact amount of entropy, but that’s not really the issue: insisting on mixed upper/lower and/or non-alpha and/or numerical components to a user password does not really improve security, and definitely makes life more difficult for users.

So basically, the functions that do a “is this a strong password” should seriously reconsider their approach, particularly if they’re used to have the app decide whether to accept the password as “good enough” at all.

I’ll be joining the NovaRUG (Northern Virginia Ruby Users’ Group) on Thursday to talk about MySQL performance. See their blog for the details and how to RSVP.

Related posts:

1. Speaking at MySQL Meetup in Northern Virginia
2. Speaking at EdUI Conference 2009
3. Speaking at Surge 2010
4. Speaking at CPOSC 2009
5. Speaking at Enterprise LAMP Summit 2009

Before we get started here, let me state that I am using Ruby 1.9.1 (I refuse to look back!), and that I have not tested this solution on Ruby 1.8.6, but it should work there as well, though I may have some 1.9-isms in my code. Should be easy enough to spot.

I am working on writing an application in Ruby that can talk to an Windows application that has an ActiveX COM Automation object exposed. Ruby is basically the wrapper so that I can access the application from the Linux side of the world. So, I am using Ruby’s DRb to bridge those worlds because, after all, I am the Linux Bloke!

Well, as you may have guessed, I ran into problems with this approach. I simply could not call the COM objects from a call initiated with DRb, though I could call them directly just fine. After scratching my head a bit, I figured it out.

The win32ole module that runs on the Windows side of the world in Ruby only wants to run in the same thread that it was started in. win32ole is simply not thread-safe, and this has to do in large part to how ActiveX works under Windows. No need to delve into the gory details as we want code that works already!

DRb is very much all about threads. The DRb Server runs in a separate thread, and threads are launched each time a DRb request comes in. Threads abound like crazy! After all, it is very clear that the implementation of DRb was based, in part, on the Java threading model and Java’s RMI. But we knew that. We know that Ruby Threads parrot Java Threads. And I’ve done a lot of work with Java Threads in the past and almost feel a bit of “déjà vu” in working with them in Ruby. Oh the days…

But I digress.

We have a major problem here. How do we get around it, without having to throw out DRb and doing something funky like writing some custom RPC bit just to make Windows happy?

Well, as you may have guess, the Linux Bloke created the very solution you need!! Funnel!

Funnel works by wrapping a given object with a “meta” object that can then be called from any thread. All the calls are actually queued up and processed by the thread the target object wants to run in. The calling threads block until the target object returns the call, and the result objects are stuffed somewhere so that the calling thread can find them.

It’s all very transparent and you need not do anything special — much. You will need to call process_funnel_messages() in the funneled thread. And you may do this once in which case process_funnel_messages() will loop forever and never return, or you can call it at regular intervals if you need to do other processing in that same thread.

You, of course, can use Funnel anywhere you need to funnel calls from multiple threads to a single thread to access something that is not inherently thread-safe or thread-aware.

The downloadable code is posted here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

=begin rdoc
Funnel created by Fred Mitchell (LinuxBloke.com) on 2010-06-05                         
 
=Funnel -- funnel calls to an object to a specific thread that created said object.    
 
With some systems, like win32ole, the system basically wants to run on the same thread
the system was started on. To facilitate that need in a multi-threaded environment,
we create the Funnel.                                                                  
 
The Funnel wrapper on an object will basically intercept all method calls and
funnel those calls to the wrapped object in the thread it was created in. The
caller thread will basically block until the Funnel calls the target object's method
and will be given, as a return, the result object of that call.                        
 
The Funnel thread will basically sit in a loop waiting for something to come in,
and wake up to process the entries, then go back to sleep until the next ones come
in.                                                                                    
 
Any exceptions (or errors) that occur in the Funnel shall be
thrown to the caller thread, as though the exception took place in that thread.        
 
This code is released under the GPLv3.                                                 
 
=end                         
 
module Funnel
  class Wrapper
    def initialize(target)
      @targetOb = target
      @targetThr = Thread.current
      @targetThr[:methQueue] = [] if @targetThr[:methQueue].nil?
    end                                                                                
 
    def method_missing(meth, *parms)
      Thread.current[:methResult] = :nothing_yet
      @targetThr[:methQueue] << [@targetOb, meth, Thread.current, parms]               
 
      # Thing is, we may have gotten a response already!
      while Thread.current[:methResult] == :nothing_yet
        if @targetThr.stop?
          @targetThr.wakeup
          # Thread.stop
        end
        Thread.pass
      end
      Thread.current[:methResult]
    end
  end                                                                                  
 
  # Called by the orginal thread to process object messages.
  # This function never returns.
  def process_funnel_messages(loop_forever = true)
    begin
      meth = nil
      (ob, meth, thr, parms) = Thread.current[:methQueue].shift unless Thread.current[\
:methQueue].nil?
      unless meth.nil?
        begin
          thr[:methResult] = ob.send(meth, *parms)
          thr.run
        rescue
          thr.raise($!)
        end
      else
        Thread.stop if loop_forever
      end
    end while loop_forever
  end                                                                                  
 
  def wrap(target)
    Wrapper.new(target)
  end
end

And here is an example of its use:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

require 'funnel'                                                                       
include Funnel                                                                         
 
class StupidThreadUnsafeThing                                                          
  def callme                                                                           
    puts "*** I've been called. My thread is"                                          
    p Thread.current                                                                   
    puts                                                                               
  end                                                                                  
end                                                                                    
 
stut = StupidThreadUnsafeThing.new                                                     
 
# This is the easy to use wrapper                                                      
fstut = wrap stut                                                                      
 
stut.callme                                                                            
 
Thread.new do                                                                          
  10.times do |i|                                                                      
    sleep 1                                                                            
    Thread.new {                                                                       
      puts "XXX #{i} calling stut from thread"                                         
      p Thread.current                                                                 
      fstut.callme                                                                     
    }                                                                                  
  end                                                                                  
  exit                                                                                 
end                                                                                    
 
# Here we loop forever processing messages.                                            
# Optionally, we could call this repeateady                                            
# to process messages by using a parameter of                                          
# "false".                                                                             
process_funnel_messages

This code is fairly straightforward, as you can see. If there is enough interest, I’ll consider turning this into a gem.

Back in November 2009 I was working on a project to port Scribd.com code base to Rails 2.2 and noticed that some old plugins we were using in 2.1 were abandoned by their authors. Some of them were just removed from the code base, but one needed a replacement – that was an old plugin called acts_as_readonlyable that helped us to distribute our queries among a cluster of MySQL slaves. There were some alternatives but we didn’t like them for one or another reasons so we’ve decided to go with creating our own ActiveRecord plugin, that would help us scale our databases out. That’s the story behind the first release of DbCharmer.

Today, six months after the first release of the gem and we’ve moved it to gemcutter (which is now the official gems hosting) and we’re already at version 1.6.11. The gem was downloaded more than 2000 times. There are (at least) 10+ large users that rely on this gem to scale their products out. And (this is the most exciting) we’ve added tons of new features to the product.

Here are the main features added since the first release:

• Much better multi-database migrations support including default migrations connection changing.
• We’ve added ActiveRecord associations preload support that makes it possible to move eager loading queries to the same connection where your finder queries go to.
• We’ve improved ActiveRecord’s query logging feature and now you can see what connections your queries executed on (and yes, all those improvements are colorized ).
• We’ve added an ability to temporary remap any ActiveRecord connections to any other connections for a block of code (really useful when you need to make sure all your queries would go to some non-default slave and you do not want to mess with all your models).
• The most interesting change: we’ve implemented some basic sharding functionality in ActiveRecord which currently is being used in production in our application.

As you can see now DbCharmer helps you to do three major scalability tasks in your Rails projects:

1. Master-Slave clusters to scale out your Rails models reads.
2. Vertical sharding by moving some of your models to a separate (maybe even dedicated) servers and still keep using AR associations
3. Horizontal sharding by slicing your models data to pieces and placing those pieces into different databases and/or servers.

So, If you didn’t check DbCharmer out yet and you’re working on some large rails project that is (or going to be) facing scalability problems, go read the docs, download/install the gem and prove them that Rails CAN scale!