Archive for the ‘sphinxsearch’ Category

how to setup gitorious on ubuntu server 11.10

Апрель 4th, 2012

In this tutorial I will describe how to setup gitorious on Ubuntu 11.10. Gitorious – a Ruby on Rails web application – can be used to conclude git projects in an easy to manage user interface. In the README of the gitorious repository I found the evidence “One of the main challenges in Gitorious is its installation process. It is anything but trivial.” – It appears correct ;) During the installation I got some errors by the sphinx search engine and some ruby gems. You will find the workarounds in this tutorial.

This tutorial is based on the gitorious – UbuntuInstallation tutorial.

In this setup I will use the hostname gitorious.linux-adm.de – change this hostname in your installation.

software installation

Required packages

apt-get install build-essential zlib1g-dev tcl-dev libexpat-dev \
    libcurl4-openssl-dev postfix apache2 mysql-server mysql-client \
    apg geoip-bin libgeoip1 libgeoip-dev sqlite3 libsqlite3-dev \
    imagemagick libpcre3 libpcre3-dev zlib1g zlib1g-dev libyaml-dev \
    libmysqlclient15-dev apache2-dev libonig-dev ruby-dev rubygems \
    libopenssl-ruby phpmyadmin libdbd-mysql-ruby libmysql-ruby \
    libmagick++-dev  zip unzip memcached git-core git-svn git-doc \
    git-cvs irb libxslt1.1 libxslt1-dev

Ruby, sphinxsearch, stompserver

apt-get install libchronic-ruby librmagick-ruby libdaemons-ruby \
ruby-hoe ruby-echoe libyadis-ruby ruby-mime-types libopenid-ruby \
libdifflcs-ruby libjson-ruby librack-ruby libhmac-ruby rake stompserver \
libapache2-mod-passenger rails sphinxsearch sphinxbase-utils

fetch gitorious

git clone git://gitorious.org/gitorious/mainline.git /var/www/gitorious

install missing ruby gems

cd /var/www/gitorious/
gem install bundler
bundle install

git user

Create a git user with the homedir in /var/www/gitorious/ and set the correct ownership.

adduser --system --home /var/www/gitorious/ --no-create-home --group --shell /bin/bash git
chown git:git -R /var/www/gitorious/

Create a directory for the repositories and a cache directory. Gitorious will add the SSH public keys of the registered users to the authorized_keys file, therefore we have to create a .ssh file and an empty authorized_keys file with the correct permissions.

su - git
git submodule init && \
git submodule update
mkdir -p tmp/pids
mkdir repositories
mkdir tarballs
mkdir .ssh
touch .ssh/authorized_keys
chmod 700 .ssh/
chmod 600 .ssh/authorized_keys

Put gitorious binary on path:

cat >> /var/www/gitorious/.bash_profile <<EOF
export PATH="$PATH:/var/www/gitorious/script/"
EOF

gitorious configuration

prepare configuration files

cd /var/www/gitorious
cp config/database.sample.yml config/database.yml
cp config/gitorious.sample.yml config/gitorious.yml
cp config/broker.yml.example config/broker.yml

create a secret key

Create a secret key for the session cookie. (we need this key later in the next step.

apg -m 32

setup config/gitorious.yml

Setup gitorious in /var/www/gitorious/config/gitorious.yml:


repository_base_path: "/var/www/gitorious/repositories"

cookie_secret: xxx # add a secret key here the command "apg -m 32"

gitorious_client_port should be 80

gitorious_host: gitorious.linux-adm.de # change to your hostname

archive_cache_dir: "/var/www/gitorious/tarballs"

archive_work_dir: "/tmp/tarballs-work"

hide_http_clone_urls: true

is_gitorious_dot_org: false

Create the logroate configuration and the init scripts.

cp /var/www/gitorious/doc/templates/ubuntu/gitorious-logrotate /etc/logrotate.d/

cp /var/www/gitorious/doc/templates/ubuntu/git-ultrasphinx /etc/init.d/
cp /var/www/gitorious/doc/templates/ubuntu/git-daemon /etc/init.d/
cp /var/www/gitorious/doc/templates/ubuntu/git-poller /etc/init.d/

Set in /etc/init.d/git-daemon and in /etc/init.d/git-poller the RUBY_HOME variable to “/usr”

#RUBY_HOME="/opt/ruby-enterprise"
RUBY_HOME="/usr"

Activate the new gitorious init scripts

update-rc.d git-daemon defaults && \
    update-rc.d git-poller defaults && \
    update-rc.d git-ultrasphinx defaults

fix gem spec files

Try to start /etc/init.d/git-ultrasphinx
If you get the following error message you have to fix the spec files:

root@ubuntu03:/var/www/gitorious# /etc/init.d/git-ultrasphinx start
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
(in /var/www/gitorious)
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
rake aborted!
Bundler couldn't find some gems. Did you run <code>bundle install</code>?
/var/www/gitorious/Rakefile:4
(See full trace by running task with --trace)

Change the “s.date” in the following spec files:

/var/lib/gems/1.8/specifications/json-1.5.4.gemspec
/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec
/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec

from:

s.date = %q{2011-08-31 00:00:00.000000000Z}

in:

s.date = %q{2012-08-31}

fix git-ultrasphinx

You have to fix the /var/www/gitorious/config/boot.rb file,
if you get the following error message after starting /etc/init.d/git-ultrasphinx.

root@ubuntu03:/var/www/gitorious# /etc/init.d/git-ultrasphinx start
(in /var/www/gitorious)
rake aborted!
uninitialized constant ActiveSupport::Dependencies::Mutex
/var/www/gitorious/Rakefile:10:in `require'
(See full trace by running task with --trace)

Add the following line in /var/www/gitorious/config/boot.rb.


# Don't change this file!
# Configure your app in config/environment.rb and config/environments/*.rb

require 'thread'

Apache configuration

Disable the default vhost configuration

a2dissite default
a2dissite default-ssl

create a new vhost configuration

cat >> /etc/apache2/sites-available/gitorious <<EOF
<VirtualHost *:80>
    ServerName gitorious.linux-adm.de         # change the hostname!
    DocumentRoot /var/www/gitorious/public
</VirtualHost>
EOF

create a new SSL vhost configuration

cat >> /etc/apache2/sites-available/gitorious-ssl <<EOF
<IfModule mod_ssl.c>
    <VirtualHost _default_:443>
        ServerName gitorious.linux-adm.de   # change the hostname!
        DocumentRoot /var/www/gitorious/public
        SSLEngine on
        SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
        SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
        BrowserMatch ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
    </VirtualHost>
</IfModule>
EOF

a2ensite gitorious
a2ensite gitorious-ssl
a2enmod ssl   # enable SSL

create MySQL User for gitorious

mysql -u root -p
GRANT ALL PRIVILEGES ON *.* TO 'gitorious'@'localhost' IDENTIFIED BY 'XXXX' WITH GRANT OPTION;
FLUSH PRIVILEGES;

change the password XXXX!

Add the MySQL User to the production section in /var/www/gitorious/config/database.yml


production:
adapter: mysql
database: gitorious_production
username: gitorious
password: XXXX # change the password!
host: localhost
encoding: utf8

Starting gitorious

Run the following commands as user git.

export RAILS_ENV=production && 
bundle exec rake db:create && 
bundle exec rake db:migrate && 
bundle exec rake ultrasphinx:bootstrap

fix ultrasphinx/production.conf

If you get the following error message, you have to fix the /var/www/gitorious/ultrasphinx/production.conf file.

export RAILS_ENV=production && 
bundle exec rake db:create && 
bundle exec rake db:migrate && 
bundle exec rake ultrasphinx:bootstrap

Sphinx 0.9.9-release (r2117)
Copyright (c) 2001-2009, Andrew Aksyonoff

using config file '/var/www/gitorious/config/ultrasphinx/production.conf'...
WARNING: key 'address' is deprecated in /var/www/gitorious/config/ultrasphinx/production.conf line 11; use 'listen' instead.
listening on all interfaces, port=3312
WARNING: index 'main': preload: failed to open /var/www/gitorious/db/sphinx//sphinx_index_main.sph: No such file or directory; NOT SERVING
FATAL: no valid indexes to serve
Failed to start
Done
Please restart your application containers

Change in /var/www/gitorious/config/ultrasphinx/production.conf the word “base_tags” to “tags”.

git-ultrasphinx cronjob

Add the following cronjob for the git user.

crontab -e -u git


* * * * * cd /var/www/gitorious && /usr/local/bin/bundle exec rake ultrasphinx:index RAILS_ENV=production

create an admin user

git@ubuntu03:/var/www/gitorious$ env RAILS_ENV=production ruby1.8 script/create_admin
Type in Administrator's e-mail: 
admin@gitorious.linux-adm.de
Type in Administrator's password: 
test
Admin user created successfully.

After a reboot of your system the gitorious web front-end should be available.


PlanetMySQL Voting: Vote UP / Vote DOWN

how to setup gitorious on ubuntu server 11.10

Апрель 4th, 2012

In this tutorial I will describe how to setup gitorious on Ubuntu 11.10. Gitorious – a Ruby on Rails web application – can be used to conclude git projects in an easy to manage user interface. In the README of the gitorious repository I found the evidence “One of the main challenges in Gitorious is its installation process. It is anything but trivial.” – It appears correct ;) During the installation I got some errors by the sphinx search engine and some ruby gems. You will find the workarounds in this tutorial.

This tutorial is based on the gitorious – UbuntuInstallation tutorial.

In this setup I will use the hostname gitorious.linux-adm.de – change this hostname in your installation.

software installation

Required packages

apt-get install build-essential zlib1g-dev tcl-dev libexpat-dev \
    libcurl4-openssl-dev postfix apache2 mysql-server mysql-client \
    apg geoip-bin libgeoip1 libgeoip-dev sqlite3 libsqlite3-dev \
    imagemagick libpcre3 libpcre3-dev zlib1g zlib1g-dev libyaml-dev \
    libmysqlclient15-dev apache2-dev libonig-dev ruby-dev rubygems \
    libopenssl-ruby phpmyadmin libdbd-mysql-ruby libmysql-ruby \
    libmagick++-dev  zip unzip memcached git-core git-svn git-doc \
    git-cvs irb libxslt1.1 libxslt1-dev

Ruby, sphinxsearch, stompserver

apt-get install libchronic-ruby librmagick-ruby libdaemons-ruby \
ruby-hoe ruby-echoe libyadis-ruby ruby-mime-types libopenid-ruby \
libdifflcs-ruby libjson-ruby librack-ruby libhmac-ruby rake stompserver \
libapache2-mod-passenger rails sphinxsearch sphinxbase-utils

fetch gitorious

git clone git://gitorious.org/gitorious/mainline.git /var/www/gitorious

install missing ruby gems

cd /var/www/gitorious/
gem install bundler
bundle install

git user

Create a git user with the homedir in /var/www/gitorious/ and set the correct ownership.

adduser --system --home /var/www/gitorious/ --no-create-home --group --shell /bin/bash git
chown git:git -R /var/www/gitorious/

Create a directory for the repositories and a cache directory. Gitorious will add the SSH public keys of the registered users to the authorized_keys file, therefore we have to create a .ssh file and an empty authorized_keys file with the correct permissions.

su - git
git submodule init && \
git submodule update
mkdir -p tmp/pids
mkdir repositories
mkdir tarballs
mkdir .ssh
touch .ssh/authorized_keys
chmod 700 .ssh/
chmod 600 .ssh/authorized_keys

Put gitorious binary on path:

cat >> /var/www/gitorious/.bash_profile <<EOF
export PATH="$PATH:/var/www/gitorious/script/"
EOF

gitorious configuration

prepare configuration files

cd /var/www/gitorious
cp config/database.sample.yml config/database.yml
cp config/gitorious.sample.yml config/gitorious.yml
cp config/broker.yml.example config/broker.yml

create a secret key

Create a secret key for the session cookie. (we need this key later in the next step.

apg -m 32

setup config/gitorious.yml

Setup gitorious in /var/www/gitorious/config/gitorious.yml:


repository_base_path: "/var/www/gitorious/repositories"

cookie_secret: xxx # add a secret key here the command "apg -m 32"

gitorious_client_port should be 80

gitorious_host: gitorious.linux-adm.de # change to your hostname

archive_cache_dir: "/var/www/gitorious/tarballs"

archive_work_dir: "/tmp/tarballs-work"

hide_http_clone_urls: true

is_gitorious_dot_org: false

Create the logroate configuration and the init scripts.

cp /var/www/gitorious/doc/templates/ubuntu/gitorious-logrotate /etc/logrotate.d/

cp /var/www/gitorious/doc/templates/ubuntu/git-ultrasphinx /etc/init.d/
cp /var/www/gitorious/doc/templates/ubuntu/git-daemon /etc/init.d/
cp /var/www/gitorious/doc/templates/ubuntu/git-poller /etc/init.d/

Set in /etc/init.d/git-daemon and in /etc/init.d/git-poller the RUBY_HOME variable to “/usr”

#RUBY_HOME="/opt/ruby-enterprise"
RUBY_HOME="/usr"

Activate the new gitorious init scripts

update-rc.d git-daemon defaults && \
    update-rc.d git-poller defaults && \
    update-rc.d git-ultrasphinx defaults

fix gem spec files

Try to start /etc/init.d/git-ultrasphinx
If you get the following error message you have to fix the spec files:

root@ubuntu03:/var/www/gitorious# /etc/init.d/git-ultrasphinx start
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
(in /var/www/gitorious)
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/json-1.5.4.gemspec]: invalid date format in specification: "2011-08-31 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec]: invalid date format in specification: "2011-08-11 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec]: invalid date format in specification: "2011-10-21 00:00:00.000000000Z"
rake aborted!
Bundler couldn't find some gems. Did you run <code>bundle install</code>?
/var/www/gitorious/Rakefile:4
(See full trace by running task with --trace)

Change the “s.date” in the following spec files:

/var/lib/gems/1.8/specifications/json-1.5.4.gemspec
/var/lib/gems/1.8/specifications/childprocess-0.2.1.gemspec
/var/lib/gems/1.8/specifications/paperclip-2.4.5.gemspec

from:

s.date = %q{2011-08-31 00:00:00.000000000Z}

in:

s.date = %q{2012-08-31}

fix git-ultrasphinx

You have to fix the /var/www/gitorious/config/boot.rb file,
if you get the following error message after starting /etc/init.d/git-ultrasphinx.

root@ubuntu03:/var/www/gitorious# /etc/init.d/git-ultrasphinx start
(in /var/www/gitorious)
rake aborted!
uninitialized constant ActiveSupport::Dependencies::Mutex
/var/www/gitorious/Rakefile:10:in `require'
(See full trace by running task with --trace)

Add the following line in /var/www/gitorious/config/boot.rb.


# Don't change this file!
# Configure your app in config/environment.rb and config/environments/*.rb

require 'thread'

Apache configuration

Disable the default vhost configuration

a2dissite default
a2dissite default-ssl

create a new vhost configuration

cat >> /etc/apache2/sites-available/gitorious <<EOF
<VirtualHost *:80>
    ServerName gitorious.linux-adm.de         # change the hostname!
    DocumentRoot /var/www/gitorious/public
</VirtualHost>
EOF

create a new SSL vhost configuration

cat >> /etc/apache2/sites-available/gitorious-ssl <<EOF
<IfModule mod_ssl.c>
    <VirtualHost _default_:443>
        ServerName gitorious.linux-adm.de   # change the hostname!
        DocumentRoot /var/www/gitorious/public
        SSLEngine on
        SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
        SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
        BrowserMatch ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
    </VirtualHost>
</IfModule>
EOF

a2ensite gitorious
a2ensite gitorious-ssl
a2enmod ssl   # enable SSL

create MySQL User for gitorious

mysql -u root -p
GRANT ALL PRIVILEGES ON *.* TO 'gitorious'@'localhost' IDENTIFIED BY 'XXXX' WITH GRANT OPTION;
FLUSH PRIVILEGES;

change the password XXXX!

Add the MySQL User to the production section in /var/www/gitorious/config/database.yml


production:
adapter: mysql
database: gitorious_production
username: gitorious
password: XXXX # change the password!
host: localhost
encoding: utf8

Starting gitorious

Run the following commands as user git.

export RAILS_ENV=production && 
bundle exec rake db:create && 
bundle exec rake db:migrate && 
bundle exec rake ultrasphinx:bootstrap

fix ultrasphinx/production.conf

If you get the following error message, you have to fix the /var/www/gitorious/ultrasphinx/production.conf file.

export RAILS_ENV=production && 
bundle exec rake db:create && 
bundle exec rake db:migrate && 
bundle exec rake ultrasphinx:bootstrap

Sphinx 0.9.9-release (r2117)
Copyright (c) 2001-2009, Andrew Aksyonoff

using config file '/var/www/gitorious/config/ultrasphinx/production.conf'...
WARNING: key 'address' is deprecated in /var/www/gitorious/config/ultrasphinx/production.conf line 11; use 'listen' instead.
listening on all interfaces, port=3312
WARNING: index 'main': preload: failed to open /var/www/gitorious/db/sphinx//sphinx_index_main.sph: No such file or directory; NOT SERVING
FATAL: no valid indexes to serve
Failed to start
Done
Please restart your application containers

Change in /var/www/gitorious/config/ultrasphinx/production.conf the word “base_tags” to “tags”.

git-ultrasphinx cronjob

Add the following cronjob for the git user.

crontab -e -u git


* * * * * cd /var/www/gitorious && /usr/local/bin/bundle exec rake ultrasphinx:index RAILS_ENV=production

create an admin user

git@ubuntu03:/var/www/gitorious$ env RAILS_ENV=production ruby1.8 script/create_admin
Type in Administrator's e-mail: 
admin@gitorious.linux-adm.de
Type in Administrator's password: 
test
Admin user created successfully.

After a reboot of your system the gitorious web front-end should be available.


PlanetMySQL Voting: Vote UP / Vote DOWN

Speaking at Linux Conference in Wellington, New Zealand

Январь 18th, 2010

I’ll be speaking at the data retrieval miniconf at Linux Conference in Wellington, New Zealand (Full Text Search with MySQL, Program)
I’ll cover some new sphinx search features (online updates)


PlanetMySQL Voting: Vote UP / Vote DOWN