Join us for a live and interactive webinar session where we will demonstrate how to start an evaluation of the MySQL Cluster database in 5 easy steps, and then how to expand your deployment for web & telecoms-scale services.

Just register here:

http://www.mysql.com/news-and-events/web-seminars/display-566.html

Getting Started will describe how to:

• Get the software
• Install it
• Configure it
• Run it
• Test it

Scaling for HA and the web will describe how to:

• Review the requirements for a HA configuration
• Install the software on more servers
• Update & extend the configuration from a single host to 4
• Roll out the changes
• On-line scaling to add further nodes

When: Wednesday, September 08, 2010: 09:00 Pacific time (America)

The presentation will be approximately 45 minutes long followed by Q&A.

export PKG_PATH="ftp://openbsd.mirrors.tds.net/pub/OpenBSD/4.7/packages/amd64/"
pkg_add -i -v wget
wget http://dbbenchmark.googlecode.com/files/dbbenchmark-version-0.1.beta_rev26.tar.gz
pkg_add -i -v python
Ambiguous: choose package for python
 a       0:
         1: python-2.4.6p2
         2: python-2.5.4p3
         3: python-2.6.3p1
Your choice: 2

pkg_add -i -v py-mysql
pkg_add -i -v mysql
pkg_add -i -v mysql-server
ln -s /usr/local/bin/python2.5 /usr/bin/python
gzip -d dbbenchmark-version-0.1.beta_rev26.tar.gz
tar -xvf dbbenchmark-version-0.1.beta_rev26.tar
cd dbbenchmark-version-0.1.beta_rev26
./dbbenchmark.py --print-sql
 - login to mysql and execute sql commands
./dbbenchmark.py

sync_binlog=1 is a problem in itself, because it makes the server not just do one fsync per commit, but several and that’s serious overhead. sync_binlog is actually not a boolean but a “fsync binlog every N commits” where 0 meaning “never”. So you could set it to 10 (fsync every 10 commits) and thus reduce the loss a little bit while not doing too much harm to performance. But it’s not ideal and won’t always prevent the above error.

So if we have system with single hard drive doing 200 IO/ssc, 48G buffer pool which is 90% dirty and completely random page writes we’ll look at 13500 seconds or about 5min per 1GB of Buffer pool size.
This is worse case scenario though it is quite common in practice to see shutdown time of about 1min per GB of buffer pool per hard drive.

Baron has written a nice post how to decrease innodb shutdown time which you may want to read on this topic.

Entry posted by peter | No comment

Add to: | | | |

Measuring Load Performance

We generated several different datasets to measure the performance of TokuDB when doing a LOAD DATA INFILE … command. To characterize performance, we vary

• rows to load
• keys per row
• row length (including keys)

All generated keys, including the primary, are random, 8-byte values. The remaining data, needed to pad out the row length to specified length, is text.

Two files files are produced as part of data generation.

1. data file, containing ‘|’ separated fields
2. sql file, containing the CREATE TABLE command corresponding to the generated data

For instance, if the number of keys is 3 and the row length is 256 bytes, the following SQL statement is produced:

     CREATE TABLE load_table (\
         val0 BIGINT UNSIGNED NOT NULL,\
         val1 BIGINT UNSIGNED NOT NULL,\
         val2 BIGINT UNSIGNED NOT NULL,\
         pad VARCHAR(232) NOT NULL,\
         PRIMARY KEY (val0),\
         KEY valkey1 (val1),\
         KEY valkey2 (val2)\
         ) ENGINE=tokudb

We can make the data generation program available if anyone is interested.

Load Test

A simple shell script

• creates the test table
• performs a LOAD DATA INFILE <datafile> INTO TABLE load_table FIELDS TERMINATED BY ‘|’
• returns execution time

For the experiments to be meaningful, we created datasets that do not fit in memory.

Results

We ran our benchmark on an Amazon Web Services c1.large node with 8 cores and 7 GB of memory. The test loads 100M rows (NOT pre-sorted). The data file was on a 2 disk RAID-0, the MySQL DB files on a different 2 disk RAID-0.

TokuDB Version 3 (~single-threaded) v. TokuDB Version 4 (multi-threaded)

Other metrics

Several metrics can be used to measure performance:

• rows per second : data insert rate
• key-value pairs per second : indicates how fast the primary and secondary indexes are being created
• MB/s : how much raw data is being added to the database

Metrics for TokuDB v4:

These results show

1. significant parallelization (we believe larger CPU core count machines will see even larger benefits)
2. a significant jump in absolute load performance
3. speed-ups are not limited to tables with many keys – even the 1 key tables are 5-7x faster

We will report further results, especially speedups on larger CPU count machines, as they become available.