Archive for Июль, 2009

MySQL Midpoint Stored Function

Июль 31st, 2009
MySQL GIS functions don't seem to include a Midpoint function; it's trivial to write one, but I thought posting this might save someone a few minutes. Enjoy!

DROP FUNCTION IF EXISTS midpoint;
DELIMITER //
CREATE FUNCTION midpoint(line GEOMETRY)
RETURNS POINT DETERMINISTIC
BEGIN

IF NumPoints(line) != 2 THEN
RETURN NULL;
END IF;

RETURN GeomFromText(CONCAT('POINT(',(X(StartPoint(line)) +X(EndPoint(line))) / 2,' ',(Y(StartPoint(line))+Y(EndPoint(line))) / 2,')'));
END //
DELIMITER ;

Copying InnoDB tables between servers

Июль 31st, 2009

The feature I announced some time ago http://www.mysqlperformanceblog.com/2009/06/08/impossible-possible-moving-innodb-tables-between-servers/ is now available in our latest releases of XtraBackup 0.8.1 and XtraDB-6.

Now I am going to show how to use it (the video will be also available on percona.tv).
Let's take tpcc schema and running standard MySQL ® 5.0.83, and assume we want to copy order_line table to different server. Note I am going to do it online, no needs to lock or shutdown server.

To export table you need XtraBackup, and you can just specify table or table by mask:

CODE:
  1. xtrabackup  --defaults-file=/etc/my.reg.cnf --backup --tables=tpcc.order_line* --target-dir=/data/vadim/mysql/export/
  2.  
  3. xtrabackup: tables regcomp(): Success
  4. xtrabackup  Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64)
  5. xtrabackup: uses posix_fadvise().
  6. xtrabackup: cd to /bench/mysqldata
  7. xtrabackup: Target instance is assumed as followings.
  8. xtrabackup:   innodb_data_home_dir = ./
  9. xtrabackup:   innodb_data_file_path = ibdata1:10M:autoextend
  10. xtrabackup:   innodb_log_group_home_dir = ./
  11. xtrabackup:   innodb_log_files_in_group = 3
  12. xtrabackup:   innodb_log_file_size = 536870912
  13. xtrabackup: use O_DIRECT
  14. >> log scanned up to (3 767617628)
  15. Copying ./ibdata1
  16.      to /data/vadim/mysql/export//ibdata1
  17. >> log scanned up to (3 769009554)
  18.         ...done
  19. Copying ./mysql/ibbackup_binlog_marker.ibd is skipped.
  20. Copying ./tpcc/stock.ibd is skipped.
  21. Copying ./tpcc/warehouse.ibd is skipped.
  22. Copying ./tpcc/new_orders.ibd is skipped.
  23. Copying ./tpcc/order_line.ibd
  24.      to /data/vadim/mysql/export//tpcc/order_line.ibd
  25. >> log scanned up to (3 770393658)
  26. ...
  27. >> log scanned up to (3 844882683)
  28.         ...done
  29. Copying ./tpcc/district.ibd is skipped.
  30. Copying ./tpcc/orders.ibd is skipped.
  31. Copying ./tpcc/item.ibd is skipped.
  32. Copying ./tpcc/customer.ibd is skipped.
  33. Copying ./tpcc/history.ibd is skipped.
  34. xtrabackup: The latest check point (for incremental): '3:763362037'
  35. >> log scanned up to (3 845737724)
  36. xtrabackup: Stopping log copying thread.
  37. xtrabackup: Transaction log of lsn (3 763355707) to (3 845737724) was copied.

Now in /data/vadim/mysql/export we have backup but only with ibdata1 and order_line.ibd files

Second step is to prepare backup, but with special option "export"

CODE:
  1. xtrabackup  --defaults-file=/etc/my.reg.cnf --prepare --export --use-memory=8G --target-dir=/data/vadim/mysql/export/
  2. xtrabackup  Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64)
  3. xtrabackup: cd to /data/vadim/mysql/export/
  4. xtrabackup: This target seems to be not prepared yet.
  5. xtrabackup: xtrabackup_logfile detected: size=92684288, start_lsn=(3 763355707)
  6. xtrabackup: Temporary instance for recovery is set as followings.
  7. xtrabackup:   innodb_data_home_dir = ./
  8. xtrabackup:   innodb_data_file_path = ibdata1:10M:autoextend
  9. xtrabackup:   innodb_log_group_home_dir = ./
  10. xtrabackup:   innodb_log_files_in_group = 1
  11. xtrabackup:   innodb_log_file_size = 92684288
  12. xtrabackup: Starting InnoDB instance for recovery.
  13. xtrabackup: Using 8589934592 bytes for buffer pool (set by --use-memory parameter)
  14. InnoDB: Log scan progressed past the checkpoint lsn 3 763355707
  15. 090730 23:22:43  InnoDB: Database was not shut down normally!
  16. InnoDB: Starting crash recovery.
  17. InnoDB: Reading tablespace information from the .ibd files...
  18. InnoDB: Doing recovery: scanned up to log sequence number 3 768598528 (6 %)
  19. ....
  20. InnoDB: Doing recovery: scanned up to log sequence number 3 845737724 (99 %)
  21. InnoDB: 12 transaction(s) which must be rolled back or cleaned up
  22. InnoDB: in total 107 row operations to undo
  23. InnoDB: Trx id counter is 0 1560320
  24. ....
  25.  
  26. id 0 1559932, 13 rows to undo
  27.  
  28. InnoDB: Rolling back of trx id 0 1559932 completed
  29. 090730 23:23:35  InnoDB: Rolling back trx with id 0 1559890, 30 rows to undo
  30.  
  31. InnoDB: Rolling back of trx id 0 1559890 completed
  32. 090730 23:23:35  InnoDB: Rollback of non-prepared transactions completed
  33. 090730 23:25:32  InnoDB: Shutdown completed; log sequence number 3 852825486

When it's done we have two files order_line.ibd and order_line.exp in directory /data/vadim/mysql/export/tpcc. ibd is regular InnoDB ® file, and exp is file with special export information. Both files should be copied to remote server.

Now on remote server you have to run MySQL server ® with XtraDB6 storage engine, you can take one of our binary builds with 5.1.36.

On new server we run
set global innodb_expand_import=1; to put XtraDB into extended import mode, and now we need to create empty table with the same table definition as on old servers:

CODE:
  1. CREATE DATABASE "testimport";
  2. USE "testimport"
  3. CREATE TABLE `order_line` (
  4.   `ol_o_id` int(11) NOT NULL,
  5.   `ol_d_id` tinyint(4) NOT NULL,
  6.   `ol_w_id` smallint(6) NOT NULL,
  7.   `ol_number` tinyint(4) NOT NULL,
  8.   `ol_i_id` int(11) default NULL,
  9.   `ol_supply_w_id` smallint(6) default NULL,
  10.   `ol_delivery_d` datetime default NULL,
  11.   `ol_quantity` tinyint(4) default NULL,
  12.   `ol_amount` decimal(6,2) default NULL,
  13.   `ol_dist_info` char(24) default NULL,
  14.   PRIMARY KEY  (`ol_w_id`,`ol_d_id`,`ol_o_id`,`ol_number`),
  15.   KEY `fkey_order_line_2` (`ol_supply_w_id`,`ol_i_id`)
  16. ) ENGINE=InnoDB DEFAULT CHARSET=latin1

Next step is to remove created .ibd file, it's done by command:

CODE:
  1. alter table order_line discard tablespace;

And now you copy both order_line.ibd and order_line.exp to MysqlDataDir/testimport dir. When it's done, final import command:

CODE:
  1. alter table order_line import  tablespace;

now some magic happens and you can see progress of import in error.log

CODE:
  1. InnoDB: import: extended import of testexport/order_line is started.
  2. InnoDB: import: 2 indexes are detected.
  3. InnoDB: Progress in %: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 done.

When finished quick check

CODE:
  1. mysql> select count(*) from order_line;
  2. +----------+
  3. | count(*) |
  4. +----------+
  5. | 32093604 |
  6. +----------+
  7. 1 row in set (3 min 29.32 sec)

New table has been imported!


Entry posted by Vadim | No comment

Add to: delicious | digg | reddit | netscape | Google Bookmarks

capttofu @ 2009-07-31T12:23:00

Июль 31st, 2009
I'm pleased to announce the release of the Memcached Functions for MySQL, version 1.0.

This release contains several changes, per the ChangeLog:

1.0 Thursday, July 30, 2009 12:00:00 EST 2009
* Fixed issue of setting NULLs with user-defined variables (Thanks to
Jean-Jacques Moortgat at aol dot com !)
* Fixed issue of obtaining a NULL value FROM memcached
* All set functions now return 0 (failure) or 1 (success)
* Other cleanups
* More tests

Importantly, there was an issue that I blogged about several days ago where in the UDF API, if you pass a user-defined variable that is set to NULL to memc_set(), the length of the argument is 8192 even though the value of the argument itself is NULL, which caused much unhappiness in MySQL (crash). That is fixed by setting the length to 0 if the argument itself is NULL. Also fixed is obtaining the NULL value from memcached using the memc_get() UDF. If the value is NULL (meaning libmemcached sees a value stored but the length is 0), the "is_null" boolean pointer in the value function must be set to true.

This bug is a great find and possible due to the work of Jean-Jacques Moortgat from AOL. I added a test containing this bug and can happily report that there is no longer a crash.

Another change, and this is one you may want to check any triggers you use or functionality where you rely on the return value of any set function - set, append, prepend, delete, add, replace. Formerly, the return value from the C underlying C libmemcached function was returned. This resulted in success being 0, and any > 0 number was the error code from the libmemcached library. While I was working on the memcached UDFs in Drizzle, I decided to use 1 for success, 0 for error, which makes more sense logically.

You can get the latest source via Launchpad Bazaar repo:

lp:memcached-udfs

There is also a source download at Launchpad:

https://launchpad.net/memcached-udfs/+download

As well as:

http://patg.net/downloads/memcached_functions_mysql-1.0.tar.gz

Have fun, and thanks for any feedback or bug reports!

Drizzle may now sleep!

Июль 31st, 2009
I realized the other day that I need the sleep() function, which up until this morning, Drizzle did not have, for testing the memcached Drizzle UDFs. Well, now it does:

drizzle> select sleep(3);
+----------+
| sleep(3) |
+----------+
| 0 |
+----------+
1 row in set (3 sec)

drizzle> select sleep(0);
+----------+
| sleep(0) |
+----------+
| 0 |
+----------+
1 row in set (0 sec)

drizzle> select sleep(20);
+-----------+
| sleep(20) |
+-----------+
| 0 |
+-----------+
1 row in set (20 sec)

As you see, it returns zero, just as the MySQL sleep() function does.

The code can be had at:

lp:~capttofu/drizzle/sleep

Drizzle and Gearman PHP talk in Portland

Июль 31st, 2009

I’m going to be giving a talk at the PHP user group here in Portland, OR on August 11th. Details can be found here. If you’re in the Portland area please join the group and come check it out! This will be similar to the Boston MySQL Meetup group talk I gave earlier this month with Patrick, but with more focus on PHP.

Gearman Updates

Июль 31st, 2009

There is a lot happening with Gearman right now. Last week at OSCON we received quite a bit of good feedback with the project which will certainly help direct our priorities moving forward. In the past couple weeks we’ve had the following Gearman releases:

Monty Taylor has a new SWIG wrapper for the Gearman C library that will give us new APIs for things like Python, Ruby, and LUA. Dustin Sallings created a twisted module for Gearman and a OCaml API. I’m also helping a group of students at Portland State University with a Java Messenging Service (JMS) provider that uses Gearman. At the last meeting the group reported it is now functional so I hope to see a 0.1 release sometime soon. :)

Brian and I received good feedback on our OSCON talks as well. Many people were excited, but a few folks thought the presentations lacked detailed examples. The goal of the talks were to give a good overview of the server and APIs, but we’ll certainly try to provide better examples both through the website and in future presentations. Thanks to everyone who attended!

Tool of the day: inotify

Июль 31st, 2009

I was actually exploring inotify-tools for something else, but they can also be handy for seeing what goes on below a mysqld process. inotify hooks into the filesystem handlers, and sees which files are accessed. You can then set triggers, or just display a tally over a certain period.

It has been a standard Linux kernel module since 2.6.13 (2005, wow that’s a long time ago already) and can be used through calls or the inotify-tools (commandline). So with the instrumentation already in the kernel, apt-get install inotify-tools is all you need to get started.

 # inotifywatch -v -t 20 -r /var/lib/mysql/* /var/lib/mysql/zabbix/*
Establishing watches...
Setting up watch(es) on /var/lib/mysql/mysql/user.frm
OK, /var/lib/mysql/mysql/user.frm is now being watched.
[...]
Total of 212 watches.
Finished establishing watches, now collecting statistics.
Will listen for events for 60 seconds.
total  modify  filename
2371   2371    /var/lib/mysql/relay-log.info
2148   2148    /var/lib/mysql/master.info
1157   1157    /var/lib/mysql/ib_logfile0
24     24      /var/lib/mysql/zabbix/
24     24      /var/lib/mysql/zabbix/history.ibd
8      8       /var/lib/mysql/zabbix/trends_uint.ibd
6      6       /var/lib/mysql/zabbix/items.ibd
5      5       /var/lib/mysql/ibdata1

This is just a limited example from a dev box, but you can see the benefit. You can see which files have been accessed, in what way, and how many times over the specified period. Consequently this provides the most insight if you’re using innodb-file-per-table (or MyISAM) rather than a single InnoDB tablespace. But of course it depends a bit on what you’re looking for.

Decimal Math Precision

Июль 31st, 2009
I ran into this little problem today; when dividing two whole number the result MySQL gave me was nowhere near as precise as I needed it to be. The values were truncated which caused me some errors in my application.

Example:
mysql> select 1*0.00001;
+-----------+
| 1*0.00001 |
+-----------+
| 0.00001 |
+-----------+

Ok, that looks fine. Let's do the same thing using division:

mysql> select 1/100000;
+----------+
| 1/100000 |
+----------+
| 0.0000 |
+----------+

Oops! What happened?

The issue is that the maximum precision of the result value depends on the number of decimal places in the arguments. Since the second version uses two whole numbers, the result uses the default number of decimal places, which is 4. So you really have to take care to make sure you're getting the precision you want out of your math operations!

To MySQL's credit,
the behavior is clearly documented (however, I maintain it's still a bit troublesome because how often would you expect the division operator to have a list of instructions and caveats?)

There are at least a couple simple solutions:
mysql> SELECT CAST(1/100000 AS DECIMAL(8,5) );
+---------------------------------+
| CAST(1/100000 AS DECIMAL(8,5) ) |
+---------------------------------+
| 0.00001 |
+---------------------------------+

Or

mysql> SET div_precision_increment=5;
mysql> SELECT 1/100000;
+----------+
| 1/100000 |
+----------+
| 0.00001 |
+----------+

Explain statements that aren’t SELECTs

Июль 31st, 2009

I’ve been editing a task description in our worklog:
WL#706 Add EXPLAIN support for other statements (UPDATE/DELETE).

Currently MySQL supports EXPLAIN SELECT but there have been requests for EXPLAIN UPDATE, EXPLAIN DELETE, and so on. In fact it’s my impression that the proposition in WL#706 (transform the statement to a SELECT and then say EXPLAIN SELECT) is not what people want. But that’s okay, while a worklog is an early stage we allow simple ’solutions’ that might disappear later.

Kontrollbase 2.0.1 revision 91 released!

Июль 31st, 2009

The latest release has a lot of great improvements that you’ll be sure to enjoy.

  • ExtJS upgraded to version 3
  • Host, User, Client, Alerts pages all updated to use GridPanels
  • Documentation on the VMware Virtual Appliance added to docs
  • More JSON responders to replace static page status events
  • Login no longer has an “OK” on successful authentication
  • New CSS UI theme – Slate

As usual you can download here: http://kontrollsoft.com/software-downloads