PlanetMysql.ru — информация о СУБД MySQL

Remember the idea behind renaming Maria, the storage engine? People regularly get confused with Maria the storage engine, and MariaDB the database. We got a whole lot of good names, and here’s a short-listed amount of names (short-listed by Monty Program employees down to about 15), and now its up to YOU to help us choose what the new name should be. Quick, go take the survey.

Voting ends on Friday, July 9, 2010, at 23:59UTC.

The winner will be announced on the first day of OSCON on July 19 2010. The winner walks away with a Meerkat net-top from System76.

Related posts:

1. Quick notes: Monty Program Group Blog; Rename Maria
2. Monty speaks about Maria
3. Trying to reliably make MyISAM crash; Maria is sturdy as

PlanetMySQL Voting: Vote UP / Vote DOWN

I recently granted ALTER access in MySQL so a user could run the ALTER TABLE command . However after I granted the necessary privileges, the user was still not able to perform the tasks needed. Reproducing the issue using a test instance, I granted a test user the required privileges and MySQL reported no errors or warnings when the ALTER TABLE was run:

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 15
Server version: 5.1.41-log MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> grant alter,create,insert on *.* to 'test'@localhost;
Query OK, 0 rows affected (0.00 sec)

mysql> show warnings;
Empty set (0.00 sec)

mysql> show errors;
Empty set (0.00 sec)

mysql>

The reason I granted the addition CREATE and INSERT privileges is that according to the MySQL documentation (http://dev.mysql.com/doc/refman/5.1/en/privileges-provided.html), they are required by the ALTER privilege:

The ALTER privilege enables use of ALTER TABLE to change the structure of or rename tables. (ALTER TABLE also requires the INSERT and CREATE privileges.)

The user was attempting to rename a table with the ALTER TABLE [NAME] RENAME [NAME2] command and seeing the following error:

mysql> alter table test1 rename test2;
ERROR 1142 (42000): DROP command denied to user 'test'@'localhost' for table 'test1'

mysql> show grants;
+----------------------------------------------------------+
| Grants for test@localhost                                |
+----------------------------------------------------------+
| GRANT INSERT, CREATE, ALTER ON *.* TO 'test'@'localhost' |
+----------------------------------------------------------+
1 row in set (0.00 sec)

mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| test1          |
+----------------+
1 row in set (0.00 sec)

Of course I immediately realized what the issue was. In MySQL, ALTER TABLE [NAME] RENAME [NEW_NAME] is done as a DROP (which requires the DROP privilege and CREATE TABLE (which requires the CREATE privilege). So I granted the user the DROP privilege. This time the user was able to successful rename the table as shown below:

mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| test1          |
+----------------+
1 row in set (0.00 sec)

mysql> alter table test1 rename test2;
Query OK, 0 rows affected (0.00 sec)

mysql> show grants;
+----------------------------------------------------------------+
| Grants for test@localhost                                      |
+----------------------------------------------------------------+
| GRANT INSERT, CREATE, DROP, ALTER ON *.* TO 'test'@'localhost' |
+----------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| test2          |
+----------------+
1 row in set (0.00 sec)

The MySQL documentation on the ALTER privilege states that it only need CREATE and INSERT but makes no mention of the requirement for DROP privilege. The RENAME TABLE documentation at http://dev.mysql.com/doc/refman/5.1/en/rename-table.html does mention:

When you execute RENAME, you cannot have any locked tables or active transactions. You must also have the ALTER and DROP privileges on the original table, and the CREATE and INSERT privileges on the new table.

However, that should be explicit in the ALTER TABLE documentation, because if you use ALTER TABLE...RENAME you will not necessarily read the documentation for the RENAME TABLE syntax.

To make the documentation better for everyone, I have filed a MySQL Bug Report on this documentation – Bug 51593.

PlanetMySQL Voting: Vote UP / Vote DOWN

At Days of Wonder we are huge fans of MySQL (and since about a year of the various Open Query, Percona, Google or other community patches), up to the point we’re using MySQL for about everything in production.

But since we moved to 5.0, back 3 years ago our production databases which hold our website and online game systems has a unique issue: the mysqld process uses more and more RAM, up to the point where the kernel OOM decide to kill the process.

You’d certainly think we are complete morons because we didn’t do anything in the last 3 years to fix the issue

Unfortunately, I never couldn’t replicate the issue in the lab, mainly because it is difficult to replicate the exact same load the production server sees (mainly because of the online games activity).

During those 3 years, I tried everything I could, from using other allocators, valgrind, debug builds and so on, without any success.

What is nice, is that we moved to an OurDelta build about a year ago, where InnoDB is able to print more memory statistics than the default MySQL version.

For instance it shows

Internal hash tables (constant factor + variable factor)
    Adaptive hash index 1455381240      (118999688 + 1336381552)
    Page hash           7438328
    Dictionary cache    281544240       (89251896 + 192292344)
    File system         254712  (82672 + 172040)
    Lock system         18597112        (18594536 + 2576)
    Recovery system     0       (0 + 0)
    Threads             408056  (406936 + 1120)
    innodb_io_pattern   0       (0 + 0)

Back several month ago, I analyzed this output just to see what figures were growing, and found that the Dictionary Cache variable part was increasing (slowly but definitely).

Sure fine MySQL experts would have been able to tell me exactly what, when and where the problem was, but since I’m not familiar with the code-base, I looked up what this number was and where it was increased (all in dict0dict.c) and added some logs each time it was increased.

I then installed this version for a quite long time (just to check it wouldn’t crash on production) on a slave server. But this server didn’t print anything interesting because it doesn’t see the exact same load the production masters.

A couple of months after that, I moved this code to one of the master and bingo! I found the operation and the tables exhibiting an increase:

mysqld[8131]: InnoDB: dict_table_rename_in_cache production/rank_tmp2 193330680 + 8112
mysqld[8131]: InnoDB: dict_table_rename_in_cache production/rank 193338792 + 8112

As soon as I saw the operation and table (ie rank), I found what the culprit is. We have a daemon that every 10s computes the player ranks for our online games.

To do this, we’re using the following pattern:

-- compute the ranks
SELECT NULL, playerID
FROM game_score as g
ORDER BY g.rankscore DESC
INTO OUTFILE "/tmp/rank_tmp.tmp"

-- load back the scores
LOAD DATA INFILE "/tmp/rank_tmp.tmp" INTO TABLE rank_tmp

-- swap tables so that clients see new ranks atomatically
RENAME TABLE rank TO rank_tmp2 , rank_tmp TO rank, rank_tmp2 TO rank_tmp

-- truncate the old ranks for a new pass
TRUNCATE TABLE rank_tmp

-- go back to the select above

You might ask why I’m doing a so much convoluted system, especially the SELECT INTO OUTFILE and the LOAD DATA. It’s just because INSERT … SELECT with innodb and binlog enabled can produce transactions abort (which we were getting tons of).

Back to the original issue, apparently the issue lies in the RENAME part of the daemon.

Looking at the dict0dict.c dict_table_rename_in_cache function we see:

ibool
dict_table_rename_in_cache(...)
...
  old_name = mem_heap_strdup(table->heap, table->name);
  table->name = mem_heap_strdup(table->heap, new_name);
...
}

Looking to mem_heap stuff, I discovered that each table has a heap associated in which InnoDB allocates various things. This heap can only grow (by block of 8112 bytes it seems), since the allocator is not a real one. This is done for performance reasons.

So each time we rename a table, the old name (why? since it is already allocated) is duplicated, along with the new name. Each time.

This heap is freed when the table is dropped, so there is a possibility to reclaim the used memory. That means this issue is not a memory leak per-se.

By the way, I’ve filed this bug on mysql bug system.

One work-around, beside fixing the code itself, would be to drop the rank table instead of truncating it. The issue with dropping/creating InnoDB table on a fast pace is that the dictionary cache itself will grow, because it can only grow as there is no way to purge it from old tables (except running one of the Percona patches). So the more tables we create the more we’ll use memory - back to square 0, but worst.

So right now, I don’t really have any idea on how to really fix the issue. Anyone having an idea, please do not hesitate to comment on this blog post

And please, don’t tell me to move to MyISAM…

PlanetMySQL Voting: Vote UP / Vote DOWN