Many database management tasks become difficult as you move from millions of rows and gigabytes of data to billions of rows and terabytes of data. Such tasks include ingesting data while maintaining indexes; changing schemas without downtime; and supporting connections, replication, and backup. For some scaling problems (connections and replication), MySQL^® is better than most of the competition. For others, such as indexing, schema changes, and backup, MySQL has typically been harder to use. Fortunately, the tasks MySQL does well are in its core, whereas the tasks that are more difficult can be solved with storage engine plug-ins.

I recently gave a talk at IOUG Collaborate, a copy of which can be found here. This presentation discusses how MySQL’s storage engines have recently made dramatic progress in large database manageability.

A complete list of MySQL talks from the show can be found here.

Many database management tasks become difficult as you move from millions of rows and gigabytes of data to billions of rows and terabytes of data. Such tasks include ingesting data while maintaining indexes; changing schemas without downtime; and supporting connections, replication, and backup. For some scaling problems (connections and replication), MySQL^® is better than most of the competition. For others, such as indexing, schema changes, and backup, MySQL has typically been harder to use. Fortunately, the tasks MySQL does well are in its core, whereas the tasks that are more difficult can be solved with storage engine plug-ins.

I recently gave a talk at IOUG Collaborate, a copy of which can be found here. This presentation discusses how MySQL’s storage engines have recently made dramatic progress in large database manageability.

A complete list of MySQL talks from the show can be found here.

Top 5 Most Popular

For more articles like these go to iHeavy, Inc +1-212-533-6828

Top 5 Most Popular

For more articles like these go to iHeavy, Inc +1-212-533-6828

XtraDB cluster (and Galera, the technology on which it is based) have been attracting a lot of interest in the community and we want to start presenting information about both what we know is essential, but also what we suspect will become essential as more and more production experience is logged with these important new technologies.

As such, I put together a talk to cover the baseline required knowledge for getting you into a working knowledge XtraDB Cluster that I’m presenting as a webinar on June 6th at 10AM PDT.  You can find details and register for it here.  This session will cover:

• Essential and Important configuration settings for Galera, but also changes to your existing MySQL settings
• Migration strategies for moving from a traditional Master/Slave topology into XtraDB Cluster
• Managing Application -> Cluster connectivity in a highly available way
• Monitoring the cluster and individual nodes
• Performing cluster maintenance
• Node deployment architectures and Quorum arbitration

One of our customers was complaining that the database would lock up at random times and then it would go back to normal in just a few seconds. This was MySQL 5.0 running MyISAM/InnoDB mix, not heavily loaded. We used pt-stalk (at that time it was aspersa stalk) trying to figure out what is happening, however all we found was a spike in writes, many queries piled up and looking at the system process list it was quite obvious that page flush daemon was acting out. I/O Pattern was rather awkward – here is an output from pt-diskstats (again, aspersa diskstats at that time):

 rd_s rd_avkb rd_mb_s rd_mrg rd_cnc   rd_rt    wr_s wr_avkb wr_mb_s wr_mrg wr_cnc   wr_rt busy in_prg
 24.0    13.0     0.2     0%    1.4    59.3  1089.2   195.8   104.1    95%  100.1    91.9 100%      1
 14.0     8.0     0.1     0%    0.1     5.4    10.0   246.4     1.2    97%    0.0     2.0   8%      0
 36.0     9.3     0.2     0%    0.2     4.2     6.0   109.3     0.3    93%    0.0     0.2  15%      0
 12.0     8.7     0.1     0%    0.1     4.2     6.0   101.3     0.3    92%    0.0     0.0   5%      0

You can see that (a) reads are affected – response time dropped significantly and (b) the reason appears to be enormous amount of writes. For a while I was really puzzled, looking for answers in MySQL status counters, innodb status, memory info and what not. It seemed that the problem was gone before pt-stalk would kick-off and all I could see was symptoms. Indeed the case it was. I was not allowed to run pmp or oprofile on that system due to potential performance impact. However, somehow I have noticed that pt-stalk kick-off time matched that of binary logs modification time:

-rw-r--r-- 1 root root 91 Nov 2 15:15 2011_11_02_15_15_13-trigger
-rw-r--r-- 1 root root 91 Nov 2 16:17 2011_11_02_16_17_20-trigger
-rw-r--r-- 1 root root 91 Nov 2 17:38 2011_11_02_17_38_22-trigger
-rw-r--r-- 1 root root 91 Nov 3 08:36 2011_11_03_08_36_00-trigger
...

-rw-rw---- 1 mysql mysql 1073742171 Nov 2 15:15 /var/lib/mysqllogs/db1-bin-log.003229
-rw-rw---- 1 mysql mysql 1073742976 Nov 2 16:17 /var/lib/mysqllogs/db1-bin-log.003230
-rw-rw---- 1 mysql mysql 1073742688 Nov 2 17:38 /var/lib/mysqllogs/db1-bin-log.003231
-rw-rw---- 1 mysql mysql 1073742643 Nov 2 20:06 /var/lib/mysqllogs/db1-bin-log.003232
-rw-rw---- 1 mysql mysql 1073742974 Nov 3 00:10 /var/lib/mysqllogs/db1-bin-log.003233
-rw-rw---- 1 mysql mysql 1073742566 Nov 3 08:35 /var/lib/mysqllogs/db1-bin-log.003234

What is happening here?

Apparently, every time current binary log gets close to max_binlog_size in size, new binary log is created and, if expire_logs_days is not zeroish, log files older than expire_logs_days are removed. &LOCK_log mutex is held for the duration of the binary log removal and if your file system is slow at deleting files (both customers systems ran on ext3, which can be very slow deleting large files), then write operations in MySQL will suffer since that very same mutex is used inside MYSQL_LOG::write() call. Even though it only lasts 1-2s, it still gives users unstable response time – here’s how that looks on one of the customer system – this one was doing ~10k qps at a time:

# mysqladmin ext -i1 | grep Threads_running
...
| Threads_running                   | 2              |
| Threads_running                   | 2              |
| Threads_running                   | 2              |
| Threads_running                   | 2              |
| Threads_running                   | 3              |
| Threads_running                   | 3              |
| Threads_running                   | 2              |
| Threads_running                   | 2              |
| Threads_running                   | 2              |
| Threads_running                   | 3              |
| Threads_running                   | 5              |
| Threads_running                   | 4              |
| Threads_running                   | 3              |
| Threads_running                   | 2              |
| Threads_running                   | 3              |
| Threads_running                   | 108            |
| Threads_running                   | 174            |
| Threads_running                   | 5              |
| Threads_running                   | 2              |
| Threads_running                   | 2              |
| Threads_running                   | 3              |
| Threads_running                   | 2              |

A small blip, yet over 100 users are backlogged for a couple of seconds – not great user experience, considering this happens every 10-15 minutes.

Solution?

Here’s what we did: we have reduced the size of binary log file from default 1GB (some systems have it set to 100MB in their default my.cnf) down to 50MB and we never saw this problem ever again. Now the files were removed much more often, they were smaller and didn’t take that long to remove.

Note that on ext4 and xfs you should not have this problem as they would remove such files much faster.

A striped view lets authorized users see only part of a table, and is how Oracle Database 11g sets up Virtual Private Databases. Oracle provides both schema (or database) level access and fine-grained control access. Fine grained control involves setting a special session variable during a user’s login. This is typically done by checking the rights in an Access Control List (ACL) and using an Oracle built-in package.

You can do more or less the same thing in MySQL by using stored functions. One function would set the session variable and the other would fetch the value for comparison in a view.

Most developers who try this initially meet failure because they try to embed the session variable inside the view, like this trivial example with Hobbits (can’t resist the example with the first installment from Peter Jackson out later this year):

1
2

CREATE VIEW hobbit_v AS
SELECT * FROM hobbit WHERE hobbit_name = @sv_login_name;

The syntax is disallowed, as explained in the MySQL Reference 13.1.20 CREATE VIEW Syntax documentation. The attempt raises the following error message:

ERROR 1351 (HY000): VIEW's SELECT contains a variable or parameter

The fix is quite simple, you write a function that sets the ACL value for the session and another that queries the ACL session value. For the example, I’ve written the SET_LOGIN_NAME and a GET_LOGIN_NAME functions. (If you’re new to stored programs, you can find a 58 page chapter on writing them in my Oracle Database 11g & MySQL 5.6 Developer Handbook (Oracle Press) or you can use Guy Harrison’s MySQL Stored Procedure Programming.)

You would call the SET_LOGIN_NAME when you connect to the MySQL database as the first thing to implement this type of architecture. You would define the function like the following. (Please note that the example includes all setup statements from the command line and should enable you cutting and pasting it. ):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

-- Change the delimiter to something other than a semicolon.
DELIMITER $$
 
-- Conditionally drop the function.
DROP FUNCTION IF EXISTS set_login_name$$
 
-- Create the function.
CREATE FUNCTION set_login_name(pv_login_name VARCHAR(20)) RETURNS INT UNSIGNED
BEGIN
 
  /* Declare a local variable to verify completion of the task. */
  DECLARE  lv_success_flag  INT UNSIGNED  DEFAULT FALSE;
 
  /* Check whether the input value is something other than a null value. */
  IF pv_login_name IS NOT NULL THEN
 
    /* Set the session variable and enable the success flag. */
    SET @sv_login_name := pv_login_name;
    SET lv_success_flag := TRUE;
 
  END IF;
 
  /* Return the success flag. */
  RETURN lv_success_flag;
END;
$$
 
-- Change the delimiter back to a semicolon.
DELIMITER ;

You can use a query to set and confirm action like this:

SELECT IF(set_login_name('Frodo')=1,'Login Name Set','Login Name Not Set') AS "Login Name Status";

A more practical example in an API would be this, which returns zero when unset and one when set:

SELECT set_login_name('Frodo') AS "Login Name Status";

The getter function for this example, simply reads the current value of the MySQL session variable. Like the prior example, it’s ready to run too.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

-- Change the delimiter to something other than a semicolon.
DELIMITER $$
 
-- Conditionally drop the function.
DROP FUNCTION IF EXISTS get_login_name$$
 
-- Create the function.
CREATE FUNCTION get_login_name() RETURNS VARCHAR(20)
BEGIN
  /* Return the success flag. */
  RETURN @sv_login_name;
END;
$$
 
-- Change the delimiter back to a semicolon.
DELIMITER ;

Before you test it, lets create a HOBBIT table, seed it with data, and create a HOBBIT_V view. They’re bundled together in the following microscript:

-- Conditionally drop the table.
DROP TABLE IF EXISTS hobbit;
 
-- Create the table.
CREATE TABLE hobbit
( hobbit_id    INT UNSIGNED
, hobbit_name  VARCHAR(20));
 
-- Seed two rows.
INSERT INTO hobbit VALUES ( 1,'Bilbo'),( 1,'Frodo');
 
-- Conditionally drop the view.
DROP VIEW IF EXISTS hobbit_v;
 
-- Create the function-enabled view.
CREATE VIEW hobbit_v AS
SELECT * FROM hobbit WHERE hobbit_name = get_login_name();

A query to the table after setting the session variable will only return one row, the row with Frodo in the HOBBIT_NAME column. It also guarantees an unfiltered UPDATE statement against the view only updates the single row returned, like this:

UPDATE hobbit_v SET hobbit_id = 2;

In a real solution, there are more steps. For example, you’d want your tables in one database, views in another, and functions and procedures in a library database. However, I hope this helps seed some ideas for those interested in creating fine-grained virtual private databases in MySQL with user-authenticated application controls.

mysql> SELECT * FROM p;
+------+------+
| a    | b    |
+------+------+
|    1 |    1 |
|   11 |    1 |
|   21 |    1 |
+------+------+
3 rows in set (0.00 sec)

OK, try to write an UPDATE statement that increments b for the row where a = 21.  Easy, right?

UPDATE p SET b = b+1 WHERE a = 21;

OK, now do it without a WHERE clause.  Maybe that made you think for a minute, but perhaps you came up with something like this:

UPDATE p SET b = b+1 ORDER BY a = 11 DESC LIMIT 1;

Nice!  Now do it without ORDER BY or LIMIT.  Can you do it?

There’s a hint in the table structure:

mysql> SHOW CREATE TABLE p\G
*************************** 1. row ***************************
Table: p
Create Table: CREATE TABLE `p` (
`a` int(11) DEFAULT NULL,
`b` int(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE (a)
(PARTITION p0 VALUES LESS THAN (10) ENGINE = InnoDB,
PARTITION p1 VALUES LESS THAN (20) ENGINE = InnoDB,
PARTITION p2 VALUES LESS THAN (30) ENGINE = InnoDB) */
1 row in set (0.00 sec)

You’ve probably figured out that 5.6 adds the ability to explicitly restrict statements to specific partitions, even if you haven’t found or reviewed the excellent documentation in the manual.  In 5.6, the following works:

mysql> UPDATE p PARTITION(p1) SET b = b+1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> SELECT * FROM p;
+------+------+
| a    | b    |
+------+------+
|    1 |    1 |
|   11 |    2 |
|   21 |    1 |
+------+------+
3 rows in set (0.00 sec)

The documentation on this is very good, and provides some useful examples of how this might prove useful, including showing how this works not only with UPDATE statements, but INSERT, SELECT, DELETE, REPLACE and LOAD DATA/XML statements.  There’s also the WorkLog and associated bugs.mysql.com feature requests.  The documentation says the following about UPDATE statements:

UPDATE statements using explicit partition selection behave in the same way; only rows in the partitions referenced by the PARTITION option are considered when determining the rows to be updated, as can be seen by executing the following statements.

It then provides an example that is similar to the solution for the problem posed above.  I still had one question, though – what would happen if I did an UPDATE on a partition key column that moves a row from one partition to another?  I get an error:

mysql> UPDATE p PARTITION(p1) SET a = 22 WHERE a = 11;
ERROR 1745 (HY000): Found a row not matching the given partition set

But if I include both the source partition and the target partition, all’s well:

mysql> UPDATE p PARTITION(p1,p2) SET a = 22 WHERE a = 11;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

How do I know the data was actually moved from p1 to p2, and that mysqld isn’t just pulling my leg?  Easy:

mysql> SELECT * FROM p PARTITION(p2);
+------+------+
| a    | b    |
+------+------+
|   21 |    1 |
|   22 |    1 |
+------+------+

Awesome.  You might want to download the 5.6.5 Development Milestone Release yourself and try it out (you’ll need to click on the “Development Releases” tab to access it, something which took me longer to figure out than I care to admit).