Most of MySQL’s memory is really used just as a cache, in one form or another, information that otherwise is on disk. So ensuring you have as large a cache as possible is important. However, making these memory sizes too large will trigger the server to start swapping and possibly can cause it to crash or cause the kernel to kill the process when it runs out of memory.  So that’s something we want to avoid.

Certain settings affect memory allocation on a per connection/thread basis, being bounded by thread_cache_size and max_connections.  If you configure for the worst behaviour (max_connections) you may end up not actually using all the memory you have available, memory which normally could be used for other purposes.

Recently a server I managed was configured incorrectly with a large sort_buffer_size (4M to 256M) and larger read_buffer_size (4M to 20M).  The change in configuration on first glance looks quite innocent, and not noticing that these are per-connection settings this got rolled out. max_connections on this server was set to 1000, while normally there were ~40 connections of which only a few were active. The mysqld memory footprint on startup looked fine. In fact under normal usage it also worked fine. However spiky load suddenly changed this nice behaviour: as configured mysqld ramped up the thread count and hence memory usage, resulting in swapping and finally server death…

The fault of course was mine for not noticing, but this has been an issue with MySQL forever.  Most other RDBMSes manage memory slightly differently: you define how much memory you want to use (maximum), this is optinally locked into memory, and further requests for different buffers are taken from this global pool.  That is much safer, avoids unexpected swapping, or memory over-allocation, but does raise the question of what happens when a memory request can not be honoured.

Initially I would expect two different behaviours: either

1. kill a thread whose memory can not be allocated, or
2. wait a certain time to allocate that memory, after which it gets killed anyway.

Option (2) is probably saner, and possibly some sort of deadlock detection can kick if in all threads are waiting for memory, perhaps killing the younger thread, or thread which has done least work first. Possibly there are other better ways of doing this?

I can imagine that changing MySQL’s current behaviour to do something like this could be quite hard, especially as ideally the engines would also use the same memory management mechanisms, but I see this as being a good thing and would make MySQL more robust, especially under load, which is after all what counts.  Of course this will not happen in today’s 5.5 GA version, or tomorrow’s 5.6 version which is probably likely to appear some time this year. That’s a major change. It would be nice if Oracle look at this for 5.7 as a way of ensuring that when resource usage does come under pressure MySQL does not go heads up, but attempts to use the allocated resources as best as possible.

In the meantime what would help would be:

• better documentation so we can see clearly how all mysql memory is allocated. There are several web pages commenting ways to calculate this, but certainly no definitive guide.
• The InnoDB engine’s documentation talks about memory usage and most people think that the innodb_buffer_pool_size is the main setting. yet read further and there’s talk of an overhead of perhaps 1/8th. I have recently been playing with innodb_buffer_pool_instances settings > 1 (using values in the range of 20-40) and am inclined to think that this increases that overhead somewhat more, yet there’s no documentation on this and whether my guess is right or not. Please InnoDB developers improve your documentation, if only to prove me wrong.
• Ideally some tools to tell you if you server is possibly misconfigured. Coming from a Sybase environment I’d be tempted to suggest a stored procedure in the mysql database which can tell you total memory usage and how it is broken down as doing this with a single SELECT is going to be tricky.

Then once that is done consider adding some extra variable to enable total memory usage to be controlled. I made a feature request for this at http://bugs.mysql.com/?id=64108. If you think this feature might interest you please let Oracle know.

I’ve only tested my test case thus far, but it seems fine.

Unfortunately, the fix is not mentioned in the 5.5.18 changelogs though.

And there is no mention (yet, anyway) of a fix in the bug report I filed (though it was designated a ‘duplicate’, so it wouldn’t necessarily be updated).

I’m trying to get confirmation from the MySQL Dev Team on this (via the bug report), and will update this post if/when I hear anything.

I’ll also perform some of the other tests on my end to see if they all pass as well.

All in all, at least the initial results look very promising!

This simple crash (if only one row is in the table) will crash mysqld.

Of course I’ve filed a bug report, but that has been nearly 3 months and no updates yet.

Here is the bug I filed (which you won’t be able to view):

http://bugs.mysql.com/bug.php?id=61842

Really, the only thing that happened to my bug report was that it was designated a duplicate of another bug (which we also cannot view):

http://bugs.mysql.com/bug.php?id=61101

Based on the id, and the submitted dates of bugs 61100 and 61102, this initial bug (61101) was filed on May 9, 2011. So, in fact, this bug has been present for over 5 months, and not one breath of an update to the public!

Therefore, I felt it necessary to warn others about this bug, (or possibly you’ll run across this if you’re searching on SELECT COUNT(DISTINCT) in the future).

All I can say is please watch out for it!

It is extremely easy to reproduce:

CREATE TABLE t (a int(1), b int(1), PRIMARY KEY (a,b)) ENGINE=InnoDB;
INSERT INTO t VALUES (1, 1);
SELECT COUNT(DISTINCT a) FROM t WHERE b = 1;

–> crash <--

For those interested, this was filed against 5.5.14. However, with each new release, I've continued testing, and this bug is present in 5.5.15, 5.5.16, and thus far in 5.5.17 (built from the latest bzr tree).

Hopefully we don't go too many more months before this is finally fixed.

And for reference (and those searching on the stack trace / error log messages), here is my full error log snippet from 5.5.16:

111017 10:54:47 [Note] C:\Program Files\MySQL\MySQL Server 5.5\bin\mysqld: ready for connections.
Version: ’5.5.16′  socket: ”  port: 3308  MySQL Community Server (GPL)
 len 128; hex f8aec9037d803805f017fc03189ddc030000000…
111017 10:55:12  InnoDB: Assertion failure in thread 5000 in file btr0pcur.c line 236
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
111017 10:55:12 – mysqld got exception 0xc0000005 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=26214400
read_buffer_size=65536
max_used_connections=1
max_threads=100
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 58325 K
bytes of memory
Hope that’s ok; if not, decrease some variables in the equation.

Thread pointer: 0x3c98428
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
00CE92EC    mysqld.exe!btr_pcur_restore_position_func()[btr0pcur.c:236]
00CA62FB    mysqld.exe!sel_restore_position_for_mysql()[row0sel.c:3081]
00CA6CEA    mysqld.exe!row_search_for_mysql()[row0sel.c:3820]
00C5FE20    mysqld.exe!ha_innobase::general_fetch()[ha_innodb.cc:5918]
00C5FEDD    mysqld.exe!ha_innobase::index_next()[ha_innodb.cc:5956]
00C20DDA    mysqld.exe!index_next_different()[opt_range.cc:11038]
00C249BC    mysqld.exe!QUICK_GROUP_MIN_MAX_SELECT::next_prefix()[opt_range.cc:11099]
00C26BE7    mysqld.exe!QUICK_GROUP_MIN_MAX_SELECT::get_next()[opt_range.cc:10824]
00B68D01    mysqld.exe!rr_quick()[records.cc:344]
00BC1B9A    mysqld.exe!sub_select()[sql_select.cc:11723]
00BD10A7    mysqld.exe!do_select()[sql_select.cc:11483]
00BD37BD    mysqld.exe!JOIN::exec()[sql_select.cc:2370]
00BD3A29    mysqld.exe!mysql_select()[sql_select.cc:2581]
00BD3D4B    mysqld.exe!handle_select()[sql_select.cc:297]
00ACD76E    mysqld.exe!execute_sqlcom_select()[sql_parse.cc:4511]
00ACF816    mysqld.exe!mysql_execute_command()[sql_parse.cc:2118]
00AD2D1F    mysqld.exe!mysql_parse()[sql_parse.cc:5548]
00AD3848    mysqld.exe!dispatch_command()[sql_parse.cc:1037]
00AD43BB    mysqld.exe!do_command()[sql_parse.cc:771]
00AF2DB6    mysqld.exe!do_handle_one_connection()[sql_connect.cc:789]
00AF2F44    mysqld.exe!handle_one_connection()[sql_connect.cc:708]
00C33DE4    mysqld.exe!pthread_start()[my_winthread.c:61]
00D9C6F3    mysqld.exe!_callthreadstartex()[threadex.c:348]
00D9C79B    mysqld.exe!_threadstartex()[threadex.c:326]
765F3823    kernel32.dll!BaseThreadInitThunk()
77CAA9BD    ntdll.dll!LdrInitializeThunk()

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (03DC0F10): SELECT COUNT(DISTINCT a) FROM t WHERE b = 1
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
InnoDB: Thread 5980 stopped in file os0sync.c line 781
InnoDB: Thread 6820 stopped in file os0sync.c line 474
InnoDB: Thread 7532 stopped in file os0sync.c line 474

101120 8:29:27 InnoDB: Operating system error number 22 in a file operation.
InnoDB: Error number 22 means ‘Invalid argument’.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html
InnoDB: File name /tmpfs/#sql6cf3_5c_0.ibd
InnoDB: File operation call: ‘aio write’.
InnoDB: Cannot continue operation.

The work around was to change the tmpdir=/tmpfs (which was a 16G tmpfs filesystem) to a physical disk.

The referenced URL didn’t provide any more information of help. Unlike Bug #26662 O_DIRECT is not specified as the flush method.

• Ensure data consistency between master and slave servers
• Immediately detect if replication is not working
• Allow a crashed slave to automatically recover from the master relay log
• Allow users to filter events for specific servers
• Correctly convert data types between master and slave servers

• Rpl_semi_sync_master_status - indicates status of when master is using asynchronous or semi-sycnhronous replication.
• Rpl_semi_sync_master_clients - shows how many slaves are configured for semi-synchronous replication.
• Rpl_semi_sync_master_yes_tx - shows number of successfully acknowledged commits by slaves.
• Rpl_semi_sync_master_no_tx - shows number of unsuccessfully acknowledged commits by slaves.

• Rpl_semi_sync_slave_status - indicates if semi-synchronous replication is enabled on slave.

• SET SLAVE_TYPE_CONVERSIONS="ALL_LOSSY' - enables conversions to types with smaller domain (INT to TINY for example)
• SET SLAVE_TYPE_CONVERSION="ALL_NON_LOSSY" - enables conversions to types with larger domain (TINY to INT for example)