• Perform a warmup of the database using custom scripts and external information that the init_file configuration may not have access to.
• Change configuration settings such as innodb_log_file_size and adjust the on disk files that this needs
• Change configuration settings such as log_bin (I try to use a relative paths to ensure that different instances use a common setting even if datadir is different, and changing from an absolute to relative path often requires adjusting the binlog index file)
• Shutting down MySQL may need to be delayed if the SQL thread has open temporary tables or open transactions are still being processed
• etc…

You may have others and many of these are likely to be site specific. I could ask Oracle to add lots of configuration options to cover many of these special cases but that is likely to not work as my needs may not match others. However asking them to add hooks to allow me to put in the infrastructure that I need, especially if those hooks are normally disabled and require minimal changes to the current init scripts, might be acceptable. That’s why I’m curious as to what others think about this.

Some may be asking: why do you need this at all? I have certainly seen several issues such as those indicated above which I need to resolve. Also the DBA is not the only person who may have to restart a server. Sometimes this may be done by a system administrator, and on some servers a by developer. These people are not necessarily expected to know about the specifics of one MySQL server compared to another or to know the server’s current state. It is also true that often a server may go for a long time without needing to be restarted, so many configuration changes may be pending. They only need to be done when the server is restarted.

So I was considering the idea of some hooks which would work in a similar way to a system init script, probably composed of the following parts:

• An /etc/mysql/init.d directory containing any scripts that might need to be run. they can be placed there manually or by system configuration tools
• An /etc/mysql/rc-start and /etc/mysql/rc-stop directory which would have files named: PRE-nn-scriptname, and POST-nn-scriptname linked to the ../init.d/scriptname file. Again similar to the system init scripts. the nn would like the init script define the relative ordering of the execution of any scripts in that directory
• A modification to /etc/init.d/mysql so that these scripts would be run (if present) at 4 stages:
  • pre-start: before starting mysqld
  • post-start: after starting mysqld
  • pre-stop: before stopping msyqld
  • post-stop: after stopping mysqld
• Not absolutely necessary but for completeness a tool which would manage the linking / unlinking of these files

I can certainly see the use of something like this for my work, but do not see that someone like Oracle will take this unless they see a clear business sense to that, or technical advantage. If you have a single mysqld to manage this may seem like overkill. That’s fine: by default it would not be used. However, as MySQL is gradually being used in larger sites I believe this sort of functionality could be quite helpful.

Have you solved the problem already? If so how? If not does this seem like a reasonable idea?

When a data node restarts, it first attempts to load the data into memory from the local log files and then it will catch up with any subsequent changes by retrieveing them from the surviving node(s) in its node group.

 Based on this, you would expect the time taken to restart a data node to be influenced by:

1. The amount of data that was stored on the data node before the restart
2. Rate of updates being made to the data during the restart
3. Network performance (assuming the data is being updated during recovery)

The times will also be influenced bycertain configuration parameters, performance of the host machine and whether the multi-threaded data node (ndbmtd) is being used.

To provide some insight into how these factors impact restart times, tests have been performed where the following factors are varied:

• Database size (Each Gbyte is made up of 1,000,000 tuples in each of 5 tables)
• Whether traffic is running or not (a single thread using the NDB API to send in up to 2K tps (10K updates/second))
• Whether the 2 data nodes in the node group are on the same host or separated by a Gbit Ethernet network

The following factors are kept constant:

• Physical hosts: Intel Core 2 Quad Q8200@2.33 GHz; 7.7 GBytes RAM
• NoOfFragmentLogFiles: 300
• MaxNoOfExecutionThreads=4

Here are the observed results:

Data Node restart times

There are a couple of things to note from these results:

• Using the multi-threaded data node (ndbmtd) greatly improves the restart time (in this case, 4 threads were available, improvements could be even greater on an 8 core/thread system)
• Results become less predictable when heavy update traffic is being processed (in this case, up to 10,000 updated rows/second on a single node group). In the tests, no attempt was made to regulate this traffic and the test application was run on the same host as the one of the data nodes. Changes to the rate of updates will vary how long it takes for the restarting node to catch-up as it’s a moving target.

There is another recovery/restart scenario. The measurements shown above assumed that the file system on the data node’s host was intact and could be used to recover the in-memory copy – if that were not the case (or the data nodes were restarted with the “initial” option) then all of the data would have to be recovered from the surviving data node(s) in the same node group. As a comparison restarting a 6 Gbyte data node with the “initial” option took 20 minutes compared to 8 minutes without it (ndbmtd, over Gbit n/w).