$ make_cluster.py ~/Downloads/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64.tar.gz

++ tar -xzf ~/Downloads/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64.tar.gz

++ low_level_make_sandbox --basedir=$HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64 --sandbox_directory=mcluster --install_version=5.1 --sandbox_port=5144 --no_ver_after_name --no_run --force --my_clause=log-error=msandbox.err --my_clause=ndbcluster 
    The MySQL Sandbox,  version 3.0.12
    (C) 2006-2010 Giuseppe Maxia
installing with the following parameters:
upper_directory                = $HOME/sandboxes
sandbox_directory              = mcluster
sandbox_port                   = 5144
check_port                     = 
no_check_port                  = 
datadir_from                   = script
install_version                = 5.1
basedir                        = $HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64
tmpdir                         = 
my_file                        = 
operating_system_user          = gmax
db_user                        = msandbox
db_password                    = msandbox
my_clause                      = log-error=msandbox.err ; ndbcluster
prompt_prefix                  = mysql
prompt_body                    =  [\h] {\u} (\d) > 
force                          = 1
no_ver_after_name              = 1
verbose                        = 
load_grants                    = 1
no_load_grants                 = 
no_run                         = 1
no_show                        = 
loading grants
.. sandbox server started
stopping server
Your sandbox server was installed in $HOME/sandboxes/mcluster

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgmd -f $HOME/sandboxes/mcluster/my_cluster/conf/config.ini --initial --configdir=$HOME/sandboxes/mcluster/my_cluster/conf/
2010-06-28 21:29:57 [MgmtSrvr] INFO     -- NDB Cluster Management Server. mysql-5.1.44 ndb-7.1.4b
2010-06-28 21:29:57 [MgmtSrvr] INFO     -- Reading cluster configuration from '$HOME/sandboxes/mcluster/my_cluster/conf/config.ini'

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndbd -c localhost:1186
2010-06-28 21:29:57 [ndbd] INFO     -- Configuration fetched from 'localhost:1186', generation: 1

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndbd -c localhost:1186
2010-06-28 21:29:57 [ndbd] INFO     -- Configuration fetched from 'localhost:1186', generation: 1

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgm -e show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 (not connected, accepting connect from localhost)
id=4 (not connected, accepting connect from localhost)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)


++ $HOME/sandboxes/mcluster/clear

++ $HOME/sandboxes/mcluster/start
... sandbox server started
Please wait. Giving the cluster time to catch up
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)


It may take up to 2 minutes to initialize ... ( 0 )
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)


It may take up to 2 minutes to initialize ... ( 5 )
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)

[...]

It may take up to 2 minutes to initialize ... ( 100 )
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4)


++ $HOME/sandboxes/mcluster/use -vvv -e "create table test.t1(i int not null primary key)engine=ndb"
--------------
create table test.t1(i int not null primary key)engine=ndb
--------------
Query OK, 0 rows affected (0.45 sec)

++ $HOME/sandboxes/mcluster/use -vvv -e "show create table test.t1\G"
--------------
show create table test.t1
--------------

*************************** 1. row ***************************
       Table: t1
Create Table: CREATE TABLE `t1` (
  `i` int(11) NOT NULL,
  PRIMARY KEY (`i`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

to shut down the cluster, type:
$HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgm -e shutdown

• Custom monitoring: monitoring & charting for user defined queries
• HTML reports for custom monitoring
• Process list dump upon alert notifications

Custom monitoring & charts

Custom monitoring allows the user to supply with a query, the results of which will be monitored.

That is, mycheckpoint monitors the status variables, replication status, OS metrics. But it cannot by itself monitor one’s application. Which is why a user may supply with such query as:

SELECT COUNT(*) FROM shopping_cart WHERE is_pending=1

Such a query will tell an online store how many customers are in the midst of shopping. There is no argument that this number is worth monitoring for. Given the above query, mycheckpoint will execute it per sample, and store the query’s result along with all sampled data, to be then aggregated by complex views to answer for:

• What was the value per given sample?
• What is the value difference for each sample?
• What is the change per second, i.e. the rate?

mycheckpoint goes one step forward, and explicity records another metric:

• How much time did it take to take that sample?

As another example, a query worth testing for rate:

SELECT MAX(shopping_cart_id) FROM shopping_cart

The above will provide with the last id. Assuming this is AUTO_INCREMENT, and assuming we’re on auto_increment_increment=1, two samples will allow us to get the number of created carts between those samples. Now, here’s a metric I’d like to read:

• How many carts are created per second, for each hour of the day?

We get all these for free with mycheckpoint, which already does this analysis. All we need to provide is the query, and how we would like it to be visualized (visualization is optional, it is not the only way to diagnose monitored data) graphically:

INSERT INTO
 custom_query (custom_query_id, enabled, query_eval, description, chart_type, chart_order)
 VALUES (0, 1, 'SELECT COUNT(*) FROM store.shopping_cart WHERE is_pending=1', 'Number of pending carts', 'value', 0);

INSERT INTO
 custom_query (custom_query_id, enabled, query_eval, description, chart_type, chart_order)
 VALUES (1, 1, 'SELECT MAX(shopping_cart_id) FROM store.shopping_cart', 'Created carts rate', 'value_psec', 0);

We can later query for these values, just like we do for normal monitored values:

mysql> SELECT id, ts, created_tmp_tables_psec, custom_0, custom_1_psec FROM sv_sample WHERE ts >= NOW() - INTERVAL 1 HOUR;
+-------+---------------------+-------------------------+----------+---------------+
| id    | ts                  | created_tmp_tables_psec | custom_0 | custom_1_psec |
+-------+---------------------+-------------------------+----------+---------------+
| 50730 | 2010-05-21 19:05:01 |                   16.64 |      448 |          3.02 |
| 50731 | 2010-05-21 19:10:02 |                   20.97 |       89 |          1.73 |
| 50732 | 2010-05-21 19:15:01 |                   15.70 |      367 |          3.56 |
| 50733 | 2010-05-21 19:20:01 |                   18.32 |       54 |          1.43 |
| 50734 | 2010-05-21 19:25:01 |                   16.42 |       91 |          1.96 |
| 50735 | 2010-05-21 19:30:02 |                   21.93 |      233 |          2.11 |
| 50736 | 2010-05-21 19:35:02 |                   14.58 |      176 |          1.91 |
| 50737 | 2010-05-21 19:40:01 |                   21.61 |      168 |          1.93 |
| 50738 | 2010-05-21 19:45:01 |                   16.05 |      241 |          2.44 |
| 50739 | 2010-05-21 19:50:01 |                   19.70 |       46 |          1.19 |
| 50740 | 2010-05-21 19:55:01 |                   15.85 |      177 |          2.28 |
| 50741 | 2010-05-21 20:00:01 |                   19.04 |        8 |          0.82 |
+-------+---------------------+-------------------------+----------+---------------+

Of course, it is also possible to harness mycheckpoint’s views power to generate charts:

mysql> SELECT custom_1_psec FROM sv_report_chart_sample\G

The rules are:

• There can (currently) only be 18 custom queries.
• The custom_query_id must range 0-17 (to be lifted soon).
• A custom query must return with exactly one row, with exactly one column, which is a kind of integer.

Please read my earlier post on custom monitoring to get more background.

Custom monitoring HTML reports

Custom monitoring comes with a HTML reports, featuring requested charts. See a sample custom report.

In this sample report, a few queries are monitored for value (pending rentals, pending downloads) and a few for rates (downloads per second, emails per second etc.).

Custom HTML reports come in two flavors:

• Brief reports, featuring last 24 hours, as in the example above. These are handled by the sv_custom_html_brief view.
• Full reports, featuring last 24 hours, last 10 days, known history. These take longer to generate, and are handled by the sv_custom_html view.

The sample report was generated by issuing:

SELECT html FROM sv_custom_html_brief;

I won’t go into details here as for how this view generates the HTML code. There is a myriad of view dependencies, with many interesting tricks on the way. But do remember it’s just a view. You don’t need an application (not even mycheckpoint itself) to generate the report. All it takes is a query.

Processlist dump

When an alert notification fires (an email is prepared to inform on some alert condition), a processlist dump summary is taken and included in email report. It may be useful to understand why the slave is lagging, or exactly why there are so many active threads.

The dump summary presents the processlist much as you would see it on SHOW PROCESSLIST, but only lists the active threads, noting down how many sleeping processes there are (PS, thread & process are the same in the terminology of MySQL connections). An example dump looks like this:

PROCESSLIST summary:

     Id: 3
   User: system user
   Host:
     db: NULL
Command: Connect
   Time: 3168098
  State: Waiting for master to send event
   Info: NULL
-------

     Id: 4
   User: system user
   Host:
     db: prod_db
Command: Connect
   Time: 612
  State: Updating
   Info: UPDATE user SET is_offline = 1 WHERE id IN (50440010,50440011)
-------

     Id: 8916579
   User: prod_user
   Host: localhost
     db: prod_db
Command: Query
   Time: 1
  State: Sending data
   Info: INSERT IGNORE INTO archive.stat_archive (id, origin, path, ts, content
-------

     Id: 8916629
   User: mycheckpoint
   Host: localhost
     db: NULL
Command: Query
   Time: 0
  State: NULL
   Info: SHOW PROCESSLIST
-------
Sleeping: 3 processes

Future plans

Work is going on. These are the non-scheduled future tasks I see:

• Monitoring InnoDB Plugin & XtraDB status.
• Interactive charts. See my earlier post.
• Monitoring for swap activity (Linux only).
• Enhanced custom queries handling, including auto-deploy upon change of custom queries.
• A proper man page.
• Anything else that interests me.

Try it out

Try out mycheckpoint. It’s a different kind of monitoring solution. You will need basic SQL skills, and in return you’ll get a lot of power under your hands.

• Download mycheckpoint here
• Visit the project’s homepage
• Browse the documentation
• Report bugs

mycheckpoint is released under the New BSD License.

You can download MySQL Connector/Python from Launchpad.

Highlights:

• It was impossible to retrieve big result sets. (bug lp:551533 and lp:586003)
• Changing copyright from Sun to Oracle (also fixing silly typo)

A very Big Thanks goes to the reporters of bug lp:551533 and lp:586003. Apologies for not being able to reproduce the bug earlier, before releasing 0.1.4.

About MySQL Connector/Python: MySQL Connector/Python is implementing the MySQL Client/Server protocol completely in Python. No MySQL libraries are needed, and no compilation is necessary to run this Python DB API v2.0 compliant driver. It is compatible with Python v2.5 and later as well as Python v3.1 and later.

Highlights:

• Reading from network was broken for bigger packages.
• Reimplementing protocol.MySQLProtocol marking simpler and easier to maintain.
• It is now possible to send multiple statements to MySQL using MySQLCursor.execute(). The results are accessible by calling the method next_resultset().
• MySQLCursor.callproc() will now store all result sets as a MySQLCursorBuffered. They are accessible using the next_proc_resultset() method. The result of the stored procedure is returned by callproc() itself as defined by PEP249.
• MySQL DATETIME conversion to Python datetime.datetime is now much faster.
• Some overall performance improvements.
• Copyright notice changes.

Big thanks to everyone using and reporting bugs found in MySQL Connector/Python. Don't hesitate to ask questions and report problems or feature requests using Launchpad.

About MySQL Connector/Python: MySQL Connector/Python is implementing the MySQL Client/Server protocol completely in Python. No MySQL libraries are needed, and no compilation is necessary to run this Python DB API v2.0 compliant driver. It is compatible with Python v2.5 and later as well as Python v3.1 and later.

The book starts out very quickly with details about the various methods that are available, the means of automation for testing, and of course the environment you’d want to be in for working on the subjects that the book covers. It then, in the second chapter, moves into the guts of testing by describing the basics of doctest via syntax and some simple examples, and then moves on to a real world example via the AVL tree. It’s all very basic testing until chapter three where the author gets into unit testing, which is probably the most useful method in my opinion, and he goes to prove it’s usefulness with examples of it’s use in different parts and stages of the development process. Later in the book the python mocker is used to separate unit sections, and then the actual unittest framework is discussed with more examples and a enough details that if you don’t understand it by then, you may never. By chapter six we are into the Nose app that drives the unittest, which is very useful of course.

The most useful part of the book comes toward the end where the author discusses and the walks through the method used to create a test-driven application and then even shows examples via the whole chapter dedicated towards making a testable web application frontend. Very impressive for such a quick read. Integration testing and System testing is also covered, thankfully. The final chapter covers some useful tools and techniques of which I particularly enjoyed the section on version control hooks. If you are not using version control in your development process you need to start now, as such the hooks for integration with the test framework are rather useful to know.

Overall this is a very nice book that discusses python application testing from the ground up. It’s perfect for a beginner or an intermediate python programmer that has little to no experience in automated testing methods. More advanced programmers that have already used these methods will probably not find the book too useful except for the last chapter that covers extra tools and techniques that they might not have seen before. If I didn’t have this book and needed to learn about python testing, it would be my first choice and my only recommendation so far. Well written and very useful.

If there is one thing I do not like about the book, it would be the reliance on the python CLI for running commands. I am a CLI kind of person and I keep lots of terminals open at the same time, so I prefer to write my code in an editor or IDE in one term tab, then switch to another and execute the script; I do not use the python command line to do much of anything. So following some of the steps in the book require that you follow the CLI method and that gets old for me. It’s a personal preference but one worth noting as there is a lot of it in the book. That’s the only thing I did not enjoy in a book that was otherwise basically perfect for the subject.

• Conditional alerts
• Email notifications
• Revised HTML reports, including 24/7 reports.
• Updated documentation

With this new revision mycheckpoint turns into a monitoring solution for MySQL. One can now:

• Store measure metrics
• Query for raw, aggregated or digested metrics
• Generate charts for selected metrics
• View HTML reports for selecetd metrics
• Define alerts conditions, query for pending alerts
• Be notified via email on raised or resolved alerts.

Conditional alerts

mycheckpoint is SQL oriented. As such, it allows for creation of alert conditions, which are nothing more than SQL conditions.

For example, we wish to raise an alerts when the slave stops replicating (just ping us with an email one this happens):

INSERT INTO alert_condition (condition_eval, description, alert_delay_minutes)
  VALUES ('seconds_behind_master IS NULL', 'Slave not replicating', 0);

Or is too far behind (but since we do maintenance work during the night, it’s OK on those hours). We only want to be notified if this goes on for 10 minutes:

INSERT INTO alert_condition (condition_eval, description, alert_delay_minutes)
  VALUES ('(seconds_behind_master > 60) AND (HOUR(ts) NOT BETWEEN 2 AND 4)', 'Slave lags too far behind', 10);

We want to be notified when the datadir mount point disk quota exceeds 95% usage. Oh, and please keep nagging us about this, as long as it is unresolved:

INSERT INTO alert_condition (condition_eval, description, repetitive_alert)
  VALUES ('os_datadir_mountpoint_usage_percent > 95', 'datadir mount point is over 95%', 1);

There’s much more to alert conditions. You can generate a pending alerts report, get a textual presentation of raised and pending alerts, view the query which determines what alerts are currently raised, and more.

Read more on the alerts documentation page.

Email notifications

Introducing email notifications, mycheckpoint now:

• Sends email notification on alert conditions meeting. See sample email screenshot.
• Sends email notification when it is unable to access the database.
• Sends report via mail. Currently only HTML brief report is supported. Report is attached as HTML file in email message.

Alert notifications are automatically sent by mail (once SMTP configuration is in place, see following) when an alert is raised (alert condition becomes true) or resolved (alert condition turns false).

Email notifications require simple configuration for SMTP host, SMTP-from-address, SMTP-to-address. These can be made in the defaults file (revised), or through the command line. The following example shows how one can manually send an HTML brief report:

mycheckpoint --defaults-file=/etc/mycheckpoint.cnf --smtp-from=monitor@my-server-company.com --smtp-to=dba@my-server-company.com --smtp-host=mail.my-server-company.com email_brief_report

One should generally set up these parameters in the configuration file (aka defaults file) and forget all about it. mycheckpoint now has a default for the defaults file, which is /etc/mycheckpoint.cnf.

Read more on the emails documentation page.

Revised HTML reports

• The brief HTML reports has been updated, see sample.
• An HTML 24/7 report as been added, see sample. This report shows the distribution of popular metrics throughout the weekdays and hours.

Full HTML reports remain slow to load. I’m putting some work into this, but I’m not sure I can work around the optimizer’s limitations of using indexes for GROUPing through views.

Updated documentation

The documentation has been revised, with more details put into the pages. Since mycheckpoint gains more and more features, I saw fit to write a Quick HOWTO page which gets you up to speed, no fuss around, with mycheckpoint’s usage and features.

Read the mycheckpoint Quick HOWTO here.

Future plans

Work is going on. These are the non-scheduled future tasks I see:

• Custom monitoring + notifications. See my earlier post.
• Monitoring InnoDB Plugin & XtraDB status.
• PROCESSLIST dump on alerts.
• Interactive charts. See my earlier post.
• A proper man page…

Try it out

Try out mycheckpoint. It’s a different kind of monitoring solution. It does not require to to have a web server or complicated dependencies. To the experienced DBA it can further provide with valuable, raw or digested information in the form of SQL accessible data. I have used it to find anomalies in passing months, doing SQL search for periods of time where several conditions applied — it really gives you some extra power.

• Download mycheckpoint here
• Visit the project’s homepage
• Browse the documentation
• Report bugs

mycheckpoint is released under the New BSD License.