Archive for the ‘MySQL Enterprise Monitor’ Category

Configuring MySQL Enterprise Monitor to authenticate from LDAP

Октябрь 12th, 2010

In the last post, we saw how to create a test OpenLDAP server, populate it and secure it with SSL certificates. Now we are going to have a look at how to configure MySQL Enterprise Manager (MEM) to authenticate against LDAP. We will be examining a few different kinds of setup methods.

1. Using LDAP to fetch just the user password

The simplest form is to configure a user with MEM and set it to the LDAP type. The user’s role is setup in MEM during user creation time and is not fetched from LDAP. Below you can see the user definition page:

How to create a LDAP user in MEM - password only

How to create a LDAP user in MEM - password only

The username is user1 as specified in MEM, but where do we get the password from? We need to tell MEM how to find this user. Here is a sample user as defined by the LDIF file used in my previous post:

# user1, People, example.com
dn: uid=user1,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: top
cn: Aaren Atp
sn: Atp
description: This is the description for Aaren Atp.
employeeNumber: 1
givenName: Aaren
homePhone: +1 280 375 4325
initials: ALA
l: New Haven
mail: user.1@maildomain.net
mobile: +1 680 734 6300
ou: admin
pager: +1 850 883 8888
postalAddress: Aaren Atp$70110 Fourth Street$New Haven, OH  93694
postalCode: 936942
st: OH
street: 70110 Fourth Street
telephoneNumber: +1 390 103 6917
uid: user1
userPassword:: e1NTSEF9Z0tsZjU4cm50Wit4b045N0U4cWlldVJQK1RMOVAzTGw=

Now tell MEM to look for the uid by navigating to Settings > Global Settings > LDAP Authentication:

MEM settings for LDAP password lookup only

MEM settings for LDAP password lookup only

The password is specified in the userPassword attribute and uses standard LDAP supported formats. Technically what MEM does is an LDAP simple bind operation (Authentication Mode = Bind as User) with the given username/password and then attempts to search for the given user using the DN pattern.  If the binding and search succeeds, we assume the password is correct. With this in place, you can now login with user1 without having to specify the password in MEM.

2. Adding MEM Roles from LDAP

If we want to fetch the user’s role from LDAP in addition to the password, we need to enable the Map LDAP Roles to Application Roles check box in the LDAP settings page. We’ll see the two ways of doing this shortly.

The good thing about enabling role mapping is that manual user creation in the Web dashboard is no longer required. If the user does not already exist within MEM, it will be “copied” from LDAP to MEM when that user first logs in. The lookup order is Built-in users first, then LDAP users. If you want user authentication to be exclusively done against LDAP then make LDAP authoritative by ticking the check box:

MEM authoritative LDAP authentication

MEM authoritative LDAP authentication

WARNING: Enabling this prevents Built-in users from being authenticated. Make sure that you test with non authoritative authentication first and make sure role mapping is working. Also, ensure that you have a valid LDAP user who has admin privileges or risk locking yourself out.

Below is an example of how my test user table looked before logging in with user1:

User table before LDAP login

User table before LDAP login

And this is how it looks after a successful login:

User table after LDAP login

User table after LDAP login

Notice how the LDAP user1 entry was created automatically in the MEM user table with values from the LDAP directory.

2.2.1 MEM roles in a LDAP user’s attribute

The first way of achieving role mapping is to simply have an attribute in the user’s entry, whose attribute value will be the role. In our example, we use the ou attribute. The relevant attributes for our sample users are:

uid: user1
ou: admin
....
uid: user2
ou: dba

The relevant settings in MEM are:

MEM settings for LDAP role mapping with user attributes

MEM settings for LDAP role mapping with user attributes

Now when you login with user2 for example, its role will be fetched from LDAP. We can verify this in the User Preferences section as shown by the screenshot below:

A user that has authenticated via LDAP with role mapping

A user that has authenticated via LDAP with role mapping

If we try user1, then we will get an admin (manager) role.

Note that some versions have a bug that prevents this setup from working. Check if you are affected.

2.2.2 MEM roles specified by a LDAP group

There is another way to specify a user’s role. Instead of each user having an attribute representing the role, a roles group is created, which enumerates the users that are members of that role. In our example, this is represented in the LDIF file as:

dn: ou=groups,dc=example,dc=com
objectClass: organizationalUnit
ou: groups

dn: cn=admin,ou=groups,dc=example,dc=com
objectClass: groupOfUniqueNames
cn: admin
uniqueMember: uid=user1,ou=People,dc=example,dc=com

dn: cn=dba,ou=groups,dc=example,dc=com
objectClass: groupOfUniqueNames
cn: dba
uniqueMember: uid=user2,ou=People,dc=example,dc=com

So again, we need to tell MEM how to do role lookups from the LDAP directory.  This is how the LDAP settings in MEM would look like for our particular case:

MEM settings for LDAP role mapping in groups

MEM settings for LDAP role mapping in groups

Note, that due this bug, you might not get this particular setup to work.

2.3 Configuring LDAP lookups to use SSL and STARTTLS

Once plain text authentication is working, you can choose to secure communications between MEM and the LDAP server with either STARTTLS or SSL. Obviously the LDAP server must support it. Note that SSL (ldaps) is deprecated in the latest LDAP v3 and STARTTLS is prefered. MEM supports both types. To use STARTTLS, simply select it in the menu as follows:

MEM with STARTTLS LDAP authentication

MEM with STARTTLS LDAP authentication

There is no need to change the default port. Using SSL (ldaps) however, implies using the ldaps port, which by default is 636, and setting MEM to use SSL(ldaps) :

MEM with SSL (ldaps) LDAP authentication

MEM with SSL (ldaps) LDAP authentication

One thing to note is that the (Primary) Server Hostname should match the SSL certificate of the LDAP server.

If you are using self signed certificates that were not issued by a commercial SSL company, MEM will not have the root CA (Certificate Authority) in it’s repository. You should see a message in the $INSTALL_DIR/monitor/apache-tomcat/logs/catalina.out logfile:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

To fix this, you will need to import the your root CA certificate that was used to generate the LDAP server’s certificate. This can be achieve with the Java Keytool utility as follows:

shell> cd  /opt/mysql/enterprise/monitor/java
shell> bin/keytool -import -trustcacerts -alias ldapssl \
       -file /etc/openldap/ssl/ca-cert.pem -keystore lib/security/cacerts

This needs to be run under MEM’s Java installation directory. Unless you have previously modified it, the default password is changeit . We saw how to create a CA certificate (ca-cert.pem) in the previous post. Finally, restart the MEM monitor service.

In case you have problems importing your root CA certificate, make sure it is correctly format and acceptable by the Java keytool utility:

keytool can import X.509 v1, v2, and v3 certificates, and PKCS#7 formatted certificate chains consisting of certificates of that type. The data to be imported must be provided either in binary encoding format, or in printable encoding format (also known as Base64 encoding) as defined by the Internet RFC 1421 standard. In the latter case, the encoding must be bounded at the beginning by a string that starts with ‘—–BEGIN’, and bounded at the end by a string that starts with ‘—–END’.”

http://download.oracle.com/docs/cd/E17409_01/javase/6/docs/technotes/tools/windows/keytool.html

To troubleshoot, the best place is to look at the catalina.out logfile. If you need extra debugging information, you can start MEM’s JVM as follows:

JAVA_OPTS="$JAVA_OPTS -Djavax.net.debug=ALL"

In the current version (2.2.1.1721) this should be added at around line 180 of $INSTALL_DIR/monitor/apache-tomcat/bin/catalina.sh. Note that catalina.sh will get overwritten when you next upgrade so if you want JAVA_OPTS changes to be persistent, then use setenv.sh (or setenv.bat for Windows.). A Tomcat restart is needed. Note that a lot of information is logged and you’re log files will grow rather quickly. Please remember to turn this off in regular production mode.

Hopefully this will help when configuring MEM to authenticate it’s users against a LDAP directory.


PlanetMySQL Voting: Vote UP / Vote DOWN

Setting up OpenLDAP for MySQL Enterprise Monitor

Октябрь 12th, 2010

The latest 2.2 release of MySQL Enterprise Monitor (MEM) has the ability to authenticate against LDAP. I decided to test this setup and for that, I had to create and populate an OpenLDAP server, including STARTTLS/SSL certificates. This guide was done on CentOS 5.5 but it shouldn’t be much different in other Linux/Unix distributions. First, start off by installing the packages with:

root@shell> yum install openldap openldap-clients openldap-servers

Then head to /etc/openldap where you can set you domain and the DN for the LDAP manager user. I’ve inserted some useful comments into the slapd.conf file. Lines without comments have not been changed from the default slapd.conf file.

shell> grep -v "^#" /etc/openldap/slapd.conf | grep -v "^$"
include		/etc/openldap/schema/core.schema
include		/etc/openldap/schema/cosine.schema
include		/etc/openldap/schema/inetorgperson.schema
include		/etc/openldap/schema/nis.schema
allow bind_v2
pidfile		/var/run/openldap/slapd.pid
argsfile	/var/run/openldap/slapd.args
#The lines below are for SSL and STARTTLS.
#I'll show you how to generate certs later on
TLSCipherSuite HIGH:MEDIUM:-SSLv2
TLSCACertificateFile /etc/openldap/ssl/ca-cert.pem
TLSCertificateFile /etc/openldap/ssl/server-cert.pem
TLSCertificateKeyFile /etc/openldap/ssl/server-key.pem
#This allows ldapsearch command to connect without a client cert
TLSVerifyClient never
database	bdb
# this is your domain. I used example.com for my tests.
suffix		"dc=example,dc=com"
# this is the "username" of the LDAP admin for this domain
rootdn		"cn=Manager,dc=example,dc=com"
# this is the encripted password. To generate a SSHA password use slappasswd.
rootpw {SSHA}8diJsdIYFRr/wt7vqk3SGj6b/ZZZ21eno
directory	/var/lib/ldap
index objectClass                       eq,pres
index ou,cn,mail,surname,givenname      eq,pres,sub
index uidNumber,gidNumber,loginShell    eq,pres
index uid,memberUid                     eq,pres,sub
index nisMapName,nisMapEntry            eq,pres,sub

If for some reason, you need to compile your own version of OpenLDAP, see this guide:
http://www.openldap.org/doc/admin24/quickstart.html

The next step is to generate our certificates. First we generate the Certificate Authority and the LDAP Server certificates. One important thing is to set the CN attribute to your server’s hostname in both certificates. You can run the hostname command in the shell to find that out. This is the same hostname you will be using in the MEM setup or to test with the ldapsearch command line utility.

# Create CA and Server Certs
shell> openssl genrsa 2048 > ca-key.pem
shell> openssl req -new -x509 -nodes -days 1000 -key ca-key.pem > ca-cert.pem
shell> openssl req -newkey rsa:2048 -days 1000 -nodes -keyout server-key.pem > server-req.pem
shell> openssl x509 -req -in server-req.pem -days 1000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 > server-cert.pem

These files should be located under /etc/openldap/ssl and the server key needs to be chmoded:

shell> chmod 600 /etc/openldap/server-key.pem

Then we create the certificate for our client utilities, and let openldap know about it:

# Create client certificate
shell> openssl req -newkey rsa:2048 -days 1000 -nodes -keyout client-key.pem > client-req.pem
shell> openssl x509 -req -in client-req.pem -days 1000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 > client-cert.pem
shell> cat /etc/openldap/ldap.conf
#
# LDAP Defaults
#
# See ldap.conf(5) for details
# This file should be world readable but not world writable.
#BASE	dc=example, dc=com
#URI	ldap://ldap.example.com ldap://ldap-master.example.com:666
#SIZELIMIT	12
#TIMELIMIT	15
#DEREF		never
#TLS_CACERTDIR /etc/openldap/cacerts
TLS_CACERT /etc/openldap/ssl/ca-cert.pem

Here are how the permissions on these files look like for me:

shell> ls -la /etc/openldap/ssl/
total 40K
drwxr-xr-x 2 root root 4.0K Jun 23 13:53 .
drwxr-xr-x 5 root root 4.0K Jun 25 20:59 ..
-rw-r--r-- 1 root root 1.5K Jun 23 13:51 ca-cert.pem
-rw-r--r-- 1 root root 1.7K Jun 23 13:51 ca-key.pem
-rw-r--r-- 1 root root 1.2K Jun 23 13:53 client-cert.pem
-rw-r--r-- 1 root root 1.7K Jun 23 13:53 client-key.pem
-rw-r--r-- 1 root root 1.1K Jun 23 13:53 client-req.pem
-rw-r--r-- 1 root root 1.2K Jun 23 13:52 server-cert.pem
-rw------- 1 ldap root 1.7K Jun 23 13:52 server-key.pem
-rw-r--r-- 1 root root 1.1K Jun 23 13:52 server-req.pem

Next enable SSL for OpenLDAP and start up the server:

shell> grep -v "^#" /etc/sysconfig/ldap | grep -v "^$"
ULIMIT_SETTINGS=
STOP_DELAY=3s
SLAPD_LDAP=yes
SLAPD_LDAPS=yes
SLAPD_LDAPI=no
shell> /etc/init.d/ldap start
Checking configuration files for slapd:  bdb_db_open: Warning - No DB_CONFIG file found in directory /var/lib/ldap: (2)
Expect poor performance for suffix dc=example,dc=com.
config file testing succeeded                                                           [  OK  ]
Starting slapd:                                            [  OK  ]
shell> ps aux | grep slapd
ldap     25224  0.0  2.3 392036 191288 ?       Ssl  21:01   0:00 /usr/sbin/slapd -h ldap:/// ldaps:/// -u ldap

Don’t worry about the warning. Let’s try and query the LDAP directory now:

shell> ldapsearch -x -h localhost -b 'dc=example,dc=com'
# extended LDIF
#
# LDAPv3
# base  with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# search result
search: 2
result: 32 No such object
# numResponses: 1

As we can see, there are no entries yet. So let’s populate the directory with this LDIF file. You will be prompted for your password. It’s the one used to setup /etc/openldap/slapd.conf.

shell> ldapadd -x -D "cn=Manager,dc=example,dc=com" -W -f mem-ldap.ldif.txt
Enter LDAP Password:
adding new entry "dc=example,dc=com"
adding new entry "ou=People,dc=example,dc=com"
adding new entry "uid=user1,ou=People,dc=example,dc=com"
adding new entry "uid=user2,ou=People,dc=example,dc=com"
adding new entry "ou=groups,dc=example,dc=com"
adding new entry "cn=admin,ou=groups,dc=example,dc=com"
adding new entry "cn=dba,ou=groups,dc=example,dc=com"

And now we can see the results with:

shell> ldapsearch -x -H ldap:///localhost -b 'dc=example,dc=com'
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# example.com
dn: dc=example,dc=com
objectClass: domain
objectClass: top
dc: example
# People, example.com
dn: ou=People,dc=example,dc=com
objectClass: organizationalUnit
objectClass: top
ou: People
# user1, People, example.com
dn: uid=user1,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: top
cn: Aaren Atp
sn: Atp
description: This is the description for Aaren Atp.
employeeNumber: 1
givenName: Aaren
homePhone: +1 280 375 4325
initials: ALA
l: New Haven
mail: user.1@maildomain.net
mobile: +1 680 734 6300
ou: admin
pager: +1 850 883 8888
postalAddress: Aaren Atp$70110 Fourth Street$New Haven, OH  93694
postalCode: 936942
st: OH
street: 70110 Fourth Street
telephoneNumber: +1 390 103 6917
uid: user1
userPassword:: e1NTSEF9Z0tsZjU4cm50Wit4b045N0U4cWlldVJQK1RMOVAzTGw=
# user2, People, example.com
dn: uid=user2,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: top
cn: Aaren Atp
sn: Atp
description: This is the description for Aaren Atp.
employeeNumber: 2
givenName: Aaren
homePhone: +1 280 375 4325
initials: ALA
l: New Haven
mail: user2@maildomain.net
mobile: +1 680 734 6300
ou: dba
pager: +1 850 883 8888
postalAddress: Aaren Atp$70110 Fourth Street$New Haven, OH  93694
postalCode: 936941
st: OH
street: 70110 Fourth Street
telephoneNumber: +1 390 103 6917
uid: user2
userPassword:: e1NTSEF9Z0tsZjU4cm50Wit4b045N0U4cWlldVJQK1RMOVAzTGw=
# groups, example.com
dn: ou=groups,dc=example,dc=com
objectClass: organizationalUnit
ou: groups
# admin, groups, example.com
dn: cn=admin,ou=groups,dc=example,dc=com
objectClass: groupOfUniqueNames
cn: admin
uniqueMember: uid=user1,ou=People,dc=example,dc=com
# dba, groups, example.com
dn: cn=dba,ou=groups,dc=example,dc=com
objectClass: groupOfUniqueNames
cn: dba
uniqueMember: uid=user2,ou=People,dc=example,dc=com
# search result
search: 2
result: 0 Success
# numResponses: 8
# numEntries: 7

You should also test encrypted connections, first with STARTTLS, by adding the -ZZ option:

shell> ldapsearch -x -h localhost -ZZ -b 'dc=example,dc=com'

You can also do it with SSL (ldaps), but this has been deprecated in LDAPv3:

shell> ldapsearch -x -H ldaps:///localhost -b 'dc=example,dc=com'

If you need to look at the OpenLDAP log files, you should enable them in syslog.conf:

# Log LDAP stuff
local4.*                        /var/log/ldap.log

By default, OpenLDAP writes to the LOG_LOCAL 4 facility. You should also set the loglevel to the amount of detail needed. Either check man slapd.conf or the documentation for the available levels.

At the moment, we have a running openldap server which is populated with two test users, ready to be used by MEM. In the next post, I will discuss how to setup MEM to authenticate against our LDAP server. If you need to manipulate your LDAP directory and need a graphical tool, I found Apache Directory Studio to be very nice.


PlanetMySQL Voting: Vote UP / Vote DOWN

Connecting JBDC to MySQL Enterprise Monitor’s Query Analyzer

Октябрь 11th, 2010

With the release of MySQL Enterprise Monitor (MEM) 2.2, there is now the ability to monitor queries using the Query Analyzer (QUAN) without needing the agent proxy to be running. You can use a .NET or JDBC connector plugin to directly gather the query statistics. In the example below, we will use the MySQL Enterprise Plugin for Connector/J.

First, make sure both the Connector/J, the Connector/J plugin and the Apache Commons Logging jars are in the $CLASSPATH. At the time of writing, these are the files needed:

mysql-connector-java-5.1.12-bin.jar
c-java-mysql-enterprise-plugin-1.0.0.42.jar
required/commons-logging-1.1.1.jar

Then, add the plugin to the connection string so that it changes from something like this:

conn =	DriverManager.getConnection("jdbc:mysql://localhost:3306/test?"+"user=root&amp;password=PASSWORD");

to something like this:

conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/test?"
       +"user=root&amp;password=PASSWORD"
       +"&amp;statementInterceptors=com.mysql.etools.jdbc.StatementPerformanceCounters"
       +"&amp;serviceManagerUrl=http://memserverhost.com:18080/"
       +"&amp;serviceManagerUser=agent&amp;serviceManagerPassword=AGENTPASSWORD");

Here is the full example code:

import java.sql.*;
 
// Notice, do not import com.mysql.jdbc.*
// or you will have problems!
 
public class LoadDriver {
    public static void main(String[] args) {
 
        try {
            // The newInstance() call is a work around for some
            // broken Java implementations
 
           	Class.forName("com.mysql.jdbc.Driver").newInstance();
 
	        Connection conn = null;
		conn =	DriverManager.getConnection("jdbc:mysql://127.0.0.1:33306/test?"+"user=root&amp;password=PASSWORD"+"&amp;statementInterceptors=com.mysql.etools.jdbc.StatementPerformanceCounters&amp;serviceManagerUrl=http://memserverhostname.com:18080/&amp;serviceManagerUser=agent&amp;serviceManagerPassword=AGENTPASSWORD");
 
		//Get a Statement object
	      	Statement stmt = conn.createStatement();
    		ResultSet rs = stmt.executeQuery("SELECT SLEEP(10)");
		System.out.println("Closing connection");
		rs.close();
		conn.close();
        } catch (Exception ex) {
            // handle the error
	    ex.printStackTrace();
        }
    }
}

Now, when you Java application runs queries, they will be analyzed by QUAN. In my example code, there is a sleep call for 10 seconds. You can clearly see that this query was caught by QUAN:

Slow queries fed to QUAN using the Connector/J plugin

This now eliminates the need for the monitoring agent to be running it’s own instance of mysql-proxy for QUAN. Java and .NET applications can simply use the interceptor, saving on resources.

Of course, prior to setting this up, you should always check the online documentation.


PlanetMySQL Voting: Vote UP / Vote DOWN

Careful how you monitor MySQL

Сентябрь 7th, 2010

I was recently struck by a problem which is unusual. In order to keep an eye on the database server I use nagios, cacti, merlin and some local scripts to monitor the database instance and ensure that it is working properly.  That normally works fine.  The different monitor processes do various things, one of which is to monitor the replication status of a slave, and warn me if the replication is not working or if it’s behind. This is done with the command SHOW SLAVE STATUS.

The server I was looking at runs some large local batch jobs aggregating data. Unfortunately, I was experiencing that replication was interfering with these batch jobs so decided to see if things would perform better if I stopped replication and let the batch jobs complete, restarting replication afterwards.

So the command STOP SLAVE was sent to the server, and this took some time. In the meantime SHOW SLAVE STATUS hangs. The STOP SLAVE command waits for the SQL replication thread to finish it’s task but that thread was waiting on the tables being used by the batch process, which was already running. The consequence of this was that STOP SLAVE waited, all calls to SHOW SLAVE STATUS blocked and with nagios, cacti, merlin and the local scripts all doing periodic SHOW SLAVE STATUS commands which would hang I ended up with the mysqld running out of user connections.  Nasty!

While I see if MySQL can do something about the behaviour of stopping the slave and showing the slave status I’m going to implement grants which limit the monitor users so they are only allowed to have a user concurrent connections open.  This is done using the syntax

GRANT whatever ON whereever TO some_user WITH MAX_USER_CONNECTIONS 5;

5 looks like a good number for nagios, merlin and the local scripts and 10 is probably sufficient as I collect a lot of graph data for the mysql server and the cacti requests are often done in parallel.

So if you monitor MySQL it may be worth you applying some sort of connection limit on your monitoring scripts, if not on other database users.


PlanetMySQL Voting: Vote UP / Vote DOWN

Careful how you monitor MySQL

Сентябрь 7th, 2010

I was recently struck by a problem which is unusual. In order to keep an eye on the database server I use nagios, cacti, merlin and some local scripts to monitor the database instance and ensure that it is working properly.  That normally works fine.  The different monitor processes do various things, one of which is to monitor the replication status of a slave, and warn me if the replication is not working or if it’s behind. This is done with the command SHOW SLAVE STATUS.

The server I was looking at runs some large local batch jobs aggregating data. Unfortunately, I was experiencing that replication was interfering with these batch jobs so decided to see if things would perform better if I stopped replication and let the batch jobs complete, restarting replication afterwards.

So the command STOP SLAVE was sent to the server, and this took some time. In the meantime SHOW SLAVE STATUS hangs. The STOP SLAVE command waits for the SQL replication thread to finish it’s task but that thread was waiting on the tables being used by the batch process, which was already running. The consequence of this was that STOP SLAVE waited, all calls to SHOW SLAVE STATUS blocked and with nagios, cacti, merlin and the local scripts all doing periodic SHOW SLAVE STATUS commands which would hang I ended up with the mysqld running out of user connections.  Nasty!

While I see if MySQL can do something about the behaviour of stopping the slave and showing the slave status I’m going to implement grants which limit the monitor users so they are only allowed to have a user concurrent connections open.  This is done using the syntax

GRANT whatever ON whereever TO some_user WITH MAX_USER_CONNECTIONS 5;

5 looks like a good number for nagios, merlin and the local scripts and 10 is probably sufficient as I collect a lot of graph data for the mysql server and the cacti requests are often done in parallel.

So if you monitor MySQL it may be worth you applying some sort of connection limit on your monitoring scripts, if not on other database users.


PlanetMySQL Voting: Vote UP / Vote DOWN

Breakfast seminar on what’s new with MySQL – London

Июнь 17th, 2010

If you’re in London on Thursday 24th June then there’s a great chance to find out what’s new in MySQL.

Join us for an Oracle MySQL Breakfast Seminar to better understand Oracle’s MySQL strategy and what’s new with MySQL!
Agenda:
09:00 a.m.    Welcome Coffee/Tea
09:30 a.m.    Oracle’s MySQL Strategy
10:00 a.m.    What’s New – The MySQL Server & MySQL Cluster
10.45 a.m.    Coffee/Tea Break
11:00 a.m.    What’s New – MySQL Enterprise & MySQL Workbench
11:45 a.m.    Q&A
12:00 noon    End of the Breakfast Seminar

Cost?
None, it’s a free event! But places are limited and the seminar is held on a first come first served basis, so register quickly!

Location:

Sun Microsystem’s Customer Briefing Center
Regis House
45 King William Street
London EC4R 9AN
Tel: (020) 7628 3000

Image courtesy of Anirudh Koul.

Join us for an Oracle MySQL Breakfast Seminar in London, Thursday June 24th 2010, to better understand Oracle’s MySQL strategy and what’s new with MySQL!

Agenda:
09:00 a.m. Welcome Coffee/Tea
09:30 a.m. Oracle’s MySQL Strategy
10:00 a.m. What’s New – The MySQL Server & MySQL Cluster
10.45 a.m. Coffee/Tea Break
11:00 a.m. What’s New – MySQL Enterprise & MyQL Workbench
11:45 a.m. Q&A
12:00 noon End of the Breakfast Seminar

* Agenda subject to change

Cost?
None, it’s a free event! But places are limited and the seminar is held on a first come first served basis, so register quickly!


PlanetMySQL Voting: Vote UP / Vote DOWN

Enable MySQL Enterprise Plugin for Connector/NET

Июнь 15th, 2010

Figure 1. Source Location

In a prior post ( Trace SQL From Database to Source Code ), I showed how to enable SQL trace capabilities for java/MySQL application to trace SQL statements from the database to the exact line of code from which the statement was executed (see Figure 1).  In this post, I’ll enable SQL tracing in the sample C# application, which is included with the MySQL Connector/NET (MySQL’s ADO.NET provider ) install.

The following instructions assume that the MySQL Enterprise Agent and Monitor is already installed.  The Monitor is available for support customers at http://customer.mysql.com or a trial is available at http://www.mysql.com/trials .

Step #1.  Download and install the Connector/NET and the MySQL Enterprise Plugin for Connector/NET

  • Connector/NET 6.2.3 or newer version. The assembly is MySQL.Data.dll  Select the option to install the C# samples.
  • MySQL Enterprise Plugin for Connnector/NET ( http://customer.mysql.com or www.mysql.com/trials ). The assembly is MySQL.MonitorPlugin.dll

Step #2.  Copy the plugin, MySQL.MonitorPlugin.dll to your application directory.  In my case, I’m building and debugging the sample application using Microsoft Visual C# 2008 Express Edition.  The MySQL.MonitorPlugin.dll needs to reside in the same directly as the application’s executable.  Copy the dll to C:\Program Files\MySQL\MySQL Connector Net 6.2.3\Samples\Table Editor\cs\bin\Debug

Step #3.  Open the sample project TableEditor.csproj ( C:\Program Files\MySQL\MySQL Connector Net 6.2.3\Samples\Table Editor\cs\TableEditor.csproj ) in Microsoft Visual Studio and add an App.config file to the project.  For this exercise, I’m using Microsoft Visual C# 2008 Express Edition.

To add the App.config file, select Solution Explorer on the View menu, right-click on TableEditor, the project name.  Point to Add, New Item, and choose Application Configuration File.

In Solution Explorer, double-click App.config to open the file.  Replace the contents of the file with the XML below.  Note: replace the items in brackets [ ] with the appropriate information for your server.  (Omit the brackets) Likewise, the UserID and Password should match the credentials that the agent uses to authenticate to the Monitor.

Copy and paste the following into App.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.diagnostics>
    <sources>
      <source name="mysql" switchName="SourceSwitch"    switchType="System.Diagnostics.SourceSwitch">
        <listeners>
          <add name="EMTrace" type="MySql.EMTrace.EMTraceListener, MySql.MonitorPlugin"
                    initializeData=""
            Host="[http://yourServer.com:18080]"
            PostInterval="60"
            UserId="[agent]"
            Password="[mysql]"/>
          </listeners>
      </source>
    </sources>
    <switches>
      <!-- You can set the level at which tracing is to occur -->
      <add name="SourceSwitch" value="All"/>
    </switches>
  </system.diagnostics>
</configuration>

[Note: if you're following the readme or documentation, you'll notice that I omitted the <system.data> section.  I believe this to be an error in the documentation.  In my testing, it never works with the <system.data> entry.]

Step #4.  Modify the connection string to include “logging = true”

Here’s the full connection string that I used while running the sample application:

string connStr = String.Format(“server={0};user id={1}; password={2}; database=mysql; port=3307; pooling=false; logging=true”,server.Text, userid.Text, password.Text );

Step #5.  Compile the application ( Debugging | Start Debugging (F5) ).   On Form1, enter the appropriate Server, User ID and Password for the MySQL Server and toggle the databases and tables.  Watch the Output window, select Output from the View menu in the IDE, and you should start seeing SQL debug information, which will be passed to the MySQL Enterprise Monitor.  In the App.config, the PostInterval is set to 30, meaning that it will take 30 seconds before the data is sent to the monitor.

If you’re a .NET developer, please give it a try.  The query analyzer provides a great view into your application and the database.



PlanetMySQL Voting: Vote UP / Vote DOWN

Configure MySQL Enterprise Monitor to monitor MySQL Cluster

Май 20th, 2010

MySQL Cluster 7.1 introduced the ndbinfo database which contains views giving real-time access to a whole host of information that helps you monitor and tune your MySQL Cluster deployment. Because this data can be accessed through regular SQL, various systems can be configured to monitor the Cluster. This post gives one example, extending MySQL Enterprise Monitor to keep an eye on the amount of free memory on the data nodes (through a graph) and then raise an alarm when it starts to run low – even generating SNMP traps if that’s what you need.

One of the features of MySQL Enterprise Monitor is that you can define custom data collectors and that those data collectors can run SQL queries to get the data. The information retrieved by those custom data collectors can then be used with rules that the user defines through the MySQL Enterprise Monitor GUI to create warning/alarms.

In this example, I create two new data collectors in the file”<MySQL Enterprise Monitor installation directory>/agent/share/mysql-proxy/items/cluster.xml” before starting up the MySQL Enterprise Monitor agent (note that these should be created for the agent of each MySQL Server in the Cluster that you would like to use to present the information from the data nodes):

cluster.xml:

<?xml version="1.0" encoding="utf-8"?>
<classes>
  <class>
    <namespace>mysql</namespace>
    <classname>cluster_max_used</classname>
    <query><![CDATA[SELECT MAX(used) AS Used FROM ndbinfo.memoryusage WHERE memory_type = 'Data Memory';]]></query>
  </class>
  <class>
    <namespace>mysql</namespace>
    <classname>cluster_min_avail</classname>
    <query><![CDATA[SELECT MIN(total) AS Total FROM ndbinfo.memoryusage WHERE memory_type = 'Data Memory';]]></query>
  </class>
</classes>

So that the agent picks up this file, it should be referenced within <MySQL Enterprise Monitor installation directory>/agent/mysql-monitor-agent.ini:

agent-item-files = share/mysql-monitor-agent/items/quan.lua,share/mysql-monitor-agent/items/items-mysql-monitor.xml,
share/mysql-monitor-agent/items/custom.xml,share/mysql-monitor-agent/items/cluster.xml

In MySQL Enterprise Monitor, events are raised by rules. Rules are grouped together into Advisors and so I create a new Advisor called “MySQL Cluster” and then create just one new rule within that Advisor group.

As shown in Fig. 1 the rule is called “Data Node Low Memory”. The “Variable Assignment” section is used to define 2 variables %used_mem% and %config_mem% which are populated from the Used and Total results from the 2 new data collectors. The “Expression” section is used to test “((Total - Used)/Total)x100< THRESHOLD” and then the values to be substituted for THRESHOLD are defined in the “Thresholds” section – indicating at what points the Info, Warning and Critical Alters should be raised.

There are then a number of optional sections that you can use to add useful information to the person investigating the alert.

Once the rule has been created, the next step is to schedule and (if desired) tag that the alerts should also result in SNMP traps being raised. This is standard MySQL Enterprise Monitor practice and so it isn’t explained here except to point out that this rule is monitoring information from the data nodes but the rule has to be applied to a MySQL Server in the Cluster (MySQL Enterprise Monitor has no idea what a data node is) and so you need to schedule the rule against one or more arbitrary MySQL Server instances in the Cluster).

Fig. 2 Warning alert

To test the functionality, start adding more data to your MySQL Cluster until the Warning alert is triggered as shown in Fig. 2. As you can see, the optional information we included is shown – including values from Used and Total.

 

 

 

 

 

Fig. 3 Major alert

I then add more data to the database until the critical alert is raised and confirm that it’s displayed on the main monitoring panel of the MySQL Enterprise Monitor dashboard. Note that if you requested these alerts be included with the SNMP feed then SNMP traps will also be raised.

Please note that this example is intended to illustrate the mechanics of setting up monitoring on an arbitrary piece of data from ndbinfo and obviously in the real world you would want to monitor more than just the memory and even for the memory, you might want to use a more sophisticated rule.

Fig. 4 Custom graph for memory usage

 

 

 

 

It is sometimes more useful to see how a value changes over time. For this, MySQL Enterprise Monitor provides graphs. The data collectors created for the rule can also be used to add a new graph to Enterprise monitor. The graph is defined by creating the following file:

<com_mysql_merlin_server_graph_Design>
  <version>1.0</version>
  <uuid>b0bc2bba-ea9b-102b-b396-94aca32b0b28</uuid>
  <tag></tag>
  <name>Per Data Node Data Memory Use</name>
  <rangeLabel>MB</rangeLabel> <frequency>00:01:00</frequency>
  <series>
    <label>Used</label>
    <expression>cluster_data_node_used_data_memory/1024/1024</expression>
  </series>
  <series>
    <label>Avail</label>
    <expression>cluster_data_node_config_data_memory/1024/1024</expression>
  </series>
  <variables>
    <name>cluster_data_node_used_data_memory</name>
    <dcItem>
      <nameSpace>mysql</nameSpace>
      <className>cluster_max_used</className>
      <attribName>Used</attribName>
    </dcItem>
    <instance>local</instance>
  </variables>
  <variables>
    <name>cluster_data_node_config_data_memory</name>
    <dcItem>
      <nameSpace>mysql</nameSpace>
      <className>cluster_min_avail</className>
      <attribName>Total</attribName>
    </dcItem>
    <instance>local</instance>
  </variables>
</com_mysql_merlin_server_graph_Design>
 

Fig. 5 MySQL Enterprise Monitor dashboard

Click on Import/Export in the Graphs tab in Enterprise Monitor (2.2) and then import the file defining the graph.

The graph will then appear on the graphs tab and can also be configured to appear on the main dashboard as shown in Fig. 5


PlanetMySQL Voting: Vote UP / Vote DOWN

Using NDBINFO – example of monitoring data nodes with MySQL Enterprise Monitor

Февраль 4th, 2010

You may have read Berd’s recent post that explained how to try out some new beta functionality for MySQL Cluster and wondered what kind of use you could put the new ndb$info to. ndb$info uses tables/views to give real-time access to a whole host of information that helps you monitor and tune your MySQL Cluster deployment. This article gives one example, extending MySQL Enterprise Monitor to keep an eye on the amount of free memory on the data nodes and then raise an alarm when it starts to run low – even generating SNMP traps if that’s what you need.

One of the features of MySQL Enterprise Monitor is that you can define custom data collectors and that those data collectors can run SQL queries to get the data. The information retrieved by those custom data collectors can then be used with rules that the user defines through the MySQL Enterprise Monitor GUI to create warning/alarms.

In this example, I create two new data collectors and store the files in the “<MySQL Enterprise Monitor installation directory>/agent/share/mysql-proxy/items/” directory before starting up the MySQL Enterprise Monitor agents:

cluster_max_used.xml:

<?xml version="1.0" encoding="utf-8"?>
<classes>
  <class>
    <namespace>mysql</namespace>
    <classname>cluster_max_used</classname>
    <query><![CDATA[SELECT MAX(used) as Used FROM ndbinfo.memoryusage
      WHERE DATA_MEMORY = 'DATA_MEMORY']]>
    </query>
  </class>
</classes>

cluster_min_avail.xml:

<?xml version="1.0" encoding="utf-8"?>
<classes>
  <class>
    <namespace>mysql</namespace>
    <classname>cluster_min_avail</classname>
    <query><![CDATA[SELECT MIN(max)as Max FROM ndbinfo.memoryusage
       WHERE DATA_MEMORY = 'DATA_MEMORY']]>
    </query>
  </class>
</classes>

Fig. 1 Rule definition

In MySQL Enterprise Monitor, rules are grouped together into Advisors and so I create a new Advisor called “MySQL Cluster” and then create just one new rule within that Advisor group.

As shown in Fig. 1 the rule is called “Data Node Low Memory”. The “Variable Assignment” section is used to define 2 variables %used_mem% and %config_mem% which are populated from the Used and Max results from the 2 new data collectors. The “Expression” section is used to test “(Max - Used) < THRESHOLD” and then the values to be substituted for THRESHOLD are defined in the “Thresholds” section – indicating at what points the Info, Warning and Critical Alters should be raised.

There are then a number of optional sections that you can use to add useful information to the person investigating the alert.

Once the rule has been created, the next step is to schedule it and (if desired) tag that the alerts should also result in SNMP traps being raised. This is standard MySQL Enterprise Monitor practice and so it isn’t explained here except to point out that this rule is monitoring information from the data nodes but the rule has to be applied to a MySQL Server (MySQL Enterprise Monitor has no idea what a data node is) and so you need to schedule the rule against one or more arbitrary MySQL Server instances in the Cluster.

Fig. 2 Warning alert


 
 
To test the functionality, start adding more data to your MySQL Cluster until the Warning alert is triggered as shown in Fig. 2. As you can see, the optional information we included is shown – including values from Used and Max.
 
 
 
 
 
 

Fig. 3 Critical alarm

I then add more data to the database until the critical alert is raised and confirm that it’s displayed on the main monitoring panel of the MySQL Enterprise Monitor dashboard. Note that if you requested these alerts be included with the SNMP feed then SNMP traps will also be raised.

Please note that this example is intended to illustrate the mechanics of setting up monitoring on an arbitrary piece of data from ndbinfo and obviously in the real world you would want to monitor more than just the memory and even for the memory, you might want to use a more sophisticated rule.


PlanetMySQL Voting: Vote UP / Vote DOWN

Less time finding, more time fixing! Enterprise Monitor 2.1, Updated Query Analyzer Now GA!

Сентябрь 9th, 2009
I just wanted to tip my hat to the MySQL Enterprise Tools Engineering team for another great release of the Enterprise Monitor. Not to name names, but I want to give a special thanks to a team that always over delivers on a collective commitment to producing quality software. So, a mega thanks to:

Andy Bang, Sloan Childers, Darren Oldag, Eric Herman, Jan Kneschke, Kay Roepke, Mark Matthews, Bill Weber, Diego Medina, Marcos Palacios, Carsten "Pino" Segieth, Josh Sled, Keith Russell, Mark Leith, Heidi Bergh-Hoff, and Gary Whizin (and also welcome Michael Schuster!)

Yet another great job guys!

The new version, 2.1, was posted as GA early on Tuesday and it is quite possibly the best release of the Enterprise Monitor to date.

For those not familiar with the Enterprise Monitor, it is included in a MySQL Enterprise subscription and is a distributed web-based app that users deploy in their environment to monitor and tune the security, performance and availability of their dev, QA and production MySQL servers. It is comprised of :

  • An agent, written in C, that is installed on each monitored data source, which collects MySQL and OS metrics, SQL code and exec results
  • A central server (aka "service manager"), written in Java, that collects, monitors and alerts on the data collected by each agent. The service manager uses MySQL Best Practice Advisors to measure the collected data against user-defined thresholds and to proactively notify DBAs of problems or tuning opportunities. Alerts are sent to a central console or via SMTP or SNMP notifications.
  • A repository that holds the data collected by each agent. The service manager monitors the data stored in repository vs maintaining a persistent connection to each MySQL server.
The Enterprise Monitor also provides MySQL specific monitoring features. There is a Replication Monitor that proactively monitors replication topologies for synch and performance issues and a new Query Analyzer that made its debut in release 2.0 in late 2008. The Query Analyzer is designed to save DBAs/Devs time in finding the most expensive queries (by total exec time, exec count, amount of data returned, etc) running across all dev, QA and prod MySQL servers without any dependence on the MySQL logs or things like SHOW PROCESSLIST. You can learn more about the Query Analyzer here.

Based on customer interviews and years spent in the field, we understand the pain associated with finding bottlenecks rooted in poorly written or inefficient SQL code (this is consistently the #1 problem we hear when talking with MySQL DBAs and devs.) This new release helps a DBA/Dev spend less time "finding" and more time "fixing" poorly performing queries. The key new features in the Query Analyzer include:

  • Clickable MySQL and OS graphs that visual correlation system and query activity. Mouse-over a spike in any graph and drill into the queries that were running at the same time. Big time saver.
  • Drill down capability for query specific executions – drill into any query and see execution specific graphs for exec time, count, data returned. Helps you see the "normal" exec pattern for a query and identify outliers that may occur during specific windows or time or for specific variable combinations.
  • Counts for SQL errors/warnings - help you quickly identify queries that may have never finished or that finished in error. These routinely go undetected by the MySQL logs, etc.
  • UI support for EXPLAIN generation threshold - gone are the days of hacking the quan.lua script to set this value!
We have also added new Advisor Rules and Graphs around connections, stale table statistics, tables without indexes, PKS and locked and long running processes.

The new release also includes an updated "What's New" page that allows you to optionally subscribe to live feeds for your open support issues (nice, especially for the issues with a status of "waiting on customer") and for MySQL product updates and alerts. Nice time saver, especially for getting updates on new releases of the Enterprise Server and for Monitor Advisors and graphs.

So how do you get it? Glad you asked...if you are an Enterprise subscriber you can grab the new release and all of the updated docs from the Enterprise Customer download page. If you are interested in learning more or want to try the new Monitor and Query Analyzer for yourself, you can register for a 30 day trial subscription which includes a fully functioning version of the Monitor and Query Analyzer. You also learn everything about MySQL Enterprise by visiting the MySQL web site.

Look forward to hearing about your experience with the new release!

PlanetMySQL Voting: Vote UP / Vote DOWN