Archive for the ‘monitor’ Category

Monitoring Related OpenWorld Talks

Октябрь 12th, 2011

I gave two monitoring related talks at OpenWorld, thanks to all that came along!

Both were monitoring related, the first an introduction to MySQL Enterprise Monitor, and the second a look at some of the new instrumentation that is getting developed, primarily within the MySQL 5.6 release. 

If you'd like to get a quick overview of how MySQL Enterprise Monitor works, then take a look through the "Getting to Know MySQL Enterprise Monitor" talk. This gives you a high level view of how the different pieces fit together, and then each of the important factors within the user interface:

And if you are interested in seeing how the instrumentation and monitoring landscape will look when 5.6 hits the streets, then you can get a sneak peak at the "MySQL Monitoring Mechanisms" talk, which runs through the new instrumentation in Performance Schema, the new INFORMATION_SCHEMA tables for monitoring InnoDB, and the new ways to trace how a statement executes with the enhanced EXPLAIN plan, and the new Optimizer Trace:


PlanetMySQL Voting: Vote UP / Vote DOWN

Monitoring Related OpenWorld Talks

Октябрь 12th, 2011

I gave two monitoring related talks at OpenWorld, thanks to all that came along!

Both were monitoring related, the first an introduction to MySQL Enterprise Monitor, and the second a look at some of the new instrumentation that is getting developed, primarily within the MySQL 5.6 release. 

If you'd like to get a quick overview of how MySQL Enterprise Monitor works, then take a look through the "Getting to Know MySQL Enterprise Monitor" talk. This gives you a high level view of how the different pieces fit together, and then each of the important factors within the user interface:

And if you are interested in seeing how the instrumentation and monitoring landscape will look when 5.6 hits the streets, then you can get a sneak peak at the "MySQL Monitoring Mechanisms" talk, which runs through the new instrumentation in Performance Schema, the new INFORMATION_SCHEMA tables for monitoring InnoDB, and the new ways to trace how a statement executes with the enhanced EXPLAIN plan, and the new Optimizer Trace:


PlanetMySQL Voting: Vote UP / Vote DOWN

Careful how you monitor MySQL

Сентябрь 7th, 2010

I was recently struck by a problem which is unusual. In order to keep an eye on the database server I use nagios, cacti, merlin and some local scripts to monitor the database instance and ensure that it is working properly.  That normally works fine.  The different monitor processes do various things, one of which is to monitor the replication status of a slave, and warn me if the replication is not working or if it’s behind. This is done with the command SHOW SLAVE STATUS.

The server I was looking at runs some large local batch jobs aggregating data. Unfortunately, I was experiencing that replication was interfering with these batch jobs so decided to see if things would perform better if I stopped replication and let the batch jobs complete, restarting replication afterwards.

So the command STOP SLAVE was sent to the server, and this took some time. In the meantime SHOW SLAVE STATUS hangs. The STOP SLAVE command waits for the SQL replication thread to finish it’s task but that thread was waiting on the tables being used by the batch process, which was already running. The consequence of this was that STOP SLAVE waited, all calls to SHOW SLAVE STATUS blocked and with nagios, cacti, merlin and the local scripts all doing periodic SHOW SLAVE STATUS commands which would hang I ended up with the mysqld running out of user connections.  Nasty!

While I see if MySQL can do something about the behaviour of stopping the slave and showing the slave status I’m going to implement grants which limit the monitor users so they are only allowed to have a user concurrent connections open.  This is done using the syntax

GRANT whatever ON whereever TO some_user WITH MAX_USER_CONNECTIONS 5;

5 looks like a good number for nagios, merlin and the local scripts and 10 is probably sufficient as I collect a lot of graph data for the mysql server and the cacti requests are often done in parallel.

So if you monitor MySQL it may be worth you applying some sort of connection limit on your monitoring scripts, if not on other database users.


PlanetMySQL Voting: Vote UP / Vote DOWN

Careful how you monitor MySQL

Сентябрь 7th, 2010

I was recently struck by a problem which is unusual. In order to keep an eye on the database server I use nagios, cacti, merlin and some local scripts to monitor the database instance and ensure that it is working properly.  That normally works fine.  The different monitor processes do various things, one of which is to monitor the replication status of a slave, and warn me if the replication is not working or if it’s behind. This is done with the command SHOW SLAVE STATUS.

The server I was looking at runs some large local batch jobs aggregating data. Unfortunately, I was experiencing that replication was interfering with these batch jobs so decided to see if things would perform better if I stopped replication and let the batch jobs complete, restarting replication afterwards.

So the command STOP SLAVE was sent to the server, and this took some time. In the meantime SHOW SLAVE STATUS hangs. The STOP SLAVE command waits for the SQL replication thread to finish it’s task but that thread was waiting on the tables being used by the batch process, which was already running. The consequence of this was that STOP SLAVE waited, all calls to SHOW SLAVE STATUS blocked and with nagios, cacti, merlin and the local scripts all doing periodic SHOW SLAVE STATUS commands which would hang I ended up with the mysqld running out of user connections.  Nasty!

While I see if MySQL can do something about the behaviour of stopping the slave and showing the slave status I’m going to implement grants which limit the monitor users so they are only allowed to have a user concurrent connections open.  This is done using the syntax

GRANT whatever ON whereever TO some_user WITH MAX_USER_CONNECTIONS 5;

5 looks like a good number for nagios, merlin and the local scripts and 10 is probably sufficient as I collect a lot of graph data for the mysql server and the cacti requests are often done in parallel.

So if you monitor MySQL it may be worth you applying some sort of connection limit on your monitoring scripts, if not on other database users.


PlanetMySQL Voting: Vote UP / Vote DOWN

Don’t forget to monitor your nameservers

Август 24th, 2010

As I mentioned in a past article I got my IPv6 connectivity working so started working on setting up various IPv6 services. One of these was to setup my name server so it also worked on IPv6.

This worked fine, but recently I lost my IPv6 connectivity but thought no more about it. I’m trying to get it working again but that’s not happened yet.

Over the last few days I’ve had some intermittent issues logging in to my home pc from my laptop and I couldn’t figure out why. Eventually I had a look at the DNS requests and of course some requests were being made to my non-reachable name server. This generated some network delays while the resolver timed out and then tried talking to one of the IPv4 name servers and correctly retrieving the information.

So conclusion, if you setup yourself with a name server for your domain and include access on IPv6 then ensure that you monitor it. I do run nagios at home and intend to add some extra checks so that I can see if there are problems. Even when playing with IPv6 if you don’t have things setup properly it can affect non-IPv6 activity…

Once the unreachable IPv6 AAAA entry was removed from my name server things went back to normal.


PlanetMySQL Voting: Vote UP / Vote DOWN

Enable MySQL Enterprise Plugin for Connector/NET

Июнь 15th, 2010

Figure 1. Source Location

In a prior post ( Trace SQL From Database to Source Code ), I showed how to enable SQL trace capabilities for java/MySQL application to trace SQL statements from the database to the exact line of code from which the statement was executed (see Figure 1).  In this post, I’ll enable SQL tracing in the sample C# application, which is included with the MySQL Connector/NET (MySQL’s ADO.NET provider ) install.

The following instructions assume that the MySQL Enterprise Agent and Monitor is already installed.  The Monitor is available for support customers at http://customer.mysql.com or a trial is available at http://www.mysql.com/trials .

Step #1.  Download and install the Connector/NET and the MySQL Enterprise Plugin for Connector/NET

  • Connector/NET 6.2.3 or newer version. The assembly is MySQL.Data.dll  Select the option to install the C# samples.
  • MySQL Enterprise Plugin for Connnector/NET ( http://customer.mysql.com or www.mysql.com/trials ). The assembly is MySQL.MonitorPlugin.dll

Step #2.  Copy the plugin, MySQL.MonitorPlugin.dll to your application directory.  In my case, I’m building and debugging the sample application using Microsoft Visual C# 2008 Express Edition.  The MySQL.MonitorPlugin.dll needs to reside in the same directly as the application’s executable.  Copy the dll to C:\Program Files\MySQL\MySQL Connector Net 6.2.3\Samples\Table Editor\cs\bin\Debug

Step #3.  Open the sample project TableEditor.csproj ( C:\Program Files\MySQL\MySQL Connector Net 6.2.3\Samples\Table Editor\cs\TableEditor.csproj ) in Microsoft Visual Studio and add an App.config file to the project.  For this exercise, I’m using Microsoft Visual C# 2008 Express Edition.

To add the App.config file, select Solution Explorer on the View menu, right-click on TableEditor, the project name.  Point to Add, New Item, and choose Application Configuration File.

In Solution Explorer, double-click App.config to open the file.  Replace the contents of the file with the XML below.  Note: replace the items in brackets [ ] with the appropriate information for your server.  (Omit the brackets) Likewise, the UserID and Password should match the credentials that the agent uses to authenticate to the Monitor.

Copy and paste the following into App.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.diagnostics>
    <sources>
      <source name="mysql" switchName="SourceSwitch"    switchType="System.Diagnostics.SourceSwitch">
        <listeners>
          <add name="EMTrace" type="MySql.EMTrace.EMTraceListener, MySql.MonitorPlugin"
                    initializeData=""
            Host="[http://yourServer.com:18080]"
            PostInterval="60"
            UserId="[agent]"
            Password="[mysql]"/>
          </listeners>
      </source>
    </sources>
    <switches>
      <!-- You can set the level at which tracing is to occur -->
      <add name="SourceSwitch" value="All"/>
    </switches>
  </system.diagnostics>
</configuration>

[Note: if you're following the readme or documentation, you'll notice that I omitted the <system.data> section.  I believe this to be an error in the documentation.  In my testing, it never works with the <system.data> entry.]

Step #4.  Modify the connection string to include “logging = true”

Here’s the full connection string that I used while running the sample application:

string connStr = String.Format(“server={0};user id={1}; password={2}; database=mysql; port=3307; pooling=false; logging=true”,server.Text, userid.Text, password.Text );

Step #5.  Compile the application ( Debugging | Start Debugging (F5) ).   On Form1, enter the appropriate Server, User ID and Password for the MySQL Server and toggle the databases and tables.  Watch the Output window, select Output from the View menu in the IDE, and you should start seeing SQL debug information, which will be passed to the MySQL Enterprise Monitor.  In the App.config, the PostInterval is set to 30, meaning that it will take 30 seconds before the data is sent to the monitor.

If you’re a .NET developer, please give it a try.  The query analyzer provides a great view into your application and the database.



PlanetMySQL Voting: Vote UP / Vote DOWN

Trace SQL From the Database to the Source Code with MySQL Enterprise Monitor

Июнь 11th, 2010

OK, you found the problem SQL statement that was affecting your server’s performance, now where did it originate?

The new MySQL Enterprise Plugins for Connector/J and Connector/NET send query statistics, including the source location for each query, directly to the MySQL Enterprise Monitor.

Figure 1 is a screenshot of new source location feature.

Figure 1. Source Location

Figure 2 shows the standard query statistics, which are collected in the query analyzer.  In both cases, the statistics are gathered by the MySQL Connector and the Plugin, not MySQL proxy.

Figure 2. Query Analyzer

If you’re a MySQL Enterprise customer, you can download the new monitor and plugins from customer.mysql.com.  If you’d like to try it out, a 30 day trial is available at http://www.mysql.com/trials/

Please review the help file and readme for full instructions, but here are a few of the highlights for setting up monitoring for Java applications.  (I’ll provide further instructions for .NET in a future post.)

1.  Set up the MySQL Enterprise Monitor and Agent.  The agent resides on the MySQL production server and the monitor should reside on a non-production server within your environment.  Although the Plugin gathers the query statistics, the agent needs to run and connect to the monitor at least once so the Monitor can associate the data with the appropriate server.

2.  Download the following files from the customer portal (customer.mysql.com) and add them to your classpath:

  • Connector/J version 5.1.12 or newer
  • Plugin for Connector/J c-java-mysql-enterprise-plugin-1.0.0.42.jar

3. Change your JDBC url.  In my case, I changed it from the standard JDBC url, “jdbc:mysql:127.0.0.1:3306/sakila , to

“jdbc:mysql://127.0.0.1:3306/sakila?statementInterceptors=com.mysql.etools.jdbc.StatementPerformanceCounters& amp;serviceManagerUrl=http://[mymonitorURL.com]&serviceManagerUser=[myagentusername]& serviceManagerPassword=[myagentpassword]l”;

Full instructions are in the readme.  Substitute appropriate URL and credentials for the bracketed [ ] items above.


4. Finally, the mysql user account that is used by the application needs to have SELECT privilege to the mysql.inventory table.  This table includes a unique identifier which the monitor uses to match the data with the MySQL instance.  I executed the following:

mysql> GRANT SELECT ON mysql.inventory TO ‘elgato’@'%’;

I found it helpful to monitor the application log file for error messages.   The readme includes additional logging information.  In my tomcat test application, the log information was sent to /var/logs/tomcat6/daemon.log file.

I hope this is helpful information.  If you have any issues, feel free to post a comment on this blog post or post on the forums:  http://forums.mysql.com/list.php?166



PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL University: Customizing MySQL Enterprise Monitor

Сентябрь 10th, 2009
Today (September 10, 13:00 UTC), Mark Leith will show how to customize the MySQL Enterprise Monitor: Customizing MySQL Enterprise Monitor. Mark is a MySQL support manager, and knows Enterprise Monitor by heart.

For MySQL University sessions, point your browser to this page. You need a browser with a working Flash plugin. You may register for a Dimdim account, but you don't have to. (Dimdim is the conferencing system we're using for MySQL University sessions. It provides integrated voice streaming, chat, whiteboard, session recording, and more.) All MySQL University sessions are recorded, that is, slides and voice can be viewed as a Flash movie (.flv). You can find those recordings on the respective MySQL University session pages which are listed on the MySQL University home page.

MySQL University is a free educational online program for engineers/developers. MySQL University sessions are open to anyone, not just Sun employees. Sessions are recorded (slides and audio), so if you can't attend the live session you can look at the recording anytime after the session.

Here's the schedule for the upcoming weeks:

  • September 17: Architecture of MySQL Backup (Lars Thalmann)
  • September 24: Concurrency Control: How It Really Works (Heikki Tuuri)
  • October 1: InnoDB Internals: InnoDB File Formats and Source Code Structure (Calvin Sun)
  • October 8: Building MySQL Releases on Unix (Jörg Brühe)
The schedule is not engraved in stone at this point. Please visit http://forge.mysql.com/wiki/MySQL_University#Upcoming_Sessions for the up-to-date list. On that page, you can also find the starting time for many time zones.

PlanetMySQL Voting: Vote UP / Vote DOWN