Archive for the ‘mysql monitoring’ Category

M3 code refactor & DBI support

Январь 28th, 2012

Pluggable M3 (Monitis Monitor Manager) Framework

Who needs an introduction about M3? – Perhaps no one!
After gaining some reputation with M3, providing extra-easy integration of any monitor into Monitis it was time to take it to the next level.

Generally speaking, the work flow of M3 was described in detail in this article.

After some thought and design, we’ve decided it’d be best if M3 was pluggable. Pluggable in terms of being able to easily add execution and parsing plugins.
The interface and behavior of M3 stayed exactly the same, however now it is much easier to obtain data from any source and parse the data the way you want it.
Saying that, it was time to put the new design for a test. We tried to integrate the DBI support into M3.
Guess what – it was much easier than expected!

The general idea of DB integration with Monitis

DB (as in Database) integration with Monitis has shown to be invaluable. Being able to extract application related data from a DB and load it to Monitis means you can graph anything going on in your database related application. Monitoring your DB is an integral part of server monitoring in today’s world.

Be it maximum number of users at peak hours, gathering statistics about record usage or just checking if your DB is serving requests quickly enough – graphing performance and uptime. Your imagination is the limit – considering your application is DB related of course.

How it’s done – the technicalities beyond

The sharp-eyed among you would notice that M3 got beefed up with a directory named “Execution” and a directory named “Parsing”.
These directories hold the plugins M3 uses for each.

All that I had to do to add DBI support is to create a new execution plugin called “DBI.pm” and fill it up with code from the Monitis DBI module which already existed.

And that my friends, that, was easy. Mission accomplished – Pluggable M3 eases the job for us.

Another way to integrate MySQL

MySQL could have been integrated previously into M3 by executing mysql and feeding it with a query, having something similar to:

 <exectemplate>echo "select count(*) from mysql.user;" | mysql</exectemplate> 

This form of execution will invoke the ‘mysql’ executable, feeding it the above query. Execution of this will result in the response of the query being returned with some other useless output.

Please avoid this form of execution, as now DBI support is fully integrated into M3. In the next paragraph I’ll show how it should be done.

Lets fetch

While conveying ideas about M3, I decided to prefer to provide very simple examples of operation, however, I encourage you, the end user to let your imagination go wild, adding interesting counters which show how your application really perform and behave.

In this example we’ll just count the number of users in an installed MySQL database. An example anyone should be able to imitate and then scale to his own need.

We’ll examine the following example. I suggest opening the mentioned link in a new window while continuing the read.
Looking at this example it is quite straight-forward to understand. The query to execute is over there, together with some other parameters which are mandatory to connect to a database, such as the database name, user name, password, etc.

An explanation should be provided however for the odd db_statistics parameter. Those of you who have read the HTTP extraction article about M3 would have probably remembered the http_statistics parameter. Then, the db_statistics one does more or less the same, providing some meta counters for the query:

  1. How long did the query take to process (in seconds)
  2. The success/failure of the query (1 for success, 0 for failure)

I encourage to do the parsing of query results, or actually the lack-of-parsing of query results in a way that takes the whole output returns from the query, hence the regular expression of (.*).

If your query returns more than what you need, please tune your query better.
Test your execution with:

 # ./DryRun.pl dbi_sample.xml DB: 'root@DBI:mysql:mysql:localhost', Query: 'select count(*) from mysql.user;' at MonitisMonitorManager.pm line 223 Matched '(.*)'=>'6' in '6' at MonitisMonitorManager.pm line 237 OK This is a dry run, data for monitor 'Sample MySQL Monitor' was not really updated. at ./DryRun.pl line 18 

It should work. Then you can work your query to fit your own needs.

Next in line

We’re looking for some more new ideas to add as plugins for M3. If you have any idea about any plugin (execution or parsing plugin) that you would like to see in M3 – we want to hear you!

Integrating software into the powerful Monitis has never been easier, now that you have M3!

Monitis can monitor anything! Follow us also on Twitter and GitHub.

Share Now:
  • del.icio.us
  • Digg
  • Facebook
  • LinkedIn
  • BlinkList
  • DZone
  • Google Bookmarks
  • Reddit
  • StumbleUpon
  • Twitter
  • RSS

PlanetMySQL Voting: Vote UP / Vote DOWN

M3 code refactor & DBI support

Январь 28th, 2012

Pluggable M3 (Monitis Monitor Manager) Framework

Who needs an introduction about M3? – Perhaps no one!
After gaining some reputation with M3, providing extra-easy integration of any monitor into Monitis it was time to take it to the next level.

Generally speaking, the work flow of M3 was described in detail in this article.

After some thought and design, we’ve decided it’d be best if M3 was pluggable. Pluggable in terms of being able to easily add execution and parsing plugins.
The interface and behavior of M3 stayed exactly the same, however now it is much easier to obtain data from any source and parse the data the way you want it.
Saying that, it was time to put the new design for a test. We tried to integrate the DBI support into M3.
Guess what – it was much easier than expected!

The general idea of DB integration with Monitis

DB (as in Database) integration with Monitis has shown to be invaluable. Being able to extract application related data from a DB and load it to Monitis means you can graph anything going on in your database related application. Monitoring your DB is an integral part of server monitoring in today’s world.

Be it maximum number of users at peak hours, gathering statistics about record usage or just checking if your DB is serving requests quickly enough – graphing performance and uptime. Your imagination is the limit – considering your application is DB related of course.

How it’s done – the technicalities beyond

The sharp-eyed among you would notice that M3 got beefed up with a directory named “Execution” and a directory named “Parsing”.
These directories hold the plugins M3 uses for each.

All that I had to do to add DBI support is to create a new execution plugin called “DBI.pm” and fill it up with code from the Monitis DBI module which already existed.

And that my friends, that, was easy. Mission accomplished – Pluggable M3 eases the job for us.

Another way to integrate MySQL

MySQL could have been integrated previously into M3 by executing mysql and feeding it with a query, having something similar to:

 <exectemplate>echo "select count(*) from mysql.user;" | mysql</exectemplate> 

This form of execution will invoke the ‘mysql’ executable, feeding it the above query. Execution of this will result in the response of the query being returned with some other useless output.

Please avoid this form of execution, as now DBI support is fully integrated into M3. In the next paragraph I’ll show how it should be done.

Lets fetch

While conveying ideas about M3, I decided to prefer to provide very simple examples of operation, however, I encourage you, the end user to let your imagination go wild, adding interesting counters which show how your application really perform and behave.

In this example we’ll just count the number of users in an installed MySQL database. An example anyone should be able to imitate and then scale to his own need.

We’ll examine the following example. I suggest opening the mentioned link in a new window while continuing the read.
Looking at this example it is quite straight-forward to understand. The query to execute is over there, together with some other parameters which are mandatory to connect to a database, such as the database name, user name, password, etc.

An explanation should be provided however for the odd db_statistics parameter. Those of you who have read the HTTP extraction article about M3 would have probably remembered the http_statistics parameter. Then, the db_statistics one does more or less the same, providing some meta counters for the query:

  1. How long did the query take to process (in seconds)
  2. The success/failure of the query (1 for success, 0 for failure)

I encourage to do the parsing of query results, or actually the lack-of-parsing of query results in a way that takes the whole output returns from the query, hence the regular expression of (.*).

If your query returns more than what you need, please tune your query better.
Test your execution with:

 # ./DryRun.pl dbi_sample.xml DB: 'root@DBI:mysql:mysql:localhost', Query: 'select count(*) from mysql.user;' at MonitisMonitorManager.pm line 223 Matched '(.*)'=>'6' in '6' at MonitisMonitorManager.pm line 237 OK This is a dry run, data for monitor 'Sample MySQL Monitor' was not really updated. at ./DryRun.pl line 18 

It should work. Then you can work your query to fit your own needs.

Next in line

We’re looking for some more new ideas to add as plugins for M3. If you have any idea about any plugin (execution or parsing plugin) that you would like to see in M3 – we want to hear you!

Integrating software into the powerful Monitis has never been easier, now that you have M3!

Monitis can monitor anything! Follow us also on Twitter and GitHub.

Share Now:
  • del.icio.us
  • Digg
  • Facebook
  • LinkedIn
  • BlinkList
  • DZone
  • Google Bookmarks
  • Reddit
  • StumbleUpon
  • Twitter
  • RSS

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Monitoring – What’s really needed

Май 18th, 2010

The implementation of MySQL Monitoring is critical for any organization that uses a database and wants to avoid the inevitable disaster. There are 3 important components that all serve a key purpose to “MySQL Monitoring” in general:

  • Monitoring – Historical and graphical information
  • Alerting – Tell me when something is wrong
  • Dashboard – The State of NOW

Monitoring

There is no one option for Monitoring that is significantly better then another. A short list of what’s on offer can be found at http://monitoring-mysql.com/monitoring-products. What’s important is you have monitoring in place so historically you can review situations and compare across your servers and enabling the better identification of physical or database bottlenecks. My recommendations for products are Cacti which is packaged with most popular Linux distrubtions, so can be installed via a single apt-get/yum/rpm command, and the MySQL Cacti Templates. This is not the best product solution or combination, it’s in my opinion the most common and covers all the essential bases.

It’s best to define a different web server (i.e. publically accessible) to be the monitoring server rather then installing the web interface on any single DB server.

Alerting

Alerting is key to notify you of a problem without the need for somebody to be viewing a screen and see it happening via your monitoring. The identification of high CPU load, a disk nearing capacity, database locking etc often helps avoid a current problem before it becomes some level of disaster. Almost all companies use Nagios or a derivative such as the main fork Icigna or products that include Nagios like Opsview or Groundworks.

Dashboard

While both Monitoring and Alerting are necessary, they both however lack a key component necessary for successful administration. That is timing. Both of these earlier options sample, e.g. 1 minute or generally 5 minutes (by default), but problems can happen quickly. This is why each organization needs to have a Dashboard. I don’t know of any products here unless you try and adjust a monitoring product, but want’s needed is a very lightweight and very business centric single page of Green/Yellow/Red status’s of your environment including databases, webservers, response time and traffic etc. This is for the state of NOW. A Dashboard should sample every 5 to 10 seconds. I have seen larger and more successful companies have various home grown implementations. I developed a product for one company and it included the following on a single page. You can see the screen output in a presentation at http://www.slideshare.net/ronaldbradford/10x-performance-improvements. This included

  • 5 DB servers monitoring load average, ping time, database connections, active,free,locked, replication availability and lag
  • 5 web servers monitoring load average, ping time, apache connections
  • Application metrics monitoring 3 different page load times, and page size

It is often important to be able to identify a key problem and then drill down to this more quickly rather then the usual “the website is slow” question and having to investigate the same repetitive tasks. You need to automate, and be more pro-active in response especially to load and locking issues.

Advanced Monitoring

Information is only have the requirement, it is what you do with this information that determines how to be proactive rather then reactive. If I was the DBA of a company I’d do even more then these initial 3 steps which are a necessary base. I would also monitor for example:

  • Database size and growth. This is important to be preemptive about your capacity. Example SQL at http://ronaldbradford.com/mysql-dba/#allschemas
  • Error Log changes
  • Backup timing. This is important as your DB grows as it affects recovery.
  • Recovery timing
  • Gather raw MySQL status information because monitoring tools only capture what you ask it to do, not everything. While you may not analyze all now, you may want to in the future look back in time. Example scripts at http://ronaldbradford.com/mysql-dba/#log-stats
  • Hourly/Daily text reports. Producing a easy readable SHOW GLOBAL status report such as with statpack will for example enable me to know network throughtput in the DB, transaction throughput and key indicators of locking, disk access etc. While you may have a graphic interface, it’s a lot easier to automate and grep text reports.
  • Proactive restrictions. The Twitter failed whale is a great example of when the system moves closer to known limits, but before those limits they start limiting load. This includes for example to disable less critical but resource intensive functionality, e.g. people search. The also start rejecting connections so they do not reach a crash state. This could be proactively changing timeout values so the DB fails queries, and the webservers respond accordingly with a try again approach.

While these are important if you have only limited resources, too much information can be just as much of a burden then people just start ignoring the information and miss what’s important.


PlanetMySQL Voting: Vote UP / Vote DOWN

Monitoring MySQL Product Options

Октябрь 16th, 2009

I’ve had plenty of comments on specific products to Monitoring MySQL Options before providing the completed list. Here are the results from my survey to give everybody a more complete list.

Nagios 25 xxxxxxxxxxxxxxxxxxxxxxxxx
MONyog 8 xxxxxxxx
Cacti 4 xxxx
Munin 3 xxx
MySQL Enterprise Monitor/Merlin 3 xxx
Hyperic 2 xx
KontrolBase 2 xx
Zabbix 2 xx
Big Brother 1 x
iGlass 1 x
MyDBA 1 x
MySQL AR 1 x
pacemaker 1 x
Panopta 1 x
Opsview 1 x
Monit 1 x
Tivoli 1 x

NOTE: Some answers included multiple products, these are all counted separately in the above figures.

There are a few products that are not listed at Monitoring MySQL in this list.

If you want to list what you use, please continue to use the MySQL Alert Monitoring Survey. Thanks to all those that replied.


PlanetMySQL Voting: Vote UP / Vote DOWN

Monitoring MySQL options

Октябрь 16th, 2009

My recent poll What alert monitoring do you use? showed 25% of the 58 respondents to bravely state they had no MySQL monitoring. I see 1 in 3, ~33% in my consulting so this is consistent.


There is no excuse to not have some MySQL Monitoring on your production system. At the worse case, you should be logging important MySQL information for later analysis. I use my own Logging and Analyzing scripts on every client for an immediate assessment regardless of what’s available. I combine that with my modified statpack to give me immediate text based analysis, broken down by hour chunks for quick reference. These help me in troubleshooting, but they are not a complete solution.

The most popular options I see and are also reflected in the results are:

There is a good list, including some products I did not know. My goal is to get this information included in the Monitoring-MySQL information site.

I have some additional information on Cacti and MONyog, and I’ll be sharing this information in upcoming posts.


PlanetMySQL Voting: Vote UP / Vote DOWN

What do you monitor in MySQL?

Сентябрь 3rd, 2009

If you are unfamiliar with what to monitor in MySQL, starting with looking at what popular Monitoring products monitor. For example, the following is the list of MySQL Cacti Plugin measurements.

Innodb Buffer Pool Activity

  • Pages Created
  • Pages Written
  • Pages Read

Innodb Buffer Pool Pages

  • Pool Size
  • Database Pages
  • Free Pages
  • Modified Pages

Inoodb File I/O

  • File Reads
  • Files Writes
  • Log Writes
  • File Fsyncs

Innodb Pending I/O

  • Aio Log Ios
  • Aio Sync ios
  • Buffer Pool Flushes
  • Chkp Writes
  • Ibuf Aio Reads
  • Log Flushes
  • Log Writes
  • Normal Aio Reads
  • Normal Aio Writes

Innodb Insert Buffer

  • Inserts
  • Merged
  • Merges

Innodb Log

  • Log Buffer Size
  • Log Bytes Written
  • Log Bytes Flushed
  • Unflushed Log

Innodb Row Operations

  • Rows Read
  • Rows Deleted
  • Rows Updated
  • Rows Inserted

Innodb Semaphores

  • Spin Rounds
  • Spin Waits
  • OS Waits

Innodb Transactions

  • Innodb Transactions
  • Current Transactions
  • History List
  • Read Views

MySQL Binary/Relay Logs

  • Binlog Cache use
  • Binlog Cache Disk Use
  • Binary Log Space
  • Relay Log Space

MySQL Command Counters

  • Questions
  • SELECT
  • DELETE
  • INSERT
  • UPDATE
  • REPLACE
  • LOAD
  • DELETE MULTI
  • INSERT SELECT
  • UPDATE MULTI
  • REPLACE SELECT

MySQL Connections

  • Max Connections
  • Max Used Connections
  • Aborted Clients
  • Aborted Connects
  • Threads Connected
  • Connections

MySQL Files and Tables

  • Table Cache
  • Open Tables
  • Open Files
  • Opened Tables

MySQL Network Traffic

  • Bytes Received
  • Bytes Sent

MySQL Processlist

  • State Closing Tables
  • State Copying to Tmp Table
  • State End
  • State Freeing Items
  • State Init
  • State Locked
  • State Login
  • State Preparing
  • State Reading From Net
  • State Sending Data
  • State Sorting Result
  • State Statistics
  • State Updating
  • State Writing to Net
  • State None
  • State Other

MySQL Query Cache

  • Queries In Cache
  • Hits
  • Inserts
  • Not Cached
  • Lowmem Prunes

MySQL Query Cache Memory

  • Query Cache Size
  • Free Memory
  • Total Blocks
  • Free Blocks

MySQL Replication

  • Slave Running
  • Slave Stopped
  • Slave Lag
  • Slave Open Temp Tables
  • Slave Retried Transactions

MySQL Select Types

  • Select Full Join
  • Select Full Range Join
  • Select Range
  • Select Range Check
  • Select Scan

MySQL Sorts

  • Sort Rows
  • Sort Range
  • Sort Merge Passes
  • Sort Scan

MySQL Table Locks

  • Table Locks Immediate
  • Table Locks Waited
  • Slow Queries

MySQL Temporary Objects

  • Created Tmp Tables
  • Created Tmp Disk Tables
  • Created Tmp Files

MySQL Threads

  • Thread Cache Size
  • Threads Created

PlanetMySQL Voting: Vote UP / Vote DOWN