PlanetMysql.ru – информация о СУБД MySQL

Most of this stuff is not PHP specific, and Python or Ruby or Java or .NET developers can use the tools in this talk.

The session on joind.in, with user comments/feedback, is at http://joind.in/talk/view/1320.

Slides are at http://talks.php.net/show/confoo10

“My name is Rasmus, I’ve been around for a long time. I’ve been doing this web stuff since 1992/1993.”

“Generally performance is not a PHP problem.” Webservers not config’d, no expire headers on images, no favicon.

Tools: Firefox/Firebug extension called YSlow (developed by yahoo) gives you a grade on your site.

Google has developed the Firefox/Firebug pagespeed tool.

Today Rasmus will pick on wordpress. He checks out the code, then uses Siege to do a baseline benchmark — see the slide for the results.

Before you do anything else install an opcode cache like APC. Wordpress really likes this type of caching, see this slide for the results. Set the timezone, to make sure conversions aren’t being done all the time.

Make sure you are cpu-bound, NOT I/O bound. Otherwise, speed up the I/O.

Then strace your webserver processs. There are common config issues that you can spot in your strace code. grep for ENOENT which shows you “No such file or directory” errors.

AllowOverride None to turn off .htaccess for every directory, just read settings once from your config file….(unless you’re an ISP).

Make sure DirectoryIndex is set appropriately, watch your include_path. All this low-hanging fruit has examples on the common config issues slide.

Install pecl/inclued and generate a graph – here is the graph image (I have linked it because you really want to zoom in to the graph…)

In strace output check the open() calls. Conditional includes, function calls that include files, etc. need runtime context before knowing what to open. In the example, every request checks to see if we have the config file, once we have config’d we can get rid of that stuff. Get rid of all the conditionals and hard-code “include wp-config.php”. Examples are on the slide.

His tips to change:
Conditional config include in wp-load.php (as just mentioned)
Conditional did-header check in wp-blog-header.php
Don’t call require_wp_db() from wp-settings.php
Remove conditional require logic from wp_start_object_cache

Then check strace again, now all Rasmus sees is theming and translations, which he decided to keep, because that’s the good benefit of Wordpress – Performance is all about costs vs. flexibility. You don’t want to get rid of all of your flexibility, but you want to be fast.

Set error_reporting(-1) in wp-settings.php to catch all warnings — warnings slow you down, so get rid of all errors. PHP error handling is very slow, so getting rid of errors will make you faster.

The slide of warnings that wordpress throws.

Look at all C-level calls made, using callgrind, which sits under valgrind, a CPU emulator used for debugging. See the image of what callgrind shows.

Now dive into the PHP executor, by installing XDebug.

Check xhprof – Facebook open sourced this about a year ago, it’s a PECL extension. The output is pretty cool, try it on your own site, Rasmus does show you how to use it. It shows you functions sorted by the most expensive to the least expensive.

For example, use $_SERVER[REQUEST_TIME] instead of time(). Use pconnect() if MySQL can handle the amount of webserver connections that will be persistent, etc.

After you have changed a lot of the stuff above, benchmark again with siege to see how much faster you are. In this case there is not much gained so far.

So keep going….the blogroll is very slow — Rasmus gets rid of it by commenting out in the sidebar.php file. I’d like to see something to make it “semi-dynamic” — that is, make it a static file that can be re-generated, since you might want the blogroll but links are not changed every second…..

At this point we’re out of low-hanging fruit.

HipHop is a PHP to C++ converter & compiler, including a threaded, event-driven server that replaces apache. Rasmus’ slide says “Wordpress is well-suited for HipHop because it doesn’t have a lot of dynamic runtime code. This is using the standard Wordpress-svn checkout with a few tweaks.”

Then, of course, benchmark again.

The first time you compile Wordpress with HipHop, you give it a list of files to add to the binary, it will complain about php code that generate file names, so you do have to fix that kind of stuff. There’s a huge mess of errors the first time you run it (”pages and pages”), and Rasmus had to patch HipHop (and Wordpress) but the changes in HipHop have been put back into HipHop, so you should be good for the most part.

Check out the errors, lots of them show logical errors like $foo.”bar” instead of $foo.=”bar” and $foo=”bar” instead of $foo==”bar” in an if statement. Which of course is nice for your own code, to find those logical errors.

(Wordpress takes in a $user_ID argument and immediately initializes a global $user_ID variable, which overwrites the argument passed in, so you can change the name of the argument passed in….)

You can also get rid of some code, things that check for existence of the same thing more than once. So it will take a bit of tweaking, but it’s worth it.

There are limitations to HipHop, for example:

• It doesn’t support any of the new PHP 5.3 language features
  
• Private properties don’t really exist under HipHop. They are treated as if they are protected instead.
  
• You can’t unset variables. unset will clear the variable, but it will still be in the symbol table.
  
• eval and create_function are limited
  
• Variable variables $$var are not supported
  
• Dynamic defines won’t work: define($name,$value)
  
• get_loaded_extensions(), get_extension_funcs(), phpinfo(), debug_backtrace() don’t work
  
• Conditional and dynamically created include filenames don’t work as you might expect
  
• Default unix-domain socket filename isn’t set for MySQL so connecting to localhost doesn’t work

and HipHop does not support all extensions — see the list Rasmus has of extensions HipHop supports.

Then Rasmus showed an example using Twit (which he wrote) including the benchmarks. He shows that you can see what’s going on, like 5 MySQL calls on the home page and what happens when you don’t have a favicon.ico (in yellow).

In summary, “performance is all about architecture”, “know your costs”.

Be careful, because some tools (like valgrind and xdebug) you don’t want to put it on production systems, you could capture production traffic and replay it on a dev/testing box, but “you just have to minimize the differences and do your best”.

Most of this stuff is not PHP specific, and Python or Ruby or Java or .NET developers can use the tools in this talk.

The session on joind.in, with user comments/feedback, is at http://joind.in/talk/view/1320.

Slides are at http://talks.php.net/show/confoo10

“My name is Rasmus, I’ve been around for a long time. I’ve been doing this web stuff since 1992/1993.”

“Generally performance is not a PHP problem.” Webservers not config’d, no expire headers on images, no favicon.

Tools: Firefox/Firebug extension called YSlow (developed by yahoo) gives you a grade on your site.

Google has developed the Firefox/Firebug pagespeed tool.

Today Rasmus will pick on wordpress. He checks out the code, then uses Siege to do a baseline benchmark — see the slide for the results.

Before you do anything else install an opcode cache like APC. Wordpress really likes this type of caching, see this slide for the results. Set the timezone, to make sure conversions aren’t being done all the time.

Make sure you are cpu-bound, NOT I/O bound. Otherwise, speed up the I/O.

Then strace your webserver processs. There are common config issues that you can spot in your strace code. grep for ENOENT which shows you “No such file or directory” errors.

AllowOverride None to turn off .htaccess for every directory, just read settings once from your config file….(unless you’re an ISP).

Make sure DirectoryIndex is set appropriately, watch your include_path. All this low-hanging fruit has examples on the common config issues slide.

Install pecl/inclued and generate a graph – here is the graph image (I have linked it because you really want to zoom in to the graph…)

In strace output check the open() calls. Conditional includes, function calls that include files, etc. need runtime context before knowing what to open. In the example, every request checks to see if we have the config file, once we have config’d we can get rid of that stuff. Get rid of all the conditionals and hard-code “include wp-config.php”. Examples are on the slide.

His tips to change:
Conditional config include in wp-load.php (as just mentioned)
Conditional did-header check in wp-blog-header.php
Don’t call require_wp_db() from wp-settings.php
Remove conditional require logic from wp_start_object_cache

Then check strace again, now all Rasmus sees is theming and translations, which he decided to keep, because that’s the good benefit of Wordpress – Performance is all about costs vs. flexibility. You don’t want to get rid of all of your flexibility, but you want to be fast.

Set error_reporting(-1) in wp-settings.php to catch all warnings — warnings slow you down, so get rid of all errors. PHP error handling is very slow, so getting rid of errors will make you faster.

The slide of warnings that wordpress throws.

Look at all C-level calls made, using callgrind, which sits under valgrind, a CPU emulator used for debugging. See the image of what callgrind shows.

Now dive into the PHP executor, by installing XDebug.

Check xhprof – Facebook open sourced this about a year ago, it’s a PECL extension. The output is pretty cool, try it on your own site, Rasmus does show you how to use it. It shows you functions sorted by the most expensive to the least expensive.

For example, use $_SERVER[REQUEST_TIME] instead of time(). Use pconnect() if MySQL can handle the amount of webserver connections that will be persistent, etc.

After you have changed a lot of the stuff above, benchmark again with siege to see how much faster you are. In this case there is not much gained so far.

So keep going….the blogroll is very slow — Rasmus gets rid of it by commenting out in the sidebar.php file. I’d like to see something to make it “semi-dynamic” — that is, make it a static file that can be re-generated, since you might want the blogroll but links are not changed every second…..

At this point we’re out of low-hanging fruit.

HipHop is a PHP to C++ converter & compiler, including a threaded, event-driven server that replaces apache. Rasmus’ slide says “Wordpress is well-suited for HipHop because it doesn’t have a lot of dynamic runtime code. This is using the standard Wordpress-svn checkout with a few tweaks.”

Then, of course, benchmark again.

The first time you compile Wordpress with HipHop, you give it a list of files to add to the binary, it will complain about php code that generate file names, so you do have to fix that kind of stuff. There’s a huge mess of errors the first time you run it (”pages and pages”), and Rasmus had to patch HipHop (and Wordpress) but the changes in HipHop have been put back into HipHop, so you should be good for the most part.

Check out the errors, lots of them show logical errors like $foo.”bar” instead of $foo.=”bar” and $foo=”bar” instead of $foo==”bar” in an if statement. Which of course is nice for your own code, to find those logical errors.

(Wordpress takes in a $user_ID argument and immediately initializes a global $user_ID variable, which overwrites the argument passed in, so you can change the name of the argument passed in….)

You can also get rid of some code, things that check for existence of the same thing more than once. So it will take a bit of tweaking, but it’s worth it.

There are limitations to HipHop, for example:

• It doesn’t support any of the new PHP 5.3 language features
  
• Private properties don’t really exist under HipHop. They are treated as if they are protected instead.
  
• You can’t unset variables. unset will clear the variable, but it will still be in the symbol table.
  
• eval and create_function are limited
  
• Variable variables $$var are not supported
  
• Dynamic defines won’t work: define($name,$value)
  
• get_loaded_extensions(), get_extension_funcs(), phpinfo(), debug_backtrace() don’t work
  
• Conditional and dynamically created include filenames don’t work as you might expect
  
• Default unix-domain socket filename isn’t set for MySQL so connecting to localhost doesn’t work

and HipHop does not support all extensions — see the list Rasmus has of extensions HipHop supports.

Then Rasmus showed an example using Twit (which he wrote) including the benchmarks. He shows that you can see what’s going on, like 5 MySQL calls on the home page and what happens when you don’t have a favicon.ico (in yellow).

In summary, “performance is all about architecture”, “know your costs”.

Be careful, because some tools (like valgrind and xdebug) you don’t want to put it on production systems, you could capture production traffic and replay it on a dev/testing box, but “you just have to minimize the differences and do your best”.

MySQL Connector/Net 6.3.1, a new version of the all-managed .NET driver
for MySQL has been released. This is an alpha release and is intended to
introduce you to the new features and enhancements we are planning. This
release should not be used in a production environment.

It is now available in source and binary form from
[http://dev.mysql.com/downloads/connector/net/6.3.html] and mirror sites
(note that not all mirror sites may be up to date at this point of time
- if you can’t find this version on some mirror, please try again later
or choose another download site.)

The new features or changes in this release are:

• Visual Studio 2010 RC support
• Nested transaction scope support

What we know may be broken
—————————-

• Documentation is not updated yet and is not integrated into VS 2010
• Some users are having trouble installing.  We are working hard to determine what is causing this problem.  These users may continue having trouble with 6.3.1.  Please be patient as we work through this.

Please let us know what else we broke and how we can make it better!

MySQL Connector/Net 6.3.0, a new version of the all-managed .NET driver
for MySQL has been released. This is an alpha release and is intended to
introduce you to the new features and enhancements we are planning. This
release should not be used in a production environment.

It is now available in source and binary form from
[http://dev.mysql.com/downloads/connector/net/6.3.html] and mirror sites
(note that not all mirror sites may be up to date at this point of time
- if you can’t find this version on some mirror, please try again later
or choose another download site.)

New features or changes:

• Visual Studio 2010 RC support
• Nested transaction scope support

What we know may be broken

• Documentation is not updated yet and is not integrated into VS 2010
• Data editing view (in VS) does not function in this build

Please let us know what else we broke and how we can make it better!

In our last installment we found our hero investigating the tracing changes found in Connector/Net 6.2.  This time we’ll take a closer look at the format of the trace data and how developers can use that information to write new and interesting trace listeners.

Understanding the plumbing

The first thing we need to understand is a little about how the trace messages are routed.  The main method we are interested in is TraceSource.TraceEvent.  Here is the signature.

public void TraceEvent(
    TraceEventType eventType,
    int id,
    string format,
    params Object[] args
)

All the other TraceSource methods like TraceInformation eventually boil down to a call to TraceEvent.  TraceEvent will run through all the attached listeners and call the TraceEvent method on each.

The default behavior of listeners is to eventually call String.Format using the format given.  It plugs the parameters in and comes out with a string that can be sent to the console, to a file, or some other output.  Thankfully Microsoft saw fit to make TraceEvent virtual allowing derivative listeners to override and do interesting things.

How we use TraceEvent

Here is the signature for TraceListener.TraceEvent.

public virtual void TraceEvent(
    TraceEventCache eventCache,
    string source,
    TraceEventType eventType,
    int id,
    string format,
    params Object[] args
)

eventCache is provided by the framework and contains information such as process id, thread id, timestamp, etc.
source  is the name of the TraceSource that provided the event
eventType is the .NET defined type of this event.  This can have values such as Information, Warning, Error, etc.
id  is the application provided event id.  This value is application defined and we’ll explain how we use it later in this post.
format is the parameters string message that listeners such as ConsoleTraceListener would use for output
args is the array of arguments available to plug into the format string.  This is the actual data for the event.

The key to using the data provided in TraceEvent is simply to know that the data always comes in a specific order and format.  The following information only applies to events coming from the mysql source.  The eventCache, source, eventType, and format parameters are self explanatory so we’ll start with id.

The id parameter is our MySql-centric event id.  We wanted to give very specific information about the type of event.  We have a public enum available in the 6.2 assembly called MySqlTraceEventType.  Here is the definition.

public enum MySqlTraceEventType : int
{
    ConnectionOpened = 1,
    ConnectionClosed,
    QueryOpened,
    ResultOpened,
    ResultClosed,
    QueryClosed,
    StatementPrepared,
    StatementExecuted,
    StatementClosed,
    NonQuery,
    UsageAdvisorWarning,
    Warning,
    Error
}

Before we talk about the event-specific data points, let me mention about a small problem we needed to solve.  We wanted to have a numeric id attached to each event that would allow all the events belonging to a given query to be gathered together.  The only way we could accomplish that was to use a counter that gets incremented each time a driver is opened.  (remember we needed a value that is unique on the same thread so we can’t use process id, thread id, or event MySQL server thread id). We’ll call this the driver id and every event that comes from the mysql source has this driver id as the first member in the args array.

Here’s a table that lists each mysql event and what the arguments array looks like.  Please keep in mind that these arguments start in the second element (index == 1)

Here’s the definition of the publicly available UsageAdvisorWarningFlags enum.

public enum UsageAdvisorWarningFlags
{
    NoIndex = 1,
    BadIndex,
    SkippedRows,
    SkippedColumns,
    FieldConversion
}

So that’s it.  This information will be documented and may change slightly as we find problems.  In fact we’ve already identified a small issue and may add a “current database” parameter to the connection opened event since the current database can be different than what is set on the connection string.  Be sure, when we make these changes they’ll be documented and by checking the version of the assemblies involved you can do the right thing.

Armed with this information you should be able to go forth and make very interesting trace sniffing apps for MySQL.  If you accept such a challenge, drop me a line.  I would love to know what you think!

For years, Connector/Net has been a key part of any MySQL & .NET developer’s toolbox.  Tracing is also a key part of a developer’s life and Connector/Net has always output trace messages. 

Recently we decided to extend our Enterprise Monitoring product by allowing connectors to collect performance data and push that data into Enterprise Monitor for analysis.  We decided to make use of tracing data to implement this in Connector/Net but this required a complete overhaul to how we output trace data.  This first post is a review of .NET tracing systems and how we changed our trace output.  The second post will cover how we used this new tracing format to implement our enterprise monitor collector.

Tracing in .NET 1.x

.NET shipped with a very simple tracing system.  You have a static class named Trace that has static methods such as Write and WriteLine.  An application can use code like the following to output a message to the trace log. 

Now that we have output our message, how do we direct it somewhere?  You do that with listeners.  There are a few standard listeners included in the framework (ConsoleTraceListener, XmlTraceListener, EventLogTraceListener) but you are free to create your own.  You can attach a listener at runtime or via an application configuration file.  There’s no need to show you how that is done.  You can easily find it on the web.  What we really want to talk about are changes in .NET 2 tracing and how Connector/Net makes use of it.

Tracing in .NET 2.x

Developers usually want to just trace a particular part of an application.  They may just want to get the network tracing or maybe just the file I/O.  To accommodate this, Microsoft created a new class named TraceSource.  You can really think of it as just a Trace class with a name.  Now when you add listeners via the config file you have to add them to a source with a particular name.  Here’s what it might look like.

   1:  <configuration>

   2:    <system.diagnostics>

   3:      <sources>

   4:        <source name="TraceTest" switchName="SourceSwitch" 

   5:          switchType="System.Diagnostics.SourceSwitch" >

   6:          <listeners>

   7:            <add name="console" />

   8:            <remove name ="Default" />

   9:          </listeners>

  10:        </source>

  11:      </sources>

  12:      <switches>

  13:        <!-- You can set the level at which tracing is to occur -->

  14:        <add name="SourceSwitch" value="Warning" />

  15:          <!-- You can turn tracing off -->

  16:          <!--add name="SourceSwitch" value="Off" -->

  17:      </switches>

  18:      <sharedListeners>

  19:        <add name="console" 

  20:          type="System.Diagnostics.ConsoleTraceListener" 

  21:          initializeData="false"/>

  22:      </sharedListeners>

  23:    </system.diagnostics>

  24:  </configuration>

You can also specify a switch level, a filter level, and trace options (like should the output include process id, thread id, timestamp, etc). 

Starting with Connector/Net 6.2, we have started using TraceSource to output our trace data.  Our source is named, cleverly enough, mysql. 

Using the mysql trace source

Using our new tracing is very simple. Just use a configuration block similar to what I have above except use “mysql” as the name of the source instead of “TraceTest”.  One question you may have, though, is how to add listeners to our source at runtime.  You do that with the new MySqlTrace class.  Here’s how.

MySqlTrace.Listeners.Add(new ConsoleTraceListener());

Difference is in the details

We’ve also increased the level of detail included in our trace messages.  Each trace message includes a counter value that attaches it to other trace messages belong to the same connection.  We did this to enable the development of applications such as a powerful log viewer that will also serve as a visual trace listener.  With this app, a developer would be able to query the log output for various scenarios.  Examples would be “show me all queries where all the rows were not read

So where to from here?

There’s much more to say about our new trace output and, in my next post, I’ll go into the exact format of the data, and we (and you) can use that data to create new and exciting trace listeners.

MySQL Connector/Net 6.2.2, a new version of the all-managed .NET driver for MySQL has been released.This is our latest GA release and is suitable for use in all scenarios against servers ranging from version 4.1 to 5.4!

It is now available in source and binary form from [http://dev.mysql.com/downloads/connector/net/6.2.html] and mirror sites
(note that not all mirror sites may be up to date at this point of time - if you can't find this version on some mirror, please try again later or choose another download site.)

The new features or changes in this release are:

• Connection pool cleanup timer.  We now utilize a timer that cleans idle connections that are no longer connected every 3 minutes
• We are now using stream and TCP-based timeouts to handle command timeouts.  This is more inline with what SqlClient does and should be more reliable than our old timer based approach
• Completely refactored MySqlConnectionStringBuilder
• We now support the TableDirect query type
• Completely refactored our logging system.  Our trace data is now published using TraceSource and has a specific format so to allow third party listeners to be created.
• Lots of bug fixes

Please let us know what else we broke and how we can make it better!

MySQL Connector/Net 6.2.0, a new version of the all-managed .NET driver for MySQL has been released. This is an alpha release and is intended to introduce you to the new features and enhancements we are planning. This release should not be used in a production environment.

It is now available in source and binary form from [http://dev.mysql.com/downloads/connector/net/6.2.html] and mirror sites (note that not all mirror sites may be up to date at this point of time - if you can't find this version on some mirror, please try again later or choose another download site.)

The new features or changes in this release are

• Connection pool cleanup timer. We now utilize a timer that cleans idle connections that are no longer connected every 3 minutes
• We are now using stream and TCP-based timeouts to handle command timeouts. This is more inline with what SqlClient does and should be more reliable than our old timer based approach
• Completely refactored MySqlConnectionStringBuilder
• We now support the TableDirect query type
• Completely refactored our logging system. As part of this, query logs now keep all information relevant to that query together to make it easier to find and read.

What we know may be broken
Documentation is not updated yet. Please let us know what else we broke and how we can make it better!

MySQL Connector/Net 6.1.2, a new version of the all-managed .NET driver for MySQL has been released. This is our latest GA release and is suitable for use in all scenarios against servers ranging from version 4.1 to 5.4!

It is now available in source and binary form from [http://dev.mysql.com/downloads/connector/net/6.1.html] and mirror sites (note that not all mirror sites may be up to date at this point of time - if you can't find this version on some mirror, please try again later or choose another download site.)

This release introduces a couple of exciting new features.

Website Configuration Dialog -- This is a new wizard that is activated by clicking a button on the toolbar at the top of the solution explorer. It is meant to work in conjunction with the ASP.Net administration pages.  It makes it easier to activate and set advanced options for the different web providers we include.

Session State Provider -- We are now including a session state provider that allows you to store your websites state in a MySql server.

Support for native output parameters -- With this release we now support native output parameters when connected to a server that supports them. Currently we know that includes 6.0.8 and later servers.  It also will include 5.4 servers.

Changed GUID type -- We have changed our backend representation of a guid type to be CHAR(36).  We are doing this because we feel users will want to use the server UUID() function to populate a guid table and UUID generates a 36 character string.  Developers of older applications can add 'old guids=true' to the connection string and the old binary(16) type will be used.

This release also includes a slew of bug fixes made to the 5.2 and 6.0 trees.  Please check the release notes and change log for a complete list of those changes.  Please let us know what else we broke and how we can make it better!

For MySQL University sessions, point your browser to this page. You need a browser with a working Flash plugin. You may register for a Dimdim account, but you don't have to. (Dimdim is the conferencing system we're using for MySQL University sessions. It provides integrated voice streaming, chat, whiteboard, session recording, and more.) All MySQL University sessions are recorded, that is, slides and voice can be viewed as a Flash movie (.flv). You can find those recordings on the respective MySQL University session pages which are listed on the MySQL University home page.

MySQL University is a free educational online program for engineers/developers. MySQL University sessions are open to anyone, not just Sun employees. Sessions are recorded (slides and audio), so if you can't attend the live session you can look at the recording anytime after the session.

Here's the schedule for the upcoming weeks:

• September 10: Customizing MySQL Enterprise Monitor (Mark Leith)
• September 17: Architecture of MySQL Backup (Lars Thalmann)
• September 24: Concurrency Control: How It Really Works (Heikki Tuuri)
• October 1: InnoDB Internals: InnoDB File Formats and Source Code Structure (Calvin Sun)
• October 8: Building MySQL Releases on Unix (Jörg Brühe)