PlanetMysql.ru — информация о СУБД MySQL

We all knew that we are risking with MMM. Risking, and placing availability as a more important like consistency.  But non of us can risk loosing data forever but we show using it, regarding to our conversations think:  "I can fix my data later on, but I can’t turn back time and prevent the downtime. (Pascal Hoffman@xaprb.com)".

As I wrote before about staying online, now let me write about how to stay consistent.

We all know, mmm is not like a key of salvation, but its getting close to it . While MySQL doesn't support multi-master-slave environments from it's source code, we will sleep badly wondering on the safety of our precious databases.

But its not just about MMM, a few days ago we ran in to a well known InnoDB "feature". Its about the auto increment counter determination on restart. InnoDB try to count the next auto increment value on MySQL restart what can screw up things in the replication as in your data integrity too (http://dev.mysql.com/doc/refman/5.1/en/innodb-auto-increment-handling.html) what could be risky when you are about to use mmm, because you can restart your masters whenever you just want to.

In this post I'm using maatkit to verify and restore the rows without locking or downtime.

Maatkit:

This is not a success of MMM, but success of maatkit.
I don't want to write about maatkit. Everybody who is able to use MMM were met with maatkit. Great stuff and if you didn't met with it yet, this is the right time. http://www.maatkit.org

Verification between your masters:

This is the hardest part, you CAN'T and/or you SHOULD NOT to avoid manual overview.

I'm using mk-table-sync for this scenerio. As the about say: mk-table-sync finds and fixes data differences between MySQL tables.
This is what we are going to do, but lets consider the possibilities. If we goes trough the help menu, we can see there is a lots of options. Worth going trough of them.

Here is the command I used to execute.

mk-table-sync --chunksize 100 --databases mytestdb --lock 1 --transaction --skipbinlog --print  db02 db01

And here is the explain:

--chunksize 100 : Chunksize means the size of the chunks maatkit compare to each other. Mean rows. In this command maatkit compare the tables by chunks which are contain 100 rows. Its an affordable size, Its wont be faster and as I benchmarked neider become slower, but as you will see later, its start a transaction and lock the table. More about locking and transactions later but to understand why I recommend a small chunksize, you have to know that this means that the locking of the table is it in will be a short lock.

--database mytestdb: Allows you to give a comma-separated list of databases should be taken care of. In this example I use only one database but as I mentioned you can pass a list of databases you want to scan or what you don't want to scan.

--lock 1: As I mentioned before, maatkit provide you two kind of "locking". The first one is a real table lock and the second one is the way of using transactions. With innodb tables using transaction is a good choice, more about later, but with MyIsam tables, I think the best way is lock 1 regarding to what I want to do here, syncing online masters to each other. mk-table-sync provide 3 different ways of locking here: "0=none, 1=per sync cycle, 2=per table, or 3=globally" as far as I know, 0 means no locks, thats not a good way if you are about to sync online masters under traffic. Locking each table could be a wrong decision too, in case of a larger table, and globally is the same under usage. So this is why I just picked lock 1 what will lock while it's take care of my chunk (this is why I lowered the chunk size to 100).

--transaction: This is what i was talking about above. Instead of locking, mk-table-sync able to use transactions . Starting a transaction with a larger chunk size could cause lock-wait-timeout messages as locking can. This is why there is a small chunk size again. You have to listen to your isolation level too. As you know, its could be unsafe under read-committed.

--skipbinlog: This is the most important part of it. In a multi-master environment syncing without this option is like a suicide. With this, mk-table-sync will execute the fixing queries without logging in to binlog. I should not to mention, that logging to binlog on the slave could really mess up your databases as the other master will replicate and execute it.

--print: mk-table-sync provide three different (and some of them able to combine) ways to run. First I would recommend is --print. Print will just print to the stdout (what you can forward to file with 1>/your/file/path) but when you redirect the stdout, don't forget when using with --ask-pass that this message won't show up for your . You can use --execute instead. That will execute the queries immediately. I'm always scared by that a little bit so I rather get the output, go over it and execute what I think I should.

db02/db01: In my example these two hosts are the writable master (at first) and the passive master as a second one.You can use whatever hosts you want. Commands will be executed on the second host.

What to do with your output: When its done and its find differences, you can find where ever your stdout goes. In thet file or on your screen you can see the queries what mk-table-sync would execute to make your passive master consistent. Now its up to you what to do with this information. You can execute each of the or re-run the sync with --execute instead of --print.

Syncing slaves

If you done with your masters, you can sync the slaves on the same way, but you can use the already consistent passive master of your to sync from. Both maatkit and mmm provide some very useful scripts to rebuild slaves and you always got a passive master to do from. With this you are free to rebuild your cluster whenever you want.

TOTD #98 showed how to create a simple Metro/JAX-WS compliant Web service using that bundle and deploy on GlassFish.

This Tip Of The Day (TOTD) shows how to create a simple Java EE 6 application that reads data from a MySQL database using JPA 2.0 and Servlet 3.0 and display the results. A more formal support of Java EE 6/Servlet 3.0 is coming but in the meanwhile the approach mentioned below will work.

Lets get started!

1. Configure database connection - The key point to notice here is that the MySQL Connector/J driver is already built into the tool so there is no need to configure it explicitly.
   1. From "Window", "Show Perspective", change to the database perspective as shown below:
      
      
   2. In the "Data Source Explorer", right-click and click on "Database Connections" and select "New ...":
      
      
   3. Search for "mysql" and type the database name as "sakila":
      
      
      
      This blog uses MySQL sample database sakila. So please download and install the sample database before proceeding further.
   4. Click on "Next >" and specify the database configuration:
      
      
      
      Notice the "Drivers" indicate that the JDBC driver is pre-bundled so there is no extra configuration required. If you are using a stand-alone Eclipse bunde and installing the plugin separately, then you need to configure the MySQL JDBC driver explictily.
      
      The URL indicates the application is connecting to the sakila database. Click on "Test Connection" to test connection with the database and see the output as:
      
      
      
      and click on "Finish" to complete. The expanded database in the explorer looks like:
      
      
      
      The expanded view shows all the tables in the database.
2. Create the Web project & configure JPA
   1. Switch to JavaEE perspective by clicking "Window", "Choose Perspective", "Other ..." and choosing "Java EE".
   2. Create a new dynamic web project with the following settings:
      
      
      
      Only the project name needs to be specified and everything else is default. Notice the target runtime indicates that this is a Java EE 6 application. Click on "Finish".
   3. Right-click on the project, search for "facets" and enable "Java Persistence" as shown below:
      
      
   4. Click on "Further configuration available ..." and modify the facet as shown below:
      
      
      
      Make sure to disable "orm.xml" since we are generating a standard Java EE 6 web application. Choose "sakila" as the database. Click on "OK" and again on "OK" to complete the dialog.
3. Generate the JPA entities
   1. Right-click on the project, select "JPA Tools", "Generate Entities" as shown:
      
      
   2. Choose the schema "sakila":
      
      
      
      and click on "Next >". If no values are shown in the schema drop-down, then click on "Reconnect ...".
   3. Specify a package name for the generated entities as "model" and select "film" and "language" table:
      
      
      
      and click on "Finish". The "film" and "language" table are related so it would be nice if all the related tables can be identified and picked accordingly.
      
      Anyway this generates "model.Film" and "model.Language" classes and "persistence.xml" as shown below:
      
      
      
      Also notice that "web.xml" and "sun-web.xml" have been explicitly removed since they are not required by a Java EE 6 application.
   4. "model.Film" class needs to modified slightly because one of the columns is mapped to "Object" which is not a Serializable obect. So change the type of "specialFeatures" from Object to String and also change the corresponding getters/setters accordingly. The error message clearly conveyed during the initial deployment and so could be fixed. But it would be nice to generate the classes that will work out-of-the-box.
4. Create a Servlet client to retrieve/display data from the database
   1. Right-click on the project, select "New", "Class" and specify the values as:
      
      
      
      and click on "Finish". This class will be our Servlet client.
   2. Change the class such that it looks like:
      

      @WebServlet(urlPatterns="/ServletClient")
      public class ServletClient extends HttpServlet {
        @PersistenceUnit
        EntityManagerFactory factory;
      
        protected void doGet(HttpServletRequest req, HttpServletResponse resp)
               throws ServletException, IOException {
          ServletOutputStream out = resp.getOutputStream();
          List list = factory.createEntityManager().createQuery("select f from Film f where f.title like 'GL%';").getResultList();
          out.println("<html><table>");
          for (Object film : list) {
            out.print("<tr><td>" + ((Film)film).getTitle() + "</tr></td>");
          }
          out.println("</table></html>");
        }
      }
      

      
      and the imports as:
      

      import java.io.IOException;
      import java.util.List;
      
      import javax.persistence.EntityManagerFactory;
      import javax.persistence.PersistenceUnit;
      import javax.servlet.ServletException;
      import javax.servlet.ServletOutputStream;
      import javax.servlet.annotation.WebServlet;
      import javax.servlet.http.HttpServlet;
      import javax.servlet.http.HttpServletRequest;
      import javax.servlet.http.HttpServletResponse;
      
      import model.Film;
      
      

      Basically, this is a Servlet 3.0 specification compliant Servlet that uses @WebServlet annotation. It uses @PersistenceUnit to inject the generated JPA Persistence Unit which is then used to query the database. The database query return all the movies whose title start with "GL" and the response is displayed in an HTML formatted table.
   3. Right-click on the project and select "Run As", "Run on Server" and select GlassFish v3 latest promoted build (this blog used build 61) as:
      
      
      
      and click on "Finish". The output at "http://localhost:8080/HelloJPA/ServletClient" looks like:
      
      

Simple, easy and clean!

How are you using Eclipse and GlassFish - the consolidated bundle or standalone Eclipse + GlassFish plugin ?

Download GlassFish Tools Bundle for Eclipse now.

Technorati: glassfish eclipse mysql jpa database

As all you Mac users (and probably many non-Macies) know Apple released Snow Leopard (Mac OS X 10.6) recently, even though this release was announced for September previously. Since a large part of the MySQL Workbench user base works on OS X it was clear that many of you will test Workbench on the new OS. However, so far we haven’t had the opportunity to do the same (too busy fixing bugs on released OSes) and hence we did not know about incompatibility problems there.

In the meantime several users did tests and reported us a crash on startup of the application, which means you cannot use MySQL Workbench on Snow Leopard for the time being. We are currently preparing build, test and developer machines with it and will hand out a fixed WB release as soon as possible. So, please stay patient. It’s only a matter of days.

I’ve been thinking recently about the failure scenarios of MySQL replication clusters, such as master-master pairs or master-master-with-slaves. There are a few tools that are designed to help manage failover and load balancing in such clusters, by moving virtual IP addresses around. The ones I’m familiar with don’t always do the right thing when an irregularity is detected. I’ve been debating what the best way to do replication clustering with automatic failover really is.

I’d like to hear your thoughts on the following question: what types of scenarios require what kind of response from such a tool?

I can think of a number of failures. Let me give just a few simple examples in a master-master pair:

Problem: Query overload on the writable master makes mysqld unresponsive

Do nothing. Moving the queries to another server will cause cascading failures.

Problem: The writable master is completely unreachable

Fence the writable master and promote the standby master.

Problem: The writable master is reachable but unresponsive due to overload-induced swapping

Do nothing. Moving the load to another server will cause cascading failures.

I don’t want to bias the jury, so I’ll stop there and ask you to contribute your failure scenarios and what you think the correct action should be.

Related posts:

1. MySQL replication breaks single-threaded limitation? It’s
2. How to check MySQL replication integrity continually I have rec
3. Pop quiz: how can one slave break another slave Suppose yo

Related posts brought to you by Yet Another Related Posts Plugin.

A short time ago in a galaxy nearby, Domas Mituzas wrote about contention profiling with GDB stack traces. Mark Callaghan found the technique useful, and contributed an awk script (in the comments) to aggregate stack traces and identify which things are blocking most threads. I’ve used it myself a time or five. But I’ve found myself wanting it to be fancier, for various reasons. So I wrote a little utility that can aggregate and pretty-print backtraces. It can handle unresolved symbols, and aggregate by only the first N lines of the stack trace. Here’s an example of a mysqld instance that’s really, really frozen up:

bt-aggregate -4 samples/backtrace.txt | head -n12
2396 threads with the following stack trace:
        #0  0x00000035e7c0a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1  0x00000000005f2bd8 in open_table ()
        #2  0x00000000005f3fb4 in open_tables ()
        #3  0x00000000005f4247 in open_and_lock_tables_derived ()

4 threads with the following stack trace:
        #0  0x00000035e7c0a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1  0x0000000000780099 in os_event_wait_low ()
        #2  0x000000000077de42 in os_aio_simulated_handle ()
        #3  0x000000000074a261 in fil_aio_wait ()

Related posts:

1. A tweak to column alignment for the mext script I tweaked
2. The unexpected consequences of SELinux I’ve

Related posts brought to you by Yet Another Related Posts Plugin.

There have been a lot of changes to the UI since the original screenshots were uploaded so it was about time that the page should reflect the current look on the application. Check them out here to see the new theme and features: http://kontrollsoft.com/screenshots

I have run into a number of cases recently that all had a similar look and feel. In most of these cases, the symptoms were very complicated, but they boiled down to just a few problems that can be prevented in very simple ways.

If you are not doing any of the following three simple things, you probably should. These are essential practices for building successful applications, which everyone should be doing. And the best part is how easy it is to do them — read on for the details.

1. Archive or Purge Your Database

Data archiving is crucial. There is a difference between data that your application needs now, and data that your application used to need. Do not co-mingle the two. Plan from the outset to archive or purge data that is no longer needed. Everybody wants to know how to deal with an application that is growing. The simplest way is to prevent it from growing. Archiving and purging unnecessary data is probably one of the highest value activities that you could ever do.

At a more technical level, archiving reduces the size of your workload’s “working set” — the data that the server really needs to keep in memory to function well. That’s because unused rows can bloat and dilute your indexes and tables. Storage engines such as InnoDB also don’t have features such as midpoint insertion algorithms for cache control, so a single table scan can flush really interesting data out of your LRU list and replace it with a bunch of stuff that’s needed only once for the table scan. Keeping those indexes small enough to fit in memory is a big step towards better performance, and archiving can help do that.

2. Monitor Your Servers

Everybody should have an automated monitoring system that is watching key elements of their systems. This can be very simple, even just an external service such as Pingdom or PagerDuty. Or it can be your own in-house hosted Nagios installation. The point is, your users should not be telling you when the service is down. Be the first to know, and have enough information to help you find the root of the problem.

Setting up monitoring can seem to be a daunting task, in part because of the steep learning curve that comes with most tools worth using. Another problem is the plethora of well-meaning but misguided advice and plugins for such tools, which will lead you to believe that you should monitor every ratio and counter there is in your systems — a sure recipe for an inbox flooded with useless noise, a monitoring screen that looks like the scene of a crime, and ultimately people who are numb to the monitoring system and ignore it when something really does go wrong. Keep it simple and monitor things that matter to your business: does the website load in the expected time and contain the expected text? Does the database server respond okay to a sample query that your application actually uses? Is the disk getting full?

3. Capture and Store Historical Data

Likewise, it is vital to have something recording metrics over time. A lot of people use an RRDTool-based system, such as Cacti. Again, it doesn’t really matter which one you use. At some point in the future, you are going to be very grateful that you have historical metrics to understand changes in your application, or understand what happened just prior to a crash. Here’s an example from a customer: the server crashed, and we could see in the log when that happened, but not why. The Cacti graph for disk space had a sharp curve — it showed the disk space suddenly filling up at the rate of gigabytes per minute. We looked at the disk and found it’d filled up with files due to an infinite loop in some code. How long would it have taken us to notice this otherwise, while checking dozens of possible causes?

Unlike with monitoring, you should measure and store everything you can, for as long as you can. You will never stay awake at night kicking yourself for measuring too much about your application.

All of the above are quite simple and fast to do once you’ve gone through the process a few times. I’d say that a practiced consultant can do a pretty comprehensive job in most applications in 8 hours or less. If it’s your first time through, it will take longer, and some good books might help. Either way, it is an investment with a very high ROI.

Entry posted by Baron Schwartz | No comment

Add to: | | | |

I’m going to present on Maatkit at the CPOSC conference in central Pennsylvania on Saturday, October 17th 2009. I’ll give an overview of the toolkit, which is no longer an easy task in a single session. I see a number of other interesting sessions have been accepted. It looks like it’ll be a good conference.

Related posts:

1. Speaking at EdUI Conference 2009 I’m
2. Learn about Maatkit at the MySQL Conference I’m
3. Presentation uploaded for Maatkit talk at MySQL Conference The slides

Related posts brought to you by Yet Another Related Posts Plugin.