Archive for the ‘mysql ha’ Category

MySQL HA reloaded by Ivan Zoratti

Февраль 5th, 2012

MySQL HA reloaded – old tricks and cool new tools to guarantee high availability to your MySQL Servers by Ivan Zoratti of SkySQL. This talk is a little longer, so check out: HA Reloaded – many ways to provide High Availability. The slides are already online.

Questions to ask: which level of high availability do I need? Do I require no loss of data? Do I need failover or is switchover enough? Can I provide a reasonable service when a component is down? Remember, five nine’s high availability also means a lot of infrastructure costs.

Other things to clarify: availability vs scalability. HA costs. HA for your entire architecture, not just for your database servers. Review your SLAs.

The best high availability solution today is combined solutions.

MySQL replication – asynchronous & semi-synchronous (lots of people use MySQL 5.5, about 4 people in room were on Percona Server – question asked due to semi-sync replication only being available in 5.5 & greater), there are pros & cons of row based replication vs statement based replication.

MySQL Replication via Multi-Master replication Manager (MMM). Features such as monitoring, automatic failover, data backup & resync. Unfortunately, it has some problems with the stability & automatic failover. The project is not improved anymore, so there are other solutions that you should consider today.

MySQL Replication with MHA is a preferred solution. Something to consider: –read-only=1 and log-bin on slaves. Master IP failover. FIltering rules. Multi-tier replication.

Tungsten Replicator – open source, heterogenous replication. Truly multi-master and fan-in with Global ID. Per-schema multi-thread. You can also use it to replicate to Postgresql, Oracle and other databases. There is also Tungsten Enterprise.

Synchronous replication with DRBD is typical for active/standby environments. People don’t really like this because they feel that there is a server doing nothing. You can always do it active/passive. It works with InnoDB only.

Synchronous replication with Galera works for InnoDB. Its multi-master with no SPOF. Its new/young technology so you may find some issues with it. Application failover must be managed, but the conflict resolution is quite tricky (when you commit a transaction you might be fine, but you may have transactions that are removed due to conflicts).

There is a commercial SchoonerSQL that provides synchronous master-slave replication for InnoDB. Its defined explicitly as a master-slave solution.

Active/Passive clusters using Shared Storage. Points to consider are the fact that redundancy & replication must be guaranteed by the shared storage. InnoDB only. What about filesystems?

Virtualized environments – data storage, high availability & load balancing are provided and managed by the virtualized software. The faults are handled by the software, not the database.

There is also geographical replication for disaster recovery, having a master-master asynchronous replication is used to update the backup data centre. There is also storage snapshots for disaster recovery (not-specific to MySQL, its storage systems based, use only InnoDB).

There is also MySQL Cluster but there is another presentation about this later at FOSDEM. Very nice closing slide, “The absolutely necessary comparison chart” which some may disagree, but Ivan thinks is the best way forward.

Related posts:

  1. MySQL at Google
  2. Ticketmaster thrives on MySQL Replication
  3. New MySQL 5.6 Features by Oli Sennhauser


PlanetMySQL Voting: Vote UP / Vote DOWN

HA Reloaded – Many ways to provide High Availability

Январь 18th, 2012

High Availability is one of the hottest topics for MySQL DBAs. As a matter of fact, when we (SkySQL) are called by users and customers, the top two questions for our PS team are:

  1. How can I make my MySQL database scalable?
  2. How can I make my MySQL database highly available

Since MySQL is the most used online database, these two questions are totally appropriate. Although the two concepts walk together, they should not be confused.

We refer to scalability when we define that a system must sustain a specific workload, measured in different ways – response time, latency, transactions or operations per second to name few, or a combination of them. So, we define the level of scalability we want to achieve and we design a system that can scale to that level.

We refer to high availability when we define that a system must be available, within the boundaries of the scalability mentioned above, also in case of malfunctions of one or more of its components. The latter, as many of you know, is called fault tolerance.

Developers and administrators tend to confuse scalability and availability and, even worse, approach the availability and the scalability of their systems all in once. This is wrong for the simple reason that the techniques to adopt to achieve high availability are different to the once to achieve scalability. What is true though, is that these techniques must be ultimately work together and a good system design is a constant reiteration of these techniques. up to the fine tuning of all the aspects that are linked together.

At SkySQL, one of the main goals is to design with our customers scalable and highly available systems. I have seen many presentations on the HA subject and I think most of them are really good – see the one that Henrik has prepared, for example, highly recommended – but I wanted to give you our perspective and show you the various options that we have investigated at SkySQL and that we evaluate for community users and for our customers. This is the reason why I have created this presentation, HA Reloaded.

HA Reloaded is a refresh of all the latest options available for MySQL in terms of HA solutions. I have presented a version of it in Seattle, in San Francisco and in Boston in December, next week I will land in New York City. People who know me, also know that I never do the same presentation twice, so if you are around NYC on Mon 23rd and you want to join us, you will find something completely new, even if you have already attended to my preso.

Here are all the details:
  • The slides of my presentation so far are on slideshare, available here.
  • The Meetup in NYC will start on Monday 23rd at 7pm – the venue is:
    General Assembly NY
    902 Broadway, 4th Floor
    New York, NY 10010
  • The URL to the MySQL Meetup in NYC is here: http://www.databasemonth.com/database/high-availability

 

For those of you who are attending the User Conference – no, I have not submitted this speech. As I said, there are already so many HA presos, I hope Henrik Ingo will do a refresh of his presentation and I am sure that our friends at Percona will talk a lot about their new solutions. But enough of this, please come and poke me at the meetup!

/iz


PlanetMySQL Voting: Vote UP / Vote DOWN

Multi-master and read slaves using Severalnines

Октябрь 13th, 2011
This blog post shows how you can use the Severalnines Configurator for MySQL Replication to deploy a Multi-master replication setup, and install ClusterControl.
You can also watch videos showing what is described below or read an even more detailed tutorial.

When the deployment is finished you have a set of tools to manage and monitor replication, as well as to add new slaves, and to perform failover.
You can choose:
  • The number of slaves you wish to connect to the master->relay master.
  • whether you want to use MySQL 5.5.x or Percona 5.5.x
  • Semi-sync replication or not (semi sync really recommended)
  • cloud provider (on premise/EC2/Rackspace
Please note that your hosts must resolve to a valid ip. If 'hostname -i' resolves to nothing or to 127.0.0.1 it will not work.

We recommend also that you don't use bi-directional replication. It may sound like a good idea, but in practice it is challenging to get it working well.

When deployed you can easily perform tasks as:
  • Add new slaves
  • Failover
  • View replication health information
  • Stop/start replication links
  • Stage slaves
The Configurator for MySQL Replication is a wizard-like application and you have to enter details about the setup you want to have. When this is done you will get a tar.gz that contains the deployment and management scripts.

ClusterControl

The deployment process will also install ClusterControl which is a set of monitoring agents and functionality to manage the database installation. The agents are deployed on each server and collects host information and information from the local MySQL server a particular agent is monitoring (such as replication statistics and status information).

One server (dedicated) is denoted the ClusterControl server. It is holding a database, CMON DB, that contains data about the monitored hosts, and the reporting data from the local agents is stored in this database. On this server a Controller is running and faciliates failover etc.

Moreover, the ClusterControl server exposes a web interface that can be used to graphically see the health of the replication cluster (after the deployment is done, take a web browser and point it to http://clustercontrolserver/cmon :

From ClusterControl you can then upload a schema, and start loading in data into a database. As well as doing GRANTs etc. In the Enterprise version you an also define performance probes (to make sure you system is not degrading over time), tune queries, try out new queries before putting them in production. You also have a Query Monitor and explains at your hands.

Failover is automatic (if the master fails, the relay will become the new master). When the master comes up again, it will connect to the new master and sync up, become the relay, and the slaves will failover to the relay. We also make use of the replication features of MySQL 5.5 (such as relay recovery) and a bunch of other techniques.

You can also add slaves or remove slaves with the click of a button, as well as scheduling backups.

Installation
When you have finished the wizard you get a package that you should deploy on the ClusterControl server:
tar xvfz s9s-mysql-55.tar.gz
cd s9s-mysql-55/mysql/scripts/install
./deploy.sh

The 'deploy.sh' script will create data directories, install initd scripts, create mysql users, apply database GRANTs (those you defined in the Configurator), and install ClusterControl.

A while in 'deploy.sh' replication will be started, and it can look like this:

*******************************************************************************
* Starting replication *
*******************************************************************************
Master hosts
------------
10.30.30.31: up
10.30.30.32: up
Slave hosts
------------
10.30.30.33: up
10.30.30.34: up
starting replication between 10.30.30.31 --> 10.30.30.32 (change master=1, reset master=1)
replication [started] 10.30.30.31 --> 10.30.30.32
starting replication between 10.30.30.32 --> 10.30.30.33 (change master=1, reset master=0)
replication [started] 10.30.30.32 --> 10.30.30.33
starting replication between 10.30.30.32 --> 10.30.30.34 (change master=1, reset master=0)
replication [started] 10.30.30.32 --> 10.30.30.34
master_host -->slave_host status master_status slave_status [binlog|m_pos|exec_pos|lag]
10.30.30.31 10.30.30.32 ok binlog.000003:250 binlog.000003| 250| 250| 0
--- slaves follows ---
10.30.30.32 10.30.30.33 ok binlog.000001:107 binlog.000001| 107| 107| 0
10.30.30.32 10.30.30.34 ok binlog.000001:107 binlog.000001| 107| 107| 0
After this, ClusterControl will be installed. If you are using RPM based installation there can be rpm dependency problems (we install libmysqlclient.so.16), but it can conflict with what comes with your distribution. Please let us know if this happens, contact us on community@severalnines.com.

Starting/stopping Replication and basic examples


After having executed deploy.sh, you can view the status of the mysql servers:
cd s9s-mysql-55/mysql/scripts/
./status.sh -a
Master hosts
------------
10.30.30.31: up
10.30.30.32: up
Slave hosts
------------
10.30.30.33: up
10.30.30.34: up
To view the replication status:
cd s9s-mysql-55/mysql/scripts/
./repl-status.sh -a
master_host -->slave_host status master_status slave_status [binlog|m_pos|exec_pos|lag]
10.30.30.31 10.30.30.32 ok binlog.000003:250 binlog.000003| 250| 250| 0
--- slaves follows ---
10.30.30.32 10.30.30.33 ok binlog.000001:107 binlog.000001| 107| 107| 0
10.30.30.32 10.30.30.34 ok binlog.000001:107 binlog.000001| 107| 107| 0
To stop replication between 10.30.30.32 (relay master) to 10.0.30.33 you can do:
cd s9s-mysql-55/mysql/scripts/
./stop-repl.sh -m 10.30.30.32 -s 10.30.30.33
stopping replication between 10.30.30.32 --> 10.30.30.33
replication [stopped] 10.30.30.32 --> 10.30.30.33
To start replication again you can do:
cd s9s-mysql-55/mysql/scripts/
./start-repl.sh -m 10.30.30.32 -s 10.30.30.33
starting replication between 10.30.30.32 --> 10.30.30.33 (change master=0, reset master=0)
replication [started] 10.30.30.32 --> 10.30.30.33

Other options for 'start-repl.sh' is -r (for reset master, dangerous), -c (change master). Normally these options are not needed but you can use them in conjuction with -l (logpos) and -f (binlog file) if you want to start replication from a particular position.

And we can check the replication status:
ubuntu@ip-10-49-122-115:~/s9s-mysql-55/mysql/scripts$ ./repl-status.sh -a
serverid master_host -->slave_host status master_status slave_status [binlog|m_pos|exec_pos|lag]
1 10.48.207.130 10.48.139.24 ok binlog.000005:494 binlog.000005| 494| 494| 0
2 replication not activated - you must start replication on this link.
--- slaves follows ---
3 10.48.139.24 10.49.122.56 ok binlog.000005:485 binlog.000005| 485| 485| 0
4 10.48.139.24 10.49.110.183 ok binlog.000005:485 binlog.000005| 485| 485| 0

Good luck and don't hesitate to contact us at feedback@severalnines.com or community@severalnines.com if you have any problems or whatever it may be. You can also book a demo this if you want to know more.

PlanetMySQL Voting: Vote UP / Vote DOWN

High Availability & Clustering at MySQL Sunday

Сентябрь 13th, 2010
mysql-cluster-cge-logo-300x205.pngWith the MySQL Sunday event at Oracle Open World rapidly approaching, and registrations to the event 10x higher than originally forecast, I thought it would be a good time to highlight sessions that are specifically addressing MySQL high availability, including MySQL Cluster .

You can see details and logistics of all of the sessions here

MySQL Cluster
Andrew Morgan and I will be presenting the latest enhancements in the world of MySQL Cluster and cluster management at our session at 15.30 in Golden Gate C3 of the Marriott Hotel.

If you are staying around for Oracle Open World itself, then drop by our stand at Moscone West W-071 where we'll give you a live demonstration of MySQL Cluster maintaining continuous operation as we upgrade, reconfigure and scale the cluster, all on-line.

MySQL Replication
Before the MySQL Cluster session - at 13.05 (precisely...) in Golden Gate C3, Dr Lars Thalmann, Development Manager, MySQL Replication & Backup - will be presenting the latest replication enhancements  delivered with the MySQL 5.5 release, specifically covering:
- replication heartbeating
- semi-synchronous replication
- fsync tuning
- relay log corruption recovery

There is also a very interesting session planned for 14.45 where Harrison Fisk from Facebook will be presenting Advanced MySQL Replication Techniques

Great, But I'm not Going....
Of course, if you are not able to attend the MySQL Sunday event, we will be presenting many of these subjects in forthcoming live webinars, starting with Lars' session on MySQL replication enhancements, scheduled for October 12th at 0900 pacific time / 17.00 UK .  Look out for more details on how to register for that session in the next week.

If you want to learn more about MySQL Cluster, you can check-out our complimentary whitepapers:
- MySQL Cluster for web applications
- MySQL Cluster for telecoms applications

If you are attending MySQL Sunday, I hope to meet you there in person.  If not, at one of our forthcoming webinars

PlanetMySQL Voting: Vote UP / Vote DOWN

Linux Open Administration Days 2010

Апрель 20th, 2010

So about 4 monts ago there was the crazy idea to start a new FOSS event in Belgium targeted at sysadmins.

What started out as an event for local people to meet local people with some local speakers actually ended up being a small local event with some top international speakers on onfiguration mananagement and system administration mixed with a bunch of good local ones !

I had the honour to open the conference with an extremely short version of the Devops talk I gave earlier last year.. extremely short as I knew that over the course of the weekend the topic would reoccur a lot.

We had the first european talk on Chef, by Joshua Timberman, and we had Puppet talks amongst by Dan Bode from Puppetlabs and CFengine talks , devops was a frequently dropped word,

We had a book raffle where we handed out O'Reilly's .. we had a great free pizza party (got the idea from the saturday pizza event at LCA 2005) , and we had some free beer. Sounds like a good combination for a geeky weekend.

Apart from the regular talks there were plenty of Open Spaces where interesting topics were discussed ... we had spaces on Open Source vs Open Core , strong voices were heard when we discussed what we should do with the Open Core companies that claim to value Open Source , some people think we should actually list the fauxpensource ones somewhere and make sure the world knows about them

We had an awesome configuration management discussion session discussing Chef vs Puppet vs CFengine . And much much more ...

Some people owe me plenty of Sushi as I had to do my MySQL HA talk before their Managing MySQL talk , but other than that .. things just went fine..

Trackback URL for this post:

http://www.krisbuytaert.be/blog/trackback/998

PlanetMySQL Voting: Vote UP / Vote DOWN