Before you begin.

For my MySQL instances I used the following:

• Extra Large, High-Memory, and High-CPU instances. Although the instruction should work on any type of instance.
• RedHat Enterprise Linux 6.2 64-bit AMI
• For MySQL data storage, multiple identical EBS devices attached to each instance

The configuration template provided in this post assumes the new MySQL instance only needs InnoDB storage engine.

Grab the packages.

Download the appropriate packages from MySQL web page. You will need client, shared-compat, and server. For example:

MySQL-client-5.5.24-1.el6.x86_64.rpm
MySQL-server-5.5.24-1.el6.x86_64.rpm
MySQL-shared-compat-5.5.24-1.el6.x86_64.rpm

Make sure the packages you downloaded are for a 64-bit architecture. You could of course choose packages from other vendors as the change should not affect the process in any relevant way.

Install MySQL

Remove the MySQL libraries that have been shipped with the system and install the new client and library packages:

[root@ip-10-238-234-26 ~]# rpm -e --nodeps mysql-libs
[root@ip-10-238-234-26 ~]# rpm -ihv MySQL-client-5.5.24-1.el6.x86_64.rpm MySQL-shared-compat-5.5.24-1.el6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:MySQL-shared-compat    ########################################### [ 50%]
   2:MySQL-client           ########################################### [100%]

Then install the server package:

[root@ip-10-238-234-26 ~]# rpm -ihv MySQL-server-5.5.24-1.el6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:MySQL-server           ########################################### [100%]

Search for EBS volumes.

Examine system to find the EBS volumes you will use for MySQL storage.

[root@ip-10-32-2-145 ~]# dmesg | grep 'unknown partition'
 xvds: unknown partition table
 xvdt: unknown partition table
 xvds: unknown partition table
 xvdt: unknown partition table

Please keep in mind some of the disks reported may be EC2 ephemeral storage, which you typically do not want to use for MySQL data, so make sure to only use EBS volumes in the following steps. If you are uncertain what is what, you can call fdisk -l and check the disk sizes:

[root@ip-10-32-2-145 ~]# fdisk -l

Disk /dev/xvde1: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
[..]

Create partitions.

Create a new partition on the first disk.

[root@ip-10-32-2-145 ~]# fdisk /dev/xvds
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x3774dc60.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-54823, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-54823, default 54823):
Using default value 54823

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): wq
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Assuming you created several identical EBS volumes, duplicate the partition table onto each.

[root@ip-10-32-2-145 ~]# sfdisk -d /dev/xvds |  sfdisk /dev/xvdt
Checking that no-one is using this disk right now ...
OK

Disk /dev/xvdt: 54823 cylinders, 255 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/xvdt: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/xvdt1            63 880731494  880731432  fd  Linux raid autodetect
/dev/xvdt2             0         -          0   0  Empty
/dev/xvdt3             0         -          0   0  Empty
/dev/xvdt4             0         -          0   0  Empty
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)

Verify partitions.

[root@ip-10-32-2-145 ~]# fdisk -l | grep -E '^/dev/xvd'
[..]
/dev/xvds1               1       54823   440365716   fd  Linux raid autodetect
/dev/xvdt1               1       54823   440365716   fd  Linux raid autodetect

Create RAID volume.

Create a new RAID level 0 volume over all EBS partitions. Use relatively large chunk size – 128KB or 256KB should be good choices.

[root@ip-10-32-2-145 ~]# mdadm -C /dev/md0 --chunk=256 -n 2 -l 0 /dev/xvds1 /dev/xvdt1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Save the md array configuration.

[root@ip-10-32-2-145 ~]# echo 'DEVICE /dev/xvds1 /dev/xvdt1' >> /etc/mdadm.conf
[root@ip-10-32-2-145 ~]# mdadm --examine --scan >> /etc/mdadm.conf

Here is what the file should look like

[root@ip-10-32-2-145 ~]# cat /etc/mdadm.conf
DEVICE /dev/xvds1 /dev/xvdt1
ARRAY /dev/md/0 metadata=1.2 UUID=f7d5b2e7:495efeb3:26297d34:7be87159 name=ip-10-32-2-145:0

Rebuild the initramfs image.

If this is not done, system won’t see the new /etc/mdadm.conf contents during boot and md device may be renamed to something else.

[root@ip-10-32-2-145 ~]# mkinitrd --force /boot/initramfs-2.6.32-220.el6.x86_64.img 2.6.32-220.el6.x86_64

You have to use the initramfs file and kernel version that are appropriate for your system. You can list /boot directory contents to figure this out:

[root@ip-10-226-155-162 ~]# ls -1 /boot/
config-2.6.32-220.el6.x86_64
efi
grub
initramfs-2.6.32-220.el6.x86_64.img
symvers-2.6.32-220.el6.x86_64.gz
System.map-2.6.32-220.el6.x86_64
vmlinuz-2.6.32-220.el6.x86_64

Create filesystem.

This part may take a while depending on the volume size.

[root@ip-10-32-2-145 ~]# mkfs.ext4 /dev/md0
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=64 blocks, Stripe width=128 blocks
55050240 inodes, 220182272 blocks
11009113 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
6720 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Create a mount point.

I do not like mounting to /var/lib/mysql directly. Typically I create a new directory inside /vol, or /mnt, or /data.

[root@ip-10-32-2-145 ~]# mkdir -p /vol/mysql

Store the filesystem information.

Always use noauto option during installation. It skips automatic filesystem mounting during system init, which can save you from a lot of trouble if something does not work for any reason and when the boot process hangs on this step. EC2 does not come with a system console that would allow rescuing a system from such state.

[root@ip-10-32-2-145 ~]# echo '/dev/md0   /vol/mysql   ext4   rw,nobarrier,noatime,nodiratime,noauto   0 0' >> /etc/fstab

It may actually be very reasonable to keep noauto there forever and either mount the database volume manually each time the server reboots (it implies you also have to start MySQL manually) or implement volume mounting in a way that does not block the instance in case of some errors.

Mount the new volume.

This command can now use the configuration from /etc/fstab.

[root@ip-10-32-2-145 ~]# mount /vol/mysql

The command should not return anything, but to make sure it worked, just run a quick check whether the volume was acutally mounted or not:

[root@ip-10-32-2-145 ~]# df -h /vol/mysql
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              827G  201M  785G   1% /vol/mysql

Mounted on should point the new mount point and not to / for example.

Move the default data directory.

Move into the new location the default MySQL data directory created by the MySQL-server package installtion. Also create a directory for MySQL logs.

[root@ip-10-32-2-145 ~]# mv /var/lib/mysql /vol/mysql/
[root@ip-10-32-2-145 ~]# ln -s /vol/mysql/mysql /var/lib/mysql
[root@ip-10-32-2-145 ~]# mkdir /vol/mysql/log
[root@ip-10-32-2-145 ~]# chown mysql:mysql /vol/mysql/log

Set swappiness to prevent unnecessary swapping.

You can refer to this article for more details on swappiness.

[root@ip-10-32-2-145 ~]# echo 'vm.swappiness = 0' >> /etc/sysctl.conf
[root@ip-10-32-2-145 ~]# sysctl -p /etc/sysctl.conf

Configure time synchronization.

Create a new file called ntpdate in /etc/cron.daily with the following contents:

#!/bin/sh

/usr/sbin/ntpdate pool.ntp.org 1> /dev/null 2>&1

Then set the file permissions:

[root@ip-10-32-2-145 ~]# chmod +x /etc/cron.daily/ntpdate

Disable or configure SELinux.

RedHat comes with SELinux enabled, but often SELinux will be disabled during a database server installation process. To disable, perform the following:

[root@ip-10-32-2-145 ~]# cd /etc/sysconfig/
[root@ip-10-32-2-145 sysconfig]# replace 'SELINUX=enforcing' 'SELINUX=disabled' -- selinux
selinux converted

Restart the system.

It is time to restart the system. You should do this for two reasons:

• You want to verify if the RAID volume loads correctly after a restart and that MySQL data volume can be mounted again without any problems.
• You disabled SELinux and it is also the only way to make the change effective.

Verify md array stats and SELinux status.

Once the system boots again, verify the md array status.

[root@ip-10-32-2-145 ~]# cat /proc/mdstat
Personalities : [raid0]
md0 : active raid0 xvds1[0] xvdt1[1]
      880729088 blocks super 1.2 256k chunks

unused devices: 

It should still say md0 if you regenerated the initramfs image.

If you disabled SELinux, verify that as well.

[root@ip-10-32-2-145 ~]# selinuxenabled
[root@ip-10-32-2-145 ~]# echo $?
1

selinuxenabled indicates whether SELinux is enabled or disabled. It returns 0 if SELinux is enabled and 1 if it is not enabled.

Mount data volume.

[root@ip-10-32-2-145 ~]# mount /vol/mysql
[root@ip-10-32-2-145 ~]# df -h /vol/mysql
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              827G  201M  785G   1% /vol/mysql

If you want, now you can remove noauto from /etc/fstab.

Create MySQL configuration file.

I sometimes use this simple script that writes a good default configuration into /etc/my.cnf. It auto-tunes a few parameters that commonly need customization. You could just copy-paste it into the command line and run.

(cat <<EOF
[mysqld_safe]
log-error = /vol/mysql/log/mysql-error.log

[mysqld]
user = mysql
port = 3306
socket = /vol/mysql/mysql/mysql.sock

datadir = /vol/mysql/mysql
tmpdir = /tmp
pid-file = /vol/mysql/mysql/mysql.pid

character-set-server = utf8

slow-query-log
slow_query_log_file = /vol/mysql/log/mysql-slow.log
long_query_time = 10

log-bin = /vol/mysql/log/mysql-bin
relay-log = /vol/mysql/log/mysql-relay
server-id = $(/bin/hostname | /usr/bin/md5sum | /usr/bin/awk --non-decimal-data '{ hex=substr($1, 24, 8); printf ("0x"hex)+0 }')

back_log = 50
max_connections = 500
max_connect_errors = 100
skip-name-resolve

key_buffer_size = 8M
sort_buffer_size = 2M
read_rnd_buffer_size = 4M
bulk_insert_buffer_size = 8M
max_allowed_packet = 16M
max_heap_table_size = 64M
tmp_table_size = 64M

table_open_cache = 2048
thread_cache_size = 16

innodb_file_per_table
innodb_data_file_path = ibdata1:128M:autoextend
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_log_buffer_size = 16M
innodb_buffer_pool_size = $(/usr/bin/awk '/MemTotal:/ { printf "%.0fM\n", ($2*0.70)/1024 }' /proc/meminfo)
innodb_log_file_size = 512M
innodb_log_files_in_group = 2
innodb_read_io_threads = $(($(/bin/grep -E 'md[0-9]+ : active raid0' /proc/mdstat | wc -w) - 2))
innodb_write_io_threads = $(($(/bin/grep -E 'md[0-9]+ : active raid0' /proc/mdstat | wc -w) - 2))
innodb_io_capacity = $((($(/bin/grep -E 'md[0-9]+ : active raid0' /proc/mdstat | wc -w) - 4 ) * 130))
innodb_thread_concurrency = 0

[mysql]
no-auto-rehash
socket=/vol/mysql/mysql/mysql.sock

[client]
socket=/vol/mysql/mysql/mysql.sock

EOF
) > /etc/my.cnf

Remember this is just a template. Each server may need individual adjustments to database configuration.

It’s done.

You can now start MySQL.

[root@ip-10-32-2-145 ~]# /etc/init.d/mysql start
Starting MySQL (Percona Server)........................     [OK]

I’ve just experienced this very same first-time feeling as we’ve released Yapixx – a picture sharing web application using the cloud storage.   The most amazing thing about Yapixx is that we wrote very little code to make it happen: most of its functionality is provided by WordPress, which by the way we didn’t modify at all. 

On one hand Yapixx is just WordPress, enhanced with plugins and configured to provide good picture sharing experience. 

On the other hand, Yapixx has gone where WordPress could not go before – Yapixx runs completely on top of Amazon S3, using the enormous power of S3 to make serving the users’ pictures highly scalable and storing all data extremely durable. 

As an engineer I’ve been on the never ending quest of finding new ways to do more with less.  How do I write less code and provide more functional solution?  How do I empower my customers to accomplish more with applying very little effort (or preferably no effort at all)?  Yapixx is one of those gems – it shows how unbelievably simple it is to run a beautiful website on top of Amazon S3: you don’t need to learn new APIs, you don’t have to know how to write code, you don’t even have to know that you are relying on MySQL + ClouSE to store data reliably and securely in the cloud!

Yapixx is not a toy application: it’s a fully functional ready-to-run WordPress on Amazon S3. With a few clicks Yapixx can be transformed into anything WordPress can do, while storing all website content in Amazon S3.  Just continue doing what you set to do with your Web site – to build a beautifully powerful representation of you, your company, and your cause.

Amazon S3 is a very powerful service designed for building one-of-a-kind massively distributed applications.  This is its strength but it’s also its weakness: if you are not building one-of-a-kind massively distributed application, the low-level vendor-specific APIs and eventual consistency guarantees are just the unnecessary complexities that you pay for, but don’t use.  WordPress is an example of how to make Amazon S3 a true cloud storage utility service that can be easily used by millions.  It’s a people-oriented (as opposed to technology-oriented) approach to building a distributed system – as a constellation of beautiful websites powered by creativity and uniqueness of individuals, like the Internet itself is!

‘Nuff said :-).  Get your own WordPress on S3 for free now.

Artem

OblakSoft is proud to introduce the 1^st ever dynamic WordPress site running on top of Amazon S3: Yapixx.  Now you too can launch your own beautiful website on Amazon S3.

While Yapixx stands for Yet Another Picture Sharing Site, it is actually one of a kind.  Yapixx is WordPress that was moved to run on top of Amazon S3 storage without changing a line of code in the WordPress core engine.

What is Yapixx?

Yapixx is ready-to-run WordPress on Amazon S3.  It took about one man-week to create Yapixx, thanks to power of WordPress.  WordPress is a mature and versatile platform that powers millions of websites with diverse purposes from blogging to social networking.  There are thousands of plugins and themes written for WordPress that can transform the website into pretty much anything.  And now it can run with Amazon S3 without writing a line of code.

Yapixx taps into power of Amazon S3.  Amazon S3 is inexpensive, highly reliable, available and scalable storage service.  It got even less expensive recently and its usage is growing 2.5-2.9 times year-over-year.  Using Amazon S3 to store Yapixx data has the following benefits:

•          Storage cost scales with usage, no upfront reservation is needed
•          Storage consumption scales up and down with the amount of data stored
•          Storage is extremely reliable and durable by Amazon S3 design
•          Pictures are served by Amazon S3 directly, which makes Yapixx highly scalable
•          No database backup and recovery is needed, now that Amazon S3 holds both for the website content stored in WordPress database and website’s media files

Yapixx demonstrates how easy it is to create web applications on top of Amazon S3.  Amazon S3 has been able to only power static websites; now S3 can power sophisticated dynamic web software such as WordPress.

Try out Yapixx

Here are the five easy steps to get Yapixx up and running:

1. Sign up for an AWS account.
2. Create an S3 bucket.
3. Start EC2 instance using Yapixx AMI.
4. Connect to the web application from a web browser.
5. Enter the S3 data location and authentication information.

The full step-by-step guide for launching Yapixx is available here.

All done!  You’ve got a picture sharing site running on top of Amazon S3.

Get started with Yapixx today for FREE!  It may be the boost that you need to propel your creative ideas and turn them into the next $1B acquisition.

Take Your Own WordPress Site to Amazon S3

You can use Yapixx as a starting point to create your own beautiful WordPress site.  The plugins that turn WordPress into Yapixx, such as the wp2cloud plugin, the Yapixx theme, etc. are available under the GPL completely FREE.  Creating a new AMI with your own web application is a matter of two mouse clicks in the AWS management console:

Moreover, you may even get your web application hosted completely free in the Amazon cloud, if your AWS usage doesn’t exceed the AWS free tier limits.  This is a great way to get a new web application to production quickly, absolutely FREE!  And when your web application usage grows, you can easily move it to larger cloud instance or to your own hardware: the data is stored in Amazon S3 and can be accessed from anywhere.

Launch your own WordPress site to the cloud today!  It’s easy and FREE, no writing code is required.  Start at http://www.oblaksoft.com/downloads.

Let us know what you think about Yapixx!  How can we further simplify the migration of websites to the cloud storage?

See also:

WordPress on S3: how it works.

Yapixx consists of the following software components:

• ·         WordPress configured as a picture sharing site
• ·         Yapixx theme based on zBench with modifications to look better for picture sharing
• ·         wp2cloud plugin that stores pictures and other media in Amazon S3 via ClouSE
• ·         LAMP software stack (Linux, Apache, MySQL, PHP)
• ·         ClouSE – the Cloud Storage Engine for MySQL

The Yapixx architecture looks like the following:
 

The WordPress platform provides the web user interface.  WordPress stores the web site content in a MySQL database.  MySQL uses ClouSE to store data in Amazon S3.  The end users use a web browser to interact with WordPress; web pages come from WordPress.  All the pictures (and other media) in the pages are served by Amazon S3, where they are stored as weblobs by WP2Cloud WordPress plugin.

WordPress is a mature and versatile platform that powers millions of websites with diverse purposes from blogging to social networking.  There are thousands of plugins and themes written for WordPress that can transform the website into pretty much anything.  And now with ClouSE WordPress can run on Amazon S3.

ClouSE is the key component of Yapixx application.  ClouSE is the Cloud Storage Engine for MySQL that can utilize cloud storage such as Amazon S3 to store user data.  Applications powered by ClouSE enjoy all the benefits of cloud storage – scaling cost with usage, high storage availability and reliability, quick and easy disaster recovery, etc.  And weblobs that store pictures for Yapixx can be served directly from cloud storage relying on its enormous power to make serving the content highly scalable.

ClouSE is the first and only relational database technology that overcomes cloud storage adoption barriers that prevents other databases from using cloud storage.  With ClouSE, cloud storage can be just plugged into a database server seamlessly and securely.

ClouSE and Amazon S3 provide seamless and secure data protection, scalability and availability for websites, static or dynamic.  A website on cloud storage has the following characteristics:

•          Storage cost scales with usage, no upfront reservation is needed
•          Storage consumption scales up and down with the amount of data stored
•          Storage is extremely reliable and durable by Amazon S3 design
•          Data is compressed, which significantly reduces storage size and cost
•          Data is encrypted, which provides data confidentiality
•          Pictures are served by Amazon S3 directly, which makes the website highly scalable
•          No database backup and recovery is needed, which makes the website highly available

ClouSE brings the power of Amazon S3 to websites, providing an easy and inexpensive built-in solution for scenarios that previously required expensive infrastructure and operation.  Overprovisioning primary storage, implementing onsite + offsite backup and replication are not needed any more.  There is no need to implement and routinely test disaster recovery procedures.  No need to buy more expensive software editions.  The end-to-end data continuity is in the box: no extra tools, processes or equipment is required.  That’s the true realization of the cloud promise – commoditization and democratization of solutions that previously could only be afforded by large IT organizations with sizeable budgets.

The most amazing thing about Yapixx is that WordPress platform was not initially designed to store its data in cloud storage.  Yet, now it runs on Amazon S3 without any changes to the WordPress core engine!  We like to say that Yapixx was assembled and configured for the cloud, rather than designed and implemented for the cloud.

Launch your own WordPress site to the cloud today!  It’s easy and FREE, no writing code is required.  Start at http://www.oblaksoft.com/downloads.

We’d love to know what you think of Yapixx.  Are you considering to move your website to the cloud storage?

See also:

WordPress on S3: run a beautiful website on Amazon cloud storage.

I was running the benchmarks on a High-CPU Extra Large Instance and couldn’t see any reliability in the results at all. I mean, in one moment I was getting poor throughput and horrible response times only to see it improve a lot a few minutes later. I ruled out a possibility that it could be some kind of problems with the storage performance causing this, so I started suspecting MySQL itself – perhaps some sort of contention. But no system statistics, nor any information from MySQL wanted to confirm that theory.

Finally, I went for oprofile and started it during one of the benchmarks and what I saw surprised me:

When oprofile was running MySQL performance nearly doubled. It was also reflected in the CPU utilization which increased accordingly and also got a lot more stable. After a while, I stopped oprofile and the performance dropped again.

At this point I am not sure how to explain this, however this was repeatable. Anyone?

Let me post the main part of the press release here:

SAN JOSE – April 18, 2012 – SkySQL, the first choice in affordable database solutions for the MySQL® and MariaDB® databases in the enterprise and the cloud, today announces that the company has raised $4 million in Series A funding from a number of investors, including OnCorps, an elite peer-based community of veteran technology investors and advisors committed to bringing better, cost-disruptive technologies into the mainstream. Also funding the round are European investors including Finnish Industry Investment Ltd., Spintop Ventures and Open Ocean Capital.

SkySQL will primarily use the investment to fund growth in its new product development, including adding critical positions. This is the company’s most recent move in expanding beyond a MySQL and MariaDB services-only company into a fully-fledged database products and services provider. SkySQL made its debut in October 2010 and has been funded to date with seed money from OnCorps, as well as Open Ocean Capital. Today, SkySQL is currently operating in 13 countries.

“The SkySQL strategy aligns well with OnCorps’s technology leadership vision,” said Bob Suh of OnCorps. “SkySQL will capitalize on the growing database and cloud services market. We are delighted and look forward to working with the SkySQL team to contribute to their continued success.”

“This investment will let us focus on ramping up development and efforts behind both enterprise and cloud database products that address a very real need in the marketplace,” said Ulf Sandberg, chief executive officer of SkySQL. “We also plan to build on our existing MySQL and MariaDB services offering, which offers customers unrivaled support, consulting and training.”

You can read the full press release here:

http://www.skysql.com/news-and-events/press-releases/skysql-raises-4-million-series-round

Very exciting times!