In my talk on MySQL Conference and Expo 2010 “An Overview of Flash Storage for Databases” I mentioned that most likely there are other players coming soon. I actually was not aware about any real names at that time, it was just a guess, as PCI-E market is really attractive so FusionIO can’t stay alone for long time. So I am not surprised to see new card provided by Virident and I was lucky enough to test a pre-production sample Virident tachIOn 400GB SLC card.

I think it will be fair to say that Virident targets where right now FusionIO has a monopoly, and it will finally bring some competition to the market, which I believe is good for the end users. I am looking forward to price competition ( not having real numbers I can guess that vendors still put high margin in the price) as well as high performance in general and stable performance under high load in particular, and also competition in capacity and data reliability areas.

Priceline for Virident tachIOn cards already shows the price competition: oriented price for tachIOn 400GB is 13,600$ (that is 34$/GB) , and entry-base card is 200GB with price 6,800$ (there also is 300GB card in product line). Price for FusionIO 160GB SLC ( from dell.com, price on 14-Jun-2010 ) is 6,308.99$ ( that is 39.5$/GB)

Couple words about product, I know that Virident engineering team was concentrating on getting stable write performance in long running
write activities and in cases when space utilization is close to 100%. As you may know (check my presentation) SSD design requires background
“garbage collector” activity, which requires space to operate and Virident card already has enough space reservation to get stable write performance even when the disk is almost full.

As for reliability, I think, the design of the card is quite neat. The card by itself contains bunch of replaceable flash modules, and each individual module can be changed in case of failure. Also internally modules are joined in RAID (it is fully transparent for end user).

All this guarantees good level of confidence in data reliability: if a single module fails, the internal RAID will allow to continue operations, and after the replacement of module – it will be rebuilt. It still leaves the controller on card as single point of failure, but in this case all flash modules can be safely relocated to the new card with working controller. (Note: It was not tested by Percona engineers, but taken from vendor’s specification)

As for power failures – flash modules also come with capacitors which guarantees data delivery to final media even if power is lost on the main host. (Note: It was not tested by Percona engineers, but taken from vendor’s specification)

Now to most interesting part – performance numbers. I took sysbench fileio benchmark with 16KB blocksize to see what maximal performance we can expect.

Server specification is:

• Supermicro X8DTH series motherboard
• 2 x Xeon E5520 (2.27GHz) processors w/HT enabled (16 cores)
• 64GB of ECC/Registered DDR3 DRAM
• Centos 5.3 2-6.18.164 Kernel
• Filesystem is XFS formatted with mkfs.xfs -s size=4096 option ( size=4096, sector size, is very important to have aligned IO requests) and mounted with nobarrier option
• Benchmark: sysbench fileio on 100GB file, 16KB blocksize

The raw results are available on Wiki

And the graphs for random read, writes and sequential writes:

I think very interesting to see distribution of 95% response time results ( 0 time is obviously the problem in sysbench, which has no enough time resolution for such very fast operations)

As you can see we can get about 400MB/sec random write bandwidth with 8-16 threads and
with 3.1ms (for 8 threads) and 3.8ms (16 threads) response time in 95% of cases.

As some issue here, I should mention, that despite the good response time results,
the maximal response time in some cases can jump to 300 ms per request, and I was told
it corresponds to garbage collector activity and will be fixed in the production release of driver.

I think it would be fair to get comparison with FusionIO card, especially for write pressure case
As you may know FusionIO recommends to have space reservation to get sustainable write performance
(Tuning Techniques for Writes).

I took FusionIO ioDrive 160GB SLC card, and tested fully formatted card (filesize 145GB), card formatted with 25% space reservation (file size 110GB), and Virident card 390GB filesize. It also allows us to see if Virident tachIOn card can sustain write in fully utilized card.

As disclaimer I want to mention that Virident tachIOn card was fine tuned by Virident engineers, while FusionIO card was tuned only by me and I may not have all knowledge needed for FusionIO tuning.

First graph is random reads, so see compare read performance

As you see in 1 and 4 threads FusionIO is better, while with more threads Virident card scales better

And now random writes:

You can see that FusionIO definitely needs space reservation to provide high write bandwidth, and it comes with
cost hit ( 25% space reservation -> 25% increase $/GB).

In conclusion I can highlight:

• I am impressed with architecture design with replaceable individual flash modules, I think it establishes new high-end standard for flash devices
• With single card you can get over 1GB/sec bandwidth in random reads (16-64 working threads), and it is the maximal results what I’ve seen so far ( again for single card)
• Random write bandwidth exceeds 400MB/sec (8-16 working threads)
• Random read/write mix results are also impressive, and it can be quite important in workloads like FlashCache, where card have both concurrent read and write pressure
• Quite stable sequential writes performance (important in question for log related activity in MySQL)

I am looking forward to present results in sysbench oltp, tpcc workload, and also in FlashCahce mode.

Entry posted by Vadim | No comment

Add to: | | | |

Also Facebook team released WriteThrough module for FlashCache, which could be good trade-off if you want extra warranty for data consistency and your load is mostly read-bound, so I tested this mode also.

All setup is similar to previous post, so let me just post the results with FlashCache on FusionIO in 20% dirty page, 80% dirty pages and write-through modes. I used full 80GB for caching ( total size of data is about 100GB).

Conclusions from the graph:

• with 80% dirty page we have about 4x better throughput ( comparing to RAID).
• Write-through mode is about 2x gain, but remember that load is very write intensive and all benefits in write-through mode come only from cached reads, so it is pretty good for this scenario

On this post I finish my runs on FlashCache for now and I think it may be considered for real usage, at least you may evaluate how it works on your workloads.

Entry posted by Vadim | No comment

Add to: | | | |

For initial runs I took tpcc 100 warehouses ( about 10GB of data) which fully fits into memory (32 GB on server),
and compared 1 and 16 users in MySQL-5.1.46/PBXT and Percona Server / XtraDB – 5.1.45-rel10.2. As workload is totally memory based it will show how PBXT scales in CPU-bond cases on 16 cores systems.

As storage system it was Intel SSD X25-M card.

While full results and config are on Wiki:
http://www.percona.com/docs/wiki/benchmark:pbxt:tpcc:start

there are graphs for 1 user:

and 16 users:

Interesting to see that in case with 1 user the maximal throughput in PBXT is about 1.5x better XtraDB, but
there periodical drops which are very similar to periodical drops in InnoDB without adaptive checkpointing, and I guess it is also related to checkpoint activity.
The final results are also better for PBXT: 5785.567 TpmC vs 4905.967 TpmC ( XtraDB)

For 16 threads final result is: 26129.350 TpmC for PBXT and 29485.518 TpmC for XtraDB , and from the graph you can see that the maximal throughput is about identical, while PBXT spends more time in drops area. Again it looks like PBXT are not fully keeping up with checkpoint activity and I am looking PBXT addresses this problem also. Beside this issue PBXT looks pretty good and in next round I am going to run IO intensive workloads.

Entry posted by Vadim | One comment

Add to: | | | |

But before test setup let me address some concern (which I also had). Intel X25-M has a write cache which is not battery backuped, so there is suspect you may have data loss in the case of power outage.
And in the case with FlashCache it would mean you can send your database to trash, as there is no way to recovery from that ( only restore from backup).
I personally did couple of power failure tests and there is article on this topic http://www.anandtech.com/show/2614/10. I did not see any data loss in my tests, and the article says that the write cache “..isn’t used for user data because of the risk of data loss, instead it is used as memory by the Intel SATA/flash controller for deciding exactly where to write data..”. So I assume we should be safe running Intel X25-M with enabled write cache.

Second issue I faced, and which took quite efforts to sort ( thanks Mohan Srinivasan, maintainer of FlashCache, for help with that) is 16KB alignment in XFS and FlashCache. As developers recommend to have 16KB block size in XFS and 16KB block size in FlashCache, I was not able to make it working, there too many moving parts – it’s XFS alignment by RAID strip size, XFS aligned file allocation, raid partition should be created aligned by 16K block, etc. In result I had too many cache misses
so results were not impressive, I had to move to standard for filesystems 4K blocks.

For benchmark I used sysbench oltp with 300 mln rows, which gives about 70GB of data. I allocated 35GB for cache on SSD and I used 16GB and 10GB buffer pool sizes. I run sysbench with special distribution, not uniform, the reason for that is that I do not really expect big improvement from clear uniform random hits, as in this case we will have 1/2 probability to go to disk read, and also there overhead in FlashCache on flushing and replacing pages. Special distribution is 75% of all requests are hits 1% of data, and rest data are accessed in 25% of cases. I think it also better describes real workload, as you do not expect all 100% of users coming to your website in the same time.

So full details are there
http://www.percona.com/docs/wiki/benchmark:flashcache:sysbench:300mln_rows:start

And short results:

As you see there is more 2x improvement, which is quite impressive.

Of course you do not have to believe benchmarks, so I encourage you to test yourself (my full scripts and mysql config files are on Wiki page).
There are binaries for CentOS 5.5 and Ubuntu 10.04 if you are not so much in kernel modules compiling:
http://percona.com/downloads/TESTING/FlashCache/centos-2.6.18-194.el5.tar.gz

http://percona.com/downloads/TESTING/FlashCache/ubuntu.10.04.kernel-2.6.32-21-server.tar.gz

Entry posted by Vadim | No comment

Add to: | | | |

I made binaries for FlashCache for CentOS 5.4, kernel 2.6.18-164.15, you can download it from our testing stage. It took some efforts to make binary, you may get my instructions for CentOS on FlashCache-dev mail-list, most likely it will not work for different CentOS / Kernel.

The full results, scripts and settings are on Benchmarks Wiki.

* Hardware: Dell PowerEdge R900
* IO subsystems
* RAID10: 8 disks SAS 2.5″ 15K ( main storage)
* SSD-1: Intel X25-E 32GB ( 1 gen firmware) ( Model Number: SSDSA2SH032G1GN INTEL, Firmware Revision: 045C8621)
* SSD-2: Intel X25-M 160GB ( 2 gen ) ( Model Number: INTEL SSDSA2M160G2GC, Firmware Revision: 2CV102HA )
* FlashCache: build over SSD-1 or SSD-2
* Filesystem XFS, builds as: mkfs.xfs -f -d su=16384,sw=40 /dev/sdc
* mounted with -o nobarrier option
* InnoDB files layout: ibdata1 and ib_logfile* are placed on separate RAID partition ( not on FlashCache or SSD)
* Benchmark: sysbench oltp ( read-only and read-write modes), 80mln rows (~18GB of data)

And I am comparing results with data stored:

• Directly on RAID
• Directly on SSD
• Data stored on RAID, but caching in FlashCache ( which located on SSD)

The results for Read-Only case:

As expected with FlashCache the results are very close to running directly on SSD, with small drop, which could be related to additional driver layer.

Read-Write case is more interesting. There I should mention that FlashCache uses WriteBack caching algorithm, which keeps some amount of dirty pages before physically writing to main storage. FlashCache allows you to set target percentage of dirty pages, so it was interesting to see how it affects ( expected that smaller amount of dirty pages puts more IO on main storage leading to smaller performance).

So there results for FlashCache with 20% and 80% of dirty pages and FlashCache based on Intel X25-M:

So it is pretty much impressed with 20% dirty pages we still have decent improvement.

And there are results for Intel X25-E:

By some reason the results much worse if compare to X25-M, probably X25-M in second generation has better performance characteristics ( I will post sysbench fileio benchmarks). Also what is attractive in X25-M, that even in biggest capacity 160GB you can get it by quite acceptable price.

In general FlashCache leaves pretty good impression, I did not have any significant problems ( not speaking about crashes or data loss), probably, the most challenge is to get a binary driver for your kernel . I will do more benchmarks especially when data exceeds memory, and also it will be interesting to see how it works when we put FlashCache on FusionIO cards instead of Intel SSD.

Entry posted by Vadim | No comment

Add to: | | | |

You can find this talk and other mostly technical material at http://tokutek.com/technology/.

This research was funded in part by the National Science Foundation.

Ooops.

Um, yeah.  Sorry about that.  Here’s a link to ‘The SmugMug Tale’ slides, and you can watch the video below:

Sorry for the extreme lag.  I suck.

The important highlights go something like this:

• Use transactional replication.  Without it, you’re dead in the water. You have no idea where a crashed slave was.
• Use a filesystem that lets you do snapshots.  Easily the best way to do backups, spin up new slaves, etc. I love ZFS.  You’ll need transactional replication to really make this painless.
• Use SSDs if you can. We can’t afford to be fully deployed on SSDs (terabytes are expensive), but putting them in the write path to lower latency is awesome.  The read path might help, too, depending on how much caching you’re already doing.  Love hybrid storage pools.
• Use Fishworks (aka Open Storage) if you can.  The analytics are unbeatable, plus you get SSDs, snapshots, ZFS, and tons of other goodies.
• Use transactional replication. This is so important I’m repeating it.  Patch it into MySQL (Google, Facebook, and Percona have patches) or use XtraDB if you use replication.  We use the Percona patch.

Holler in the comments if something in the presentation isn’t clear, I’ll answer.  Apologies again.

Shameless plug - we’re hiring. And it’s a blast.

Here’s the benchmark I came up with to confirm if this is the case:

• Percona-XtraDB-9.1 release
• Sysbench OLTP workload with 80 million rows (about 18GB worth of data+indexes)
• XFS Filesystem mounted with nobarrier option.
• Tests run with:
  • RAID10 with BBU over 8 disks
  • Intel SSD X25-E 32GB
  • FusionIO 320GB MLC
• For each test, run with a buffer pool of between 2G and 22G (to test performance compared to memory fit).
• Hardware was our Dell 900 (specs here).

To start with, we have a test on the RAID10 storage to establish a baseline.  The Y axis is transactions/second (more is better), the X axis is the size of innodb_buffer_pool_size:

Let me point out three interesting characteristics about this benchmark:

• The A arrow is when data fits completely in the buffer pool (best performance). It’s important to point out that once you hit this point, a further increase in memory at all.
• The B arrow is where the data just started to exceed the size of the buffer pool.  This is the most painful point for many customers – because while memory decreased by only ~10% the performance dropped by 2.6 times!  In production this usually matches the description of “Last week everything was fine.. but it’s just getting slower and slower!”.  I would suggest that adding memory is by far the best thing to do here.
• The C arrow shows where data is approximately three times the buffer pool.  This is an interesting point to zoom in on – since you may not be able to justify the cost of the memory, but an SSD might be a good fit:

Where the C arrow was, in this graph a Fusion-IO card improves performance by about five times (or 2x with an Intel SSD).  To get the same improvement with memory, you would have needed to add 60% more memory -or- 260% more memory for a 5x improvement.  Imagine a situation where your C point is when you have 32GB of RAM and 100GB of data.  Than it gets interesting:

• Can you easily add another 32G RAM (are your memory slots already filled?)
• Does your budget allow to install SSD cards? (You may still need more than one, since they are all relatively small.  There are already appliances on the market which use 8 Intel SSD devices).
• Is a 2x or 5x improvement enough?  There are more wins to be had if you can afford to buy all the memory that is required.

The workload here is designed to keep as much of the data hot as possible, but I guess the main lesson here is not to underestimate the size of your “active set” of data.  For some people who just append data to some sort of logging table it may only need to be a small percentage – but in other cases it can be considerably higher.  If you don’t know what your working set is – ask us!

Important note: This graph and these results are valid only for sysbench uniform. In your particular workload the points B and C may be located in differently.

Raw results:

Entry posted by Vadim | No comment

Add to: | | | |

To register, just click on one of the following links:

• Wednesday October 21st 2009: ZFS - Changing the Way You Look at Storage: ZFS is a new kind of file storage subsystem that provides simple administration, transactional semantics, end-to-end data integrity, and immense scalability. ZFS is not an incremental improvement to existing technology; it is a fundamentally new approach to data management. ZFS also combines, Disks, DRAM and Flash into one storage subsystem easily providing the best overall price performance for storage. Come learn about ZFS and its benefits. See a demo of how easy it is to use and administrate. Hear from a startup on their experience with ZFS.
  

• October 14th 2009: Storage for your database: how databases can take advantage of ZFS, SSDs and the Open Storage line of products from Sun to increase data security, scalability, and reduce the price/performance ratio.

  Note: Everybody can join. Useful for ISVs and startups using a database in their application.
  

• September 23rd 2009 :JavaFX for all your screens and Java Store for your distribution

Note: Everybody can join. Especially useful for ISVs and startups.

• September 9th 2009 : Mobile application design
  

Note : Everybody can join. Especially useful for ISVs and startups.

Has anyone done any more work on recompiling InnoDB with 4k pages and benchmarking under SSD?

We’re building out a new DB that uses very small records (around 32-64 bytes) so reading a whole 16k for this record should have a performance difference.

I haven’t seen any benchmarks on 16k random read IOPS on the Intel SSD but my hunch is that there will be a 20-30% penalty here.

Though even if it was a 4x penalty that would still be about 9k transactions per second which is pretty good.

On a personal note I just bought a new Mac Book Pro which will be upgraded to the Intel X-25M MLC SSD.

Needless to say I’m very excited!