PlanetMysql.ru — информация о СУБД MySQL

The Oracle Linux and Virtualization Documentation Team is seeking an experienced Technical Writer
with a focus on writing documentation for the Oracle Linux product. (The MySQL Documentation Team is part of that group as well.)

Applicants should be located in either Ireland, the UK, Sweden, Norway, Denmark, or Finland (click on the links for a detailed job description).

We're a vastly distributed team, with writers in Australia, North America, and Europe. Our infrastructure is based on DocBook XML, and we're not just writing docs, but also maintain the whole processing and publication work chain.

Key competencies you should have include:

• 3 or more years previous experience in writing software documentation (please provide URLs of your writings I can look at!)
  
• Experience with writing documentation for system level software and operating systems
• Strong knowledge of the Linux operating system
• Strong knowledge of XML, DocBook XML, and XSL style sheets (and motivation to help maintain and expand our tools and infrastructure)
  
• Ability to administer own workstation and test environment
• Good experience with distributed working environments and versioning systems such as SVN

If this sounds like something for you, follow the links above and send in your application!

PlanetMySQL Voting: Vote UP / Vote DOWN

Topics for this podcast:

*Hadoop v1.0 and year ahead
*Oracle-Cloudera deal for more Hadoop
*Oracle’s ‘Sun spot’ with Solaris
*Open Source M&A outlook for 2012
*Our new MySQL/NoSQL/NewSQL survey

iTunes or direct download (28:49, 4.9MB)

PlanetMySQL Voting: Vote UP / Vote DOWN

How To Set Up Multi-Master Replication Using Tungsten And MySQL-Proxy For MySQL High Availability On Ubuntu 10.04.3 LTS

This tutorial is based on my experience setting up Tungsten Replicator and MySQL-Proxy for a client's production setup.

PlanetMySQL Voting: Vote UP / Vote DOWN

If you’ve been tuning your MySQL database and have wondered what effect the innodb_flush_method settings have on write performance, then this information might help. I’ve recently been doing a lot of baseline load tests to show performance differences between localdisk and the new SAN we’re deploying. Since we run InnoDB for everything in production, and writes are very heavy, I decided to run comparison tests between two identical servers to find the best setting for innodb_flush_method. We have the following specs for the hardware:

• Dell R610
• 24 core Intel Xeon X5670 @ 2.93ghz
• 72GB ECC RAM
• Brocade 825 HBA
• Local disk: RAID-10 15K SAS Ext3 (ugh)
• SAN: Oracle 7420 with four Intel Xeon X7550 @ 2.00GHz, 512GB RAM, 2TB read-cache(SLC-SSD), 36GB write cache (MLC-SSD), 3 disk shelves populated with 60x2TB 7200RM SATA drives setup in mirrored format with striped logs, dual 8Gb FC links to redundant fabric, connected to Brocade DCX 8510-4.
• The my.cnf file being used for the tests: click-click

I’m using the following sysbench command to run the tests. On each server the same commands are used. I ran a 1B row prepare prior to the 1B row test.

sysbench –db-driver=mysql –num-threads=64 –max-requests=1000000000 –max-time=3600 –test=oltp –verbosity=3 –validate=off –oltp-test-mode=complex –oltp-read-only=off –oltp-table-name=sbtest –oltp-table-size=1000000000 –oltp-dist-type=special –mysql-host=localhost –mysql-port=3306  –mysql-table-engine=innodb run

On the server that is utilizing SAN paths there are two LUNS presented for MySQL use. /db/data01 for InnoDB data files, /db/logs01 for InnoDB logs. These filesystems are both formatted as XFS. The server running local-disk tests is running Ext3. I might run some more tests later with the local-disk setup as XFS if time allows.

Here are the results. Clearly a well designed SAN infrastructure is superior to even RAID-10 15K SAS drives. And of course you can see the different performance values from using O_DIRECT for the innodb_flush_method for the different data storage mediums.

1B Row Complex Transactional Test, 64 threads

• SAN O_DIRECT: read/write requests: 31560140 (8766.61 per sec.)
• SAN O_DSYNC: read/write requests: 5179457 (1438.52 per sec.)
• SAN fdatasync: read/write requests: 9445774 (2623.66 per sec.)
• Local-disk O_DIRECT: read/write requests: 3258595 (905.06 per sec.)
• Local-disk O_DSYNC: read/write requests: 3494632 (970.65 per sec.)
• Local-disk fdatasync: read/write requests: 4223757 (1173.04 per sec.)

PlanetMySQL Voting: Vote UP / Vote DOWN

If you’ve been tuning your MySQL database and have wondered what effect the innodb_flush_method settings have on write performance, then this information might help. I’ve recently been doing a lot of baseline load tests to show performance differences between localdisk and the new SAN we’re deploying. Since we run InnoDB for everything in production, and writes are very heavy, I decided to run comparison tests between two identical servers to find the best setting for innodb_flush_method. We have the following specs for the hardware:

• Dell R610
• 24 core Intel Xeon X5670 @ 2.93ghz
• 72GB ECC RAM
• Brocade 825 HBA
• Local disk: RAID-10 15K SAS Ext3 (ugh)
• SAN: Oracle 7420 with four Intel Xeon X7550 @ 2.00GHz, 512GB RAM, 2TB read-cache(SLC-SSD), 36GB write cache (MLC-SSD), 3 disk shelves populated with 60x2TB 7200RM SATA drives setup in mirrored format with striped logs, dual 8Gb FC links to redundant fabric, connected to Brocade DCX 8510-4.
• The my.cnf file being used for the tests: click-click

I’m using the following sysbench command to run the tests. On each server the same commands are used. I ran a 1B row prepare prior to the 1B row test.

sysbench –db-driver=mysql –num-threads=64 –max-requests=1000000000 –max-time=3600 –test=oltp –verbosity=3 –validate=off –oltp-test-mode=complex –oltp-read-only=off –oltp-table-name=sbtest –oltp-table-size=1000000000 –oltp-dist-type=special –mysql-host=localhost –mysql-port=3306  –mysql-table-engine=innodb run

On the server that is utilizing SAN paths there are two LUNS presented for MySQL use. /db/data01 for InnoDB data files, /db/logs01 for InnoDB logs. These filesystems are both formatted as XFS. The server running local-disk tests is running Ext3. I might run some more tests later with the local-disk setup as XFS if time allows.

Here are the results. Clearly a well designed SAN infrastructure is superior to even RAID-10 15K SAS drives. And of course you can see the different performance values from using O_DIRECT for the innodb_flush_method for the different data storage mediums.

1B Row Complex Transactional Test, 64 threads

• SAN O_DIRECT: read/write requests: 31560140 (8766.61 per sec.)
• SAN O_DSYNC: read/write requests: 5179457 (1438.52 per sec.)
• SAN fdatasync: read/write requests: 9445774 (2623.66 per sec.)
• Local-disk O_DIRECT: read/write requests: 3258595 (905.06 per sec.)
• Local-disk O_DSYNC: read/write requests: 3494632 (970.65 per sec.)
• Local-disk fdatasync: read/write requests: 4223757 (1173.04 per sec.)

PlanetMySQL Voting: Vote UP / Vote DOWN

It was a long time now that I wanted to run some benchmark tests to understand better the surprises I've met in the past with Linux I/O performance during MySQL benchmarks, and finally it happened last year, but I was able to organize and present my results only now..

My main questions were:

• what is so different with various I/O schedulers in Linux (cfq, noop, deadline) ?..
• what is wrong or right with O_DIRECT on Linux ?..
• what is making XFS more attractive comparing to EXT3/EXT4 ?..

There were already several posts in the past about impact on MySQL performance when one or another Linux I/O layer feature was used (for ex. Domas about I/O schedulers, Vadim regarding TPCC-like performance, and many other) - but I still did not find any answer WHY (for ex.) cfq I/O scheduler is worse than noop, etc, etc..

So, I'd like to share here some answers to my WHY questions ;-))
(while for today I still have more questions than answers ;-))


Test Platform

First of all, the system I've used for my tests:

• HW server: 64 cores (Intel), 128GB RAM, running RHEL 5.5
• the kernel is 2.6.18 - as it was until now the most common Linux kernel used on Linux boxes hosting MySQL servers
• installed filesystems: ext3, ext4, XFS
• Storage: ST6140 (1TB on x16 HDD striped in RAID0, 4GB cache on controller) - not a monster, but fast enough to see if the bottleneck is coming from the storage level or not ;-))

Test Plan

Then, my initial test plan:

• see what the max possible Read/Write I/O performance I can obtain from the given HW on the raw level (just RAW-devices, without any filesystem, etc.) - mainly I'm interested here on the impact of Linux I/O scheduler
• then, based on observed results, setup more optimally each filesystem (ext3, ext4, XFS) and try to understand their bottlenecks..
• I/O workload: I'm mainly focusing here on the random reads and random writes - they are the most problematic for any I/O related performance (and particularly painful for databases), while sequential read/writes may be very well optimized on the HW level already and hide any other problems you have..
• Test Tool: I'm using here my IObench tool (at least I know exactly what it's doing ;-))

TESTING RAW DEVICES

Implementation of raw devices in Linux is quite surprising.. - it's simply involving O_DIRECT access to a block device. So to use a disk in raw mode you have to open() it with O_DIRECT option (or use "raw" command which will create an alias device in your system which will always use O_DIRECT flag on any involved open() system call). Using O_DIRECT flag on a file opening is disabling any I/O buffering on such a file (or device, as device is also a file in UNIX ;-) - NOTE: by default all I/O requests on block devices (e.g. hard disk) in Linux are buffered, so if you'll start a kind of I/O write test on, say, your /dev/sda1 - you'll obtain a kind of incredible great performance ;-)) as no data probably will not yet even reach your storage and in reality you'll simply test a speed of your RAM.. ;-))

Now, what is "fun" with O_DIRECT:

• all your I/O requests (read, write, etc.) block size should be aligned to 512 bytes (e.g. be multiplier of 512 bytes), otherwise your I/O request is simply rejected and you get an error message.. - and regarding to RAW devices it's quite surprising comparing to Solaris for ex. where you're simply instead of /dev/dsk/device using /dev/rdsk/device and may use any block size you want..
• but it's not all.. - the buffer you're using within your system call involving I/O request should also be allocated aligned to 512 bytes, so mainly you have to allocate it via posix_memalign() function, otherwise you'll also get an error.. (seems that during O_DIRECT operations there is used some kind of direct memory mapping)
• then, reading the manual: "The O_DIRECT flag on its own makes at an effort to transfer data synchronously, but does not give the guarantees of the O_SYNC that data and necessary metadata are transferred. To guarantee synchronous I/O the O_SYNC must be used in addition to O_DIRECT" - quite surprising again..
• and, finally, you'll be unable to use O_DIRECT within your C code until you did not declare #define _GNU_SOURCE

Interesting that the man page is also quoting Linus about O_DIRECT:
"The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances." Linus

But we have to live with it ;-))

And if you need an example of C or C++ code, instead to show you the mine, there is a great dev page on Fusion-io site.

So far, what about my storage performance on the RAW devices now?..

Test scenario on RAW devices:

• I/O Schedulers: cfq, noop, deadline
• Block size: 1K, 4K, 16K
• Workload: Random Read, Random Write

NOTE: I'm using here 1K block size as the smallest "useful" size for databases :-)) then 4K as the most aligned to the Linux page size (4K), and 16K - as the default InnoDB block size until now.

Following graphs are representing 9 tests executed one after one: cfq with 3 different block sizes (1K, 4K, 16K), then noop, then deadline. Each test is running a growing workload of 1, 4, 16, 64 concurrent users (processes) non-stop bombarding my storage subsystem with I/O requests.


Read-Only@RAW-device:

Observations :

• Random Read is scaling well for all Linux I/O Schedulers
• Reads reported by application (IObench) are matching numbers reported by the system I/O stats
• 1K reads are running slightly faster than 4K (as expected as it's a "normal" disks, and transfer of a bigger data volume reducing an overall performance, which is normal)..

Write-Only @RAW-device:

Observations :

• looking on the graph you may easily understand now what is wrong with "cfq" I/O scheduler.. - it's serializing write operations!
• while "noop" and "deadline" are continuing to scale with a growing workload..
• so, it's clear now WHY performance gains were observed by many people on MySQL workloads by simply switching from "cfq" to "noop" or "deadline"

To check which I/O scheduler is used for your storage device:
# cat /sys/block/{DEVICE-NAME}/queue/scheduler

For ex. for "sda": # cat /sys/block/sda/queue/scheduler

Then set "deadline" for "sda": # echo deadline > /sys/block/sda/queue/scheduler
To set "deadline" as default I/ scheduler for all your storage devices you may boot your system with "elevator=deadline" boot option. Interesting that by default many Linux systems used "cfq". All recent Oracle Linux systems are shipped with "deadline" by default.



TESTING FILESYSTEMS

As you understand, there is no more reasons to continue any further tests by using "cfq" I/O scheduler.. - if on the raw level it's already bad, it cannot be better due any filesystem features ;-)) (While I was also told that in recent Linux kernels "cfq" I/O scheduler should perform much more better, let's see)..

Anyway, my filesystem test scenario:

• Linux I/O Scheduler: deadline
• Filesystems: ext3, ext4, XFS
• File flags/options: osync (O_SYNC), direct (O_DIRECT), fsync (fsync() is involved after each write()), fdatasync (same as fsync, but calling fdatasync() instead of fsync())
• Block size: 1k, 4K, 16K
• Workloads: Random Reads, Random Writes on a single 128GB file - it's the most critical file access for any database (having a hot table, or a hot tablespace)
• NOTE: to avoid most of background effects of caching, I've limited an available RAM for FS cache to 8GB only! (all other RAM was allocated to the huge SHM segment with huge pages, so not swappable)..

Also, we have to keep in mind now the highest I/O levels observed on RAW devices:

• Random Read: ~4500 op/sec
• Random Write: ~5000 op/sec

So, if for any reason Read or Write performance will be faster on any of filesystems - it'll be clear there is some buffering/caching happening on the SW level ;-))

Now, let me explain what you'll see on the following graphs:

• they are already too many, so I've tried to bring more data on each graph :-))
• there are 12 tests on each graph (x3 series of x4 tests)
• each serie of tests is executed by using the same block size (1K, then 4K, then 16K)
• within a serie of 4 tests there are 4 flags/options are used one after one (osync, direct, fsync, fdatasync)
• each test is executed as before with 1, 4, 16, 64 concurrent user processes (IObench)
• only one filesystem per graph :-))

So, let's start now with Read-Only results.


Read-Only @EXT3:

Observations :

• pretty well scaling, reaching 4500 reads/sec in max
• on 1K reads: only "direct" reads are really reading 1K blocks, all other options are involving reading of 4K blocks
• nothing unexpected finally :-)

Read-Only @EXT4:

Observations :

• same as on ext3, nothing unexpected

Read-Only @XFS:

Observations :

• no surprise here either..
• but there were one surprise anyway ;-))

While the results on Random Read workloads are looking exactly the same on all 3 filesystems, there are still some difference in how the O_DIRECT feature is implemented on them! ;-))

The following graphs are representing the same tests, but only corresponding to execution with O_DIRECT flag (direct). First 3 tests are with EXT3, then 3 with XFS, then 3 with EXT4:


Direct I/O & Direct I/O

Observations :

• the most important here the last graph showing here the memory usage on the system during O_DIRECT tests
• as you may see, only with XFS the filesystem cache usage is near zero!
• while EXT3 and EXT4 are still continuing cache buffering.. - may be a very painful surprise when you're expecting to use this RAM for something else ;-))

Well, let's see now what is different on the Write Performance.


Write-Only @EXT3:

Observations :

• the most worse performance here is with 1K blocks.. - as default EXT3 block size is 4K, on the 1K writes it involves a read-on-write (it has to read 4K block first, then change corresponding 1K on changes within it, and then write the 4K block back with applied changes..)
• read-on-write is not happening on 1K when O_DIRECT flag is used: we're really writing 1K here
• however, O_DIRECT writes are not scaling at all on EXT3! - and it explains me finally WHY I've always got a worse performance when tried to use O_DIRECT flush option in InnoDB on EXT3 filesystem! ;-))
• interesting that the highest performance here is obtained with O_SYNC flag, and we're not far from 5000 writes/sec for what the storage is capable..

Write-Only @EXT4:

Observations :

• similar to EXT3, but performance is worse comparing to EXT3
• interesting that only with O_SYNC flag the performance is comparable with EXT3, while in all other cases it's simply worse..
• I may suppose here that EXT3 is not flushing on every fsync() or fdatasync(), and that's why it's performing better with these options ;-)) need to investigate here.. But anyway, the result is the result..

What about XFS?..


Write-Only @XFS:

Observations :

• XFS results are quite different from those of EXT3 and EXT4
• I've used a default setup of XFS here, and was curios to not observe the impact of missed "nobarrier" option which was reported by Vadim in the past..
• on 1K block writes only O_DIRECT is working well, but in difference from EXT3/EXT4 it's also scaling ;-) (other options are giving poor results due the same read-on-write issue..)
• 4K block writes are scaling well with O_SYNC and O_DIRECT, but still remaining poor with other options
• 16K writes are reporting some anomalies: while with O_SYNC nothing is going wrong and it's scaling well, with O_DIRECT there is some kind of serialization happened on 4 and 16 concurrent user processes.. - and then on 64 users things then came back to the normal.. Interesting that is was not observed with 4K block writes.. Which remains me the last year discussion about page block size in InnoDB for SSD, and the gain reported by using 4K page size vs 16K.. - just keep in mind that sometimes it may be not related to SSD at all, but just to some filesystem's internals ;-))
• anyway, no doubt - if you have to use O_DIRECT in your MySQL server - use XFS! :-)

Now, what is the difference between a "default" XFS configuration and "tuned" ??..

I've recreated XFS with 64MB log size and mounted with following options:
# mount -t xfs -o noatime,nodiratime,nobarrier,logbufs=8,logbsize=32k

The results are following..


Write-Only @XFS-tuned:

Observations :

• everything is similar to "default" config, except that there is no more problem with 16K block size performance
• and probably this 16K anomaly observed before is something random, hard to say.. - but at least I saw it, so cannot ignore ;-))

Then, keeping in mind that XFS is so well performing on 1K block size, I was curious to see if thing will not go even better if I'll create my XFS filesystem with 1K block size instead of default 4K..


Write-Only @XFS-1K:

Observations :

• when XFS is created with 1K block size there is no more read-on-write issue on 1K writes..
• and we're really writing 1K..
• however, the performance is completely poor.. even on 1K writes with O_DIRECT !!!
• why?..

The answer is came from the Random Reads test on the same XFS, created with 1K block size.


Read-Only @XFS-1K:



Observations :

• if you followed me until now, you'll understand everything from the last graph, reporting RAM usage.. ;-))
• the previously 8GB free RAM is no more free here..
• so, XFS is not using O_DIRECT here!
• and you may see also that for all reads except O_DIRECT, it's reading 4K for every 1K, which is abnormal..

Instead of SUMMARY

• I'd say the main point here is - "test your I/O subsystem performance before to deploy your MySQL server" ;-))
• avoid to use "cfq" I/O scheduler :-)
• if you've decided to use O_DIRECT flush method in your MySQL server - deploy your data on XFS..
• seems to me the main reason why people are using O_DIRECT with MySQL it's a willing to avoid to deal with various issues of filesystem cache.. - and there is probably something needs to be improved in the Linux kernel, no? ;-)
• could be very interesting to see similar test results on the other filesystems too..
• things may look better with a newer Linux kernel..

So far, I've got some answers to my WHY questions.. Will be fine now to get a time to test it directly with MySQL ;-)

Any comments are welcome!

Rgds,
-Dimitri

PlanetMySQL Voting: Vote UP / Vote DOWN

Previously, I’ve called out years for non-desktop Linux in 2008, Linux in both the low and high-ends of the market in 2009, ‘hidden’ Linux in 2010 and last year, cloud computing in 2011. For 2012, I see continued growth, prevalence, innovation and impact from Linux, thus leading to a 2012 that is dominated by Linux.

I expect to see nothing but continued strength for Linux and open source in cloud computing in 2012. The cloud continues to be the biggest disruptor and opportunity for Linux providers. 2012 got off to an interesting start with Microsoft’s efforts to support for Linux on Azure, which highlights just how pervasive Linux has become in cloud computing. As detail in our special report on The Changing Linux Landscape, we also expect Linux to continue to be the basis for most offerings in IaaS and particularly PaaS, which is burgeoning across open source languages and frameworks as well as verticals and enterprise customers. Its popularity among enterprise and other developers will also bolster Linux and open source software in 2012.

We can certainly expect to see Linux continue its domination in supercomputing and the Top 500 Supercomputer List, where Linux continues to grow its share above 90% while others, such as Microsoft, Apple and BSD, fall off of the list.

I also expect Linux will grow its presence and impact on the wider, more mainstream server market, where Red Hat and SUSE continue to benefit from Unix migration, particularly from Solaris. Our analysis with survey data from 451 Research division TheInfoPro shows server spending for databases and data warehousing favoring Red Hat with Linux over Oracle with either Linux or Solaris. Out of more than 165 server professionals interviewed by TIP, 67% are planning to spend more with Red Hat on database/data-warehousing, and only 6% plan to spend less. The positive figures for Red Hat mirror negative spending intentions for Oracle, with 55% planning to spend less and only 9% planning to spend more. Spending continues to decline strongly for all of the primary Unix providers in the study, which in addition to Oracle includes IBM and Hewlett-Packard.

We may also see further expansion for Red Hat, which may be eyeing key acquisitions, and other Linux and open source vendors as they continue building their channels and wade more into midmarket and SMB customers.

In smartphones and mobile software, I also expect Linux will do quite well in 2012 with continued Android strength, diminished FUD and possibly an open source boost from a newly-open sourced WebOS. We also see Ubuntu arriving on the mobile and converged device scene, including ‘concept’ appearance at CES.

We’re also likely to see Linux in automobiles, health care and other electronics even more in 2012, though you may never hear Linux or open source. Don’t be fooled though, Linux is expanding its already impressive, wide presence and 2012 looks to be another year of significant gains.

PlanetMySQL Voting: Vote UP / Vote DOWN

The reason we are confident that the comparative decline in the use of the GNU GPL family of licenses and the increasing significance of complementary vendors in relation to funding for open source software-related vendors will continue is due to the analysis of our database of more than 400 open source software-related vendors, past and present.

We previously used the database to analyze the engagement of vendors with open source projects for our Control and Community report, plotting the strategies used by the vendors against the year in which they first began to engage with open source projects to get an approximate view of open source-related strategy changes over time.

For example, we found that the engagement of vendors with projects that used strong copyleft licenses peaked in 2006, while the engagement of vendors with projects using non-copyleft licenses had been rising steadily since 2002.

Analysis of our updated database shows that the the number of new vendors engaging with open source projects in each year has risen steadily in recent years, from 26 in 2008 to 44 in 2011. However, as noted last week, we have also seen a shift towards ‘complementary vendors’ – those that are dependent on open source software to build their products and services, even though those products and services may not themselves be open source.

2010 was the first year in which we saw more complementary vendors engage with open source projects than open source specialist, and that trend accelerated in 2011.

As previously explained, complementary vendors were responsible for over 30% of open source software-related funding raised in 2011, and we should expect that proportion to remain high given that over 57% of the vendors engaging with open source in 2011 were complementary vendors.

We have also seen that complementary vendors are more likely to engage with projects with non-copyleft licenses (38% of complementary vendors have engaged with projects with non-copyleft licenses, compared to 24% that have engaged with projects with strong copyleft licenses).

If we look at all 400+ vendors in our database in terms of open source software license preference, the trend towards new vendors engaging with non-copyleft licenses is clear.

There has been a strong shift from vendors towards non-copyleft licenses in recent years, accelerated in 2011 by the likes of Apache Hadoop and OpenStack in particular. This does not mean that the number of projects using strong copyleft licensing has decreased (although as we previously saw the proportion of projects using the GPL family of licenses has declined).

It is indicative, we believe, of the shift away from specialist open source vendors using vendor-led projects and strong copyleft licenses towards multi-vendor collaborative projects and proprietary implementations of open source code, however.

This trend should not really surprise anyone. For some time we have seen open source becoming part of the fabric of modern software development and licensing strategies, rather than a competitive differentiator. Back in 2009 we predicted the increased importance of business strategies that relied on vendor-led development communities, rather than projects dominated by a single vendor.

We called this “open source 4.0″ and later suggested that it might be considered the golden age of open source, based on our belief that vendors had learned that they stand to gain more from collaborating on open source projects and differentiating at another stage in the software stack than they do from attempting to control open source projects.

Updating the results of our analysis to the end of 2011 and 400+ vendors indicates that, from the perspective of the commercial adoption of open source business strategies at least, we were not far off.

Some might not consider the proliferation of multi-vendor open source communities and proprietary distributions of open source software as the peak of achievement for open source. Each is of course entitled to come to their own conclusions about the implications.

Our perspective, as always, is that open source methodologies present a potentially disruptive, and also valuable, asset that complements the way both vendors and enterprise IT organizations conduct their businesses.

Our analysis indicates, however, that open source methodologies are increasingly being employed by ‘complementary vendors’ with a leaning towards more permissive licensing.

PlanetMySQL Voting: Vote UP / Vote DOWN

One of the favourite blog topics on CAOS Theory blog over the years has been our quarterly and annual updates on venture capital funding for open source-related businesses, based on our database of over 600 funding deals since January 1997 involving nearly 250 companies, and over $4.8bn.

There are still a few days left for funding deals to be announced in 2011 but it is already clear that 2011 will be a record year. $672.8m has been invested in open source-related vendors in 2011, according to our preliminary figures, an increase of over 48% on 2010, and the highest total amount invested in any year, beating the previous best of $623.6m, raised in 2006.

Following the largest single quarter for funding for open source-related vendors ever in Q3, Q4 was the second largest single quarter for funding for open source-related vendors ever, as $230.4m was invested in companies including Cloudera, Hortonworks, and Rapid7.

As with Q3, however, the list of vendors presents us with something of an existential dilemma, as we see an increasing amount of activity by what we have referred to as ‘complementary vendors’ – those that are dependent on open source software to build their products and services, even though those products and services may not themselves be open source – as opposed to open source specialists.

The list of complementary vendors has grown rapidly in 2011, particularly around projects such as OpenStack and Apache Hadoop. If we examine the figures in more detail we find that over 30% of the funding raised in 2011 was raised by complementary vendors, compared to just 4% in 2006.

In fact, as the chart below indicates, VC funding for specialist open source vendors in 2011 was actually less than that in 2006 and 2008, and only marginally up on 2010, when again just 4% of funding went to complementary vendors.

The low amount of funding for complementary vendors in 2010 shows that the significance of complementary vendors is not growing at a constant rate, although for reasons that will become clear when we publish a follow-up post on the latest trends regarding the engagement of vendors with open source projects, we do expect that the proportion of funding related to complementary vendors is more likely to increase in the future, rather than decline.

This has implications for the ongoing trends related to open source software licensing, as covered yesterday. Examining our database of over 400 open source-related vendors – funded and unfunded, complementary and specialist – indicates that specialist vendors are much more likely to engage with projects using strong copyleft licenses than complementary vendors.

Specifically, our data indicates that 55% of open source specialists have engaged with projects that use strong copyleft licenses, while just 20% have engaged with projects with non-copyleft licenses. In comparison, 38% of complementary vendors have engaged with projects with non-copyleft licenses, compared to 24% that have engaged with projects with strong copyleft licenses.

Will will take a more detailed look at the trends related to the engagement of vendors with open source projects in the concluding part of this series of posts.

PlanetMySQL Voting: Vote UP / Vote DOWN

SCaLE 10X is the premier Open Source show in Southern California. This year it will feature a MySQL day with top speakers from the MySQL Community. Volunteers run SCaLE and do such an amazing job that they charge only $60 for a full pass. If you are anywhere near Southern California January 20th to the 22nd, you need to be at SCaLE.

SCaLE MySQL Day (Friday) Presentations
• MySQL Replication (Keith Larson)
• The MySQL Storage Engines Landscape (Colin Charles)
• A Closer Look at MySQL Cluster: An Architectural Overview (Max Mether)
• Playing in the Same Sandbox: MySQL and Oracle (Lynn Ferrante)
• Care and Feeding of a MySQL Database for Linux Administrators (Dave Stokes)
• MySQL Update (Dave Stokes)

Solomon Chang is going to present a four hour MySQl Replication Tutorial (but I am unsure of the date) that will be great.

Hope to see you there!

PlanetMySQL Voting: Vote UP / Vote DOWN