PlanetMysql.ru - информация о СУБД MySQL

PlanetMysql.ru - информация о СУБД MySQL » Spider http://planetmysql.ru Блог о самой популярной СУБД MySQL Thu, 24 May 2012 22:24:00 +0000 en hourly 1 http://wordpress.org/?v=3.3 [MySQL][Spider][VP]Spider-2.28 VP-0.17 released http://wild-growth.blogspot.com/2012/02/mysqlspidervpspider-228-vp-017-released.html?utm_source=rss&utm_medium=rss&utm_campaign=mysqlspidervpspider-2-28-vp-0-17-released http://wild-growth.blogspot.com/2012/02/mysqlspidervpspider-228-vp-017-released.html#comments Thu, 09 Feb 2012 16:43:00 +0000 Kentoku SHIBA http://planetmysql.ru/?guid=511bf2c97de435da4b2bdb04e1c25e9b Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

Please use the following for downloading binary file.
http://spiderformysql.com/download_spider.html

The main changes in this version are following.
Spider
- Support parallel searching. ("spider_bgs_mode > 0")
The table using the Spider bundled MySQL and table partitioning performs parallel search of each partition. However, parallel search is not performed in the case which does not make all the partitions applicable to search like sequential search with limit.
- Add server parameter "spider_use_default_database" and "spider_remote_default_database".
These parameter is added for using some replication parameters like "binlog-do-db" on data node. "spider_use_default_database=0" is same of previous versions.
- Add table parameter "access_balance".
This parameter is the weight of load balancing for Spider's redundancy feature.
- Add INFORMATION SCHEMA "spider_alloc_mem".
This INFORMATION SCHEMA plugin adds "information_schema.spider_alloc_mem" table for showing Spider's using memory status. "alloc_mem_count" and "free_mem_count" columns in "information_schema.spider_alloc_mem" table express the count of increasing and decreasing memory, so they do not necessarily become the same.
- Add value of 3 to "quick_mode"(table parameter) and "spider_quick_mode"(server parameter).
"quick_mode=3" is the mode of using temporary table for result set from data node. This mode is useful for searching a huge table.

Spider's management table was changed from previous version and add new plugin, please execute "install_spider.sql" for upgrading.

Vertical Partitioning
- Add UDF paramter "suppress_autoinc".
- Add table parameter "allow_bulk_autoinc" and "allow_different_column_type".
- Add server parameter "vp_allow_bulk_autoinc".

handlersocket
Becomes version 1.1.
- Add server parameter "handlersocket_slow_log", "handlersocket_long_exec_time" and "handlersocket_close_table_interval".
"handlersocket_slow_log" is logging handlersocket request with slow (spend over "handlersocket_long_exec_time" micro second) response into slow log with "slow_query_log=on".
"handlersocket_close_table_interval" is interval of closing table internally for releasing meta data lock for executing like "alter table". You can use this parameter "handlersocket_close_table_interval=0" (doesn't close) for normally and only change this parameter for executing like "alter table".
These parameters are global parameter and you can change them from SQL interface.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Thanks to Takafumi, Yukihiro, Jung, Adrian, leechangyeol, liuyanhong, Hisazumi, Hideyuki, Keisuke for bug reporting.
Thanks to Akira for your advice.

Enjoy!

PlanetMySQL Voting: Vote UP / Vote DOWN]]> http://planetmysql.ru/2012/02/09/mysqlspidervpspider-2-28-vp-0-17-released/feed/ 0 [MySQL][Spider][VP]Spider-2.28 VP-0.17 released http://wild-growth.blogspot.com/2012/02/mysqlspidervpspider-228-vp-017-released.html?utm_source=rss&utm_medium=rss&utm_campaign=mysqlspidervpspider-2-28-vp-0-17-released http://wild-growth.blogspot.com/2012/02/mysqlspidervpspider-228-vp-017-released.html#comments Thu, 09 Feb 2012 16:43:00 +0000 Kentoku SHIBA http://planetmysql.ru/?guid=511bf2c97de435da4b2bdb04e1c25e9b Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

Please use the following for downloading binary file.
http://spiderformysql.com/download_spider.html

The main changes in this version are following.
Spider
- Support parallel searching. ("spider_bgs_mode > 0")
The table using the Spider bundled MySQL and table partitioning performs parallel search of each partition. However, parallel search is not performed in the case which does not make all the partitions applicable to search like sequential search with limit.
- Add server parameter "spider_use_default_database" and "spider_remote_default_database".
These parameter is added for using some replication parameters like "binlog-do-db" on data node. "spider_use_default_database=0" is same of previous versions.
- Add table parameter "access_balance".
This parameter is the weight of load balancing for Spider's redundancy feature.
- Add INFORMATION SCHEMA "spider_alloc_mem".
This INFORMATION SCHEMA plugin adds "information_schema.spider_alloc_mem" table for showing Spider's using memory status. "alloc_mem_count" and "free_mem_count" columns in "information_schema.spider_alloc_mem" table express the count of increasing and decreasing memory, so they do not necessarily become the same.
- Add value of 3 to "quick_mode"(table parameter) and "spider_quick_mode"(server parameter).
"quick_mode=3" is the mode of using temporary table for result set from data node. This mode is useful for searching a huge table.

Spider's management table was changed from previous version and add new plugin, please execute "install_spider.sql" for upgrading.

Vertical Partitioning
- Add UDF paramter "suppress_autoinc".
- Add table parameter "allow_bulk_autoinc" and "allow_different_column_type".
- Add server parameter "vp_allow_bulk_autoinc".

handlersocket
Becomes version 1.1.
- Add server parameter "handlersocket_slow_log", "handlersocket_long_exec_time" and "handlersocket_close_table_interval".
"handlersocket_slow_log" is logging handlersocket request with slow (spend over "handlersocket_long_exec_time" micro second) response into slow log with "slow_query_log=on".
"handlersocket_close_table_interval" is interval of closing table internally for releasing meta data lock for executing like "alter table". You can use this parameter "handlersocket_close_table_interval=0" (doesn't close) for normally and only change this parameter for executing like "alter table".
These parameters are global parameter and you can change them from SQL interface.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Thanks to Takafumi, Yukihiro, Jung, Adrian, leechangyeol, liuyanhong, Hisazumi, Hideyuki, Keisuke for bug reporting.
Thanks to Akira for your advice.

Enjoy!

PlanetMySQL Voting: Vote UP / Vote DOWN]]> http://planetmysql.ru/2012/02/09/mysqlspidervpspider-2-28-vp-0-17-released/feed/ 0 [MySQL][Spider][VP]Spider-2.27 VP-0.16 released http://wild-growth.blogspot.com/2011/10/mysqlspidervpspider-227-vp-016-released.html?utm_source=rss&utm_medium=rss&utm_campaign=mysqlspidervpspider-2-27-vp-0-16-released http://wild-growth.blogspot.com/2011/10/mysqlspidervpspider-227-vp-016-released.html#comments Wed, 19 Oct 2011 13:35:24 +0000 Kentoku SHIBA http://planetmysql.ru/?guid=ba66d9ba085a4ae9fdf659af95c63204 Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

Please use the following for downloading binary file.
http://spiderformysql.com/download_spider.html

The main changes in this version are following.
Q4M is bundled for Linux 64bit.
Bundled Q4M supports replication, but please use it carefully.

Spider
- Support R-Tree index.
- Support direct updating for SQL access.
- Support handlersocket increment and decrement.
- Change table parameter from "net_timeout" to "connect_timeout", "net_read_timeout" and "net_write_timeout".
- Change server parameter from "spider_net_timeout" to "spider_connect_timeout", "spider_net_read_timeout" and "spider_net_write_timeout".
- Add UDF paramter "access_mode". "spider_direct_sql" support handlersocket access.
- Add server parameter "spider_hs_ping_interval", "spider_error_read_mode" and "spider_error_write_mode".
- Add table parameter "hs_write_to_read", "error_read_mode" and "error_write_mode".
- Performance improvement for "COUNT(*)", "MAX" and "MIN" without clause.
Note
- Change parameter from "net_timeout" and "spider_net_timeout" to "connect_timeout", "net_read_timeout", "net_write_timeout", "spider_connect_timeout", "spider_net_read_timeout" and "spider_net_write_timeout" from this version.
- From this version, you can't compile with static link option for Spider.

Vertical Partitioning
- Support R-Tree index.
- Support direct updating.
- Support handlersocket increment and decrement.
Note
- From this version, you can't compile with static link option for VP.

handlersocket
- Add server parameter "handlersocket_general_log".
Logging handlersocket request into general log with "general_log=on". This parameter is global parameter and you can change it from SQL interface.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

PlanetMySQL Voting: Vote UP / Vote DOWN]]> http://planetmysql.ru/2011/10/19/mysqlspidervpspider-2-27-vp-0-16-released/feed/ 0 [MySQL][Spider][VP]Spider-2.23 VP-0.12 released http://wild-growth.blogspot.com/2010/10/mysqlspidervpspider-223-vp-012-released.html?utm_source=rss&utm_medium=rss&utm_campaign=mysqlspidervpspider-2-23-vp-0-12-released http://wild-growth.blogspot.com/2010/10/mysqlspidervpspider-223-vp-012-released.html#comments Sat, 16 Oct 2010 17:02:00 +0000 Kentoku SHIBA Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql

The main changes in this version are following.
Spider
- Add server parameter "spider_udf_ct_bulk_insert_interval" and "spider_udf_ct_bulk_insert_rows".

Vertical Partitioning
This release is bug fix release.

Please see "99_change_logs.txt" in the download documents for checking other changes.

Thanks to Gen, Yoshihiko, Satoko and Makoto for testing for clustering and online schema changing.
Thanks to Frederic for bug report.

Enjoy!

PlanetMySQL Voting: Vote UP / Vote DOWN]]> http://planetmysql.ru/2010/10/16/mysqlspidervpspider-2-23-vp-0-12-released/feed/ 0 [MySQL][Spider][VP][Other]Spider-2.22 VP-0.11 BKA-for-ha_partition-0.3 released http://wild-growth.blogspot.com/2010/08/mysqlspidervpotherspider-222-vp-011-bka.html?utm_source=rss&utm_medium=rss&utm_campaign=mysqlspidervpotherspider-2-22-vp-0-11-bka-for-ha_partition-0-3-released http://wild-growth.blogspot.com/2010/08/mysqlspidervpotherspider-222-vp-011-bka.html#comments Mon, 30 Aug 2010 16:35:00 +0000 Kentoku SHIBA Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql
BKA-for-ha_partition is a patch file of supporting "Batched Key Access" for table partitioning feature.
http://launchpad.net/partitionmrrformysql

The main changes in this version are following.
Spider
- MS Windows support.
- Add UDF "spider_copy_tables".
Spider HA feature is now available!

Vertical Partitioning
- MS Windows support.

Pathced MySQL source code and compiled MySQL binary are available from this release.

Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Toru for working for supporting Windows.

Enjoy!

PlanetMySQL Voting: Vote UP / Vote DOWN]]> http://planetmysql.ru/2010/08/30/mysqlspidervpotherspider-2-22-vp-0-11-bka-for-ha_partition-0-3-released/feed/ 0 [MySQL][Spider]Spider-2.21 released http://wild-growth.blogspot.com/2010/06/mysqlspiderspider-221-released.html?utm_source=rss&utm_medium=rss&utm_campaign=mysqlspiderspider-2-21-released http://wild-growth.blogspot.com/2010/06/mysqlspiderspider-221-released.html#comments Sat, 26 Jun 2010 17:19:00 +0000 Kentoku SHIBA Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "bka_mode" and "bka_engine".
- Add server parameter "spider_bka_mode" and "spider_bka_engine".
This parameters are for improvement performance for using BKA and using a lot of conditions with IN clause and so on.

"mysql-5.1.44.slave-trx-retry.diff" is included in the source tar ball from this release.
mysql-5.1.44.slave-trx-retry.diff adds a option "slave_transaction_retry_errors". This parameter is used like "slave_transaction_retry_errors=1158,1159,2013,12701". Slave sql thread retries the transaction if slave sql thread encounters the errors at this parameter. Number of retries are at "slave_transaction_retries".

Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Mikiya for your advice.

Enjoy!

PlanetMySQL Voting: Vote UP / Vote DOWN]]> http://planetmysql.ru/2010/06/26/mysqlspiderspider-2-21-released/feed/ 0 [MySQL][Spider][VP][Other]Spider-2.20 VP-0.10 BKA-for-ha_partition-0.2 released http://wild-growth.blogspot.com/2010/05/mysqlspidervpotherspider-220-vp-010-bka.html?utm_source=rss&utm_medium=rss&utm_campaign=mysqlspidervpotherspider-2-20-vp-0-10-bka-for-ha_partition-0-2-released http://wild-growth.blogspot.com/2010/05/mysqlspidervpotherspider-220-vp-010-bka.html#comments Mon, 17 May 2010 15:38:00 +0000 Kentoku SHIBA Spider is a Storage Engine for database sharding.
http://spiderformysql.com/
Vertical Partitioning is a Storage Engine for vertical partitioning for a table.
http://launchpad.net/vpformysql
BKA-for-ha_partition is a patch file of supporting "Batched Key Access" for table partitioning feature.
http://launchpad.net/partitionmrrformysql

The main changes in this version are following.
Spider
- "spider_multi_split_read" supports dividing condition count.
From this release, there is "install_spider.sql" file for installing and updating Spider. Please use this file.

Vertical Partitioning
- Support "Batched Key Access".
This feature is now available on MariaDB 5.3 with "join_cache_level=6 and more".

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

PlanetMySQL Voting: Vote UP / Vote DOWN]]> http://planetmysql.ru/2010/05/17/mysqlspidervpotherspider-2-20-vp-0-10-bka-for-ha_partition-0-2-released/feed/ 0 [MySQL][Spider]Spider-2.19 released http://wild-growth.blogspot.com/2010/05/mysqlspiderspider-219-released.html?utm_source=rss&utm_medium=rss&utm_campaign=mysqlspiderspider-2-19-released http://wild-growth.blogspot.com/2010/05/mysqlspiderspider-219-released.html#comments Mon, 03 May 2010 16:36:00 +0000 Kentoku SHIBA Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Support "Batched Key Access".
This feature is now available on MariaDB 5.3 with "join_cache_level=6 and more".

Please see "99_change_logs.txt" in the download documents for checking other changes.

Enjoy!

PlanetMySQL Voting: Vote UP / Vote DOWN]]> http://planetmysql.ru/2010/05/03/mysqlspiderspider-2-19-released/feed/ 0 [MySQL][Spider]Spider-2.18 released http://wild-growth.blogspot.com/2010/04/mysqlspiderspider-218-released.html?utm_source=rss&utm_medium=rss&utm_campaign=mysqlspiderspider-2-18-released http://wild-growth.blogspot.com/2010/04/mysqlspiderspider-218-released.html#comments Thu, 29 Apr 2010 07:28:00 +0000 Kentoku SHIBA Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
This release is bug fix release.

Please see "99_change_logs.txt" in the download documents for more detail.

Enjoy!

PlanetMySQL Voting: Vote UP / Vote DOWN]]> http://planetmysql.ru/2010/04/29/mysqlspiderspider-2-18-released/feed/ 0 Not Only NoSQL!! Uber Scaling-Out with SPIDER storage engine http://samurai-mysql.blogspot.com/2010/03/not-only-nosql-uber-scaling-out-with.html?utm_source=rss&utm_medium=rss&utm_campaign=not-only-nosql-uber-scaling-out-with-spider-storage-engine http://samurai-mysql.blogspot.com/2010/03/not-only-nosql-uber-scaling-out-with.html#comments Tue, 30 Mar 2010 09:58:24 +0000 Mikiya Okuno
However, when a site grow large, amount of traffic may exceed the replication's capacity. In such a case, people may use memcached. It's an in-memory, very fast and well-known KVS, key value store, and its read throughput is far better than MySQL. It's been used as a cache for web applications to store 'hot' data with MySQL as a back-end storage, as it can reduce read requests to MySQL dramatically.

While 1:N replication can scale read workload and memcached can reduce read requests, it cannot ease write load well. So, write traffic gets higher and higher when a web site becomes huge. On such web sites, a technique called "Sharding" has been used; it's a technique that the application choose an appropriate MySQL server from several servers.

In that way, MySQL+memcached has been a de-fact standard data store on huge web sites for long time.

Since web applications are getting larger still, especially on social media sites, write load is getting higher and higher as people communicate in real-time. In such area, yet another technique is required to handle the write load. Then, some people have chosen NoSQL solutions instead of MySQL+memcached. NoSQL is a kind of buzz word, IMHO, which represents non-relational databases which doesn't require SQL access. Despite lack of SQL access, some NoSQL softwares are suitable for huge scale web applications, like Cassandra. Although people cannot JOIN records on NoSQL system, it is not possible on RDBMS over the shards as well. So, MySQL isn't used as a RDBMS, is used as a data store without joins in other words, on such a web application in the first place.

For further information of this kind of thoughts, I recommend you to read Mark Calleghan's post: http://mysqlha.blogspot.com/2010/03/plays-well-with-others.html
and this post: http://nosql.mypopescu.com/post/407159447/cassandra-twitter-an-interview-with-ryan-king

Technically, it is possible to handle huge amount of traffics using MySQL, but a running cost gets expensive, Twitter says. As these techniques are separate ones, so those people have to spent their time to learn all of three who implement the application over them and manage them. On the other hand, Cassandra can handle more traffics as a single database management system, so people only have to learn it instead of three. Sounds great? But, is it a really good choice?

No! They're not aware of yet another solution, say SPIDER storage engine!

SPIDER for MySQL
http://spiderformysql.com/

SPIDER is a storage engine developed by a Japanese MySQL hacker, Mr. Kentoku Shiba, it makes use of MySQL's partitioning functionality and store partitioned data onto remote servers. I may say it's a Sharding storage engine. While flexibility of MySQL's storage engine API enables such an engine, but I value Kentoku's design a lot.

The following picture depicts how SPIDER storage engine works. (This is a snippet from the site above.)

In this entry, I do not explain how to use SPIDER storage engine, but I tell you how great its ability is. If you want to try it out, please refer to Giuseppe Maxia's post.

Please look at the following graph, which represents an INSERT performance comparing a single MySQL server (InnoDB), 2 SPIDER node + 2 backend MySQL server and 4 SPIDER node + 4 backend MySQL Server. You can see how good it scales.

The next graph is a SELECT performance. Read scales pretty good as well.

Red circles indicate where working set sizes exceed memory sizes. While performance drops when a working set size exceeds the available memory size, SPIDER is able to expand the memory so that a working set fits in it. SPIDER can make use of memory on all remote servers, as if there is a huge buffer pool in total.

For more information about SPIDER's performance test, please refer to Kentoku's slide. It's surprising.

The most significant problem for twitter is to scale out read/write load with less running cost. Unfortunately, they had chosen NoSQL solution due to the fact that "MySQL replication + memcached + sharding" cannot handle write intensive workload well. However, such a problem can be resolved using SPIDER storage engine with MySQL!

Generally, KVS cannot solve certain problems like below:

JOIN
Sort (ORDER BY)
Aggregation (GROUP BY)

When using KVS, these problems can be handled using MapReduce, however, we can process the same task using a very simple SQL in general. Thus, SQL allows us to develop a complex logic very efficiently. When I ask Kentoku permission to write an article about his storage engine, he told me his philosophy like below:

I think that the most significant benefit to use RDB is its usefulness and flexibility. It is a very important characteristic for developers in order to keep the application competitive, especially for those developers who have to add new features/functionalities day by day, like web services. I develop SPIDER storage engine in order to provide developers such useful and flexible RDB's characteristics, even on the environment where the traffic and data is huge thus Sharding is required.

I 100% agree with his opinion. If you are facing the problem caused by high traffic and huge data just like twitter, please consider to use SPIDER storage engine before migrating to NoSQL solutions.

PlanetMySQL Voting: Vote UP / Vote DOWN]]> http://planetmysql.ru/2010/03/30/not-only-nosql-uber-scaling-out-with-spider-storage-engine/feed/ 0