Archive for the ‘myconnpy’ Category

Fetching rows as dictionaries with MySQL Connector/Python

Октябрь 13th, 2010

This post describes how to make a custom cursor returning rows as dictionaries using MySQL Connctor/Python v0.2 (or later).

Problem: you want to fetch rows from the database and return them as a dictionary with keys being the column names.

First, lets check how you would do it without any custom cursor.

cnx = mysql.connector.connect(host='localhost',database='test')
cur = cnx.cursor()
cur.execute("SELECT c1, c2 FROM t1")
result = []
columns = tuple( [d[0].decode('utf8') for d in cur.description] )
for row in cur:
  result.append(dict(zip(columns, row)))
pprint(result)
cur.close()
cnx.close()

The above results in an output like this:

[{u'c1': datetime.datetime(2010, 10, 13, 8, 55, 35), u'c2': u'ham'},
 {u'c1': datetime.datetime(2010, 10, 13, 8, 55, 38), u'c2': u'spam'}]

Alternatively, you can code your own cursor which will return a row as a dictionary. Here is an example on how to do this:

import mysql.connector

class MySQLCursorDict(mysql.connector.cursor.MySQLCursor):

  def fetchone(self):
    row = self._fetch_row()
      if row:
        return dict(zip(self.column_names, self._row_to_python(row)))
      return None

What we did is subclassing MySQLCursor and overload the fetchone()-method. To use the above, you can do following:

cnx = mysql.connector.connect(host='localhost',database='test')
cur = cnx.cursor(cursor_class=MySQLCursorDict)
cur.execute("SELECT c1, c2 FROM t1")
rows = cur.fetchall()
pprint(rows)
cur.close()
cnx.close()

The trick is to pass the cursor_class-argument when creating the cursor. (At the time of writing, I realize that the cursor_class should be part of the connection arguments to set a default for all requested cursors. We’ll change that.)

It’s a bit more elegant coding and possibilities are endless. Maybe you’d like a cursor which returns a row in XML for example.


PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Connector/Python 0.2-devel available

Октябрь 13th, 2010

Next development release v0.2.0 of MySQL Connector/Python is available for download and testing. We still don’t recommend to use it in production: it is not beta or GA yet, but we are getting there.

Bug reports and feature requests are welcome through the Launchpad bug tracking tool.

Highlights:

  • .executemany() now optimizes INSERT statements using the MySQL
    multiple row syntax.
  • Setting sql_mode and time_zone when connecting as well as collation.
  • Raw Cursors can be used when you want to do the conversion yourself.
  • Unittests now bootstrap own MySQL server instance.
  • Tidying the source tree.

Full list of changes and bug fixes can be found online or in the ChangeLog-file, part of the distribution.

About MySQL Connector/Python: MySQL Connector/Python is implementing the MySQL Client/Server protocol completely in Python. No MySQL libraries are needed, and no compilation is necessary to run this Python DB API v2.0 compliant driver. It is compatible with Python v2.5 and later as well as Python v3.1 and later.


PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Connector/Python and database pooling

Август 26th, 2010

MySQL Connector/Python is (or should be) compliant with the Python DB-API 2.0 specification. This means that you can use DBUtils' PooledDB module to implement database connection pooling.

Here below you'll find an example which will output the connection ID of each connection requested through the pooling mechanism.

from DBUtils.PooledDB import PooledDB
import mysql.connector

def main():
    pool_size = 3
    pool = PooledDB(mysql.connector, pool_size,
        database='test', user='root', host='127.0.0.1')
    
    cnx = [None,] * pool_size
    for i in xrange(0,pool_size):
        cnx[i] = pool.connection()
        cur = cnx[i].cursor()
        cur.execute("SELECT CONNECTION_ID()")
        print "Cnx %d has ID %d" % (i+1,cur.fetchone()[0])
        cur.close()
    
    for c in cnx:
        c.close()

The output will be something like this:

Cnx 1 has ID 42
Cnx 2 has ID 41
Cnx 3 has ID 40

PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Connector/Python 0.1.5 release: critical bug fix

Май 27th, 2010

We just released MySQL Connector/Python 0.1.5 which includes a critical bug fix. It was impossible to read big result sets. The files for 0.1.4-release have been removed.

You can download MySQL Connector/Python from Launchpad.

Highlights:

  • It was impossible to retrieve big result sets. (bug lp:551533 and lp:586003)
  • Changing copyright from Sun to Oracle (also fixing silly typo)

A very Big Thanks goes to the reporters of bug lp:551533 and lp:586003. Apologies for not being able to reproduce the bug earlier, before releasing 0.1.4.

About MySQL Connector/Python: MySQL Connector/Python is implementing the MySQL Client/Server protocol completely in Python. No MySQL libraries are needed, and no compilation is necessary to run this Python DB API v2.0 compliant driver. It is compatible with Python v2.5 and later as well as Python v3.1 and later.


PlanetMySQL Voting: Vote UP / Vote DOWN

MySQL Connector/Python 0.1.4-devel available

Май 21st, 2010

Next development release 0.1.4 of MySQL Connector/Python is now available for download. This will be the last in the 0.1-series as we move on to 0.2. The aim is to release more often to get to v1.0. Hurray!

Highlights:

  • Reading from network was broken for bigger packages.
  • Reimplementing protocol.MySQLProtocol marking simpler and easier to maintain.
  • It is now possible to send multiple statements to MySQL using MySQLCursor.execute(). The results are accessible by calling the method next_resultset().
  • MySQLCursor.callproc() will now store all result sets as a MySQLCursorBuffered. They are accessible using the next_proc_resultset() method. The result of the stored procedure is returned by callproc() itself as defined by PEP249.
  • MySQL DATETIME conversion to Python datetime.datetime is now much faster.
  • Some overall performance improvements.
  • Copyright notice changes.

Big thanks to everyone using and reporting bugs found in MySQL Connector/Python. Don't hesitate to ask questions and report problems or feature requests using Launchpad.

About MySQL Connector/Python: MySQL Connector/Python is implementing the MySQL Client/Server protocol completely in Python. No MySQL libraries are needed, and no compilation is necessary to run this Python DB API v2.0 compliant driver. It is compatible with Python v2.5 and later as well as Python v3.1 and later.


PlanetMySQL Voting: Vote UP / Vote DOWN

Simulating server-side cursors with MySQL Connector/Python

Апрель 26th, 2010

Last week, my colleague Massimo and I discussed how to handle big result sets coming from MySQL in Python. The problem is that MySQL doesn't support server-side cursors, so you need to select everything and then read it. You can do it either buffered or not. MySQL Connector/Python defaults to non-buffered, meaning that you need to fetch all rows after issuing a SELECT statement. You can also turn on the buffering, mimicking what MySQL for Python (MySQLdb) does.

For big result sets, it's better to limit your search. You can do this using an integer primary key or some temporal field for example. Or you can use the LIMIT keyword. The latter solution is what is used in the MySQLCursorServerSide cursor-class. Using the SELECT it creates a temporary table from which the fetch-methods will get the information. It is something people have probably implemented in their applications, but I hope this new class will make it easier since it's done transparently.

The code is not pushed yet, but expect it to be available in next release. Here is an example how you could use it. This code selects cities staring with Z, loops over the result getting the country (yes, this is a simple join made difficult):

cnx = db.connect(user='root',db='world')
    cur = cnx.cursor()
    curCity = cnx.cursor(db.cursor.MySQLCursorServerSide)
    
    curCity.execute("SELECT ID,Name,CountryCode FROM City "\
        "WHERE NAME LIKE 'Z%' ORDER BY ID")
    
    for city in curCity:
        cur.execute("SELECT Code,Name FROM Country WHERE CODE = %s",
            (city[2],))
        country = cur.fetchone()
        print "%s (%s)" % (city[1], country[1])
    
    cur.close()
    cnx.close()

I guess the main advantage is that you can use two or more cursor objects with the same connection without the need of buffering everything in Python. On the MySQL side, the temporary table could go to disk when to big. It's maybe slower, but keeping big result sets in memory ain't good either.

Comments are welcome!


PlanetMySQL Voting: Vote UP / Vote DOWN

Multiple result sets in MySQL Connector/Python

Апрель 24th, 2010

Latest code of MySQL Connector/Python on launchpad has support for multiple result sets when you execute a stored procedure. We also changed the way the actual result of the routine is returned to conform to PEP249.

Here is some example code: it creates a stored procedure which generates 2 result sets. You can get the result by calling next_resultset(), which is returning a MySQLCursorBuffered.

    cur = cnx.cursor()

    cur.execute("DROP PROCEDURE IF EXISTS multi")
    proc = """
      CREATE PROCEDURE multi(IN pFac1 INT, IN pFac2 INT, OUT pProd INT)
      BEGIN
        SELECT 1,'a' as FooBar;
        SELECT 2;
        SET pProd := pFac1 * pFac2;
      END"""

    cur.execute(proc)
    result = cur.callproc("multi", (5, 6, 0))
    print "Result:", result
    
    extcur = cur.next_resultset()
    i = 1
    while extcur:
        rows = extcur.fetchall()
        print "Result set #%d:" % i
        print rows
        extcur = cur.next_resultset()
        i += 1
        
    cur.close()

The output:

Result: ('5', '6', 30)
Result set #1:
[(1, u'a')]
Result set #2:
[(2,)]

As mentioned above: this will be part of 0.1.4-devel release due next week. Comments are welcome through the associated bug report.


PlanetMySQL Voting: Vote UP / Vote DOWN

Status report No.2 on SQLAlchemy and MySQL Connector/Python

Февраль 19th, 2010

Few days ago, the folks at SQLAlchemy pushed some proposed modification to the MySQL Connector/Python dialect. Before this patch, previous report yielded 72 errors and 11 failures. Now we got down to 9 errors, but the failures are still lingering. Is this an improvement? Yes and no, failures should go down, but there are some SQLAlchemy tests I just can't figure out, yet.. clues are welcome!

Here are some detailed results which also included MySQLdb and oursql. I used SQLAlchemy revision 6788 (i.e. from svn trunk) which has now 2143 unittests:

  • mysql.connector rev216: SKIP=1, errors=9, failures=11 (355.707s)
  • MySQLdb 1.2.3c1: SKIP=2, errors=8, failures=1 (315.884s)
  • oursql 0.9.1: SKIP=1, errors=8, failures=2 (322.318s)

Software used: MacOSX v10.6.2, MySQL v5.1.42 and Python v2.6.1.

Using SQLAlchemy might not be the best way to messure how mature MySQL Connector/Python is, but it sure helps lots.


PlanetMySQL Voting: Vote UP / Vote DOWN

Stuffing the gaps in the COLLATIONS table using a stored procedure

Февраль 17th, 2010

To far fetched (pun inteded), some might think.. Below you'll find a procedure to get a list of MySQL supported character sets and their collations. The output is Python and can be used to build a (big) tuple.

The problem is that character set IDs in MySQL have 'gaps'. For example hebrew has ID 16, and the next character set in the COLLATIONS-table, tis620, has ID 18. Not a big problem, just a bit annoying. This list is hardcoded in MySQL Connector/Python and I needed something to easily maintain it.

Solutions were using text editing skills, parsing it in Python to produce a list, etc.. But why not a Stored Routine? The following stored procedure is looping over a cursor, calculating the number of rows the gap has, and inserts blanks as needed.

DROP PROCEDURE IF EXISTS python_getcharsets;
DELIMITER //
CREATE PROCEDURE python_getcharsets()
BEGIN
  DECLARE i,diff,cid,done INT DEFAULT 0;
  DECLARE chname,coname VARCHAR(32);
  DECLARE cur CURSOR FOR SELECT ID,CHARACTER_SET_NAME,
    COLLATION_NAME FROM INFORMATION_SCHEMA.COLLATIONS
    ORDER BY ID;
  DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
  OPEN cur;
  loop_cur: LOOP
    FETCH cur INTO cid,chname,coname;
    IF done THEN LEAVE loop_cur; END IF;
    SET i = i + 1; SET diff = cid - i;
    WHILE diff DO
      SELECT "None,";
      SET diff = diff - 1; SET i = i + 1;
    END WHILE;
    SELECT CONCAT('("',chname,'","',coname,'"), # ',cid);
  END LOOP loop_cur;
  CLOSE cur;
END//
DELIMITER ;

CALL python_getcharsets();
DROP PROCEDURE IF EXISTS python_getcharsets;

You would save the above to a file called getcharsets.sql for example, and execute it like this:

shell> mysql -N test < getcharsets.sql 
..
("cp1251","cp1251_bulgarian_ci"), # 14
("latin1","latin1_danish_ci"), # 15
("hebrew","hebrew_general_ci"), # 16
None,
("tis620","tis620_thai_ci"), # 18
("euckr","euckr_korean_ci"), # 19
..

I'll spare you the complete output, but as you can see from above sample: the gap has been stuffed with a None-Python value. Taking this output, you'll inserted it your code:

desc = (
    None,
    ("big5","big5_chinese_ci"), # 1
    ("latin2","latin2_czech_cs"), # 2
    ("dec8","dec8_swedish_ci"), # 3
    ("cp850","cp850_general_ci"), # 4
..

Silly? Definitely basic stuff, but I got my code a bit faster, and cleaner!


PlanetMySQL Voting: Vote UP / Vote DOWN

Don’t forget the COMMIT in MySQL

Февраль 9th, 2010

Yes, MySQL has transactions if you use InnoDB or NDB Cluster for example. Using these transactional storage engines, you'll have to commit (or roll back) your inserts, deletes or updates.

I've seen it a few times now with people being surprised that no data is going into the tables. It's not so a silly problem in the end. If you are used to the defaults in MySQL you don't have to commit anything since it is automatically done for you.

Take the Python Database Interfaces for MySQL. PEP-249 says that, by default, auto-commit should be turned off. You could turn it back on, but it's good practice to be explicit and commit in your code. Remember the Zen of Python!

Here is just a small example to show it. Uses MySQL Connector/Python, but it does work also with others:


import mysql.connector
cnx = mysql.connector.connect(db='test')
cur = cnx.cursor()
cur.execute("""CREATE TABLE innodb_t1 (
id INT UNSIGNED NOT NULL,
c1 VARCHAR(128),
PRIMARY KEY (id)
) ENGINE=InnoDB""")
ins = "INSERT INTO innodb_t1 (id,c1) VALUES (%s,%s)"
cur.execute(ins,
(1,'MySQL Support Team _is_ already the best',))
cnx.commit()
cur.close()
cnx.close()

PlanetMySQL Voting: Vote UP / Vote DOWN