Yesterday when I was looking into the code for an answer to the question of one of my customers this research came into my mind again. Today I was a bit more curious and I tried to find a way to patch the MySQL code to get a logoff trigger. Luckily I was successful right away and I created the exit_connect variable which acts as the logoff trigger.

The patches for the logoff trigger you can find here.

What you can do with these patches you will see in the following example. First we create an audit schema with an audit table:

CREATE SCHEMA audit;
USE audit;

-- thread_id is no good PK, because of restart!
CREATE TABLE audit_connect (
  id              BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY
, thread_id       INT UNSIGNED NOT NULL DEFAULT 0
, user            VARCHAR(64) NOT NULL DEFAULT 'unknown'
, login_ts        TIMESTAMP NULL DEFAULT NULL
, logout_ts       TIMESTAMP NULL DEFAULT NULL
, com_select      INT UNSIGNED NOT NULL DEFAULT 0
, bytes_received  BIGINT UNSIGNED NOT NULL DEFAULT 0
, bytes_sent      BIGINT UNSIGNED NOT NULL DEFAULT 0
, KEY (thread_id)
);

Then we create a stored procedure:

DROP PROCEDURE IF EXISTS audit.login_trigger;
DROP PROCEDURE IF EXISTS audit.logoff_trigger;

DELIMITER //

CREATE PROCEDURE audit.login_trigger()
SQL SECURITY DEFINER
BEGIN
  INSERT INTO audit.audit_connect (thread_id, user, login_ts)
  VALUES (CONNECTION_ID(), USER(), NOW());
END;

CREATE PROCEDURE audit.logoff_trigger()
SQL SECURITY DEFINER
BEGIN

  DECLARE com_select INT DEFAULT 0;
  DECLARE bytes_received INT DEFAULT 0;
  DECLARE bytes_sent INT DEFAULT 0;

  SELECT variable_value
    INTO com_select
    FROM INFORMATION_SCHEMA.session_status
   WHERE variable_name = 'COM_SELECT';

  SELECT variable_value
    INTO bytes_received
    FROM INFORMATION_SCHEMA.session_status
   WHERE variable_name = 'BYTES_RECEIVED';

  SELECT variable_value
    INTO bytes_sent
    FROM INFORMATION_SCHEMA.session_status
   WHERE variable_name = 'BYTES_SENT';

  UPDATE audit.audit_connect
     SET logout_ts = NOW(), com_select = com_select
       , bytes_received = bytes_received, bytes_sent = bytes_sent
   WHERE thread_id = CONNECTION_ID();
END;

//
DELIMITER ;

Then we grant the EXECUTE privilege to ALL users which have to connect to this database:

GRANT EXECUTE ON PROCEDURE audit.login_trigger TO 'testuser'@'%';
GRANT EXECUTE ON PROCEDURE audit.logoff_trigger TO 'testuser'@'%';

And last we have to hook our login and logoff triggers into MySQL:

mysql> SET GLOBAL init_connect="CALL audit.login_trigger()";
mysql> SET GLOBAL exit_connect="CALL audit.logoff_trigger()";

This you should also make permanent in the the my.cnf.

Then you can start and connecting and running some statements against you database and some reports against you audit table:

Which user connected most

SELECT user, COUNT(user) AS count
  FROM audit_connect
 GROUP BY user
 ORDER BY count DESC;

+-----------+-------+
| user      | count |
+-----------+-------+
| u3@master |   169 |
| u2@master |     2 |
| u1@master |     1 |
+-----------+-------+

Total, average, max and min connect time per user

SELECT user, MAX(logout_ts-login_ts) AS max, MIN(logout_ts-login_ts) AS min
     , AVG(ROUND(logout_ts-login_ts, 0)) AS avg, SUM(logout_ts-login_ts) AS total
  FROM audit_connect
 GROUP BY user;

+-----------+------+------+----------+-------+
| user      | max  | min  | avg      | total |
+-----------+------+------+----------+-------+
| u1@master |  220 |  220 | 220.0000 |   220 |
| u2@master |   17 |    0 |   8.5000 |    17 |
| u3@master |    2 |    0 |   0.0414 |     7 |
+-----------+------+------+----------+-------+

Which user did the most SELECT queries

SELECT user, SUM(com_select) AS cnt
  FROM audit_connect
 GROUP BY user
 ORDER BY cnt DESC;

+-----------+------+
| user      | cnt  |
+-----------+------+
| u3@master |  503 |
| u2@master |   29 |
| u1@master |    6 |
+-----------+------+

Which user sent the most traffic over the network

SELECT user, SUM(bytes_received) AS rcvd, SUM(bytes_sent) AS sent
  FROM audit_connect
 GROUP BY user;

+-----------+-------+-----------+
| user      | rcvd  | sent      |
+-----------+-------+-----------+
| u1@master |   242 | 358488916 |
| u2@master |  1046 |     16753 |
| u3@master | 23259 |     70808 |
+-----------+-------+-----------+

Which user was doing what in a certain time range

SELECT user, COUNT(*) AS cnt, SUM(com_select) AS sel
     , SUM(bytes_received) AS rcvd, SUM(bytes_sent) AS sent
  FROM audit_connect
 WHERE login_ts <= '2010-12-10 22:54:59' and logout_ts >= '2010-12-10 22:54:00'
 GROUP BY user
;

+-----------+-----+------+------+-----------+
| user      | cnt | sel  | rcvd | sent      |
+-----------+-----+------+------+-----------+
| u1@master |   1 |    6 |  242 | 358488916 |
| u3@master |  13 |   37 | 1721 |      5119 |
+-----------+-----+------+------+-----------+

How do InnoDB and Postgres store large columns? Both use overflow pages for rows with large columns and support compression. Postgres is more likely to store large columns in overflow pages and InnoDB is likely to use more space in overflow pages. Tables can be queried in Postgres to determine the per-table space used by overflow pages. MySQL does not have tables for that but the data can be extracted with a lot more work and time.

InnoDB

Peter wrote an excellent post for InnoDB. There are two file formats, Antelope and Barracuda, and the general approach is the same. A row can use about 1/2 of a page which is ~8k for 16k pages. When the row is too big and has large columns, the largest of them are moved to overflow pages until the row fits. If the row is not small enough after all have been moved then an error is raised.

When a column is moved for the Antelope format the first 768 bytes are stored inline. If 10 large columns are moved they can still consume 7680 bytes inline, excluding the metadata bytes. Thus you can't have a row with 11 BLOB columns each with length 10,000 as that would not fit.

The difference with Barracuda is that when a large column is moved to an overflow page, only a 20-byte pointer is stored inline rather than a 768-byte prefix.

Large columns do not share overflow pages. When five columns each from two rows are moved to overflow pages, then at least ten overflow pages will be allocated. This can use much more space than expected and it is difficult to monitor. You can get the data on space used but it is not fun and it can take a long time (see xtrabackup --stats, upcoming patch to innochecksum in Facebook MySQL patch, InnoDB tablespace monitor).

The Barracuda format also supports compression for the overflow pages and with key_block_size=16 compression is only done for overflow pages (not for the inline columns).

Documentation can be tricky:

• This accurately describes  Antelope behavior but does not state that this is Antelope behavior as the official docs describe the built-in behavior and the Antelope/Barracuda distinction occurs with the plug-in.
• This page confuses me and incorrectly describes the 768-byte prefix approach.
• This is a good but brief description of the changes with Barracuda.
• This is the best description and covers both the strategy for moving columns and the amount of space used by external pages.

Postgres

Disclaimer - I do not use Postgres, but I read the manuals, mailing lists and blog posts. What I have written below is a summary. There are many more details in the docs that I cite. They are worth reading.

Postgres uses TOAST (The Oversized Attribute Storage Technique) for large columns, has an 8k page size by default and rows cannot span pages. Large columns are compressed and then split into pieces (usually 2k each) where each piece is stored as a separate row in the per-table TOAST table. Queries can be done to determine the space used for each TOAST table.

Differences

Postgres is more likely than InnoDB to use overflow pages (the TOAST table) because large columns are stored in the TOAST table when a row is wider than TOAST_TUPLE_TARGET (2k by default). InnoDB does not use overflow pages until rows are wider than ~8k.

But InnoDB is likely to use more space in the overflow pages because each column that uses overflow pages requires its own pages. If 5 columns each from rows A and B use overflow pages then at least 10 pages are allocated by InnoDB. This can waste a lot of space. Postgres splits each large column into compressed pieces and treats each piece as a row in a TOAST table.

There are three active projects to add support for group commit to MySQL/InnoDB. Monty Program and MySQL appear to have work in progress. The Facebook version is done but we will gladly undo it when official versions are ready.

PostgreSQL has group commit today. It is enabled by the commit_delay configuration parameter and the feature is described in the manual.

The commit_delay options provides a constraint on when group commit is done that has yet to be considered in the MySQL implementations. When commit_siblings is set to a non-zero value then a transaction attempting to commit will only wait at least that many transactions are active. This is a great option because for workloads with limited or intermittent concurrency there is no reason to wait when it is unlikely that another commit will be attempted before the commit_delay time expires.

It is harder to implement group commit for MySQL/InnoDB because two resource managers must be kept in sync: InnoDB and the binlog. This looks a lot like XA. But that is required until someone implements storage-engine replication (which is another hard problem).

In PostgreSQL there is but one log to worry about, the WAL. That simplifies group commit, replication and incremental backup. But those are topics for another post or another author.

• Pick up all SQL Statements executing in MySQL. This is done using the MySQL Audit plugin interface.
• The SQL Statement is then normalized, which in this case means that all literals in the statements are taken away, so that SELECT * FROM orders WHERE orderid = 57 becomes SELECT * FROM orders WHERE orderid = ?. Then we place this statement in a list of most recently executed statements, if the normalized version is already in the list, the execute counter is incremented and the statement is popped to the top (as it is most recently executed), else it is inserted into the list (and popped to the top), in which case, if the list of full, the least recently executed statement leaves the list.
• Then we check a second list, the toplist of most executed statements. If the execute count of the normalized statement will make it appear on this list, it is added there in the same way.
  As a sidenote, both the recently executed list as well as the top executed list are simple in-memory datastructures. They are not stored on disk or anything, they are just lists of C-structs allocated dynamically.
• The last thing the Plugin do is to expose two INFORMATION_SCHEMA plugins, so that the recently executed list and the toplist can be examined as normal MySQL tables.

• STATEMENT - The normalized statement.
• SAMPLE_STATEMENT - The first real statement that was turned into the normalized statement in the STATEMENT column.
• NUM_EXECUTES - The number of times the statement has been executed.
• NUM_ROWS_TOTAL - The total number of rows retrieved by all NUM_EXECUTES executions of the statement.