High Availability is the #1 requested enhancement to the MySQL & MariaDB servers, even more popular than scalability and performance.  And with SkySQL's expertise at hand, it is now easier than ever before for customers to achieve the level of High Availability that they want.

SkySQL™ Enterprise HA is SkySQL's 360° answer to providing a ready-to-go solution for MySQL & MariaDB High Availability – in no more than 3 days.

Check out the following resources for more information:

Visit the SkySQL Enterprise HA product page

Including:

• SkySQL™ Enterprise HA Options Table
• SkySQL™ Enterprise HA Statement of Work

Download the SkySQL High Availability whitepaper

Contact your local SkySQL representative to discuss your HA needs

Finally, if you are in New York City today, join Ivan Zoratti, SkySQL CTO, at the MySQL Meetup for a discussion about cool new tools & tricks to achieve High Availability of your MySQL servers!

Fore more information, visit the New York City MySQL Group webpage.

We look forward to helping you achieve your High Availability objectives for your MySQL & MariaDB databases!

• ndb_conflict_trans_row_conflict_count
  This is essentially the same as ndb_conflict_epoch_trans - the number of row events with conflict detected.
• ndb_conflict_trans_row_reject_count
  The number of row events which were handled as in-conflict. It will be at least as large as ndb_conflict_trans_row_count, and will be higher if other rows are implicated by being in a conflicting transaction, or being dependent on a row in a conflicting transaction.
  A separate ndb_conflict_trans_row_implicated_count could be constructed as ndb_conflict_trans_row_reject_count - ndb_conflict_trans_row_conflict_count
• ndb_conflict_trans_reject_count
  The number of discrete user transactions detected as in-conflict.
• ndb_conflict_trans_conflict_commit_count
  The number of epoch transactions which had transactional conflicts detected during application.
• ndb_conflict_trans_detect_iter_count
  The number of iterations of the three-pass algorithm that have occurred. Each set of passes counts as one. Normally this would be the same as ndb_conflict_trans_conflict_commit_count. Where further conflicts are found on the third pass, another iteration may be required, which would increase this count. So if this count is larger than ndb_conflict_trans_conflict_commit_count then there have been some conflicts generated concurrently with conflict detection, perhaps suggesting a high conflict rate.
  

• Conflict detection logic is pushed down to data nodes for execution
  Minimising extra data transfer + locking
• Slave operation batching is preserved
  Multiple row events are applied together, saving MySQLD <-> data node round trips, using data node parallelism
  For both algorithms, one extra MySQLD <-> data node round-trip is required in the no-conflicts case (best case)
• NDB$EPOCH : One extra MySQLD <-> data node round-trip is required per *batch* in the all-conflicts case (worst case)
• NDB$EPOCH : Minimal impact to Binlog sizes - one extra row event per epoch.
• NDB$EPOCH : Minimal overhead to Slave SQL CPU consumption
• NDB$EPOCH_TRANS : One extra MySQLD <-> data node round-trip is required per *batch* per *pass* in the all-conflicts case (worst case)
• NDB$EPOCH_TRANS : One round of two passes is required for each conflict newly created since the previous pass.
• NDB$EPOCH_TRANS : Small impact to Binlog sizes - one extra row event per epoch plus one user transaction id per row event.
• NDB$EPOCH_TRANS : Small overhead to Slave SQL CPU consumption in no-conflict case
  

• Reads from the Secondary are tentative
  Data committed on the secondary may later be rolled back. The window of potential rollback is limited, after which Secondary data can be considered stable. This is described in more detail here.
• Writes to the Secondary may be rolled back
  If this occurs, the fact will be recorded on the Primary. Once a committed write is stable it will not be rolled back.
• Out-of-band dependencies between transactions are out-of-scope
  For example direct communication between two clients creating a dependency between their committed transactions, not observable from their database footprints.
  

• Detected transaction dependencies are limited to dependencies between binlogged writes (Insert, Update, Delete)
  Reads are not currently included.
• Delete vs Delete+Insert conflicts risk data divergence
  Delete vs Delete conflicts are detected, but currently do not result in conflict handling, so that Delete vs Delete + Insert can result in data divergence.
• With NDB$EPOCH_TRANS, unplanned Primary outages may require manual steps to restore Secondary consistency
  With pending multiple, time spaced, non-overlapping transactional conflicts, an unexpected failure may need some Binlog processing to ensure consistency.
  

• Increment conflict counters
• Optionally insert a row into an exceptions table
  

1. Lock the affected row/primary key until transaction commit time, even if it does not exist (much as an Insert would).
2. Set the affected row's author metacolum to 0
   The refresh is logically a local change
3. On commit
   - Row exists case : Set the row's last committed epoch to the current epoch
   - Cause a WRITE_ROW (row exists case) or DELETE_ROW (no row exists) event to be generated by attached Binlogging MySQLDs.
   

• Time between commit of transaction, and next Primary Cluster epoch boundary
  (Worst = 1 * TimeBetweenEpochs, Best = 0, Avg = 0.5 * TimeBetweenEpochs)
• Time required to log event in Primary Cluster's Binlogging MySQLDs Binlog (~negligible)
• Time required for Secondary Slave MySQLD IO thread to
  - Minimum : Detect new Binlog data - negligible
  - Maximum : Consume queued Binlog prior to the new data - unbounded
  - Pull new epoch transaction
  - Record in Relay log
  
• Time required for Secondary Slave MySQLD SQL thread to
  - Minimum : Detect new events in relay log
  - Maximum : Consume queued Relay log prior to new data - unbounded
  - Read and apply events
  - Potentially multiple batches.
  - Commit epoch transaction at Secondary
• Time between commit of replicated epoch transaction and next Secondary Cluster epoch boundary
  (Worst = 1 * TimeBetweenEpochs, Best = 0, Avg = 0.5 * TimeBetweenEpochs)
• After this point a Secondary-local commit on the data is possible without conflict
• Time required to log event in Secondary Cluster's Binlogging MySQLDs Binlog (~negligible)
• Time required for Primary Slave MySQLD IO thread to
  - Minimum : Detect new Binlog data
  - Maximum : Consume queued Binlog data prior to the new data - unbounded
  - Pull new epoch transaction
  - Record in Relay log
• Time required for Primary Slave MySQLD SQL thread to
  - Minimum : Detect new events in relay log
  - Maximum : Consume queued Relay log prior to new data - unbounded
  - Read and apply events
  - Potentially multiple batches.
  - For NDB$EPOCH_TRANS, potentially multiple passes
  - Commit epoch transaction
  - Update max replicated epoch to reflect new maximum.
• Further Secondary sourced modifications to the rows are now considered not-in-conflict
  

  Cluster_conflict_window_length = EpochDelay +
                                  P_Binlog_lag +
                                  S_Relay_lag +
                                  S_Binlog_lag +
                                  P_Relay_lag

 Where
    EpochDelay minimum is 0
    EpochDelay avg     is TimeBetweenEpochs
    EpochDelay maximum is 2 * TimeBetweenEpochs

     EpochDelay minimum is 0
    EpochDelay avg     is 100 millis
    EpochDelay maximum is 200 millis

• The time taken to detect a conflict with a Primary transaction
• The time taken for a committed Secondary transaction to become stable or be reverted

 Primary->Secondary

  Client_conflict_window_length = EpochDelay +
                                  P_Binlog_lag +
                                  S_Relay_lag +
                                  EpochDelay

or

Secondary->Primary

  Client_conflict_window_length = EpochDelay +
                                  S_Binlog_lag +
                                  P_Relay_lag

Where EpochDelay is defined as above.

 Tb - Ta > Client_conflict_window_length(Primary->Secondary)

• The Client_conflict_window_length in either direction will on average not be less than the EpochDelay (100 millis by default)
• Clients racing against replication to update both clusters need only beat the current Client_conflict_window_length to cause a conflict
• Replication latencies in either direction are potentially independent
• Detected conflict rates partly depend on replication latencies

The MySQL Cluster product team is always interested to see new and innovative uses of the database. Last week, a team of students at the KTH Royal Institute of Technology in Sweden blogged about their use of MySQL Cluster in creating a scalable and highly available HDFS Namenode.

There are many established use cases of MySQL Cluster in the web, cloud/SaaS, telecoms and even flight control systems – you can see those we are allowed to talk about publicly here. 

The KTH team has been working on a project to move all of the metadata from the HDFS / Hadoop nameenode to MySQL Cluster. Why did they want to do this, you may ask? Well…:

- The namenode is a single point of failure. If it goes down, so too does the file system

- As a single server, the namenode becomes a bottleneck within heavily loaded HDFS / Hadoop deployments. As server resources are consumed and write volumes increase, so the system can grind to a halt. (And with data volumes growing around 40% per year, this will only become more common!)

So KTH decided to move metadata storage to MySQL Cluster. Why, you may ask? Well….

- MySQL Cluster already offered them a replicated, shared-nothing database, distributed across commodity hardware.

- MySQL Cluster is widely deployed with proven stability

- The metadata can be distributed across nodes to scale out capacity, while retaining complete consistency to the clients and eliminating any Single Point of Failure

- Linear scaling of operations per second across the cluster, as new namenodes are added.

Access to the cluster is via the MySQL Cluster Connector for Java, providing a NoSQL, Java based ORM with very low latency. You can learn more about this ClusterJ API here. 

Of course, the work at KTH is on-going with future optimizations planned – which we will follow with interest.

So how can you determine if MySQL Cluster is the right choice for your new project? We have just updated our MySQL Cluster Evaluation Guide (note, this will directly open the pdf).

This update is based around the latest MySQL Cluster 7.2 Development Release which includes a series of enhancements to further broaden the use case of MySQL Cluster, including:

- 70x higher JOIN performance with Adaptive Query Localization pushing JOIN operations down to MySQL Cluster’s data

- Native Key-Value Memcached interface to the cluster allowing schema and schemaless storage

- New cross-data center scalability enhancements

MySQL Cluster is not a fit for every use-case, but by downloading the Evaluation Guide, you’ll get a clear picture of where MySQL Cluster can be useful to you, and best practices in planning and executing your evaluation.

Let us know of other interesting use-cases in the comments below

1. Slaves should 'reflect' information about replicated epochs they have applied
   Applied epoch numbers should be included in the Slave Binlog events returning to the originating cluster, in a Binlog position corresponding to the commit time of the replicated epoch transaction relative to Slave local transactions.
   
2. Masters should maintain a maximum replicated epoch
   A cluster should use the reflected epoch information to track which of its epochs has been applied by a Slave cluster. This will be the maximum of all epochs applied by the Slave.
   
3. Masters should track commit-time epoch per row
   To allow per-row detection of conflicts
4. Masters should track commit-authorship per row
   To differentiate recent epochs due to replication or conflicting activity.
   

1. Downstream Clusters become aware of their position relative to all upstream Master clusters, not just their immediate Master cluster.
   They gain extra mysql.ndb_apply_status entries for all upstream Masters.
2. Circularly replicating clusters become aware of which of their epochs, and epoch transactions, have been applied to all clusters in the circle.
   They gain extra mysql.ndb_apply_status entries for all Binlogging MySQLDs in the loop
   

 
     Cluster 1 Epoch transactions in flight in
           a circular configuration
          (Ignoring Cluster 2 epochs)

                             39       38       37 
                       ->---->----->----->----->--
                      /                           \ (Queued epochs 36-26)
            Cluster 1                             Cluster 2
(Queued epochs 23,24) \                           /
                       -<---<------<----<----<----
                            25       26       27

Current epoch = 40
Max replicated epoch = 22                

@ Slave start

   max_replicated_epoch = SELECT MAX(epoch)
                            FROM mysql.ndb_apply_status
                           WHERE server_id IN @@IGNORE_SERVER_IDS;

  in_conflict = (ndb_gci64 > max_replicated_epoch)

  in_conflict = (ndb$author != change_author) && (ndb_gci64 > max_replicated_epoch)

  in_conflict = (ndb$author != 1) && (ndb_gci64 > max_replicated_epoch)

              = (ndb$author == 0) && (ndb_gci64 > max_replicated_epoch)

• @Cluster A
  Transactions modify rows, automatically setting their hidden NDB$GCI64 column to the current epoch and their NDB$AUTHOR column to 0
  
  Binlogging MySQLDs record modified rows in epoch transactions in their Binlogs, together with MySQLD generated mysql.ndb_apply_status WRITE_ROW events
  
  
• @Cluster B
  Slave MySQLDs apply replicated epoch transactions along with their generated mysql.ndb_apply_status WRITE_ROW events
  
  Other clients of Cluster B commit transactions against the same data.
  
  Binlogging MySQLDs 'reflect' the applied-replicated epoch information by recording the mysql.ndb_apply_status WRITE_ROW events in their Binlogs as a result of --ndb-log-apply-status.
  
  Binlogging MySQLDs also record the row changes made by local clients.
  
  
• @Cluster A
  Slave MySQLDs track the incoming reflected epoch mysql.ndb_apply_status WRITE_ROW events to maintain their ndb_slave_max_replicated_epoch variables
  
  Slave MySQLDs attach NdbApi interpreted programs to UPDATE and DELETE operations as they are applied to the database, comparing the row's stored NDB$GCI64 and NDB$AUTHOR columns with constant values supplied in the program.
  
  If there are no conflicts, the UPDATE and DELETE operations are applied, and the row's NDB$AUTHOR columns are set to one indicating a successful Slave modification
  
  If there are conflicts then conflict handling for the conflicting rows begins.
  

  Incoming independent transactions
   affecting multiple rows

     T3         T4         T7
     T1         T2         T5

      |         |          |
      V         V          V

   --------  --------  --------
   |  1   |  |  2   |  |  3   |
   |  TC  |  |  TC  |  |  TC  |   Data nodes with multiple
   |      |--|      |--|      |   transaction coordinators
   |------|  |------|  |------|   acting on data stored in
   |      |  |      |  |      |       different nodes
   | DATA |  | DATA |  | DATA |
   --------  --------  --------

      |         |          |
      V         V          V

     t4        t4          t3
     t1        t7          t2
     t2        t1          t7
               t5

   Outgoing row change event
    streams by causing
       transaction

 Incoming independent transactions

     T3         T4         T7
     T1         T2         T5

      |         |          |
      V         V          V

   --------  --------  --------
   |  1   |  |  2   |  |  3   |
   |  TC  |  |  TC  |  |  TC  |   Data nodes with multiple
   |      |--|      |--|      |   transaction coordinators
   |------|  |------|  |------|   acting on data stored in
   |      |  |      |  |      |      different nodes
   | DATA |  | DATA |  | DATA |
   --------  --------  --------

      |         |          |
      V         V          V

    t4(22)    t4(22)      t3(22)            Epoch 22
    ......    ......      ......
    t1(23)    t7(23)      t2(23)            Epoch 23
    t2(23)    t1(23)      t7(23)
              ......
              t5(24)                        Epoch 24

    Outgoing row change event
    streams by causing transaction
    with epoch numbers in ()

      Events      Transactions
                 contained in epoch

     t4(22)
     t4(22)      {T4,T3}
     t3(22)
   
     ......

     t1(23)
     t2(23)
     t7(23)
     t1(23)      {T1, T2, T7}
     t2(23)
     t7(23)

     ......

     t5(24)      {T5}

• Efficiency : They can be considered a kind of Binlog group commit, where multiple user transactions are recorded in one Binlog (epoch) transaction. As an epoch normally contains 100 milliseconds of row changes from a cluster, this is a significant amortisation.
  
• Consistency : Each epoch transaction contains the row operations which occurred when moving the cluster from epoch boundary consistent state A to epoch boundary consistent state B
  Therefore, when applied as a transaction by a slave, the slave will atomically move from consistent state A to consistent state B
• Inter-epoch ordering : Any row event recorded in epoch n+1 logically happened after every row event in epoch n
• Intra-epoch disorder : Any two row events recorded in epoch n, affecting different rows, may have happened in any order.
• Intra-epoch key-order : Any two row events recorded in epoch n, affecting the same row, happened in the order they are recorded.

    BEGIN
    WRITE_ROW mysql.ndb_apply_status server_id=4, epoch=6998
    WRITE_ROW ...
    UPDATE_ROW ...
    DELETE_ROW ...
    ...
    COMMIT # Consistent state of the database

    BEGIN
    WRITE_ROW mysql.ndb_apply_status server_id=4, epoch=6999
    ...
    COMMIT # Consistent state of the database

    BEGIN
    WRITE_ROW mysql.ndb_apply_status server_id=4, epoch=7000
    ...
    COMMIT # Consistent state of the database
    ...

• Epochs provide an incrementing logical clock
• Epochs are recorded in the Binlog, and therefore shipped to Slaves
• Epoch boundaries imply happened-before relationships between events before and after them in the Binlog

Those are the topics we cover in the latest episode of our “Meet The MySQL Experts” podcast.

Mat Keep and Bernd Ocklin talk about new database requirements, and walk us through what's new in the second Development Milestone Release of MySQL Cluster 7.2, including impressive performance improvements, new NoSQL access via memcached, cross data center scalability, and more...

Enjoy the podcast!

1. MX -> MY (MX happened before MY)
   
   or
   
   
2. MY -> MX (MY happened before MX)
   

MA1 -> M*

MB1 -> M*

MB1 -> MA1

MA1 -> MB1

MA1 -> M*

If Server B Binlog implies MB* -> MA1 then there has been a conflict

• It is defined per committed modification per disjoint data set
  
• It can be extended by further modifications to the same data from the same server
  The window does not close all further modifications have been fully replicated
• Window duration is dependent on the replication round-trip delay
  Which can vary greatly
  
• Once it closes, further modifications to the same data from anywhere are safe, but will each open their own window of potential conflict.
  
• From the point of view of one Server, conflicts can occur at any time until the conflict window is closed
  
• From the point of view of one Server, the duration of the window of potential conflict is similar to
  
  Replication Propagation Delay A to B + Replication Propagation Delay B to A
  
  These delays may not be symmetric.
• From the point of view of an external observer/actor, the system will detect two modifications MA1 and MB1 committed at times tMA1 and tMA2 as in-conflict if
  
  tMB1 - tMA1 < Replication Propagation Delay A to B
  
  ( A before B, but not by enough to avoid conflict )
  
  or
  
  tMA1 - tMB1 < Replication Propagation Delay B to A
  
  ( B before A, but not by enough to avoid conflict )
• The window of potential conflict can only be as short as the replication propagation delay between systems, which can tend towards, but never reach zero.
  

 Server A Binlog events    ClockVal stored in DB
                           by Modification

  ...
  MA1                       39
  MA2                       39
  MA3                       39
  ClockVal_A = 40
  MA4                       40
  MA5                       40
  ClockVal_A = 41
  MA6                       41

 Server B Binlog events

  ...
  MB1
  MB2
  ClockVal_A = 40
  MB3
  MB4
  ClockVal_B = 234
  MB5
  MB6
  ClockVal_A = 41
  MB7
  ...

in_conflict = row_clockval >= maximum_replicated_clockval

• Automatic update on commit of row modification, opening conflict window
• Automatic extension of conflict window on further modification on row with open conflict window.
  
• Automatic closure of conflict window on maximum replicated clock value exceeding row's stored value
• Efficient storage cost per row - one clock value.
  
• Efficient runtime processing cost - inequality comparison between maximum replicated clock value and row's stored clock value.