• DO 1 — this one does not retrieve a result set, and thus can be seen as a small “update”.
• SELECT 1 — the minimal SELECT query.

The test program runs a query N times, and if the query was a select, it retrieves all values from the result set, using ResultSet.getObject(i), and calculates the queries-per-second value. (The best thing is that the test program is single-threaded, and how often does one get to run single-threaded tests?  the test was run on my own workstation, which runs Windows Server 2008 R2, and I have useConfigs=maxPerformance in the URL for ConnectorJ.

Results (Queries per second,  unprepared)

MariaDB JDBC appears to be a little faster (~10%) than  ConnectorJ, and much faster (~30%) than Drizzle JDBC.

Can ConnectorJ do better? I bet it can. Looking into profiler output – CPU profiling, instrumentation mode in NetBeans – for  a test that executes “SELECT 1″ in a loop,  shows com.mysql.jdbc.StatementImpl.findStartOfStatement() taking 7.5% of runtime. Ok, instrumentation results should be taken with a grain of salt, however the single reason string search is used, is because - if an update (DML) statement is executed inside ResultSet.executeQuery(), it is rejected with an exception. This can be done differenty, I believe. If absolutely necessary, throwing an exception can be delayed, until the client finds out that the server sent an OK packet instead of a result set.

Even more interesting is the case with Drizzle JDBC. In theory, since the MariaDB driver has a Drizzle JDBC heritage, the performance characteristics should be similar, but they are not, so there must be a bug somewhere. It appears very easy to find, as according to profiler, 50.2% CPU time (take that number with a big grain of salt) is spent in a function that constructs a hexdump from a byte buffer. Looking at the source code, we find following line that is unconditionally executed:

log.finest("Sending : " + MySQLProtocol.hexdump(byteHeader, 0));

While the result of the hexdump is never used (unless logging level is FINEST), the dump string is still created, using relatively expensive Formatter routines, concatenated with the String “Sending:”, and then thrown away… In Markus’ defense, hexdump() is not his fault, it was contributed 3 years ago. But it remained undetected for 3 years. This bug is now filed  https://github.com/krummas/DrizzleJDBC/issues/21 [UPDATE: this bug was resolved within hours  after reporting]

So, let’s check how much we can gain by putting the offending code into an if (log.getLevel() == java.util.logging.Level.FINEST) condition.
The QPS from “DO 1″ raises from 15288 to 19968 (30%), and for “SELECT 1″ we have increase from 13410 to respectable 16824 (25%). Not bad for a single line fix.

While the one-liner makes the Drizzle JDBC faster, with slightly better numbers than ConnectorJ, it is still not as fast as MariaDB.

In the MariaDB JDBC connector, there were a couple of improvements to performance which were made since forking. One of the early improvements was to avoid copying data unnecessarily when sending, and to decrease the number of byte buffers.  Another improvement came recently, after profiling and finding that parsing Field packets is expensive (mostly due to the construction of Strings for column name, aliases, and etc…). The improvement was lazy parsing,  delaying string construction, and avoiding it entirely in most cases. For example, if column names are not used, and rows are accessed using integer indexes in ResultSet.getXXX(int i), the metadata won’t be fully parsed. Also, perhaps there were some other fixes that I do not remember anymore.

Can we further increase the QPS?

We can try. First, statements can be prepared. MariaDB and Drizzle so far only provide client-side prepared statements (ConnectorJ can do both client and server-side prepared statements) but using them saves having to convert the query to bytes, and JDBC escapes preprocessing. From now on I’ll stay just with “DO 1″ which proved to be the fastest query. Trying it on MariaDB driver shows some minimal QPS increase 22104 (not prepared) vs 22183 (prepared), or 0.3%. Slightly more on ConnectorJ (19543 vs 20096, or 2.9%). Nothing revolutionary so far.

But, We still have not used all of the options in this (admittedly silly) quest for maximizing the performance of “DO 1″. Recall that ConnectorJ can support named pipes on Windows, which are allegedly much faster than TCP connections. Restart server with named pipe, set JDBC URL to “jdbc:mysql:///?socketFactory=com.mysql.jdbc.NamedPipeSocketFactory&namedPipePath=\\\\.\\Pipe\\MySQL&user=root&useConfigs=maxPerformance”, and rerun the test with 1000000 prepared queries. Now the QPS grew to 29542! That is strong, and is a 33% improvement compared to the best result seen so far. Yet, unfortunately, still no cigar, since JVM dumps a stack trace when the named pipe connection is closed. This is a “Won’t fix” (chalked off as a JVM problem) MySQL bug Bug#62518, which renders named pipe support almost useless – though maybe there is a trick to shut up th JVM somehow in this case, but I do not know of such a trick.

How fast is C client library in comparison?

Out of curiosity, I also tested how the native client compares to JDBC. With the TCP protocol, it does slightly better than the fastest JDBC (MariaDB, prepared), but it is not a huge margin – 24063 QPS vs 22183 (8.5% difference), and I believe Java drivers could improve further.
With named pipe, QPS is 33122, which is ~12% better than what ConnectorJ could do, if pipes worked properly there.

Accessing benchmark program

I put the benchmark program on Launchpad, together with the drivers. If you’re on Windows, and if you have a server running on port 3306, and the ‘root’ user doesn’t have a password, you can just branch the repository and run bench_all.bat. Those of you who are using other operating systems, I trust you to be able to quickly rewrite the batch files as shell scripts.

Download MariaDB Java Client 1.1.2

Release Notes Changelog About the MariaDB Java Client

Bugs fixed in this release

MariaDB Java Client 1.1.2 is a bug fix release. Some of the bugs fixed include the following:

• PreparedStatement.getMetaData() will now return correct ResultSet metadata, also prior to statement execution. In this case, to retrieve metadata, statement will be prepared on the server side. (CONJ-21)

• Performance enhancement : Connection.getAutoCommit() will not more issue not issue “select @@autocommit” query anymore. This information is also available in on status flags sent with OK/EOF protocol packets. Also, Connection.setAutoCommit() will be a no-op, if autocommit status is already the same as desired one (CONJ-30)

• Fixed several issues with CLOB datatype and ResultSet.setCharacterStream() -non-ASCII character could get lost after setCharacterStream(). CLOB was errneously sent to server as binary (using _BINARY introducer). (CONJ-31)

• CHAR BINARY and VARCHAR BINARY were errnously handled as binary type. They are now correctly treated as CHAR/VARCHAR with binary collation. (CONJ-28)

• Blob.getBinaryStream() could return blob with incorrect size (CONJ-31)

• Fix server version string for MariaDB Server 10.0.2 (CONJ-32)

Many developers use master-slave replication to solve a number of different problems, including problems with performance, supporting the backup of different databases, and as a part of a larger solution to alleviate system failures. Traditionally, master-slave replication is done with real servers, but it can also be done with cloud database servers. This guest post from Jelastic (originally published here) describes how to set up MariaDB master-slave replication using their Jelastic PaaS (Platform as a Service).

Replication Overview

Master-slave replication enables data from one database server (the master) to be replicated to one or more other database servers (the slaves). The master logs the updates, which then ripple through to the slaves. The slave outputs a message stating that it has received the update successfully, thus allowing the sending of subsequent updates. Master-slave replication can be either synchronous or asynchronous. The difference is simply the timing of propagation of changes. If the changes are made to the master and slave at the same time, it is synchronous. If changes are queued up and written later, it is asynchronous.

Usage targets for replication in MariaDB include:

• Scale-out solutions — spreading the load among multiple slaves to improve performance. In this environment, all writes and updates must take place on the master server. Reads, however, may take place on one or more slaves. This model can improve the performance of writes (since the master is dedicated to updates), while dramatically increasing read speed across an increasing number of slaves.
• Data security — as data is replicated to the slave, and the slave can pause the replication process, it is possible to run backup services on the slave without corrupting the corresponding master data.
• Analytics — live data can be created on the master, while the analysis of the information can take place on the slave without affecting the performance of the master.
• Long-distance data distribution — if a branch office would like to work with a copy of our main data, we can use replication to create a local copy of the data for their use without requiring permanent access to the master.

Let’s examine a few examples on how we can use such replication and take advantage of it:

• Backups: to use replication as a backup solution, replicate data from the master to a slave, and then back up the data slave. The slave can be paused and shut down without affecting the running operation of the master, so we can produce an effective snapshot of “live” data that would otherwise require the master to be shut down.
• Scale-out: we can use replication as a scale-out solution; that is, where we want to split up the load of database queries across multiple database servers, within some reasonable limitations. Because replication works from the distribution of one master to one or more slaves, using replication for scale-out works best in an environment where we have a high number of reads and low number of writes/updates. Most Web sites fit into this category, where users are browsing the Web site, reading articles, posts, or viewing products. Updates only occur during session management, or when making a purchase or adding a comment/message to a forum. Replication in this situation enables us to distribute the reads over the replication slaves, while still enabling our web servers to communicate with the replication master when a write is required.
• Spreading the load: there may be situations when we have a single master and want to replicate different databases to different slaves. For example, we may want to distribute different sales data to different departments to help spread the load during data analysis.
• Increasing the performance: as the number of slaves connecting to a master increases, the load, although minimal, also increases, as each slave uses a client connection to the master. Also, as each slave must receive a full copy of the master binary log, the network load on the master may also increase and create a bottleneck. If we are using a large number of slaves connected to one master, and that master is also busy processing requests (for example, as a part of a scale-out solution), then we may want to improve the performance of the replication process. One way to improve the performance of the replication process is to create a deeper replication structure that enables the master to replicate to only one slave, and for the remaining slaves to connect to this primary slave for their individual replication requirements.
• Failover alleviating: We can set up a master and a slave (or several slaves), and then write a script that monitors the master to check whether it is up. Then instruct our applications and the slaves to change master in case of failure.
• Security: we can use SSL for encrypting the transfer of the binary log required during replication, but both the master and the slave must support SSL network connections. If either host does not support SSL connections, replication through an SSL connection is not possible. Setting up replication using an SSL connection is similar to setting up a server and client using SSL. We must obtain (or create) a suitable security certificate that we can use on the master, and a similar certificate (from the same certificate authority) on each slave.

Now let’s examine a simple example on how to configure master-slave replication on Jelastic PaaS.

Create environments

*The instructions below only mention MariaDB but they are fully suitable for MySQL database servers.

First of all we create two environments in Jelastic for our master and slave databases.

1. Go to jelastic.com and sign up if we haven’t done it yet or log in with our Jelastic credentials by clicking the Sign In link on the page.
2. Ask Jelastic to create a new environment.
   
3. In the Environment topology dialog pick MariaDB as a database we want to use. Set the cloudlet limit and type the name of our first environment, for example, masterbase.
   Wait just a minute for our environment to be created.

   

4. In the same way create one more environment with MariaDB or just clone it. Let’s name it slavebase. BTW, it will be located on the other hardnode, which is even more secure and reliable for storing our data. Now we have two identical environments with two databases.

   

Configure master database

Let’s configure our master database now.

1. Click the config button for our master database.
   
2. Navigate to my.cnf file and add the following properties:

   server-id = 1
   log-bin = mysql-bin
   binlog-format=mixed
   

   

   We use binlog format “mixed” (binlog-format=mixed) to allow a replication of operations with foreign keys.

   Note: Do not use binlog format “statement” (otherwise we will get errors later on!)

3. Save the changes and restart MariaDB in order to apply the new configuration parameters.

   

4. Click the Open in browser button for MariaDB. When we created the database, Jelastic will have sent us an email with credentials to it. Log in using these credentials.
5. Navigate to the Replication tab and click on Add slave replication user.

   

6. Specify the name and password for our slave replication user and click Go.

   

   Now our slave user is successfully created.

   

Configure slave database

Let’s go back to the Jelastic dashboard and configure our slave database.

1. Click the config button for our slave database.

   

2. Navigate to the my.cnf file and add the following strings:

   server-id = 2
   slave-skip-errors = all
   

   

   We allow our slave database to skip all errors from the master (slave-skip-errors = all) in order not to stop normal slave operations in case of errors on the master. Don’t set this option during development as we want to see and catch all bugs, large and small, before moving our code from development to production. Production code should not have any big bugs, so setting this can be useful in that it allows replication to keep working through small, innocuous errors which could stop production and cost money (downtime, lost sales, etc…). This option can be dangerous however, so proceed with caution, and make regular backups.

3. Save the changes and restart our slave database server in order to apply the new configuration parameters.

   

4. Navigate to phpMyAdmin using the credentials which Jelastic sent us when we created the environment for our slave database.
5. Go to the Replication tab click configure for Slave replication.

   

6. Configure our master server (enter the name, the password and the host of our slave replication user).

   

   Now our master server is configured.

   

7. Click on Control slave -> Full start for the slave server in order to run Slave SQL and Slave IO threads.

   

8. Check the slave status table to ensure that everything is ok.

   

Check the results

We have to ensure now that master-slave replication works for our databases.

1. Let’s create a new database (e.g. Jelastic) in our master database.

   

2. Navigate to the slave database and we’ll see that the new database was successfully replicated.

   

Connection to master-slave

Here are two examples on how to connect to our master and slave databases from Java and PHP applications.

1. Our first example is the code of a Java application which connects to our master and slave databases.

   Database_config.cfg:

   master_host=jdbc:mysql://mariadb-master-host/mysql
   master_username=root
   master_password=abcABC123
   slave_host=jdbc:mysql://mariadb-slave-host/mysql
   slave_username=root
   slave_password=abcABC123
   
   driver=com.mysql.jdbc.Driver
   

   Dbmanager.java:

   package com.jelastic.test;
   
   import java.io.FileInputStream;
   import java.io.IOException;
   import java.sql.*;
   import java.util.ArrayList;
   import java.util.List;
   import java.util.Properties;
   import java.util.logging.Level;
   import java.util.logging.Logger;
   
   public class DbManager {
   
   private final static String createDatabase = "CREATE SCHEMA IF NOT EXISTS jelastic";
   private final static String showDatabases = "SHOW DATABASES";
   
   public Connection createMasterConnection() throws IOException, ClassNotFoundException, SQLException {
   Connection masterConnection;
   Properties prop = new Properties();
   prop.load(new FileInputStream(System.getProperty("user.home") + "/database_config.cfg"));
   String master_host = prop.getProperty("master_host").toString();
   String master_username = prop.getProperty("master_username").toString();
   String master_password = prop.getProperty("master_password").toString();
   String driver = prop.getProperty("driver").toString();
   
   Class.forName(driver);
   masterConnection = DriverManager.getConnection(master_host, master_username, master_password);
   return masterConnection;
   }
   
   public Connection createSlaveConnection() throws IOException, ClassNotFoundException, SQLException {
   Connection slaveConnection;
   Properties prop = new Properties();
   prop.load(new FileInputStream(System.getProperty("user.home") + "/database_config.cfg"));
   String slave_host = prop.getProperty("slave_host").toString();
   String slave_username = prop.getProperty("slave_username").toString();
   String slave_password = prop.getProperty("slave_password").toString();
   String driver = prop.getProperty("driver").toString();
   
   Class.forName(driver);
   slaveConnection = DriverManager.getConnection(slave_host, slave_username, slave_password);
   return slaveConnection;
   }
   
   public boolean runSqlStatementOnMaster() {
   boolean execute = false;
   Statement statement = null;
   try {
   statement = createMasterConnection().createStatement();
   execute = statement.execute(createDatabase);
   } catch (IOException ex) {
   Logger.getLogger(DbManager.class.getName()).log(Level.SEVERE, null, ex);
   } catch (ClassNotFoundException ex) {
   Logger.getLogger(DbManager.class.getName()).log(Level.SEVERE, null, ex);
   } catch (SQLException ex) {
   ex.printStackTrace();
   } finally {
   if (statement != null) {
   try {
   statement.close();
   } catch (SQLException e) {
   e.printStackTrace();
   }
   }
   }
   return execute;
   }
   
   public List<String> runSqlStatementOnSlave() {
   List<String> stringList = new ArrayList<String>();
   Statement statement = null;
   ResultSet resultSet = null;
   try {
   statement = createSlaveConnection().createStatement();
   resultSet = statement.executeQuery(showDatabases);
   while (resultSet.next()) {
   stringList.add(resultSet.getString(1));
   }
   } catch (IOException ex) {
   Logger.getLogger(DbManager.class.getName()).log(Level.SEVERE, null, ex);
   } catch (ClassNotFoundException ex) {
   Logger.getLogger(DbManager.class.getName()).log(Level.SEVERE, null, ex);
   } catch (SQLException ex) {
   ex.printStackTrace();
   } finally {
   if (resultSet != null) {
   try {
   resultSet.close();
   } catch (SQLException e) {
   e.printStackTrace();
   }
   }
   if (statement != null) {
   try {
   statement.close();
   } catch (SQLException e) {
   e.printStackTrace();
   }
   }
   }
   return stringList;
   }
   }
   

2. Our second example is a connection to our master and slave databases in a PHP application:

   <?php
   /* Master settings */
   $master_server = "xx.xxx.x.xx";
   $master_username = "root";
   $master_password = "abcABC123"; /* Slave settings */
   $slave_server = "xx.xxx.x.xx";
   $slave_username = "root";
   $slave_password = "abcABC123";$link_to_master = mysqli_connect(
   $master_server,
   $master_username,
   $master_password,
   'mysql');
   
   if (!$link_to_master) {
   printf("Unable to connect master database server. Error: %sn", mysqli_connect_error());
   exit;
   }
   
   $link_to_slave = mysqli_connect(
   $slave_server,
   $slave_username,
   $slave_password,
   'mysql');
   
   if (!$link_to_slave) {
   printf("Unable to connect slave database server. Error: %sn", mysqli_connect_error());
   exit;
   }
   
   print("
   Creating database with name Jelastic on Master node ");
   
   $result = mysqli_query($link_to_master, 'CREATE DATABASE JelasticX');
   
   sleep (3);
   
   print("
   Checking if created database was replciated to slave ");
   
   if ($result = mysqli_query($link_to_slave, 'SHOW DATABASES LIKE "JelasticX"')) {
   $result_text = mysqli_fetch_array($result);
   print ("
   Replicated database is ".$result_text[0]);
   }
   
   mysqli_close($link_to_master);
   mysqli_close($link_to_slave);
   ?>
   

Conclusion

Database replication with MariaDB adds redundancy, helps to ensure high availability, simplifies certain administrative tasks such as backups, may increase performance, and much more. It’s become easy to configure database replication in the cloud, only few minutes and everything is ready. Enjoy!

When dealing with high performance, low latency storage devices, such as SSD cards, one finds bottlenecks in new places. This is a story about such a bottle neck and how to work around it.

One unique feature of InnoDB is the double write buffer. This buffer was implemented to recover from half-written pages. This can happen in case of a power failure while InnoDB is writing a page (16KB = 32 sectors) to disk. On reading that page, InnoDB would be able to discover the corruption from the mismatch of the page checksum. However in order to recover, an intact copy of the page would be needed.

The double write buffer provides such a copy. Whenever InnoDB flushes a page to disk, it is first written to the double write buffer. Only when the buffer is safely flushed to disk, InnoDB writes the page to the final destination. When recovering, InnoDB scans the double write buffer and for each valid page in the buffer checks if the page in the data file is valid too.

Both the checksum calculation and the double writing consume time and thus reduce the performance of page flushing. The effect becomes visible only with fast storage and heavy write load. For testing it is possible to disable both features. However I strongly discourage from using this in a productive environment.

Currently, in order to use atomic writes, it is necessary to use the DirectFS file system, which is a part of the Fusion IO SDK. Wlad Vaintroub from Monty Program AB, in cooperation with FusionIO developers, implemented the necessary changes in InnoDB/XtraDB to use the new feature. If the all table spaces reside on directFS/FusionIO and thus support atomic writes, the new variable innodb_use_atomic_writes=1 will switch to using atomic writes instead of the double write buffer. The patch is already in MariaDB-10.0.2 and will also be included in MariaDB 5.5.31. The user documentation of the feature is available in this Knowlegde Base article.

Now for numbers! First tests have shown that atomic writes are of little to no benefit for small data sets (data set fits into RAM). However with fast SSD one can now afford to have a much bigger data set (compared to RAM size) because reads and writes are much faster.

The following numbers are for the Sysbench OLTP benchmark, using a data set of 100GB (400 Mio rows in 16 tables) but only 16GB of RAM in the InnoDB buffer pool. Performance was of course slower than with the usual 10GB data set. Numbers:

For the big data set the following configurations have been compared:

• double write, InnoDB page checksum
• atomic write, InnoDB page checksum
• atomic write, XtraDB fast page checksum
• atomic write, no page checksum

Conclusions:

• enabling atomic writes yields 30% better write performance out of the box
• using the fast checksum from XtraDB boosts this to nearly 50%
• disabling checksums completely is only marginally better

Benchmark details:

MariaDB-5.5.30 (lp trunk). Sysbench-0.5 (Lua enabled). Complex OLTP benchmark. 400 mio rows in 16 tables. 16GB InnoDB buffer pool, 4G redo logs.

As always the benchmark scripts and results are available to anybody.

Download MariaDB 10.0.2

Release Notes  Changelog  Overview of 10.0

APT and YUM Repository Configuration Generator

About this Release

MariaDB 10.0 is the development version of MariaDB. It is built on the MariaDB 5.5 series with backported features from MySQL 5.6 and entirely new features not found anywhere else.

This is the third 10.0-based release, and we are releasing it now to get it into the hands of any who might want to test it. Not all features planned for the MariaDB 10.0 series are included in this release. Additional features will be pushed in future releases. See the release notes and changelog for details on what is new in this release.

User Feedback plugin

MariaDB includes a User Feedback plugin. This plugin is disabled by default. If enabled, it submits basic, completely anonymous MariaDB usage information. This information is used by the developers to track trends in MariaDB usage to better guide development efforts.

If you would like to help make MariaDB better, please add “feedback=ON” to your my.cnf (my.ini on Windows) file!

See the User Feedback Plugin page for more information.

Quality

The project always strives for quality, but in reality, nothing is
perfect. Please take time to report any issues you encounter at:

http://mariadb.org/jira

We hope you enjoy MariaDB!

A key takeaway is that SkySQL will work alongside & collaborate closely with the MariaDB Foundation, plus continue to invest in the development of the MariaDB server.

The Board members are now Rasmus Johansson, Andrew Katz, Simon Phipps, Michael “Monty” Widenius, and Jeremy Zawodny. The first act of the new interim Board was to appoint Johansson as Chair and Phipps as Secretary and Chief Executive Officer.

The Board bring together a wealth of experience. Johansson is an experienced leader within the MariaDB developer community. Katz is a well-known expert in open source legal matters and has been acting as COO of the Foundation until now. Phipps brings experience of IBM and Sun Microsystems as well as leadership in non-profit organisations such as the Open Source Initiative. Widenius is a veteran of the MariaDB and MySQL projects, having founded both. Zawodny is a long-time implementer of solutions using both the MySQL and MariaDB databases; he is currently at Craigslist and was formerly at Yahoo.

The interim Board plans to adopt a membership-based governance model that includes a diversity of stakeholders, similar to that used by the Eclipse Foundation, so has invited its Executive Director Mike Milinkovich to directly advise on governance matters. The Board aims to devise and publicly review suitable governance rules between now and July and then to have MariaDB Foundation members and contributors elect a new Board using those rules.

The Foundation invites companies wishing to contribute to the success of the MariaDB database project to contact them. Opportunities both for sponsorship and for engagement in the governance of the new Foundation are available. Please contact foundation at the mariadb org domain.

Press enquiries:  press at mariadb org   (release)

Team MariaDB talks:

1. (Monday) Advanced Query Optimizer Tuning & Analysis, a tutorial by Sergei Petrunia & Timour Katchaounov
2. (Tuesday) MariaDB Cassandra Interoperability, a talk by Sergei Petrunia & Colin Charles
3. (Tuesday) Engine-independent persistent statistics with histograms in MariaDB, a talk by Igor Babaev
4. (Tuesday) Replication Changes in MariaDB, a talk by Michael “Monty” Widenius & Sergei Golubchik
5. (Tuesday) a BoF at 7pm, in Ballroom F, where you can be sure to get black vodka!
6. (Wednesday) MariaDB 10.0 & What’s New With the Project, a talk by Michael “Monty” Widenius & Colin Charles
7. (Thursday) Storage Engines & Other Plugins: What’s New? a talk by Sergei Golubchik

Many others are also speaking about MariaDB, and some of the team have talks that aren’t exactly related to MariaDB and those will be covered in another post.

In the meantime, what are you waiting for, register now! Use the code SponsorSQL to get 15% off the price as well.