Last weekend Vadim from Percona published his MariaDB 5.3.4 benchmark results. As the new benchmark guy at Monty Program I take this oportunity to add some more results of my own.

One question in the comments to Vadim was if it is fair to compare MariaDB-5.3 with MySQL-5.5. Or if this comparison should be done with MySQL-5.1. The answer is: it does not matter much. MySQL-5.5 and MySQL-5.1 show very similar results in the Sysbench OLTP benchmark.

So I created a Sysbench environment pretty much like Vadims and tested the following versions of the MySQL Server:

  • MariaDB-5.3.4 – the Monty Program release candidate, both with XtraDB and the InnoDB plugin
  • Percona-Server 5.1.61 because it is based on the same XtraDB version as MariaDB-5.3
  • Percona-Server 5.5.20 – the current Percona flagship
  • MySQL-5.5.20 – the current Oracle flagship

The result:

Indeed MariaDB-5.3.4 scales significantly worse than MySQL-5.5.20. However Percona-Server 5.1.61 behaves nearly identical and Percona-Server 5.5.20 is only slightly better. If we run MariaDB-5.3.4 with the InnoDB plugin instead of the default XtraDB, then MariaDB outperforms Percona-Server 5.5.20.

As a consequence, we will deliver the InnoDB plugin with future binary releases of MariaDB. MySQL-5.1.61 with the InnoDB plugin (not shown in the graph) is between MariaDB-5.3.4(InnoDB-plugin) and MySQL-5.5.20. We are currently investigating why MySQL-5.1 is faster than MariaDB-5.3, even with the same storage engine.

Please keep in mind that Sysbench is a highly synthetical benchmark. Real world workload will probably yield different results. Also this benchmark used solid state disks. A storage system with spinning disks might saturate much earlier and would then flatten out the peaks.

If I should draw a conclusion, then I would say that for highest concurrency levels you should go for MySQL-5.5. Or wait for MariaDB-5.5 :)

If your workload includes complex (sub)queries, then you will probably benefit more from MariaDBs new optimizer features. And if you are running replication, then MariaDB 5.3 with group-commit will greatly outperform any other MySQL version.

Benchmark details: sysbench OLTP multi-table. 16 tables with 250K rows each. 20G InnoDB buffer pool. Data and logs on SSD. Dual socket hexacore Xeon with HT enabled (24 logical cores). You can download the tarball with benchmark scripts, config and raw results.

24 thoughts on “Benchmarking MariaDB-5.3.4

  1. When 5.5 came out, Oracle claimed 369% better performance on sysbench RW than 5.1, with or without plugin.
    http://downloads.mysql.com/presentations/mysql-sunday/MySQL_OOW2010_Keynote_Final.pdf
    (page 16 here)

    Percona Server claims to be 40% faster than MySQL in overall throughput, though on a different benchmark.
    http://www.percona.com/software/percona-server/benchmarks/

    This result here shows that none of this would be true, 5.1 and 5.5 performance is similar, while Percona Server is 40% *slower* in overall throughput, and XtraDB by Percona being factor #1 in slowing down MariaDB.

    That’s funny.

    I only believe benchmarks I run myself from now on..

    • In this benchmark are activated the new 5.5 features like multiple buffer pools and new I/O threads?

      • I do know, I have not seen my.cnf or sysbench parameters from Oracle, only the graph.

      • I/O threads yes (8 readers and 8 writers). Multiple buffer pools no.

  2. > This result here shows that none of this would be true, 5.1 and 5.5 performance is similar…

    369% I think is for 1024 threads, which wasn’t tested here. At 256, results appear to be similar.

  3. The Percona white paper that claims to be 40% faster over same MySQL version is focused on SSD performance. The white paper makes it clear that on spinning disks there will be much less of a benefit.

    Since this test was in fact using SSD, it’s a good question why this 40% gain can’t be seen here? The answer is, Percona adds specific SSD opitmized InnoDB flushing algorithm (and maybe something else) that needs to be specifically turned on. Since this test probably used same configuration file for all tests, it didn’t benefit from this optimization.

    Axel here doesn’t say if SSD was SAS connected or PCIe connected – makes a big difference. Also not explicit, it seems this data set actually fits into memory?

    That Percona Server would be significantly slower than the matching MySQL version seems very unlikely and suggests there is a flaw in this test. (I didn’t comment on Vadim’s test, but likewise there could obviously be some flaw in that test too.)

    • Since this test was in fact using SSD, it’s a good question why this 40% gain can’t be seen here? The answer is, Percona adds specific SSD opitmized InnoDB flushing algorithm (and maybe something else) that needs to be specifically turned on.

      It is true that no extra configuration was used to help XtraDB. But then OTOH Vadim didn’t do that too. Indeed it would be interesting to rerun the benchmark with optimized XtraDB settings.

      Axel here doesn’t say if SSD was SAS connected or PCIe connected – makes a big difference. Also not explicit, it seems this data set actually fits into memory?

      The disk used in this test is a RAID0 over 3 SAS SSDs. The data set indeed fits into memory (again: just like in Vadims test). The OLTP rw benchmark writes significant amounts of data. And while the log is consecutive I/O, the checkpointing is not. I don’t remember the number of I/Os per sec but I’ve seen up to 100M/s writes to the disk. You need a rather big disk cache and/or many disks if you want to stem this amount of random writes with spinning disks.

  4. Pingback: Benchmarking MariaDB-5.3.4 | MySQL | Syngu

  5. Did you test Oracle MySQL Enterprise or Community version?

    • This test used the Community Server from Oracle. Also everything was compiled from source with the same (release build) options to eliminate any influence of compiler and libraries used.

      In those tests it did not make a difference. Up to 5.5 there are no functional differences between Community and Enterprise. This will change when the thread pool becomes GA because Oracle (unlike MariaDB) will not put this into the Community Server.

    • Ilari, the two are identical, save for the commercial license on the Enterprise edition.

    • How does MariaDB 5.2.10 fit into the above comparisons ?

      MariaDB-5.2, MariaDB-5.3 and Percona-5.1 all behaved nearly identical (except some statistical fluctuations). In fact this was expected because all 3 use the same XtraDB version. It seems XtraDB is the limiting factor here because MariaDB-5.3.4 was so much better with the InnoDB plugin. The latter is quite strange because XtraDB is based on the InnoDB plugin code.

  6. Pingback: MariaDB 5.3.4 RC vs MySQL 5.5.20 - 5.1 XtraDB Engine vs InnoDB Plugin | vbtechsupport.com

  7. Pingback: Benchmarking MariaDB-5.3.4 | dasini.net - Journal d'un expert MySQL

  8. Great analysis and presentation of results. Providing the best performance at various sizes and loads is a real challenge, it is nice to see that MariaDB will include the InnoDB plugin going foward. It is also refreshing to see your recommendation of someone elses product (MySQL-5.5), as you say “for now”.

  9. Pingback: Percona Server vs MySQL on Intel 320 SSD | Weez.com

  10. BTW, the MariaDB group commit patch is also in Percona Server 5.5.

  11. Pingback: Some news about MySQL in obs » Michal Hrušecký

  12. Pingback: MariaDB-5.3 optimizer benchmark | Monty Program Group Blog

  13. Pingback: Percona Server vs MySQL on Intel 320 SSD - MySQL Performance Blog

Comments are closed.