This is both hilarious and sad. The new MySQL 5.7 milestone release presents a new feature — replication filters are now dynamic. This is a great and long awaited feature, no doubt about it.

In short, for years MySQL slaves could filter the incoming stream of replication events based on the database or table name these events were applicable to. These filters were configured using the my.cnf file (or command-line), in particular with the following variables:

Naturally, users wanted to be able to change the values of these options without having to restart the server. And eventually Davi Arnaut implemented it. MariaDB got it as a contribution back in version 5.5.22. Since that release these options in MariaDB have been dynamic system variables and one could change them at will. For example,

Now, I understand that sometimes we might implement a certain useful feature first. And I understand that in that case Oracle might want to have it, while not looking like they’re following MariaDB. Of course not! The feature must be totally original. With completely incompatible syntax. Most recent examples would be EXPLAIN FOR CONNECTION in MySQL 5.7.2 about a year after SHOW EXPLAIN was introduced in MariaDB, and multi-source replication using “channels”, while MariaDB has already been using named slaves for quite a while.

But really, there should be some logic in it. Admittedly, EXPLAIN FOR CONNECTION isn’t worse than SHOW EXPLAIN. And multi-source “channels” look just fine, syntax-wise. Still, if there is a command-line option, there (almost always) is a corresponding system variable, visible in SHOW VARIABLES. One (almost always) changes the value of the option by assigning a new value to the corresponding variable. There is only one way of doing it.

Alas, we have taken this way in MariaDB. So, by strictly adhering to the NIH principle (and, I suspect, internal Oracle policies), MySQL developers simply had to invent a different way of changing server options. Lo and behold, CHANGE REPLICATION FILTER statement. Examples are:

Well done. Common sense? Nope, not in this case.

I’m getting more and more concerned about the current Oracle approach to MySQL security. And the fact that I was solely responsible for the security@mysql.com for about ten years, doesn’t make it easier, on the contrary, it only emphasizes changes in the attitude.

Starting from the obvious — somewhat slower response to critical bug fixes, which can be expected, Oracle is a big company, right? Very little information about security vulnerabilities is disclosed, CPUs are carefully stripped from anything that might help to understand the problem, it takes hours to map them to code changes. Heck, even test cases are kept private now. This seriously smells Security through Obscurity to me.

But all that isn’t news. Here I want to talk about the recent wave of security vulnerabilities. If you search for, say, “mysql security full-disclosure”, you will find the original postings to the full-disclosure@lists.grok.org.uk mailing list, as well as further discussion and my replies. Not Oracle replies, though. In short, the vulnerabilities  announced in the early December were:

  1. CVE-2012-5611 MySQL (Linux) Stack based buffer overrun PoC Zeroday
  2. CVE-2012-5612 MySQL (Linux) Heap Based Overrun PoC Zeroday
  3. CVE-2012-5613 MySQL (Linux) Database Privilege Elevation Zeroday
  4. CVE-2012-5614 MySQL Denial of Service Zeroday PoC
  5. CVE-2012-5615 MySQL Remote Preauth User Enumeration Zeroday
  6. CVE-2012-5627 MySQL Local/Remote FAST Account Password Cracking

All posted to the full disclosure mailing list, accompanied by exploits in Perl. Out of these issues, the 3rd was a configuration issue (FILE privilege granted to untrusted user, no --secure-file-priv used), 4th was already fixed in the latest MariaDB and MySQL releases (the reporter used an outdated MySQL version), 1st — the most dangerous one — was already fixed in the latest MariaDB release (we were lucky to discover this issue independently few weeks ealier and immediately released a fix). Everything else was valid, exploitable, and very much public, after the original postings were quoted in various blogs and news articles.

In a due time MySQL 5.5.29 is released. The 1st issue — CVE-2012-5611 — looks fixed. The second one — CVE-2012-5612 — is fixed too. Hurra! But 5th and 6th vulnerabilities — CVE-2012-5615 and CVE-2012-5627  — are not fixed. Which, apparently, leaves MySQL installations at the mercy of any script-kiddie, who can use google. And the 1st one (the worst of all) is fixed incompletely — the fact, that is simply impossible to miss, given the amount of testing that MySQL continuous integration framework (Pushbuild2) is performing.

Really, I wouldn’t expect a database vendor to blatantly ignore publicly announced vulnerabilities, that have CVE identifiers, known and easily available exploits, and were posted and reposted all over. So, Oracle, what was that, eh?

2013/Feb/05 UPDATE: MySQL 5.5.30 doesn’t seem to fix these issues either. Two months after the vulnerabilities went public. After MariaDB 5.5.29 release (with the fixes) and this blog post! I don’t think the Hanlon’s razor can explain that.

About a week ago I was looking at MySQL 5.5.27, and noticed a curious thing. Despite the fact that the new MySQL release contained its usual share of bug fixes, not a single one of them was accompanied with a test case.

Now, let me tell you something about tests. For many years MySQL was using its own testing framework, called mysql-test. The first version was written as early as 1999. Over the years it has accumulated a lot of tests. Tests for new features and regression tests — those that guarantee that a bug, once fixed, will never ever show up again. We had pretty strict policies about it in MySQL AB (and, later, Sun Microsystems) — every new bug fix always had to come with a test case for the bug. And because these tests were always run on many platforms for every push (by the continuous integration tool called Pushbuild — developed in-house by Kristian Nielsen) we were reasonably sure that any bug, once fixed, will stay fixed forever. I’m not with MySQL anymore, but I still cannot imagine that Oracle would weaken that rule. So, it must be something else then.

One of the changes that 5.5.27 brings in an extension to the mysql-test-run script — the driver script of the mysql-test suite — which makes it look for test cases in a new directory. In addition to the usual location (that is, the mysql-test/ directory in the source tree), it will now look for test cases in the internal/mysql-test/ directory. Does this mean that test cases are no longer open source? Oracle did not reply to my question. But indeed, there is evidence that this guess is true. For example, this commit mail shows that new test cases, indeed, go in this “internal” directory, which is not included in the MySQL source distribution.

MySQL test cases were always an important part of the MySQL source tree. They were particularly useful for storage engine developers and for other people extending MySQL, for example, at Facebook, Twitter, and Taobao.  But also for Linux distributions which add their patches to the base MySQL, and even to users, who don’t modify the sources — they still want to confirm that a particular bug was fixed or that their custom-built binary has no obvious flaws.

In May, at the Ubuntu Developer Summit in Oakland, Oracle had 7 representatives there, and they promised that Oracle will be more contributor- and distribution-friendly. It is sad to see that instead of that the MySQL source tree is being closed down.

MySQL AB was never very good at building a development community around the product. There weren’t many MySQL developers or contributors to the project outside of MySQL AB, and the company didn’t do much to increase their number. But now Oracle has noticed them — and it intentionally kills whatever is left of the MySQL development community. Without test cases MySQL becomes as opaque to external developers as any piece of closed source software, and only those most experienced and familiar with the MySQL code base will be able to continue working with it.

UPDATE: It’s difficult to find anything more valuable to external developers than test cases. But arguably the revision history is. It groups changes to these millions of lines of source code into change sets, one change set per a distinct feature or a particular bug fix. It allows to see who changed a specific line of code, when, and why. And it seems that Oracle is going to keep this information to itself too. Public MySQL trees on launchpad with the revision history are not being updated.

Now, is there a single successful Open Source project with no developer community at all ?

UPDATE 2: We’re not the only ones this issue affects. See also: