Drupal’s vulnerability reports are not signs of security weakness

I’ve been tweeting back and forth with Alex Limi, one of the founders of Plone, about the validity of the security analysis from a CMS comparison report that includes Plone and Drupal. He’s proud of Plone’s infrequent vulnerability notices; it had two in the last year. Drupal had 26. Alex also cited a related IBM report on security in a later tweet.

While both reports above seem to identify Drupal (and Joomla! and WordPress, to be fair) as having notably bad security, they’re also both based on one superficial metric: self-reported vulnerabilities. Neither severity nor response time nor history of actual exploitation factored in.

The vulnerabilities in question have all (long) been fixed in Drupal, so Alex’s argument could only be that past occurrences of vulnerability reports are a predictor of future security problems. Unfortunately, he merely begs the question of correlation without answering it, and that’s only the beginning of the problems with his argument.

Even if vulnerability reports were perfect indicators of future risk, vulnerability self-reporting carries a high conflict-of-interest. This conflict is especially strong when, like Alex, you argue that the quantity of reports you issue should be held against your project.

The Drupal community (both in developers and users) is much larger than the Plone one, and the two continue to diverge.

Many of us in the free software community are familiar with Linus’s Law: “given enough eyeballs, all bugs are shallow.” Vulnerabilities are merely a special class of bugs. All other things being equal, Drupal’s larger developer and user base should be expected to find and publish more vulnerability reports than Plone’s.

But Drupal had more than just community growth in 2008; it also experienced unprecedented security review thanks to work by Barry Jaspan, who presented his findings at Drupalcon Szeged 2008. Barry subjected Drupal’s core code to static and dynamic analysis, resulting in the discovery of several vulnerabilities. Has Plone undergone similar scrutiny? A quick search on Google didn’t uncover anything of the sort.

Despite Alex’s thoughts on my stubbornness, I am open to an honest evaluation of Drupal’s security versus similar tools. I’m just not willing to base the debate on a superficial metric of such questionable importance.