Drupal's vulnerability reports are not signs of security weakness
Photo by loop_oh on Flickr.
I’ve been tweeting back and forth with Alex Limi, one of the founders of Plone, about the validity of the security analysis from a CMS comparison report that includes Plone and Drupal. He’s proud of Plone’s infrequent vulnerability notices; it had two in the last year. Drupal had 26. Alex also cited a related IBM report on security in a later tweet.
While both reports above seem to identify Drupal (and Joomla! and WordPress, to be fair) as having notably bad security, they’re also both based on one superficial metric: self-reported vulnerabilities. Neither severity nor response time nor history of actual exploitation factored in.
The vulnerabilities in question have all (long) been fixed in Drupal, so Alex’s argument could only be that past occurrences of vulnerability reports are a predictor of future security problems. Unfortunately, he merely begs the question of correlation without answering it, and that’s only the beginning of the problems with his argument.
Even if vulnerability reports were perfect indicators of future risk, vulnerability self-reporting carries a high conflict-of-interest. This conflict is especially strong when, like Alex, you argue that the quantity of reports you issue should be held against your project.
The Drupal community (both in developers and users) is much larger than the Plone one, and the two continue to diverge:
Drupal is red, and Plone is blue (to state the obvious).
Many of us in the free software community are familiar with Linus’s Law: “given enough eyeballs, all bugs are shallow.” Vulnerabilities are merely a special class of bugs. All other things being equal, Drupal’s larger developer and user base should be expected to find and publish more vulnerability reports than Plone’s.
But Drupal had more than just community growth in 2008; it also experienced unprecedented security review thanks to work by Barry Jaspan, who presented his findings at Drupalcon Szeged 2008. Barry subjected Drupal’s core code to static and dynamic analysis, resulting in the discovery of several vulnerabilities. Has Plone undergone similar scrutiny? A quick search on Google didn’t uncover anything of the sort.
Despite Alex’s thoughts on my stubbornness, I am open to an honest evaluation of Drupal’s security versus similar tools. I’m just not willing to base the debate on a superficial metric of such questionable importance.
Copyright and licensing
Comments
Thanks for your efforts in debating Drupal security.
For the record, search term volume is a lousy metric.
Drupal.org has used internal core search, and recently Apache Solr search for searches on Drupal.org. Other projects use Google search for all their site search traffic. The result is an unnatural bumps in search trends due to the outsourcing of their search. Also, the default search traffic is global. We’ve seen huge surges in search traffic from countries with few active contributors or participants. That kind of search volume is valid, but does not accurately reflect the health of an open source project.
Cheers,
Kieran
I considered attaching a disclaimer to the Google Trends chart for the reasons you’ve cited, but I removed it in the end because it muddled the flow of the post, and I figured the issue would come up in comments, anyway.
Google Trends is indeed a poor metric for open source project health in absolute terms, but it remains a reasonable indicator in the context I used it, which was to show wide pre-2008 disparity between the projects and general movement thereafter. (Also, the self-search disparity you illuminate operates only to Drupal’s detriment.)
I tried to find more accurate models of participation, but there did not seem to be any that neutrally examined both projects, and it was unhelpful to show Drupal’s changes alone without providing a fair comparison to Plone’s (or vice versa).
You could blame me for the same fault I’m pegging on Alex: use of a superficial, “least common denominator” metric of questionable validity (given the application) to make a Drupal versus Plone comparison. Fortunately, Google Trends is only one leg of my argument, and I don’t think anyone is disputing the premise I used it to establish, which is that Drupal has a much larger development and user community.
I never could understand why people often view drupal security releases a bad thing. For me, it makes me more comfortable with drupal security— I know the security team is ever vigilant, so I don’t have to be. And not just with core but with contributed modules as well! How many open source projects also constantly monitor their contributed code?
The fundamental logical flaw in almost all of these arguments, including the one you mention above, is to equate no security updates with no security flaws.
For the record: no security updates != no security flaws. It merely means 1) they haven’t been detected or 2) no one cares. Neither of which is acceptable to me.
All software has bugs, and some of those bugs will involve security vulnerabilities. To pretend otherwise is pure folly.
As far as I’m concerned, if an application, commercial or os, does not have routine security updates, then it is insecure.
The hackers are ever vigilant— legitimate programmers need to be as well.
Create post. I think its really important to both further analyze the “art” that is statistics, and to talk about web security in a meaningful manner (not just quoting a number of bugs). Thanks.
Reported Security vulnerabilities is a bad metric because it doesn’t measure what people are actually looking for. A process engineer would just shake his head at it.
But, it looks good if you are selling something that’s reported few vulnerabilities. So, It can be used for marketing to people who don’t know.
I wonder what a better metric would be?
I do criticize the fact that “Neither severity nor response time nor history of actual exploitation factored in.” The metrics would improve with consideration of those
Better metrics, IMO, are:
1) time from report to vulnerability release
2) consistency in public disclosure of vulnerabilities
The Drupal project isn’t perfect in the way it does security, but we like to take a little pride in how we do it relative to similar projects.
One other point: there was one major review of Drupal by a German security team in 2007 and another major review of Drupal by an American team (via commonplaces) in 2008 and there is an ongoing review by the NSA. All of these (except the NSA review) have found vulnerabilities in Drupal in addition to the review that Barry did. They mostly find vulnerabilities in custom code, then in contrib, and fairly few in core.
So, there are 3 completed major security audits in addition the “thousands of eyeballs” and 1 more under way. Not too shabby.
Even absent David’s valid argument about the size of the Drupal user and developer community naturally finding more bugs and weaknesses, his point that Alex’s belief is a conflict of interest is important. Such a belief distorts the statistics to the point they’re nearly irrelevant.
Plone may well be very secure. But the comparison report which rated Drupal lower simply because of more security reports is nonsense. Drupal may be as secure as Plone — or more so. But there’s no way to tell from the number of reports, nor from Alex’s claims. How many thorough security evaluations by outside parties has Plone had?
Unless you live in a perfect world, there is some time lag between discovery and/or report of a vulnerability and its patch. Each one reported in absolute terms means that there was a period of time when the software was at risk. I agree that a better metric would be time from report to fix, but still a larger number of vulnerabilities is indicative of a larger window of opportunity for ne’er-do-wells.
“…it’s striking that three [Drupal, Joomla!, Wordpress] of the Top Ten contenders on IBM’s security worry-list have PHP in common (http://www-935.ibm.com/services/us/iss/xforce/midyearreport/xforce-midye…). You can read whatever you want to into that, I suppose.” (Kas Thomas, 8/2008, http://www.intelligententerprise.com/blog/archives/2008/08/what_do_jooml…)
Actually, Drupal does a rather excellent job of synchronizing security reports and the updates that fix them. I can’t remember a vulnerability report in the last year or so where that wasn’t the case.
The National Vulnerability Database (http://web.nvd.nist.gov/view/vuln/search) shows 32 records for Drupal during the last 3 months, 7 of them of high severity. For the sake of argument, let’s say the Drupal community fixes each vulnerability in an astonishingly responsive one hour and that every Drupal site in the world applies the patch within an hour after that. That’s 64 hours of known vulnerability.
Plone shows no vulnerabilities for the same period. Again for the sake of argument, let’s say the Plone community takes 24 hours to fix each bug and another day for the patches to be applied worldwide. That still comes out zero hours of known vulnerability.
I realize I’m pulling your chain with the numbers, but there are fundamental weaknesses in the way PHP architecture is implemented and neither size of developer community nor number of external security audits will change that. True, with diligence one can lock down and secure a Drupal site, but there are more opportunities for a slip-up with a PHP-based system compared to a Python-based one.
I’m calling bullshit. Almost all of those vulnerabilities were for contributed Drupal modules. Does Plone even have a system for reporting vulnerabilities in modular, community-maintained code?
And some of the “vulnerabilities” for Drupal itself were published against releases that hadn’t been current for months.
“Almost all of those vulnerabilities were for contributed Drupal modules.”
However, almost all Drupal sites use contributed modules.
Yes, but there’s probably no Drupal site in the world that uses all of the modules cited in the issues you linked. Some of the modules cited in that listing are only used on a handful of sites, if any.
Your exercise of blaming Drupal for security flaws in contributed modules remains misleading and false to the point of dishonesty.
Plone has a robust issue tracking system at https://dev.plone.org/plone/newticket and specifically for security related matters, security=at=plone.org is constantly monitored.
I tried the link, but it required HTTP authentication with credentials I don’t have. So, I’ll just ask here: does this system track security issues with non-core code maintained by random community members?
Feel free to obtain credentials at http://plone.org/join_form, but better still, just view the entire collection at http://dev.plone.org/plone/report. To answer your question directly, yes, non-core code security issues are tracked here.
As the primary researcher and one of the primary writers of the Idealware CMS report, one thing I will say here is that our report does not in any way suggest or say that Drupal’s security is “notably bad”! The report wasn’t designed to be an in-depth analysis on security (or, really, say anything definitive about the security of any one system) - it was a broad comparison of systems.
Anyway, here’s my full response.
I posted my reply on your blog.
Okay, you got us Idealware writers all riled up! I was the other lead writer on the report, as well as the Executive Director of Idealware, and here’s my full response!
In short: there’s no question that our analysis of security could have been more rigorous. If anyone out there has the time or budget to do the analysis you suggest that factors in severity and response time and and history of actual exploitation, we’ve love that - we’d use it, and attribute the heck out of it, as we certainly don’t have the budget to do it ourselves.
That said, it doesn’t make any sense to dismiss security vulnerabilities as useless. It’s a rough measure, no question, but a useful one. Yes, a more popular system has more eyes on it to generate security reports - but also more black hat eyes on it to find and exploit known (or new) security issues.
I posted my reply on your blog.
Instead of going for the “Drupal is so popular, all bugs are shallow, that’s why we have so many security holes” rhetoric, I’d suggest addressing the list of the 10 most common security vulnerabilities in web applications from OWASP. It’s a good checklist that lists the most common attack vectors for web applications these days. If the PHP-based projects (not just Drupal :) can show how they address these, they are on their way to show that they take security seriously.
Plone’s version is here: http://plone.org/products/plone/security/overview
Please note that I’m not trying to stir up this issue, Plone has chosen to focus on security, which of course comes with its own tradeoffs — the “why can’t untrusted visitors embed a YouTube video in my site” being the most common complaint.
Oh, and let’s grab a beer the next time you’re in SF, David. :)
Thanks, Alexander, for your informative comment.
Within Drupal, we don’t have a master document mapping most common vulnerabilities to Drupal’s solution to them. We break the problems down by responsibility, and we try to ensure Drupal is keeping its part of the burden.
For module and core developers: http://drupal.org/writing-secure-code
Plone’s answer to DDoS attacks is much the same as Drupal’s: add caching layers in front of your application to handle anonymous traffic. We see this as the administrator’s responsibility, but we build support into Drupal for it. Additionally, Drupal has been shown to scale well with multiple webheads accessing a database cluster with replication, and this also mitigates the ability for high traffic to take down a site.
Some things listed on the Plone security page are platform issues, like buffer overflows. Plone requires Python to be secure for these; Drupal has the same reliance on PHP. Historically, PHP’s security issues have mostly revolved around poorly conceived features like “safe mode” that are both unnecessary to use with good systems architecture and slated for removal in PHP 6.0 (if not earlier).
Likewise, Drupal’s cryptography uses either PHP’s built-in routines or mcrypt/hmash, both of which use well-established routines.
Having looked at Plone’s security page, I noticed the CVE comparison for Drupal includes PHP and MySQL. I think this is misleading (beyond the reasons in my original post) because it implies a Drupal site would have been exposed to any vulnerability in PHP or MySQL, as well. That’s clearly false, but I think the greater error is excluding Apache and Linux from being listed beneath both the LAMP- and Zope-based applications. In such rough “insecurity by association,” MySQL’s vulnerabilities are Drupal’s as much as Linux’s are Plone’s. And, yes, Plone can run systems other than Linux (like BSD), but Drupal also doesn’t have to run on MySQL, either. (PostgreSQL is also supported.)
Regarding beer, we’re on. I’ll set something up. :-)
You should google better: http://lmgtfy.com/?q=art+of+plowning
“All other things being equal, Drupal’s larger developer and user base should be expected to find and publish more vulnerability reports than Plone’s.”
Yes, because Drupal is written in PHP.
You are absolutely right that the number of self-reported security-issues shouldn’t be held against a project, but you also shouldn’t try to reverse the argument that a system has better security because if has more reported security-issues.
Also don’t neglect that Zope+Plone is +1million lines of code, a lot bigger codebase then Drupal core, so I certainly do not follow your conclusion that it’s ‘logical’ that more security-issue are found in Drupal.
If you are developing in PHP, then you need to accept some simple consequences.
Security is one of them. This graph says enough IMHO:
http://zope2.zopyx.de/about-zope-2/six-reasons-for-using-zope/zope-is-se…
“You should google better: http://lmgtfy.com/?q=art+of+plowning”
I checked the first few results and tried to view the report, but I just get forwarded to procheckup.com, a company that provides penetration testing. Now, assuming this report showed penetration testing of Plone, that’s still entirely different from the static and dynamic security analysis that was performed on Drupal.
Additionally, I deduct 10 points for your condescending link. :-) It’s acceptable to criticize someone for not searching for the obvious, but what you’ve done here is post a link that searches for the exact title of a related report with a strange conjugation of “Plone.” Knowing what you’re looking for and constructing a Google search that finds it is pretty silly.
“Yes, because Drupal is written in PHP.”
The things that make Plone and Drupal have their most dangerous issues are almost always unrelated to the underlying platform. Neither Python nor PHP establishes any model to prevent XSS, SQL injection, etc. That’s the job of the framework.
“but you also shouldn’t try to reverse the argument that a system has better security because if has more reported security-issues.”
Lucky for me, I never did. I only said it was “expected” that we’d discover and report more issues for Drupal.
“Also don’t neglect that Zope+Plone is +1million lines of code, a lot bigger codebase then Drupal core, so I certainly do not follow your conclusion that it’s ‘logical’ that more security-issue are found in Drupal.”
Plone could have 100 million lines of code, but if no one is looking or performing extensive security analysis on them, it’s unlikely the nuanced security issues that exist will be found.
“This graph says enough IMHO”
Great. Yet another visualization of the same irrelevant data.
This is certainly true of Plone. It’s few serious vulnerabilities over the years have largely been due to getting too far out front of it’s Zope framework. Note that I said “Zope” — not “Python.” Plone’s framework is Zope, a full application server with an excellent security model that makes it very difficult for application and add-on developers to fall prey to the most common web app failings.
What WordPress, Joomla! and Drupal have in common is that their app server framework is PHP+SQL. Each has application-provided layers of abstraction that, if used properly, aid in avoiding common security problems. But, these abstractions are on the application, not the framework, level. It is very hard to enforce their correct use.
The real question about past security reports is “what have you learned from them?” That means that, if the vast majority of an application’s security problems have been in add-ons, there may be a framework problem that needs addressing. I give the Drupal community big cred for promoting good coding standards for add-ons. Whether or not that will result in fewer future vulnerabilities in add-ons is next-year’s story.
There are definitely things that the Drupal community can learn from Plone, WordPress and other communities. I’ve been using Plone as an example of open source security now, because the CIA uses it.
Does this mean that Plone is more secure than Drupal because it was adopted by a very security conscious spy agency? I don’t think so.
However, just because we can have this conversation, learn from each others best practices and look to enhance our culture’s understanding of security, the better we’ll be. It’s also only the kind of communication that can happen in an open source community.
Once you finish discussing drupal vs plone, we all can start (again) with vim vs emacs. It not fair to state anyone is better in security terms (specially if you state your’s is better than the other), but it’s unfair to say one is worse than the other when they never worked together (and this includes both drupal and plone defenders).
I’ve never heard of a drupal site compromissed, but have hacked two plone servers.
In my opinion, the one with a correct vulnerability disclosure process will win whatever this battle means, and this is currently the situation of drupal. Drupal states several bugs as ‘security issues’ even if they are just simple bugs, instead of correcting them silently. I don’t know if plone could say something like that..
I’ll ignore your doing exactly what you start by complaining others are doing ;)
Indeed I think it’s the best tool for the person, depending on the job they have before them. I use Plone for some things, drupal for others.
You’ve hacked 2 plone servers? The servers, or zope/plone?
Can you elaborate on the exploits you used? or are you just a typical emacs user? ;) (joke please, please don’t start;)
Running community servers using both platforms and many others, plus commercial hosting with clients using both platforms on VPS and Dedicated, I have seen 2 drupal sites compromised (yes by slack admins definitely, but there’s plenty of slack plone admins too believe me).. I just have never heard of a Zope/Plone site ever being ‘hacked’… I’d be interested in hearing about such a case.
Other than that, from a high level user of both systems, they are both awesome :) The weakest link in both systems would have to be maintenance practices, regardless of the amount of exploits revealed. Still, it’s a bugger having to apply them, but better safe than sorry.
You guys are all talking about reports and verbosity of the community. It is hard to get anything useful out of this discussion.
There is another thing you can look at, and this is the product itself, especially the code architecture. A good architecture will encourage programmers (core, contrib and custom) to write more secure code, and saves them from re-implementing existing stuff in a probably less secure way. You can do the same thing with less code, and a smaller chance to do damage (think about implementing hook_menu, vs explicitly doing a db query to write that stuff to the menu_router table).
OOP frameworks have the advantage that they can do more type checking on function arguments - one thing that is mostly missing in Drupal, where you pass around arrays with no type checking whatsoever.
Drupal’s hooks are a nice example of Inversion of Control (without OOP), and I think that’s beneficial for security. Implementing a hook means less chance to do things wrong.
Drupal does provide tools for argument filtering (against SQL injection or XXS), but you need to know how to use them properly, especially for custom functionality.
There is still a lot of global state that also could cause security-relevant side effects.
Is Plone any better?
Honestly, I have no idea. I assume it is more OOP, but: A lot of the OOP projects I’ve seen (in PHP) still had a lot of global state and other unpleasant things.
The security of any system rises and falls in correlation with the competence of those who administer it. I don’t know enough about Plone to comment on its security, but I will say that Drupal’s core modules are quite tight if administered properly.
Some of the contributed modules are probably vulnerable, but it is incumbent on the administrators to vet those.
Asking which is more secure is like asking which is a better guitar, a (real) Stratocaster or a Les Paul — it depends on who’s playing it, the amplifier, the cord, the acoustics of the room, the mood of the crowd, and lots of other factors.