Open-source software bug hunt results released

Coverity Inc. of San Francisco has released the of a Homeland Security Department-funded bug hunt that ranged across 40 popular open-source programs. The company found less than one-half of one bug per thousand lines of code on average, and found even fewer defects in the most widely used code, such as the Linux kernel and the Apache Web server.The results are the first deliverable of a $1.2 million, three-year grant DHS to a team consisting of Coverity, Stanford University and Symantec Corp. of Cupertino, Calif. DHS wants to reinforce the quality of open-source programs supporting the U.S. infrastructure.The agency is hoping developers will fix the defects highlighted by the team's advanced bug-hunting techniques. Such defects can pose security vulnerabilities because they could be used by malicious programs to disrupt or gain control of a system.To test the programs, Coverity deployed analysis software first developed by Stanford's computer science department. Ben Chelf, chief technology officer of Coverity, warned that this automated bug scan is not definitive, but it can point to bugs traditional in-house code review techniques can miss. Most of the 40 programs tested averaged less than one defect per thousand lines of code. The cleanest program was XMMS, a Unix-based multimedia application. It had only six bugs in its 116,899 lines of code, or .51 bugs per thousands lines of code. The buggiest program is the Advanced Maryland Automatic Network Disk Archiver, or AMANDA, a Linux backup application first developed at the University of Maryland. Coverity found 108 bugs in its 88,950 lines of code, or about 1.214 bugs per thousand lines of code. Overall, the average defect density of all the programs was .43 bugs per thousand lines of code. The most widely used programs scored well under this average. The 3 million lines of code that make up the Linux Kernel had an average of .33 bugs per thousand lines of code. Apache has .25 bugs per thousand lines of code. The open-source LAMP stack (consisting of Linux, Apache, MySQL and a scripting language of either Perl, PHP or Python), had a defect density of .29 bugs per thousand lines of code.Generally speaking, it is difficult to determine how well these open-source programs compare with their proprietary counterparts, Chelf said. Coverity has tested only a few commercial products, so direct comparisons cannot be made.The company has drawn a number of observations from the study, and elaborated upon them in a paper accompanying the results (available on the Coverity site after registration). The chief lesson is that the number of lines of code is not an indicator of quality. Smaller programs can have plenty of bugs while larger projects, such as the Linux kernel, can be tightly controlled. Quality is more accurately reflected by the ratio of developers to the size of the code base and by the number of users who use the software (and provide feedback).The maintainers of the source codes can register with Coverity to see the full results. (End users cannot see the bug lists themselves; they will be able to see how buggy a particular program may be.)Bruce Momjian, who oversees development of has used Coverity reports before and has found them useful, if not absolutely essential. The results of a previous study pointed to "a few unusual cases that weren't exploitable bugs, but were something we wanted to clean up," he said.The PostgreSQL team reviewing the new list, though, has found a number of reported bugs that they actually already fixed. "We have someone looking into the actual items now," he e-mailed.