Benchmarks for evaluating Checked C

Benchmarks for evaluating Checked C

This page lists real-world programs that we have chosen to use to evaluate Checked C. Our goal is to have a suite of 10 to 12 programs that we can use to evaluate Checked C and tools for converting existing code to use Checked C.

The criteria for considering a program for inclusion in the evaluation suite are:

• The program is used in production.
• Someone cares about the program. The program is associated with an active project and it is being developed or maintained.
• The license allows us to redistribute a modified version of the program.
• The source code is at least 10,000 lines long.

We used the following criteria for decide on the mix of programs to include in the evaluation suite:

• Prefer programs that are widely used.
• Have an interesting mix of programs. Avoid more than 2 programs that "do the same thing" (compression, image processing etc.).
• Prefer programs that run on Windows in addition to Unix over those that just run on Unix.
• Prefer programs that have well-defined testing and benchmarking processes.

Selected programs

zlib

• Pascal Cuoq of trust-in-soft is currently running a bug-finding effort on this project with tis-interpreter.
• Very permissive license
• 19687 lines of C code
• Used all over the place at a low level
• Old (1995), C systems code

libarchive

• Bugs have bound in this before using fuzzing.
• 3-clause UC Regents license
• 144545 lines of C code
• Used all over the place at a low level
• "mid-modern" (2004)

redis

• Very permissive license
• 93868 lines of C code
• Internet service in semi-popular use
• Slightly more modern (2009)

libtiff

• Very permissive license
• 65912 lines of C code
• Used all over the place at a low level
• Initially developed in 1993?

vsftpd

• GPLv2
• 16414 lines of C code
• Has a very good security track record and was written in a very defensive style with regards to memory management. It would be an interesting case study to contrast with something that isn't, like say libarchive.

icecast

• GPLv2
• 17434 lines of C code
• Internet service, probably in decline
• Has had security issues.

sqlite

• Public domain
• 130372 lines of C code
• Very robust and well tested