Challenge Sets

After the Cyber Grand Challenge (CGC), DARPA released the source code for over 100 challenge sets (CS). These programs approximate real software with enough complexity and a sufficient variety of flaws to stress both manual and automated vulnerability discovery.

But they were written for DECREE, the competition’s custom Linux-derived operating system on the 32-bit Intel x86 architecture; not readily accessible for the security software industry.

We recognized the CSs as a golden opportunity. If they became widely adopted as benchmarks,

  • How good are the CGC tools vs. existing program analysis and bug finding tools?
  • When a new tool is released, how does it stack up against the current best?
  • Do static analysis tools that work with source code find more bugs than dynamic analysis tools that work with binaries?
  • Are tools written for macOS better than tools written for Linux or Windows?

So, we ported the challenge sets to run in Windows, macOS, and Linux. It took us several attempts to find the best porting approach to minimize the amount of code changes, while preserving as much original code as possible between platforms.

The eventual solution was fairly straightforward: build each compilation unit without standard include files (as all CS are statically linked), implement CGC system calls using their native equivalents, and perform various minor fixes to make the code compatible with more compilers and standard libraries.

Now, there’s no need to set up a virtual machine just for DECREE. Users can run the CSs on the machines they already have. When researchers publish their code, we can evaluate how well their findings work for a particular OS or compiler.

Get the DARPA Challenge Sets in the most common operating systems.

Next Project