Vulnerability Tools x Benchmark Suites =

(enter "s" to see presenter text for slides)

Existing Benchmarks x Analyzers

Introducing Omnibus

Full set of standardized vulnerability classes

  • (...)

Micro-benchmarks and real-world samples for each subclass

Who should be interested in this?

  • Smart contracts developers
    • Does tool handle a vulnerability you are interested in?
  • Vulnerability/security auditors
  • Tool Developers
    • How am I doing today?
    • Have I improved over time?

Is it all that difficult to do?


  • Install Analyzer tools
  • Gather Benchmark Suite(s)
  • Write a runner that can run the tools over the benchmark suite
  • Run tools over suite(s) — takes time
  • Classify and grade results

What Information is needed for Benchmark Suite testing?

  • Suite: What does it cover?
  • Benchmark Test: What Vulnerablity does it detect?
  • Benchmark Test: What's the right answer?
  • Benchmark Test x Tool: What's the right answer for this tool? Is it even relevant?

The need for common terminology:

  • Location
  • Vulnerabilty Class
  • ...

  • Dr. Suhabe Bugrara — cool idea
  • Aleks Sobolev — revamping the reports
  • Vladimirs Timofejevs — code to cover Manticore and Oyente
  • Rocky Bernstein — and takes credit for the above
  • Steve Marx — for the win
  • ConsenSys Diligence — has been funding this to date
  • You — for getting involved