Vulnerability Tools x Benchmark Suites = Unicorns

Vulnerability Tools x Benchmark Suites =

(enter "s" to see presenter text for slides)

Hi - I am Rocky Bernstein, a long-time open-source developer. I work now in the Mythril team at ConsenSys Diligence, and specifically in developing what will go behind the Mythril Platform API which you will hear more about from Bernhard next.

Analysis tools play a big role in helping to find Smart Contract vulnerabilities. How do we know if these tools are any good? Which tools are best for finding a certain category of bug?

There’s been a recent effort to answer these questions with a benchmark suite. The goal is to have a standard set of Smart Contracts that we can use to measure each tool’s accuracy at finding vulnerabilities.

Today I’d like to convince you that this work is important and that it’s worth your time to contribute test cases to the benchmarks at the hackathon that will happen at ETHBerlin.

Let’s take a look at what we have so far...

Existing Benchmarks x Analyzers

Currently we have two different benchmark suites, one that Suhabe Bugrara wrote, and other was written by Trail of Bits folks you just heard from. Suhabe's suite contains micro-benchmarks while the "(Not so) Smart Contracts" is at the other end of the spectrum and are based on interesting contracts that have been published.

As for as analyzer tools we have 3: Mythril which you'll hear about next, Trail of Bits' Manticore, and Oyente which is now bundled in the remix IDE.

I just want to show here is that we have the ability to run several tools across a given benchmark and show the results side-by-side. Some have opined that comparing these particular tools are like comparing apples, to oranges, to bananas.

This aspect is more historical than intentional. For dynamic tracing tools, there are very few tools and they are not equivalent.

But comparison which fosters competition and that has already been beneficial. Just in publicizing this 10-minute talk, I and others have found bugs in some of the tools as a result of running the suites, and some of these have already been fixed.

Introducing Omnibus

Full set of standardized vulnerability classes

OMN-CALL-UNCHECKEDRETVAL
OMN-CALL-UNTRUSTED
OMN-DELEGATECALL-UNTRUSTED
OMN-LINT-OLDCOMPILER
OMN-LINT-VISIBILITYNOTSET
(...)

Micro-benchmarks and real-world samples for each subclass

Who should be interested in this?

Smart contracts developers

Does tool handle a vulnerability you are interested in?

Vulnerability/security auditors
Tool Developers

How am I doing today?
Have I improved over time?

Suppose you are a smart-contracts developer: how do the tools stack up against each other? What kinds of vulnerabilities do they catch? And are the things reported really vulnerabilities?

You also might be interested in how long it takes an analyzer to run over various Smart Contracts.

Suppose instead you are a Vulnerability/security auditor like we have at Diligence. For you, it is important to keep abreast of the tools are out there and assess how good they are.

Of course, things change over time, so it is important to track changes in the tools. New kinds of attacks come out, and new kinds of recommendations may surface.

Finally, suppose you are developing analyzer tools like me. I want to know how well I am doing, where I need to work harder, and I need to make sure we are continually improving.

Is it all that difficult to do?

Yes!

Install Analyzer tools
Gather Benchmark Suite(s)
Write a runner that can run the tools over the benchmark suite
Run tools over suite(s) — takes time
Classify and grade results

What Information is needed for Benchmark Suite testing?

Suite: What does it cover?
Benchmark Test: What Vulnerablity does it detect?
Benchmark Test: What's the right answer?
Benchmark Test x Tool: What's the right answer for this tool? Is it even relevant?

We want to make sure that a benchmark suite covers some interesting related set of vulnerabilities. This means having benchmark tests that detects a kind of vulnerability as well as one that doesn't but looks deceptive so it they might fool an analyzer.

For a benchmark in a benchmark suite, we want to know what it is testing, and what in a broad sense, what the analyzer tool should find. It is desirable to have a simple or the simplest Smart Contract that does this.

Knowing the vulnerabilities of a benchmark test in broad terms, isn't enough: we may need to know how each particular tool expresses the kind of vulnerability that a benchmark test expressions.

And for each benchmark and analyzer tool, we have to decide whether the benchmark test is relevant.

Analyzer results may change as the analyzer tool changes: maybe earlier versions of a tool detects the problem tersely, and later versions are more detailed. So there are several levels of "correctness".

Finally results can change depending on Solidity compiler generation and optimization level. So we need somehow to need to note and accommodate that.

The need for common terminology:

Location
Vulnerabilty Class

In a single report there are runs _n_ tool runs over a single benchmark suite for each analyzer. Here _n_ is 3.

But we allow for several benchmark suites. So we have code running _n_ analyzers run over _m_ benchmarks. Here _m_ is 2.

Fortunately _n_ and _m_ are small right now. But as you can see the multiplication leads to complexity that should be managed.

This is a standard problem in "compiler collections" like, say, the GNU compiler collection, "GCC" which accepts a number of front-end languages and produces code for a number of target machines.

The standard technique to reduce complexity here is to use a common intermediate language. So if we can use common terminology for ideas like "vulnerability class" and location in a program, we will keep the code to gather data from analyzers and the code to produce reports simpler. For example is "Transaction Order" that mythril reports the same thing as "Reentrancy" that Manticore reports?

Thanks

Dr. Suhabe Bugrara — cool idea
Aleks Sobolev — revamping the reports
Vladimirs Timofejevs — code to cover Manticore and Oyente
Rocky Bernstein — and takes credit for the above
Steve Marx — for the win
ConsenSys Diligence — has been funding this to date
You — for getting involved