Micro-benchmarks and real-world samples for each subclass
Who should be interested in this?
Smart contracts developers
Does tool handle a vulnerability you are interested in?
Vulnerability/security auditors
Tool Developers
How am I doing today?
Have I improved over time?
Is it all that difficult to do?
Yes!
Install Analyzer tools
Gather Benchmark Suite(s)
Write a runner that can run the tools over the benchmark suite
Run tools over suite(s) — takes time
Classify and grade results
What Information is needed for Benchmark Suite testing?
Suite: What does it cover?
Benchmark Test: What Vulnerablity does it detect?
Benchmark Test: What's the right answer?
Benchmark Test x Tool: What's the right answer for this tool? Is it even relevant?
The need for common terminology:
Location
Vulnerabilty Class
...
Thanks
Dr. Suhabe Bugrara — cool idea
Aleks Sobolev — revamping the reports
Vladimirs Timofejevs — code to cover Manticore and Oyente
Rocky Bernstein — and takes credit for the above
Steve Marx — for the win
ConsenSys Diligence — has been funding this to date
You — for getting involved
1
Vulnerability Tools x Benchmark Suites =
(enter "s" to see presenter text for slides)
Hi - I am Rocky Bernstein, a long-time open-source developer. I work
now in the Mythril team at ConsenSys Diligence, and specifically in
developing what will go behind the Mythril Platform API which you will hear
more about from Bernhard next.
Analysis tools play a big role in helping to find Smart Contract
vulnerabilities. How do we know if these tools are any good? Which
tools are best for finding a certain category of bug?
There’s been a recent effort to answer these questions with a
benchmark suite. The goal is to have a standard set of Smart Contracts
that we can use to measure each tool’s accuracy at finding vulnerabilities.
Today I’d like to convince you that this work is important and that
it’s worth your time to contribute test cases to the benchmarks at the
hackathon that will happen at ETHBerlin.
Let’s take a look at what we have so far...