Decompilation, Translation and Debugging

Decompilation, Translation, and Debugging

Where am I now — Really?

(enter "s" to see presenter text for slides; video is here.)

A translation of an earler version of the first part of the talk is graciously and gratefully provided by Gonzalo Peña-Castellanos is here.

Hi I am Rocky Bernstein, a long time open-source developer; one of the people Christine mentioned yesterday.

This talk is in two parts. The first part describes decompilation; and then the second part to demoes it from a debugger.

As I said I write a lot of code. I also make I make a lot of mistakes. I even make mistakes in finding my mistakes.

So when I do, I'd like to know exactly where I am at. I feel as though I'm trying to find Waldo, the guy with the glasses and red ski cap in this picture.

Computers generally excel at giving detailed information involving lots of calculation. But in this one area, just about all of the programming languages could do better at giving more precise and useful information. So let's look at a Python error...

A Python Error


Traceback (most recent call last):
  File "bug.py", line 2, in <module>
    i / j / k
ZeroDivisionError: integer division or modulo by zero

Was j zero, or was k zero?

More errors where line numbers are not good enough


x = prev[prev[0]]                      # which prev ?


[e[0] for i in d[j] if got[i] == e[i]] # lots going on here


exec(some_code % 10, namespace)        # code at runtime

An Error where no file exists


exec(some_code)


Traceback (most recent call last):
  File "bug-exec.py", line 3, in <module>
    exec(template % 10, namespace)
  File "<string>", line 1, in <module>
NameError: name 'bad' is not defined

File "<string>"?

Programmer to Computer

A programmer may write:


  x + y

In the C Python implementation, the code you write gets translated into something called Bytecode. when I write in Python x+y, next the computer sees... next

The above is for C Python implementation, and the instructions are called Bytecode. The above bytecode is for Python 3.6.

The "2" on the first line means that the following instructions come from line two, until there is another line number given. But since there are no other line numbers given, the Python interpreter assumes that all of the three instructions have something to do with line 2.

The numbers 8, 10, and 12 are the instruction addresses, or really instruction offsets from where the function or module starts. So being more precise about what is associated with line 2, the answer is instructions at offsets 8, 10, and 12. In Python versions before 3.6, the instruction size for `LOAD_NAME` is 3 bytes instead of 2, so the offsets would be 8, 11, and 14.

And this (or a more obstruse binary-encoded version of that) is what the computer "sees" and runs.

In this simple example, it looks like a single token in the source-code gets converted to a single line in the bytecode. However the order is a little bit different: the "add" comes at the end rather than in the middle.

Compare that with... next

but the C Python interpreter sees:


  2:  8 LOAD_NAME    x
     10 LOAD_NAME    y
     12 BINARY_ADD

Spanish to English

A person says in Spanish: “mango fragante”
might translate in English to: “fragrant mango”

A person says in Spanish: "mango fragante" and that might translate in English to: "fragrant mango"

This is pretty simple: you just switch the order of the words around and make some small spelling changes. That happens in bytecode translation too. Now consider: next

"Entiendo" which might translate to "I understand".

Here, one Spanish word becomes two English words. That too happens in Python translation to bytecode. next

Finally consider... next templado

In the last example, Which part of "templado" is "not hot" and which part is "and not cold"? The concepts are mismatched: there is no simple English equivalent.

This kind of thing never happens in Python translation to bytecode. next

And intentionally. Fortunately for us, the bytecode and the source code were designed to know about the same kinds of concepts.

So how do we make sense of this? Let me suggest an answer by pointing out that you've probably already seen examples of what I showed before with line numbers, and also this kind matching problem when you read a side-by-side translation:

A person says in Spanish: “Entiendo”
might translate in English to: “I understand”

A person says in Spanish: “templado”
might translate in English to: “not hot and not cold”

Side-by-side Translation

1: Whose woods these are I think I know.
2: His house is in the village though

1: A quién pertenece este bosque creo que sé
2: Aunque su casa en el pueblo está

Side-by-side Translation

1: Whose woods these are I think I know.
2: His house is in the village though

1: A quién pertenece este bosque creo que sé
2: Aunque su casa en el pueblo está

Side-by-side Translation

1: Whose woods these are I think I know.
2: His house is in the village though

1: A quién pertenece este bosque creo que sé
2: Aunque su casa en el pueblo está

Side-by-side Python translation


  x = 1  # line 1
  y = 2  # line 2


  1: 0 LOAD_CONST  1
     2 STORE_NAME  x

  2: 4 LOAD_CONST  2
     6 STORE_NAME  y

Notice again that the order of the variable name and constants have been reversed as they were in the example with addition. The mathematical name for this kind of ordering is called "reverse polish notation". English adjectives are like that. The "assignment" equal sign doesn't appear explicitly although it is implicit by the fact we are using a "STORE_NAME" instruction. This is similar I guess to the "I" being implied by the "o" in "Comido".

Now that we have seen how this resembles a human-language translation problem, rather than a programmer-versus-computer problem, we can ask: how would we get more detail in the correspondences between the two translations?

I'm sure many of you already know the answer: you break down the sentence into its grammar parts. So... next

English Sentence Parse

“I eat tripe.”

    sentence
      0. subject
        pronoun
          I
      1. predicate
         0. verb
            eat
         1. object
            tripe

Spanish Sentence Parse

“Como mondongo.”

    sentence
    0. subject
         I
      1. predicate
         0. verb
            eat
         1. object
            mondongo

Bytecode Parse

 prev[prev[0]]

parses to:

Next is the bytecode

Notice that some instructions like LOAD_NAME and LOAD_CONST have an operand or attribute. We will come back to this a little later.

For both of the LOAD_NAME the operand is "prev" however that is just a coincidence here.

For LOAD_CONST the operand value is 0. next

And finally a reconstituted parse tree from just the instructions which is created by my decompiler uncompyle6.

I will not go into how this tree was created other than to say that it uses some standard and a little non-standard parsing theory from compiler theory.

Notice how all of the leaves of the parse tree are the bytecode instructions. It might be hard to see that because the leaves are at different tree depths. But as you will see, the different depths will become important a little later.

down

translates to:


2  6 LOAD_NAME "prev"
   8 LOAD_NAME "prev"
  10 LOAD_CONST 0
  12 BINARY_SUBSCR
  14 BINARY_SUBSCR

uncompyle6 (de)parses to:

In listing form:


       Subscript
         0. Expr
            L.   2       6  LOAD_NAME 'prev'
         1. Expr
            Subscript
            0. Expr
                         8  LOAD_NAME 'prev'
            1. Expr
                        10  LOAD_CONST 0
            2.          12  BINARY_SUBSCR
         2.          14  BINARY_SUBSCR
...

Parse to Text

With Instruction Operands

With "Expr" rule

With "Subscr" rule

Parse at offset 12

Parse at offset 12 Display

instruction      12  BINARY_SUBSCR

prev[prev[0]]
     -------

Parse at offset 14

Parse at offset 14 Text display

instruction      14  BINARY_SUBSCR

prev[prev[0]]
-------------

trepan2 help

deparsing comprehension errors

Debugging code created at runtime

This example is now probably the toughest for any debugger.. debugging code generated at runtime.

I encourage people who use other debuggers and IDEs to see how well this works there.

Although in general, writing code that modifies itself at runtime, is frowned upon. For example there are security concerns. But I'm going to show you that in fact it happens a bit more than people admit. I'm going to do this using one of Python standard library routines, namedtuple.

This class creates a tuple, where you can give names to the integer index elements. Here is a typical use...

First I'll try debugging it with pdb. Where I really want to get to see is how creating the Point() class works, and how repr() is implemented. So I'll set a breakpoint.

We see that in fact the Point class as generated somewhow at runtime. And so I can't see the source code to this with a "list" command. The position though seems to be line 8 of some unnamed string. That's helpful! (not). That then we go to line 10 of some unnamed string. And then we return from that.

I step into Point's __repr__ function, and I have the same problem.

So let's try with the Python3 version of trepan called trepan3k. And I am going to adjust the highlighting style. However trepan3k did detect there was a dark background and made an adjustment for that.

Note that I can do a "break followed by a continue" in one command by giving a number after continue. The @31 is the instruction offset when I am stopped. And now I can list lines of the function. The (1, 14) at the end just says line 1 of this file maps to line 14 in whatever source this originally came from. This si probably wrong since pdb said it was line 8. I don't know which one was right.

But now that I can see what is in that function, I can start to do something meaningful. I can inspect value, and I see in what part of that one line function I am about to run. It looks like I've just am loading up the name of the function that I am about to call. Code to evaluate the parameters like _cls, x, or y, would come later.

I'll ask for help on _tuple. Ah it is the internal name for tuple. After a couple of steps I see that although I am on the same line, deparse tells me at this point I have evaluated everything on the line.

And we can do the same thing when we step into __repr__ in running print

Again, those of you how use Jetbrains, or some other debugger, I am curious how well those do in this kind of situation.

Debugging code when there is no source

The last example I am going to show is a slight variation of the last example.

We can debugging bytecode when there is no source code at all! Just the bytecode file. Reverse engineers: pay attention.

There is a command-line program, pydisasm I wrote that will disassemble bytecode. It has a --header option to give meta information about a bytecode file. In particular, it gives the version interpreter version and the magic number of the for the bytecode file. The file name Python was given for the file when it create it was "exec" which is pretty meaningless.

Even though I don't have the source code, I can still run the program Python.

But what if I try to debug it using the pdb module, I get gibberish. Python thinks the bytecode is text. I can't even issue a quit command or Ctrl-d to get out. I have to suspend and kill it.

I've been running this inside Emacs, and I'm also going to show off an emacs package I wrote which interfaces not just to this debugger, trepan3k, but also to about a dozen others including pdb, and ipdb. The package is called realgud.

When I give "trepan3k" a bytecode file it notices that and looks for the source code. It can't find that, so then it decompiles the entire program and debugs that.

You can modify, and rename that file if you want since it is a file in the filesystem now.

The emacs package tracks the location for me. Line with treen green arrow is where I am currently. The line that I had been previously stopped at has an orange mark. And the line before that has a brown mark.

In fact, the emacs interface saves all of the locations you have been stopped at even though it shows the last 3 lines. There are commands to cycle though just the stopping-point locations.

Notice that in function fib() there are two recursive calls. And again deparse can show you which one was called.

So we went up in the call stack. Run a "deparse" and you can see which of the two calls we are inside.

We step further do it again and now we can see that the call locat has changed.

And as you might expect in a debugger front-end, you highlight a region and have that evaluated.

There is more to do...

Emacs Lisp:


(/ a (/ b c)))

gives:


0	varref	  a
1	varref	  b
2	varref	  c
3	quo
4	quo
5	return

Links...

- fin