Where am I now, Really?

Where am I now — Really?

Decompilation for Precise Program Location Information

(enter "s" to see presenter text for slides)

Line numbers are not good enough


x = prev[prev[0]]                      # which prev ?
y = a / b / c                          # which divide is invalid?
[x[0] for i in d[j] if got[i] == e[i]] # lots going on here
exec("a=%s; b=y" % z)                  # code created at runtime

Python 3.6 bytecode


prev[prev[0]]

compiles to:


  2       10 LOAD_NAME           prev
          12 LOAD_NAME           prev
          14 LOAD_CONST          0
          16 BINARY_SUBSCR
          18 BINARY_SUBSCR
          20 POP_TOP

But first a little background. When you run a program, the computer always has to know which instruction to run next. In Python, the location of the instruction (which is different from the source code line) is called an "offset."

Here is the relevant part of a Python 3.6 disassembly of the above code

You don't really need to know in detail what the above means. But each line is an "instruction" and the numbers that start at 10 and run to 20 are the instruction offsets. And you'll see a couple of instructions repeated, like LOAD_NAME or BINARY_SUBSCR, but with different offsets. In Python 2.7 everything looks the same except for those offset numbers.

In the assembly listing, the "2" at the top left means that any instruction from offset 10 to 20 comes from something on line 2. And you'll see that this is vague, because any of the instructions above are on line 2 and we might hit an error on any one of those instructions.

So now let me describe a little bit about parsing and compiling. This is what transformed the first Python text into the representation of the second text shown above. Furthermore a very similar process can be used to transform the last text back into the first as well - and that's my hack!

So let me describe how parsing works.

When I talk you hear sounds. Unconsciously you break those sounds into segments, and then still unconsciously you combine those until you recognize words and sentences. And only starting at one of these higher levels do you start to understand what I am saying. If you were to read the text for this talk, the process is similar, although the lowest-level recognition is letters rather than sounds. You read letters and then form them into words, and then into sentences.

The point is that sounds and letters aren't the same - this is the reason for the notorious problems of English spelling - but the higher-level combinations do wind up in pretty much the same place.

English Sentence Parse

“I ate the mango.”


    sentence
     0. subject
        pronoun, first person
          I
     1. verb, past tense
          ate
     2. object
          noun phrase
          0. definite article
             the
          1. noun
             mango
    3. punctuation
             .

Bytecode Assembly Parse

stmts (4)
  ...
  1. stmt
    0. expr
       binary_subscr
         0. expr
            L.   2      10  LOAD_NAME 'prev'
         1. expr
            binary_subscr
            0. expr
                        12  LOAD_NAME 'prev'
            1. expr
                        14  LOAD_CONST 0
            2.          16  BINARY_SUBSCR
        2.          18  BINARY_SUBSCR
    1.          20  POP_TOP

Bytecode assembly Parse

2 10 LOAD_NAME "prev"
  12 LOAD_NAME "prev"
  14 LOAD_CONST 0
  16 BINARY_SUBSCR
  18 BINARY_SUBSCR

when parsed becomes:

    0. expr
       binary_subscr
         0. expr
            L.   2      10  LOAD_NAME 'prev'
         1. expr
            binary_subscr
            0. expr
                        12  LOAD_NAME 'prev'
            1. expr
                        14  LOAD_CONST 0
            2.          16  BINARY_SUBSCR
         2.          18  BINARY_SUBSCR
...

Displaying the results

At offset 16:

instruction      16  BINARY_SUBSCR

prev[prev[0]]
     -------

At offset 18:

instruction      18  BINARY_SUBSCR

prev[prev[0]]
-------------

This comes from the tree structure in the parse.

There is more to do...

Emacs Lisp:


(/ a (/ b c)))

gives:


0	varref	  a
1	varref	  b
2	varref	  c
3	quo
4	quo
5	return

Links...

- end