chilaxan’s blog

Symbolic Numbers In Python - Part 1

2023-12-01T00:00:00+00:00

A while ago I wanted to see if I could add symbolic numbers to Python. This would mean that something like one would equal 1 and two_hundred_and_five would equal 205.

The obvious option is to just do the following:

one = 1
two = 2
three = 3
...
forty_two_thousand_one_hundred_and_twenty_two = 42122
# and on to infinity?

But this rapidly fills up disk space (and would be a bit insane).

A better way would be to use a module level getattr like this:

# numbers.py

def parse_int(textnum, numwords={}):
    # parse symbolic number into integer
    # https://ataiva.com/how-to-convert-numeric-words-into-numbers-using-python/
    if not numwords:
        units = [
            "zero", "one", "two", "three", "four", "five", "six", "seven", "eight",
            "nine", "ten", "eleven", "twelve", "thirteen", "fourteen", "fifteen",
            "sixteen", "seventeen", "eighteen", "nineteen",
        ]
        tens = [
            "", "", "twenty", "thirty", "forty", "fifty", "sixty", "seventy", 
            "eighty", "ninety"
        ]
        scales = ["hundred", "thousand", "million", "billion", "trillion"]
        numwords["and"] = (1, 0)
        for idx, word in enumerate(units):
            numwords[word] = (1, idx)

        for idx, word in enumerate(tens):
            numwords[word] = (1, idx * 10)

        for idx, word in enumerate(scales):
            numwords[word] = (10 ** (idx * 3 or 2), 0)

    current = result = 0
    for word in textnum.split('_'):
        if word not in numwords:
          raise ValueError('could not convert to number')
        scale, increment = numwords[word]
        current = current * scale + increment
        if scale > 100:
            result += current
            current = 0
    return result + current

def __getattr__(key):
    number = parse_number(key)
    if number is None:
        raise AttributeError(key)
    return number

And then use it as follows:

# some_other_file.py
from numbers import one, forty_two_thousand

This is quite pythonic, but not quite as magic as I wanted it to be. Another obvious solution would be using a global object with a __getattr__ that performs the number parsing. Eventually I determined the best way to add the numbers as though they were constants would require some runtime trickery. My desired syntax for the literals was the different words, seperated with _, ex: one_hundred_twenty_four would equal 124. Since this syntax would be the same as a name lookup, I researched how variable lookups work in CPython.

➜  ~ echo "one_hundred_twenty_four" | python3 -m dis
  0           0 RESUME                   0

  1           2 LOAD_NAME                0 (one_hundred_twenty_four)
              4 POP_TOP
              6 LOAD_CONST               0 (None)
              8 RETURN_VALUE
➜  ~ 

We can see the important part here is the LOAD_NAME opcode. We can see the implementation in bytecodes.c. Glancing at the implementation, it is clear that LOAD_NAME is a sort of catch all, first it checks locals, then globals, then builtins. We can pull these different parts out and look for something we can influence.

Load from locals:

PyObject *mod_or_class_dict = LOCALS();
if (mod_or_class_dict == NULL) {
    _PyErr_SetString(tstate, PyExc_SystemError,
                        "no locals found");
    ERROR_IF(true, error);
}
PyObject *name = GETITEM(frame->f_code->co_names, oparg);
if (PyDict_CheckExact(mod_or_class_dict)) {
    v = PyDict_GetItemWithError(mod_or_class_dict, name);
    if (v != NULL) {
        Py_INCREF(v);
    }
    else if (_PyErr_Occurred(tstate)) {
        goto error;
    }
}
else {
    v = PyObject_GetItem(mod_or_class_dict, name);
    if (v == NULL) {
        if (!_PyErr_ExceptionMatches(tstate, PyExc_KeyError)) {
            goto error;
        }
        _PyErr_Clear(tstate);
    }
}

Load from globals:

v = PyDict_GetItemWithError(GLOBALS(), name);
if (v != NULL) {
    Py_INCREF(v);
}
else if (_PyErr_Occurred(tstate)) {
    goto error;
}

Load from builtins:

if (PyDict_CheckExact(BUILTINS())) {
    v = PyDict_GetItemWithError(BUILTINS(), name);
    if (v == NULL) {
        if (!_PyErr_Occurred(tstate)) {
            format_exc_check_arg(
                    tstate, PyExc_NameError,
                    NAME_ERROR_MSG, name);
        }
        goto error;
    }
    Py_INCREF(v);
}
else {
    v = PyObject_GetItem(BUILTINS(), name);
    if (v == NULL) {
        if (_PyErr_ExceptionMatches(tstate, PyExc_KeyError)) {
            format_exc_check_arg(
                        tstate, PyExc_NameError,
                        NAME_ERROR_MSG, name);
        }
        goto error;
    }
}

So right off the bat we have three places to inspect. The load from locals seems to be the most complicated, and locals can change based on state so it would not be easy to place a hook in there. globals is similar, and seems to just take the current frame’s GLOBALS() dictionary and pass it into PyDict_GetItemWithError. (It is possible to install a hook here using some memory corruption, more on that in part 2). But the lowest hanging fruit is the third load, from builtins. The first thing it does is check if BUILTINS() is a dictionary object. Normally, it is, but if it is not, the code continues down and uses PyObject_GetItem on BUILTINS(). From here we have a clear path. we can change the value returned by BUILTINS() by changing the __builtins__ value inside of a given frame’s globals. By inserting a custom dictionary subclass, we can install code to run when a given key is not found. Python actually provides a simple way to do this, with __missing__. On dictionary subclasses, __missing__ is called if a given key is not found within the dictionary. Knowing this, we can write a simple dictionary subclass using __missing__ to call our parse_int function.

class missing_hook(dict):
    def __init__(self, *args, missing=lambda self, arg: None, **kwargs):
        super().__init__(*args, **kwargs)
        self.missing = missing

    def __missing__(self, key):
        try:
            return self.missing(self, key)
        except Exception:
            raise KeyError(f"name {key!r} is not defined") from None

Now, all we need to do is install this missing_hook class as __builtins__ in our target frame. We can use the following to do that:

def init_symbolic_numbers():
    frame_globals = sys._getframe(1).f_globals
    frame_globals['__builtins__'] = missing_hook(builtins.__dict__, missing=parse_int)

init_symbolic_numbers when called, will grab the frame above it’s globals dictionary, and replace __builtins__ with an instance of missing_hook. Testing this shows that it works!

➜  ~/Desktop python3.12                       
Python 3.12.0 (main, Oct  5 2023, 15:44:07) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from numbers import init_symbolic_numbers
>>> init_symbolic_numbers()
>>> one_hundred_and_twenty_five
125
>>>

This works pretty well here, but we will start to notice some problems when we try to use it in a file

# main.py
from numbers import init_symbolic_numbers
init_symbolic_numbers()

print(one_hundred_and_twenty_five)

➜  ~/Desktop python3.12 main.py           
Traceback (most recent call last):
  File "/Users/chilaxan/Desktop/main.py", line 4, in <module>
    print(one_hundred_and_twenty_five)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'one_hundred_and_twenty_five' is not defined

This is because CPython copies a reference to the builtins dictionary when it is building the frame, which happens before our code runs. So while this approach works for the repl, we will need to try something different to get it working everywhere.

Developing fishhook

2023-11-24T00:00:00+00:00

Fishhook was one of my first major low level Python projects. I set out with the goal to make an easy to use, dynamic Python dunder hooking tool, similar to forbiddenfruit. However, I wanted to improve on the methods used in forbiddenfruit, where many of the offsets and details were hard coded.

Disclaimer: This post goes into a lot of internal details about how CPython (the reference implementation of Python) works. I recommend reading the CPython C documentation and chunks of referred to source code for clarity

Background

Dunder methods in Python are implemented as function pointers located on the type instance. Some are located on the base instance, and some are located on substructures pointed to by the base instance. Some of the functions on the main instance are type->tp_call and type->tp_iter, while things like tp_add are located on type->tp_as_number->tp_add. In order for my hooking strategy to be nicely dynamic, I wanted to calculate as many offsets and memory locations at runtime based off of attributes that would change with updates. Moving forward, this enabled fishhook to work on multiple versions without major code changes, and also allowed fishhook to work with no changes when Python 3.10 released.

In order to replace a C implemented dunder with a Python implementation, the given tp_* function would need to be replaced with one that called the Python implementation. Also, for ease of use, a strategy for calling the original implementation needed to be provided.

All versions of fishhook provided the same basic hook and orig functions to provide this functionality, but the underlying implementation changed dramatically.

Major Version 1

While I was building fishhook, I went through a few strategies for calculating dunder function pointer locations. In reviewing various objects available in the Python interpreter, the first method I used involved some behavior of the wrapper_descriptor class. When dunders for C types are accessed directly, like int.__add__, the interpreter creates and returns an instance of wrapper_descriptor configured as a slot wrapper.

wrapper_descriptor instances are layed out as follows:

// cpython/blob/main/Include/cpython/descrobject.h
struct wrapperbase {
    const char *name;
    int offset;
    void *function;
    wrapperfunc wrapper;
    const char *doc;
    int flags;
    PyObject *name_strobj;
};

typedef struct {
    PyDescr_COMMON;
    struct wrapperbase *d_base;
    void *d_wrapped;
} PyWrapperDescrObject;

This layout yields a property instance->d_base.offset which details the offset from the beginning of a given tp_as_type dunder methods structure. This did not give me a way of knowing where the different tp_as_type structures were, but I could at least generate the function mapping into those structures at runtime.

The way I generated this mapping was not clean. I looped over every subclass in object.__subclasses__(), and looked for dunders. Then I checked if they were instances of wrapper_descriptor, and pulled out the offsets and the name of a given dunder. Now, the interesting thing about d_base.offset is that it is actually the offset of a given dunder on a PyHeapTypeObject, not a PyTypeObject. The objects are similar, with one key difference, in PyHeapTypeObjects, the tp_as_type structures are layed out directly after the PyTypeObject data. This means, given the offsets from the wrapperbase, you could determine where exactly a slot pointer for a given dunder needed to be placed.

I used this to generate a slotmap of __dunder__ -> (_size, _location, _index), where _size = sizeof(tp_as_type) / sizeof(void *), _location = offsetof(PyTypeObject, tp_as_type) and _index = offsetof(tp_as_type, tp_dunder). This meant that later, inside of a hook function, I could allocated a needed tp_as_type structure using _size, place the pointer to the structure as needed using _location, and place the given slot pointer using _index. If _location was 0, then the tp_dunder existed on the main instance, not on a substructure.

orig at this point consisted of a basic cache of the hooked slot wrapper instances, and would use some frame inspection to determine which was the correct one when called.

This system served its purposes pretty well for the first version of fishhook, but I still wanted to find a way to calculate the tp_as_type structure locations. At this point, they, along with the offset into PyWrapperDescrObject for grabbing offset, were the only hardcoded values. However, I was stuck on that for several months, and in the meantime made many quality of life improvements and patches to fishhook.

I finally determined a way to calculate the offsets and sizes of each tp_as_type structure when looking back over the definition of PyHeapTypeObject. I realized that I could find the locations of each structure by reading the memory of an entire instance of PyHeapTypeObject and looking for pointers that pointed into the object. With those pointers, I could calculate both the sizes and locations of each substructure in one pass.

# (Notes from my research)
# A-E are unknown size arrays

'''
(PyHeapTypeObject) [
  (PyTypeObject) [
  ...
  -> A
  ...
  -> C
  -> B
  -> D
  ...
  -> E
  ]
  A
  B
  C
  D
  E
  ...
]
'''

# 1. Collect ptr values to get starting addresses of A-E
#    by looking for pointers that direct within PyHeapTypeObject
from ctypes import (
    sizeof,
    c_void_p,
    c_char
)

basic_size = sizeof(c_void_p)

def mem(addr, size):
    return (c_char*size).from_address(addr)

class HeapTypeObj:
    __slots__ = ()

size = type(HeapTypeObj).__sizeof__(HeapTypeObj)
static_size = type.__sizeof__(type)
cls_mem = mem(id(HeapTypeObj), size).raw
address = id(HeapTypeObj)
pointers = [(offset, ptr) for offset, ptr in enumerate(memoryview(cls_mem).cast('l'))
                if address < ptr < address + len(cls_mem)]

# 2. Get Differences between ptr[B]-ptr[A], ... to get sizes
sizes = []
last_addr = None
for offset, ptr in sorted(pointers, key=lambda i:i[1]):
    if last_addr is not None:
        sizes.append(ptr - last_addr)
    last_addr = ptr

sizes.append(last_addr - ptr + len(cls_mem))

# 3. We now know the offsets and sizes of ptr[A-E] in PyTypeObject
structs = [(0, static_size)] \
        + [(offset, size) for (offset, _), size in zip(pointers, sizes)]

Once I had those values, I was able to make almost all of fishhook fully dynamic, and it even worked without any changes when Python updated from 3.9 to 3.10, despite a new dunder being added to tp_as_async

Major Version 2

The second major version of fishhook came with major changes internally. When 3.10 released, there was a new type flag Py_TPFLAGS_IMMUTABLETYPE that I noticed and started researching. In the process of this research, I found another semi-similar flag, Py_TPFLAGS_HEAPTYPE. I determined that with some minor coercion, I could toggle these flags, and convince the CPython implementation to swap all of the pointers I needed, negating the need for the large subclass loop from version one. The substructure size and locations were still important however, as I needed to allocate those on-demand prior to toggling flags.

Once I implemented these flags, I was able to get rid of a lot of the code that handled raw C pointers, and I spent a lot of time redesiging the strategy I used for calling original dunders. Now my solution walks back subclasses as well, so a hooked subclass’s orig call will call its parent classes dunder automatically.

I was also able to overcome another bug in version two, that actually stems from an assumption (bug?) in CPython. In CPython it is assumed that if tp_as_type structures are set, that the class must be a subclass of object. During execution, there are places that if a given tp_as_type is set, then tp_base->*properties* will be blindly accessed. This assumption is violated by fishhook when the base type object is hooked. Normally object has no tp_as_type substructures, and a null tp_base. When hooked, it suddenly has tp_as_type structures, and eventually, you run into a nasty crash. My solution for this was to manifest a fake base class, that object->tp_base would point to. This base class would have nearly nothing set, just the bare things needed for things to skip passed it when calculating inheritance. I also added a fake property to object.__base__ that returns None, to make it so that for normal code, object still appears to be the root base class.

After that fix, there were some small fixes for 3.12 involving the new feature of managed instance dictionaries, where I have to forcibly remove invalid ones, but thats it. Version two is still the current strategy, and it works fairly well.

The source code for fishhook can be found on github here