Skip to content

Latest commit

 

History

History
197 lines (142 loc) · 7.26 KB

dev-intro.md

File metadata and controls

197 lines (142 loc) · 7.26 KB
Raw

Introduction for Mypyc Contributors

Supported Features

Mypyc supports a subset of Python. If you try to compile something that is not supported, you are not likely to get a good error message.

Here are some major things that aren't supported in compiled code:

  • Unannotated functions
  • Functions that take *args or **kwargs
  • Many dunder methods (only some work, such as __init__ and __eq__)
  • Monkey patching compiled functions or classes
  • Metaclasses
  • Async features
  • Generally Python 3.5+ only features
  • General multiple inheritance (a limited form is supported))
  • Classes or functions that use type variables with value restrictions
  • Self types
  • TypedDict
  • Named tuple defined using the class-based syntax
  • Complex numbers
  • Defining protocols
  • Defining overloaded functions
  • # type: ignore
  • del name

We aren't focused on Python feature completeness right now. Instead, we support a Python subset that is good enough to compile mypy. We are generally happy to accept contributions that implement new Python features, however.

High-level Overview

Mypyc compiles a Python module to C, and compiles that to a Python C extension module.

It has these passes:

  • Type check the code using mypy and infer types for variables and expressions.
  • Translate the mypy AST into a mypyc-specific intermediate representation (IR).
    • The IR is defined in mypyc.ops.
    • The translation happens in mypyc.genops.
  • Insert exception handling (mypyc.exceptions).
  • Insert explicit reference count inc/dec opcodes (mypyc.refcount).
  • Translate the IR into C (mypyc.emit*).
  • Compile the generated C code using a C compiler.

Tests

The test cases are defined in the same format (.test) as used in the mypy project. Look at mypy developer documentation for a general overview of how things work. Test cases live under test-data/.

Type-checking Mypyc

One of the tests (test_self_type_check) type checks mypyc using mypy.

Overview of Generated C

Mypyc uses a tagged pointer representation for integers, char for booleans, and C structs for tuples. For most other objects mypyc uses the CPython PyObject *.

Mypyc compiles a function into two functions:

  • The native function takes a fixed number of C arguments with the correct C types. It assumes that all argument have correct types.
  • The wrapper function conforms to the Python C API calling convention and takes an arbitrary set of arguments. It processes the arguments, checks their types, unboxes values with special representations and calls the native function. The return value from the native function is translated back to a Python object ("boxing").

Calls to other compiled functions don't go through the Python module namespace but directly call the target native function. This makes calls very fast compared to CPython.

The generated code does runtime checking so that it can assume that values always have the declared types. Whenever accessing CPython values which might have unexpected types we need to insert a type check. For example, when getting a list item we need to insert a runtime type check (an unbox or a cast operation), since Python lists can contain arbitrary objects.

The generated code uses various helpers defined in lib-rt/CPy.h. The header should only contain inline or static functions, since we don't compile the C helpers into a separate object file.

Other Important Limitations

All of these limitations will likely be fixed in the future:

  • We don't detect infinite recursion.

  • We don't handle Ctrl-C in compiled code.

Hints for Implementing Typical Mypyc Features

This section gives an overview of where to look for and what to do to implement specific kinds of mypyc features.

Syntactic Sugar

Syntactic sugar that doesn't need additional IR operations typically only requires changes to mypyc.genops. Test cases are located in test-data/genops-*.test and the test driver is in mypyc.test.test_genops.

You may also need to add some definitions to the stubs used for builtins during tests (test-data/fixtures/ir.py). We don't use full typeshed stubs to run tests since they would seriously slow down tests.

Adding C Helpers

If you add an operation that compiles into a lot of C code, you may also want to add a C helper function for the operation to make the generated code smaller. Here is how to do this:

  • Add the operation to lib-rt/CPy.h. Usually defining a static function is the right thing to do, but feel free to also define inline functions for very simple and performance-critical operations. We avoid macros since they are error-prone.

  • Add unit test for your C helper in lib-rt/test_capi.cc. We use Google Test for writing tests in C++. The framework is included in the repository under the directory googletest/. The C unit tests are run as part of the pytest test suite (test_c_unit_tests).

A New Primitive Type

Some types such as int and list are special cased in mypyc to generate operations specific to these types.

Here are some hints about how to add support for a new primitive type (this may be incomplete):

  • Decide whether the primitive type has an "unboxed" representation (a representation that is not just PyObject *).

  • Create a new instance of RPrimitive to support the primitive type. Make sure all the attributes are set correctly and also define <foo>_rprimitive and is_<foo>_rprimitive.

  • Update mypyc.genops.Mapper.type_to_rtype().

  • Update emit_box in mypyc.emit.

  • Update emit_unbox or emit_cast in mypyc.emit.

  • Update emit_inc_ref and emit_dec_ref in mypypc.emit if needed. If the unboxed representation does not need reference counting, these can be no-ops. If the representation is not unboxed these will already work.

  • Update emit_error_check in mypyc.emit for unboxed types.

  • Update emit_gc_visit and emit_gc_clear in mypyc.emit if the type has an unboxed representation with pointers.

The above may be enough to allow you to declare variables with the type and pass values around. You likely also want to add support for some primitive operations for the type (see Built-in Operation for an Already Supported Type for how to do this).

If you want to just test C generation, you can add a test case with dummy output to test-data/module-output.test and manually inspect the generated code. You probably don't want to commit a new test case there since these test cases are very fragile.

Add a test case to test-data/run.test to test compilation and running compiled code. Ideas for things to test:

  • Test using the type for an argument.

  • Test using the type for a return value.

  • Test passing a value of the type to a function both within compiled code and from regular Python code. Also test this for return values.

  • Test using the type as list item type. Test both getting a list item and setting a list item.

Other Hints

  • This developer documentation is not very complete and might be out of date.

  • It can be useful to look through some recent PRs to get an idea of what typical code changes, test cases, etc. look like.

  • Feel free to open GitHub issues with questions if you need help when contributing, or ask questions in existing issues. Note that we only support contributors. Mypyc is not (yet) an end-user product.