Optimizing with Cython

Ok, the quest for Python optimization continues. A friend of mine suggested that my stock options simulation in Python could be made much faster by using Cython, since it is a pure matematical simulation.

Cython is very easy to work with. Basically, it compiles a Python script into C, which then is compiled natively. Both steps are carried out automatically by distutils. Each .pyx module becomes a .so module, that can be imported by Python normally. (The main usage of Cython seems to be the writing of bindings.)

Actually, Cython understands a restricted dialect, but my script compiled with almost no changes. But — no pain, no gain: the performance boost of the compiled version was just 10% or so. Converting more and more of the code to the Cython 'dialect' (e.g. typed parameters, typed functions) increased the performance gain, but not much, not even near the gains offered by Psyco (which gives 4x for absolutely free).

The Activity Monitor's profiling showed that program spent too much time within the Python interpreter, that did not seem right, since we had compiled the thing, right? Time to take a look into generated C code. The main problem was usage of math and random modules in my code. Each time Cython code calls a Python module function, it needs to convert parameters from native to Python objects and reconvert the function returns.

Cython documentation explains how to use the C math functions directly. Also, I replaced seemingly innocent functions like max() by cdef'ed functions. Doing that increased the gain to around 1.4-1.5x.

Replacing random.gauss() was more difficult, since it does not have a C counterpart. I found a Java version in the Internet and converted it to Cython. Finally the efforts paid off: 23x performance gain over vanilla Python, 6x over Psyco.

The "modular" approach, and the fact that code needs some Cythonization work, means the programmer should profile the application and try to convert only the critical functions/classes/modules to Cython.

Since the scripts would need extensive rewrite to work under Cython, and I am happy enough with the current performance provided by Psyco, I am probably not going to use Cython in my simulations. But it is good to know that there are alternatives beyond Psyco (and before resorting to C++), should I need more performance.