Overview of what pypy is.

"I've worked on 3 implementations of new style classes, one for jython, one for python and one for pypy. Hopefully that is my last"

1.0 contains a fully compliant python interpreter, a tool chain which produces C, LLVM and .net python interpreters, JIT, optimizations, CLR friendly backend for PyPy.net and taint and proxy object spaces.

Can mix and match features in produced interpreters

Needs polishing and missing some important extension modules

Special language features and rpython already useful

Performance is getting quite good, approximately 1.2x to 3x slower on average. In one case 6x slower.

Working on GC performance

JIT true trump card

JIT huge investment in resources, hard to evolve

Created a JIt generation framework (timeshifter)

Partial evaluation techniques to generate dynamic compiler

Psyco inspired

Of course hard to implement

Types in a dynamic language are a problem

e.g. know an operation is an addition, but of what? Makes constant folding hard

Solution is to "promote" run time interpreter parts to compiler parts, get information from the evaluation

Use other tricks like lazy allocation of objects

Dynamic generation process language agnostic

E.g. someone wrote a prolog interpreter in rpython, so get JIT for prolog

ia32 and ppc backengs

Int arithmetic optimized

in speed range of gcc -o0

In some cases, e.g. demo 63x faster than CPython!

"If you need exactly this example..."

Implemented as a transformation of the low level control flow graphs, use graph "colouring" to analyse. Timeshifting, turning run time graph into compile time graph

"green" - compile time value

"red" - run time value

see graph pics

Specialize function for given values of green values, e.g. f(x, y) becomes f_3(y) when x = 3

Need to handle conditional cases, .e.g code conditional on red code.

Also need to handle merges, where can reuse code

Need to catch loops

Can manually hint promotion:

def f(x,y):
  x1 = hint(x, promote=True)
  return x1 * x1 + y * y

First generates:

def f_(x, y):
  switch x:
    pass
  default:
    compile_more(value=x)

After called with 3:

def f_(x, y):
  switch x:
    case 3:
      return 9 + y * y
  default:
    compile_more(value=x)

Obviously works well with repeated calls with same values

Adding new features to the interpreter

Integrating into different environments, e.g. clr

Handling calls to and from the target environment

Features are independent of backend, e.g. security tainting of objects

Transparent proxies, e.g. distribution or peristence

Demo of rope, behaves like a string :)

million char string constructs faster

transparent proxy of list shared between processes

frames shared transparently

e.g. remote_open("/etc/passwd").read()

can attach pdb to traceback of remote object

orthogonal persistence

taint object space allows you to control where objects get accessed. E.g. prevents sensitive object getting logged:

>>> x = 3
>>> x
3
>>> taint(x)
>>> x
...exception...
>>> l = [x]
>>> l
...exception...

tainted object can't cross i/o barrier until untainted. But can still fully manipulate.

stackless

getting faster and faster

If you're not too dependent on extension modules pypy is getting very close to being able to run your application as is

OS threads don't work very well