python / xinetd / virtualenv

So, while developing a server application for a client, my colleague Harm decided it would be a waste of our programming time to add TCP server code.

Inetd and friends can do that really well. The amount of new connects to the server would be minimal, so the overhead of spawning a new Python process for every connect was negligible.

Using xinetd as an inetd server wrapper is simple. The config would look basically like this:

service my_server
{
    ...
    port        = 20001
    server      = /path/to/virtualenv/bin/python
    server_args = /path/to/app/server.py
    ...

Yes! That’s right. We can call the python executable from the virtualenv directory and get the right environment without having to call the ‘activate’ wrapper.

We can? Yes, we can. Check this out:

$ cd /path/to/very-minimal-virtualenv
$ ls -l `find . -type f -o -type l`
-rwxr-xr-x 1 walter walter 3773512 feb 11 17:08 ./bin/python
lrwxrwxrwx 1 walter walter      24 feb 12 08:58 ./lib/python2.7/os.py -> /usr/lib/python2.7/os.py
-rw-rw-r - 1 walter walter       0 feb 12 08:57 ./lib/python2.7/site.py
-rw-rw-r - 1 walter walter     126 feb 12 09:00 ./lib/python2.7/site.pyc
$ ./bin/python -c 'import sys; print sys.prefix'
/path/to/very-minimal-virtualenv

Awesome, then we won’t need a wrapper that sources ./bin/activate.

Except… it didn’t work when called from xinetd! The python executable called from xinetd stubbornly decided that sys.prefix points to /usr: the wrong Python environment would be loaded.

If we added in any random wrapper application, things would work:

    server      = /usr/bin/env
    server_args = /path/to/virtualenv/bin/python /path/to/app/server.py

And this worked:

    server      = /usr/bin/time
    server_args = /path/to/virtualenv/bin/python /path/to/app/server.py

And it worked when we explicitly set PYTHONHOME, like this:

    server      = /path/to/virtualenv/bin/python
    server_args = /path/to/app/server.py
    env         = PYTHONHOME=/path/to/virtualenv

So, what was different?

Turns out it was argv[0].

The “what prefix should I use” code is found in the CPython sources in Modules/getpath.c calculate_path():

    char *prog = Py_GetProgramName();
    ...
    if (strchr(prog, SEP))
            strncpy(progpath, prog, MAXPATHLEN);
    ...
    strncpy(argv0_path, progpath, MAXPATHLEN);
    argv0_path[MAXPATHLEN] = '\0';
    ...
    joinpath(prefix, LANDMARK);
    if (ismodule(prefix))
        return -1; /* we have a valid environment! */
    ...

Where Py_GetProgramName() was initialized by Py_Main(), and LANDMARK happens to be os.py:

    Py_SetProgramName(argv[0]);

#ifndef LANDMARK
#define LANDMARK "os.py"
#endif

And you’ve guessed it by now: xinetd was kind enough — thanks dude — to strip the dirname from the server before passing it to execve(2). We expected it to call this:

execve("/path/to/virtualenv/bin/python",
       ["/path/to/virtualenv/bin/python", "/path/to/app/server.py"],
       [/* 1 var */])

But instead, it called this:

execve("/path/to/virtualenv/bin/python",
       ["python", "/path/to/app/server.py"],
       [/* 1 var */])

And that caused Python to lose its capability to find the closest environment.

After figuring that out, the world made sense again.