2.10. Robustness Principle

TCP implementations should follow a general principle of robustness:
be conservative in what you do, be liberal in what you accept from
others.

RFC 1122 elaborates with section 1.2.2.:

1.2.2 Robustness Principle

At every layer of the protocols, there is a general rule whose
application can lead to enormous benefits in robustness and
interoperability [IP:1]:

“Be liberal in what you accept, and
conservative in what you send”

Software should be written to deal with every conceivable
error, no matter how unlikely; sooner or later a packet will
come in with that particular combination of errors and
attributes, and unless the software is prepared, chaos can
ensue.
[...]

For anything expected to inter-operate “in the wild” with other implementations of a given standard or API this approach is optimal as both competitive advantage (how successful would a browser that only accepted valid HTML have been in a world with Netscape 1.0?) and user experience (always working as best as possible increases the chance of something working).

However, when an API is internal to a component, department, or a company it might well be better for implementations to reject fast invalid or malformed requests with informative error messages rather than attempting to proceed.

Within an organization the chances of integration testing catching an error before it affects user experience is much higher and failing immediately helps reduce the chances of future revisions of software needing to be “bug compatible” with easily avoided problems. It’s not a competitive advantage to try to muddle along with bad input in a world where there are no competitors. The entire system is more robust if each side of every API is careful that they are producing only valid requests and responses.

For example, if a company has standardized on UTF-8 as a wire format for text it is probably best if all new implementations of services validate that their input is correct UTF-8 at every edge and refuse to process anything that is invalid. Otherwise eventually some system is going to end up having to try and guess the correct encoding in order to proceed and serve valid data that came in an API that didn’t validate and has to go out an API that must be valid.

This only applies, of course, when both sides of an integration are new enough to be tested together but this applies to a lot of new development within an organization — a new capability is being added to an entire “stack” resulting in API revisions at each layer.

It’s dramatically easier to catch problems at the first opportunity and requiring the source of the incorrectness to correct it before proceeding than ending up years later with tons of special compatibility hacks where an API is versioned based on the different clients foibles rather than by design.

Internal APIs should be designed to be as easy as possible to use correctly — but once designed they should only work when used correctly.

Several Ubuntu updates later I noticed libyahoo2 had been upgraded and, sure enough, this version worked.

I updated the Git repository mentioned in that post but it is likely if I continue to work on it it will go to python-yahoo2 on GitHub.

Pyrex’s blindness where “const” is concerned is pretty annoying.

The upgrades from Ubuntu v.older to 9.04 went quite smoothly on every machine including the ones where I had to do several stepped upgrades since on various machines I skipped one or more upgrades prior to 9.04.

Rather than return to my old ways of just hosing packages (and, worse, “setup.py develop” symlinks) into /usr/local I’ve decided to use python-virtualenv to create some non-root-owned Python environments to work on my various Python-y apps. The main difference to me is being able to keep things I’m working on separate from one another.

We’ll see how it goes.

I’m afraid all these years of using package-managed software has made me soft — now a piece of software not having a nice package means I need to think a bit harder about if it’s worth dealing with the hassle of having that software outside of the package manager. (That applies to when I’d like to be using a more recent version of the package, too, though generally on Ubuntu I run into that a lot less frequently than I did on Debian.) I have mixed feelings about setuptools and easy_install.

I am pleased that now mod_wsgi is in a package so I can switch to that version rather than the one I installed by hand. mod_wsgi was worth it to install by hand to use until there was a nice package available. It’s Just Better than my previous mechanisms for running WSGI apps in, under, or behind Apache.

Prior to today every time I took the time to try and update the source tree on Sourceforge I ran into a snag such as “Sourceforge CVS is unavailable.” Months ago, when I tried to migrate from CVS to SVN on Sourceforge I ran into errors. I didn’t even try to investigate what went awry as I was low on patience and time.

As a result, my own tree (first in Perforce and now in Subversion) has gotten pretty out of sync from that tree. I decided to try GitHub and try using Git for something “real” — so I’m going to move Singleshot’s source to GitHub and then push all my changes to there.

I followed (mostly) the steps from How to export revision history from mercurial or git to cvs which describes how to use the git cvsimport command to pull Singleshot’s revision history from its Sourceforge CVS repository, testing making a tiny change to a README, and pushing that change back to the Sourceforge CVS tree.

(After backing up the source tree on Sourceforge using rsync.)

% export CVS_RSH=ssh
% git cvsimport -d :ext:xythian@singleshot.cvs.sourceforge.net:/cvsroot/singleshot -C test1 \
    -r cvs -k -A /home/fox/src/singleshot-import/authors.txt  singleshot
Initialized empty Git repository in /home/fox/src/singleshot-import/test1/.git/
[longish pause]
Counting objects: 561, done.
Compressing objects: 100% (502/502), done.
Writing objects: 100% (561/561), done.
Total 561 (delta 317), reused 0 (delta 0)

(Some poking around to see if things look reasonable, making a small change to the README, commiting it first to the git repository and then back to Sourceforge using git cvsexportcommit.)

Then I pushed the whole thing to a GitHub tree:

http://github.com/xythian/singleshot/tree/master

% git remote add origin git@github.com:xythian/singleshot.git
% git push origin master

That was pretty straightforward and appears to have worked.

My plan is to merge my changes into the out of date tree and then push the new code to GitHub. If it all works out I’ll probably indicate that the Sourceforge project is defunct there and host the code on GitHub henceforth.

It rapidly became clear that I was going to want a layer on top of the library to make interacting with it from Python more palatable.   I switched to Pyrex, since I wanted to write that wrapper in Python (or something Python-like) rather than building a straight-C wrapper so using SWIG would continue to make sense.   SWIG’s big benefit in my mind over Pyrex is easy support for more languages and better tools for defining straight wrappers.  I wasn’t going to get "free" use in other languages and it wasn’t going to be a straight wrapper now but rather a module that exposed the functionality of libyahoo2 to Python.

I kept the callbacks.c I had defined but started migrating the definitions to the Pyrex file as I implemented.  This way my library would continue to link without complaint about functions I didn’t have yet.

Following the usual pattern for Python bindings to libraries that need wrappers to be more Pythonic, I planned to have a ‘yahoo2′ module in Python and a ‘_yahoo2.so’ extension module.  The _yahoo2 module is written in Pyrex.

libyahoo2 appears not to adhere to the documentation it defines.  It’ll call ext_yahoo_remove_handler with a tag that was never returned by ext_yahoo_add_handler… (0).  It looks like an undocumented (that I found) part of the charter of ext_yahoo_add_handler is not to add a given handler more than once.

This also made defining the callbacks the way libyahoo2 expected easier.

D’oh, I got it far enough along to get this:

libyahoo2.c:620: debug: Key: 4 Value: Yahoo_Messenger
libyahoo2.c:620: debug: Key: 5 Value: hodorbot
libyahoo2.c:620: debug: Key: 14 Value: This version of Messenger expired on April 2, 2008. Please upgrade now to the latest supported version: http://messenger.yahoo.com Learn more: http://messenger.yahoo.com/eol
libyahoo2.c:620: debug: Key: 15 Value: 1225167367
libyahoo2.c:620: debug: Key: 97 Value: 1

I’ve learned what I wanted to, so instead of seeing if a more recent libyahoo2 than what’s in Ubuntu works (> 0.7.5+dfsg-3), I’m just going to call it here.  I’ll post it in case someone can learn something useful from it.

You can pull a working repository with a command like:

git clone http://notes.xythian.net/media/2008/10/pythonlibyahoo2.git/ mydirectory

There’s three files that do anything:

• yahoo2.py – wraps some of the lower level details from the Pyrex layer, including exposing the IO bits as an asyncore dispatcher
• _yahoo2.pyx – is the binding
• callbacks.c – exists to have empty functions defined to satisfy the linker until those have implementations in the Python binding

Not much works, really; there’s implementations of connect and async_connect, but only async_connect is called by libyahoo2 before it gets far enough along to not log in with the error message above. There’s a wrapper for setting the log level (which was key to discovering the above fact…).

This is a typical definition of one of the library callbacks:

cdef public int ext_yahoo_connect_async(int id, char *host, int port, \
  yahoo_connect_callback callback, void *callback_data):
   cdef ConnectionHandle handle
   handle = ConnectionHandle(id, host, port)
   handle.connect_callback = callback
   handle.connect_data = callback_data
   handle.async_connect(callback, callback_data)
   HANDLER_MAP[id].connections.append(handle)
   MANAGER.add(handle)
   return handle.fileno()

ConnectionHandle is an extension class which wraps a Python socket object and can make the callbacks the libyahoo2 IO layer expect.   MANAGER is the connection manager, which is implemented in the higher level layer (yahoo2.py) so it could be replaced with something that interacted with a GUI event loop.

I’m probably done with this for the forseeable future and hope publishing it along with a git repoistory may allow someone to build on or learn from what I’ve done.   I stumbled upon Cython while working on this, but didn’t want to derail any progress to something working by playing with it.

So what did I learn?

• Write toy bots against open protocols with existing libraries in your language of choice, such as XMPP, lest your project get hijacked working on a binding rather than the toy.
• Pyrex is pretty nice for building wrappers that have more meat than a typical SWIG binding.
• Cython is probably worth checking out.
• git is worth playing with more

The symptom is a crash when your cdb object is deallocated usually with a “double-free” memory corruption error message. Assuming a .cdb file named “foo.cdb”, the following script will repro the bug:

#!/usr/bin/python
import cdb
c = cdb.init(‘foo.cdb’)
del c

with the following message:

fox@hercules:~$ python cdbrepro.py
*** glibc detected *** python: free(): invalid pointer: 0x00002b56c25bd750 ***
======= Backtrace: =========
/lib/libc.so.6[0x2b56c300ab0a]
/lib/libc.so.6(cfree+0x8c)[0x2b56c300e6fc]
python(PyDict_DelItem+0xfa)[0x44370a]
python(PyEval_EvalFrameEx+0x2e40)[0x485140]
python(PyEval_EvalCodeEx+0×830)[0x489d60]
python(PyEval_EvalCode+0×32)[0x489da2]
python(PyRun_FileExFlags+0x10e)[0x4ab4fe]
[...]
Aborted (core dumped)

Some other searching suggests that python-cdb’s use of PyMem_DEL is no longer recommended. I haven’t verified that this doesn’t cause other problems, but replacing cdbmodule.c’s use of PyMem_DEL with PyObject_Del (and the PyObject_NEW with _New, to use consistent naming).

As soon as Ubuntu’s bug tracker (launchpad) works again for me I’ll report the bug. Launchpad is timing out with an error message for me now.

Flex uses ActionScript which bears a remarkable similarity to JavaScript. It’s not quite the same but it’s close enough that I haven’t stumbled too badly with it working from JavaScript knowledge.  As with any "new" language the learning curve isn’t the language but the environment and libraries.

The experience with Flex and also the Greasemonkey hacking I’ve been doing lately has made me rather fonder of JavaScript than I’ve been in the past.  JavaScript is particularly nice when used in an environment that is safe from worrying about cross-browser API woes — such as a Firefox-only Greasemonkey user script or an environment like Flex. It’s not ideal but it has a lot of what I like in a language, including garbage collection, closures, reasonable typing and widespread deployment.

Launchy

Free, open-source Windows app that indexes Program Files and any other directories you tell it to. Then alt-space pops up a command line box and it autocompletes as you type. I installed it a while ago and meant to mention it but it kind of faded into the background and I don’t think about using it anymore. It’s just there. No more do I need to concern myself with the program menu’s lengthy “organization” by vendor nor do I keep needing to add paths to my PATH. I used to do that so I could use Windows-Run to run things rather than rummaging in Program Files but Launchy is a much better solution. I’ve read that it resembles Quicksilver for the Mac.

SQLAlchemy

“SQLAlchemy is the Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.” — mostly it handles some tedium without getting in the way. ORMs still make me nervous since they tend to have a lot of “magic” in them but SQLAlchemy makes it pretty easy to use the convenient parts and override the “magic” if/when it’s necessary for performance. Which probably won’t happen too much but knowing it’s easy makes it easier to rely on SQLAlchemy in the meantime. I don’t know if I’ll keep using it, because like every other time I use a library like this I find myself spending more time trying to figure out how SQLAlchemy expresses something and I already know how to use SQL.

The obvious solution is to hook OnBeforeNavigate2 like so:

        self.Bind(iewin.EVT_BeforeNavigate2, self.OnBeforeNavigate2, self.ie)
...
    def OnBeforeNavigate2(self, evt):
        wx.LaunchDefaultBrowser(evt.URL)
        evt.Cancel = True

but if the link the user clicked is set to open a new window, the emebdded IE will just open a new window. So you hook NewWindow2. Oops, NewWindow2 doesn’t tell you what the URL is!

Here’s what I did, since I don’t care if my reader works on pre-SP2 Windows XP:

Define an Event to permit me to hook NewWindow3, which is a new event that includes the URL being opened, then hook it appropriately: