Several Ubuntu updates later I noticed libyahoo2 had been upgraded and, sure enough, this version worked.

I updated the Git repository mentioned in that post but it is likely if I continue to work on it it will go to python-yahoo2 on GitHub.

Pyrex’s blindness where “const” is concerned is pretty annoying.

The upgrades from Ubuntu v.older to 9.04 went quite smoothly on every machine including the ones where I had to do several stepped upgrades since on various machines I skipped one or more upgrades prior to 9.04.

Rather than return to my old ways of just hosing packages (and, worse, “setup.py develop” symlinks) into /usr/local I’ve decided to use python-virtualenv to create some non-root-owned Python environments to work on my various Python-y apps. The main difference to me is being able to keep things I’m working on separate from one another.

We’ll see how it goes.

I’m afraid all these years of using package-managed software has made me soft — now a piece of software not having a nice package means I need to think a bit harder about if it’s worth dealing with the hassle of having that software outside of the package manager. (That applies to when I’d like to be using a more recent version of the package, too, though generally on Ubuntu I run into that a lot less frequently than I did on Debian.) I have mixed feelings about setuptools and easy_install.

I am pleased that now mod_wsgi is in a package so I can switch to that version rather than the one I installed by hand. mod_wsgi was worth it to install by hand to use until there was a nice package available. It’s Just Better than my previous mechanisms for running WSGI apps in, under, or behind Apache.

Prior to today every time I took the time to try and update the source tree on Sourceforge I ran into a snag such as “Sourceforge CVS is unavailable.” Months ago, when I tried to migrate from CVS to SVN on Sourceforge I ran into errors. I didn’t even try to investigate what went awry as I was low on patience and time.

As a result, my own tree (first in Perforce and now in Subversion) has gotten pretty out of sync from that tree. I decided to try GitHub and try using Git for something “real” — so I’m going to move Singleshot’s source to GitHub and then push all my changes to there.

I followed (mostly) the steps from How to export revision history from mercurial or git to cvs which describes how to use the git cvsimport command to pull Singleshot’s revision history from its Sourceforge CVS repository, testing making a tiny change to a README, and pushing that change back to the Sourceforge CVS tree.

(After backing up the source tree on Sourceforge using rsync.)

% export CVS_RSH=ssh
% git cvsimport -d :ext:xythian@singleshot.cvs.sourceforge.net:/cvsroot/singleshot -C test1 \
    -r cvs -k -A /home/fox/src/singleshot-import/authors.txt  singleshot
Initialized empty Git repository in /home/fox/src/singleshot-import/test1/.git/
[longish pause]
Counting objects: 561, done.
Compressing objects: 100% (502/502), done.
Writing objects: 100% (561/561), done.
Total 561 (delta 317), reused 0 (delta 0)

(Some poking around to see if things look reasonable, making a small change to the README, commiting it first to the git repository and then back to Sourceforge using git cvsexportcommit.)

Then I pushed the whole thing to a GitHub tree:

http://github.com/xythian/singleshot/tree/master

% git remote add origin git@github.com:xythian/singleshot.git
% git push origin master

That was pretty straightforward and appears to have worked.

My plan is to merge my changes into the out of date tree and then push the new code to GitHub. If it all works out I’ll probably indicate that the Sourceforge project is defunct there and host the code on GitHub henceforth.

It rapidly became clear that I was going to want a layer on top of the library to make interacting with it from Python more palatable.   I switched to Pyrex, since I wanted to write that wrapper in Python (or something Python-like) rather than building a straight-C wrapper so using SWIG would continue to make sense.   SWIG’s big benefit in my mind over Pyrex is easy support for more languages and better tools for defining straight wrappers.  I wasn’t going to get "free" use in other languages and it wasn’t going to be a straight wrapper now but rather a module that exposed the functionality of libyahoo2 to Python.

I kept the callbacks.c I had defined but started migrating the definitions to the Pyrex file as I implemented.  This way my library would continue to link without complaint about functions I didn’t have yet.

Following the usual pattern for Python bindings to libraries that need wrappers to be more Pythonic, I planned to have a ‘yahoo2′ module in Python and a ‘_yahoo2.so’ extension module.  The _yahoo2 module is written in Pyrex.

libyahoo2 appears not to adhere to the documentation it defines.  It’ll call ext_yahoo_remove_handler with a tag that was never returned by ext_yahoo_add_handler… (0).  It looks like an undocumented (that I found) part of the charter of ext_yahoo_add_handler is not to add a given handler more than once.

This also made defining the callbacks the way libyahoo2 expected easier.

D’oh, I got it far enough along to get this:

libyahoo2.c:620: debug: Key: 4 Value: Yahoo_Messenger
libyahoo2.c:620: debug: Key: 5 Value: hodorbot
libyahoo2.c:620: debug: Key: 14 Value: This version of Messenger expired on April 2, 2008. Please upgrade now to the latest supported version: http://messenger.yahoo.com Learn more: http://messenger.yahoo.com/eol
libyahoo2.c:620: debug: Key: 15 Value: 1225167367
libyahoo2.c:620: debug: Key: 97 Value: 1

I’ve learned what I wanted to, so instead of seeing if a more recent libyahoo2 than what’s in Ubuntu works (> 0.7.5+dfsg-3), I’m just going to call it here.  I’ll post it in case someone can learn something useful from it.

You can pull a working repository with a command like:

git clone http://notes.xythian.net/media/2008/10/pythonlibyahoo2.git/ mydirectory

There’s three files that do anything:

• yahoo2.py – wraps some of the lower level details from the Pyrex layer, including exposing the IO bits as an asyncore dispatcher
• _yahoo2.pyx – is the binding
• callbacks.c – exists to have empty functions defined to satisfy the linker until those have implementations in the Python binding

Not much works, really; there’s implementations of connect and async_connect, but only async_connect is called by libyahoo2 before it gets far enough along to not log in with the error message above. There’s a wrapper for setting the log level (which was key to discovering the above fact…).

This is a typical definition of one of the library callbacks:

cdef public int ext_yahoo_connect_async(int id, char *host, int port, \
  yahoo_connect_callback callback, void *callback_data):
   cdef ConnectionHandle handle
   handle = ConnectionHandle(id, host, port)
   handle.connect_callback = callback
   handle.connect_data = callback_data
   handle.async_connect(callback, callback_data)
   HANDLER_MAP[id].connections.append(handle)
   MANAGER.add(handle)
   return handle.fileno()

ConnectionHandle is an extension class which wraps a Python socket object and can make the callbacks the libyahoo2 IO layer expect.   MANAGER is the connection manager, which is implemented in the higher level layer (yahoo2.py) so it could be replaced with something that interacted with a GUI event loop.

I’m probably done with this for the forseeable future and hope publishing it along with a git repoistory may allow someone to build on or learn from what I’ve done.   I stumbled upon Cython while working on this, but didn’t want to derail any progress to something working by playing with it.

So what did I learn?

• Write toy bots against open protocols with existing libraries in your language of choice, such as XMPP, lest your project get hijacked working on a binding rather than the toy.
• Pyrex is pretty nice for building wrappers that have more meat than a typical SWIG binding.
• Cython is probably worth checking out.
• git is worth playing with more

The symptom is a crash when your cdb object is deallocated usually with a “double-free” memory corruption error message. Assuming a .cdb file named “foo.cdb”, the following script will repro the bug:

#!/usr/bin/python
import cdb
c = cdb.init(‘foo.cdb’)
del c

with the following message:

fox@hercules:~$ python cdbrepro.py
*** glibc detected *** python: free(): invalid pointer: 0x00002b56c25bd750 ***
======= Backtrace: =========
/lib/libc.so.6[0x2b56c300ab0a]
/lib/libc.so.6(cfree+0x8c)[0x2b56c300e6fc]
python(PyDict_DelItem+0xfa)[0x44370a]
python(PyEval_EvalFrameEx+0x2e40)[0x485140]
python(PyEval_EvalCodeEx+0×830)[0x489d60]
python(PyEval_EvalCode+0×32)[0x489da2]
python(PyRun_FileExFlags+0x10e)[0x4ab4fe]
[...]
Aborted (core dumped)

Some other searching suggests that python-cdb’s use of PyMem_DEL is no longer recommended. I haven’t verified that this doesn’t cause other problems, but replacing cdbmodule.c’s use of PyMem_DEL with PyObject_Del (and the PyObject_NEW with _New, to use consistent naming).

As soon as Ubuntu’s bug tracker (launchpad) works again for me I’ll report the bug. Launchpad is timing out with an error message for me now.

Launchy

Free, open-source Windows app that indexes Program Files and any other directories you tell it to. Then alt-space pops up a command line box and it autocompletes as you type. I installed it a while ago and meant to mention it but it kind of faded into the background and I don’t think about using it anymore. It’s just there. No more do I need to concern myself with the program menu’s lengthy “organization” by vendor nor do I keep needing to add paths to my PATH. I used to do that so I could use Windows-Run to run things rather than rummaging in Program Files but Launchy is a much better solution. I’ve read that it resembles Quicksilver for the Mac.

SQLAlchemy

“SQLAlchemy is the Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.” — mostly it handles some tedium without getting in the way. ORMs still make me nervous since they tend to have a lot of “magic” in them but SQLAlchemy makes it pretty easy to use the convenient parts and override the “magic” if/when it’s necessary for performance. Which probably won’t happen too much but knowing it’s easy makes it easier to rely on SQLAlchemy in the meantime. I don’t know if I’ll keep using it, because like every other time I use a library like this I find myself spending more time trying to figure out how SQLAlchemy expresses something and I already know how to use SQL.

The obvious solution is to hook OnBeforeNavigate2 like so:

        self.Bind(iewin.EVT_BeforeNavigate2, self.OnBeforeNavigate2, self.ie)
...
    def OnBeforeNavigate2(self, evt):
        wx.LaunchDefaultBrowser(evt.URL)
        evt.Cancel = True

but if the link the user clicked is set to open a new window, the emebdded IE will just open a new window. So you hook NewWindow2. Oops, NewWindow2 doesn’t tell you what the URL is!

Here’s what I did, since I don’t care if my reader works on pre-SP2 Windows XP:

Define an Event to permit me to hook NewWindow3, which is a new event that includes the URL being opened, then hook it appropriately:

A brief history of this project:

• January 2005 — A proof-of-concept toy that used an IMAP store and copied posts from a single RSS feed to an IMAP server. I was still using Thunderbird and rawdog to do my feed reading at this point.
• October 2005 — began using my new aggregator. It had two components — a fetcher which ran from cron, used sqlite to keep track of state, and emitted new article messages to a Spread group and an IMAP storer which read from the Spread group and copied articles into an IMAP folder. I read the IMAP folder using Thunderbird.
• March 2006 — I collapsed the components into a single fetcher which ran from cron, used a SQLite data store, and put messages into an IMAP folder. I also improved the feed scheduling to reduce the number of fetches on relatively inactive feeds.
• January 2007 — moved from the SQLite-based aggregator to a new aggregator based on PostgreSQL. The new reader implements its own IMAP server using Twisted. It’s much faster than the old SQLite solution and keeps all of the state in the database rather than having “feed state” in a SQLite database and “user state” (flagging, read state) in an IMAP folder.

I made a very basic web UI with the SQLite-based aggreagtor which allowed me to subscribe to new feeds with a bookmarklet. I migrated that UI to the new aggregator. It was always inadequate and I supplemented it with manual SQL commands to manage feeds. I’d sit down to enhance the interface but keep getting distracted playing with Ajax toolkits because the UI I had in mind didn’t fit very well into the web browser.

This weekend I began working on a desktop client. It’ll be backed by the same database so the IMAP server will continue to work in case I want to read via IMAP for some reason such as to do offline reading. I have a very crude GUI which has already let me do some management of the feeds which was awkward to do before but the desktop client still exists more as doodles in a notebook than code.

The first version connects directly to the PostgreSQL database. Eventually I plan to split the database access out into its own tier. I don’t yet know enough of what I want that protocol to look like to do that now and want to get a better feel for what the client will need before taking that step. The plan to split the storage layer into its own tier guides the design of that layer.

There’s always been a bit of a mismatch between my needs in a mail client and my needs for a feed reader but until recently I put up with all the little cuts that resulted from using a mail client for feed reading because the benefits of a desktop experience and shared server state outweighed the pain of using a mail client (and IMAP) for feed reading.

I’m using wxPython to develop the client so at least in theory it should work on Windows, Linux, and Mac OS. I only plan to test and develop using Windows for the foreseeable future, though.

No project of mine advances as quickly as those that scratch an itch or alleviate a pain for me.

I still don’t know if I’ll ever release any of this. It would probably be academic since I imagine the demand for this kind of thing is very low given how excited people get about “no install” web readers. I think the the thought process that guides design decisions in the reader is more interesting to other folks than the actual code. I suppose the RSS fetcher part of the system might be interesting to other folks as its own component.

I’m converting my RSS aggregator to use PostgreSQL as its back-end instead of SQLite.  It has served me well for quite a while but it’s time to move on from SQLite so I can add some features such as a feed management UI and an IMAP server.

I’m working on an IMAP server for my RSS aggregator.  I want to capture the read status and message flagging from my reading habits in my database.  I’ve been using message flagging to indicate articles I want to review later or remember for some reason. The new aggregator will have its own IMAP server to serve articles directly from the database rather than copying articles into another IMAP server.  This will also permit me to make a more dynamic folder view of the data rather than freezing the state of the IMAP view into a bunch of Maildirs.  This part has been  going well.  I’ll have more to say about this, too, but the short version is “Twisted is still pretty cool and is handling all most of the grotty IMAP4rev1 protocol bits and letting me just implement a few interfaces to present a view of my data as IMAP folders and messages.”

That isn’t why I’m posting now, though.  I’m going to tell you a little story starring one of my favorite kinds of bugs with my favorite mysterious symptoms.

My project was proceeding along nicely.  Then I did another import and now mutt is segfaulting when it tries to load the folder view.  I chase down a rabbit hole convinced that the reason is I’m sending more articles than mutt expects because my folder size counting code doesn’t agree with my FETCH processing code. Deceptively, I discover a case where these can be mismatched. But that isn’t what is happening.

Eventually I set this issue aside and resume using Thunderbird to test.  Deceptively, I’ve rebuilt the database in the meantime and am no longer using the same messages. The problem no longer appears and I thus blame mutt’s shoddy IMAP implementation.

Everything is great again and my IMAP server is really shaping up. I rebuild the database to use a larger percentage of my real dataset and work through a number of issues in my article processing code. 

Now Thunderbird starts hanging.  I recheck mutt. It’s segfaulting again. Curses! Both the Windows and Linux versions of Thunderbird appear to connect and work for a while and then they start hanging, spinning, and generally losing badly.  I need to figure this out to proceed.

After some printf debugging suggests everything is fine, it’s time to see what Thunderbird is actually seeing. I bring out Ethereal Wireshark.  Wireshark rapidly shows that some of my IMAP server’s command responses look like this:

2a 00 00 00 20 00 00 00 31 00 00 00 36 00 00 00
20 00 00 00 46 00 00 00 45 00 00 00 54 00 00 00
…

Most of the responses look fine but some of them have three nulls between each real character.  Uh oh.  I peek with ptrace and confirm the process really is send()ing three nulls between each character.  Why are my characters four-byt… oh. Before jumping to the conclusion, I refine my printf debugging to also print the type of the string being added to the outgoing buffer.

WRITE <type ‘str’> : * 18 FETCH (
WRITE <type ‘str’> : UID 48319
WRITE <type ‘str’> :
WRITE <type ‘str’> : RFC822.SIZE 865
WRITE <type ‘str’> :
WRITE <type ‘str’> : FLAGS (\Unseen)
WRITE <type ‘str’> :
WRITE <type ‘unicode’> : BODY[HEADER.FIELDS (From To Cc Subject Date Message-Id Priority X-Priority References Newsgroups In-Reply-To Content-Type)] {333}

‘unicode’!  It is as I suspected.   It is idiomatic in Python when buffering writes to assemble a list of strings and then call .write(“”.join(list)).  Twisted does this.

Python doesn’t have a “raw byte buffer” type. It has unicode strings (type ‘unicode’) and “raw” strings (type ‘str’). I’m using “unicode” strings for virtually all of my data (and in the database) and raw strings to represent byte buffers.  I thought I had caught everywhere I was outputting and called an appropriate .encode() on the string.   Unfortunately, concatenating a ‘str’ and a ‘unicode’ results in a ‘unicode’ instead of a TypeError.  Python got this wrong for ‘int’ and ‘float’ so it is no wonder that it is wrong for ‘str’ and ‘unicode’. A single unicode leak will result in the entire write being unicode.  Python, it turns out, will also cheerfully write out the raw bytes of a unicode string.

Java has different types for “byte” and “char”. You just can’t pass a directly to anything that’s going to do i/o without casting or encoding. Java characters are all unicode, of course. Most of the time this is just another hoop to jump through when dealing with i/o in Java. Right now I really appreciate it. I wish Python i/o primitives threw an exception if you tried to write an unencoded ‘unicode’.  Even encoding as ‘utf-8′ would be better than writing out the raw bytes.

It’s not very robust of mutt and Thunderbird to hang, spin, or crash when they encounter unexpected nulls in the result from a network server. I pity the poor end user using one of these clients to connect to a shoddy server.

For this round, I wrote everything in Python.

matplotlib is a library for Python that tries to present a MatLab-like interface for a bunch of graphics and data manipulation tools. I don’t know how successful it is at emulating MatLab since I never used MatLab enough to become very familar with it. matplotlib is nice, though — probably easiest library of this type that I’ve used.

SQLite I have mentioned here before — it’s an embeddable SQL database engine. In this case, I’m using primarly to avoid inventing another binary file format with the added bonus of being able to query the data stored in my file. I needed to parse the many log files and store them in an intermediate form that was easy to read and query so I didn’t have to wait for a script to grovel through the log files every time I wanted to revise how I analyzed the data. For the reasons I discussed earlier, I’m still using the APSW Python binding for SQLite.

A script parsed the raw log files and output a SQLite database representing essentially all of the information in the log files in tables. Then another script processed that data into some intermediate, derived tables. Finally, the last script used matplotlib to create the graphics from both the raw data and the intermediate tables. There was much experimention in the last two steps as I generated graphics that led me to wonder about something which led me to compute more intermediate data to look at and/or graph. I used matplotlib a little interactively to browse around on the graphics but I avoided using

All these words about parsing and graphics invite posting of some pictures and scripts to do it. I can’t post the work related code so I decided to perform a similar exercise on some other information for this post.

There is a great graphic shown in one of Edward Tufte’s books of a train schedule. It represented stations vertically and time horizontally. Each train was represented as a line intersecting the horizontal of a station for the time the train would be in the station. The slope of the lines indicated how fast each train was. I will write a script to process Caltrain’s HTML schedule into such a graphic.

Weekday (800×600 PNG, 260K), Weekend/Holiday (800×600 PNG, 125K)

Starting from the Caltrain timetable and Caltrain station info in HTML, I used Beautiful Soup (a Python HTML scraping library) to extract the timetable. I’d never used Beautiful Soup before this but the thought of constructing the parser (or regexps) to extract the schedule from the HTML made me sad enough to learn how to use a new library. Beautiful Soup is nice! It was very easy to figure out how to extract the information I wanted from the HTML.

Incidentally, as of this writing that timetable has, in HTML comments, snippets of ELisp code. I couldn’t figure out what the ELisp was supposed to do that was helpful, though.

The script isn’t very pretty, but here it is anyway: schedule.py (4K). It requires Beautiful Soup and matplotlib to run. The graphic omits stations south of San Jose because so few trains run to them and it squished the stations most of the trains did run to into less space. I didn’t really put together how closely spaced the Caltrain stations on the penninsula until I saw this graphic.

The script needs more work if you wanted to use the schedule for anything as the use of express and local trains meant I had to come up with some way to indicate if a given train actually stopped at the stop it was passing. I used a black circle, but I can see places where overlapping trains and dots make which train is doing the stopping somewhat ambiguous.

The script assumes you have timetable.html and caltrain_stations.html in the current directory — these were both just pulled straight from the caltrain web site using the links above.

Now that I have all this parsing code I’ll probably finally get around to making something to print custom schedules which include only the stations I care about. I’ll just use a simple textual table, though.