Prior to today every time I took the time to try and update the source tree on Sourceforge I ran into a snag such as “Sourceforge CVS is unavailable.” Months ago, when I tried to migrate from CVS to SVN on Sourceforge I ran into errors. I didn’t even try to investigate what went awry as I was low on patience and time.

As a result, my own tree (first in Perforce and now in Subversion) has gotten pretty out of sync from that tree. I decided to try GitHub and try using Git for something “real” — so I’m going to move Singleshot’s source to GitHub and then push all my changes to there.

I followed (mostly) the steps from How to export revision history from mercurial or git to cvs which describes how to use the git cvsimport command to pull Singleshot’s revision history from its Sourceforge CVS repository, testing making a tiny change to a README, and pushing that change back to the Sourceforge CVS tree.

(After backing up the source tree on Sourceforge using rsync.)

% export CVS_RSH=ssh
% git cvsimport -d :ext:xythian@singleshot.cvs.sourceforge.net:/cvsroot/singleshot -C test1 \
    -r cvs -k -A /home/fox/src/singleshot-import/authors.txt  singleshot
Initialized empty Git repository in /home/fox/src/singleshot-import/test1/.git/
[longish pause]
Counting objects: 561, done.
Compressing objects: 100% (502/502), done.
Writing objects: 100% (561/561), done.
Total 561 (delta 317), reused 0 (delta 0)

(Some poking around to see if things look reasonable, making a small change to the README, commiting it first to the git repository and then back to Sourceforge using git cvsexportcommit.)

Then I pushed the whole thing to a GitHub tree:

http://github.com/xythian/singleshot/tree/master

% git remote add origin git@github.com:xythian/singleshot.git
% git push origin master

That was pretty straightforward and appears to have worked.

My plan is to merge my changes into the out of date tree and then push the new code to GitHub. If it all works out I’ll probably indicate that the Sourceforge project is defunct there and host the code on GitHub henceforth.

It didn’t work. Some digging indicated that Singleshot’s JPEG height/width detection was failing on the new JPEG.

Singleshot’s JPEG header parsing code was only looking for a SOF0 marker and thus didn’t work on progressive JPEGs (which used SOF2). This is why it’s important to read the spec for file formats you’re reading rather than just cribbing information like “to get the height/width from a JPEG, look for the marker 0xC0 and then pull height and width from that block” from random code

Since the last Singleshot release I’ve fixed a bunch of bugs in IPTC parsing (and added a fallback to use XMP if IPTC is not there and XMP is). I’ve also greatly enhanced the robustness of my tests for the IPTC reading code with a script that automatically compares the output of Singleshot’s reader to a couple of other parsers (exiv2, PIL) and alerts me when they differ. As usual the reality of what shows up in files (even just ones I have) seems to differ from the cold, formal world of a spec.

I’ve also added a couple a feature of dubious merit: for runs of photos tagged with an event:foo keyword, Singleshot’s RSS feed and front page treats them as a single post with a link to an album /album/events/foo. I did it so I could post lots of photos for a party without having them all show up individually in the RSS feed — it really didn’t make sense to have that. I may still have to refine how it works, though. I’m not thrilled with the magic dependence on an Album, but I wanted it for this because I wanted to link to a page with some description of the event.

I’ve also added some pretty experimental support to allow commenting on individual photos. It pretty gratutiously uses AJAX-y refreshing because I wanted to play with that. It appears to work in all the modern browses I have access to (Firefox, IE 6). If you disable JavaScript, then the comments and the UI for submitting them just don’t appear.

On the back-end, it stores all the comments in a SQLite “database” — mostly preserving the ‘file system is the true source of information’ property of Singleshot.

I compensated with an awful hack and, just to be sure, pregenerating all of the view images so I can go to bed without, hopefully, leaving photos.xythian.com woefully broken.

I think NearlyFreeSpeech is going to have to go. They seem fine for relatively static web sites, but trying to run applications there is like tearing my hair out. Testing on my home machine is completely irrelevant for determining if it’s going to work at all on NearlyFreeSpeech.

• Add ~ expansion to singleshotinit.py –root
• Fix situation when rewritebase is /
• Fix RSS feed (publish_time is now a datetime always); RSS feed now works…
• Modify month_dir so it creates datetimes at midnight rather than dates — dates can’t be sorted with datetimes.
• Change limits on recent images to take advantage of pagination.
• Fix handling in IPTC parsing to deal with more weirdo cases.
• Add a way to set page size and modified keywords template to use it.
• Fixed conditional display of keywords: label in view when there’s no keywords.

.. is now released. I also finally updated photos.xythian.com with the latest Singleshot.

It’s mostly bug fixes now based on reports from my legions of dedicated testers. In a3, the caching problem when in persistent mode is fixed. Also an issue with Apache 2 and I added pagination of the album pages to sidestep a problem when Singleshot is pointed at a huge directory of images (1000+).

It’ll be in Singleshot 2.0, though at least as experimentally as the mod_python support. My plan is to include both mod_python and FastCGI support with the 2.0 release and see if anyone other than me cares about it. I’ll be using the FastCGI support for photos.xythian.com as soon as I move it somewhere that supports FastCGI (probably Dreamhost since I already have things there but I may try TextDrive). TextDrive is appealing because it’ll let me easily host just photos.xythian.com with them without delegating xythian.com’s DNS to them. ISPs that think a domain is just a web site can be pretty irritating.

FastCGI Singleshot currently only supports ImageMagick for image resizing — it’s single threaded and it turned out having the whole process block for making thumbnails was not good for responsivness. Since Singleshot already spawned a process to run convert, it was easy enough to have it not block the main thread on the spawned converts. Since ImageMagick was also faster and still seems to have a quality edge over PIL, it was an easy decision to only support ImageMagick when deployed using FastCGI.

Singleshot FastCGI supports multiple, multiplexed connections and allows the server to reuse connections. Too bad Apache mod_fastcgi doesn’t. I still need to test the new FastCGI support with another web server, and also figure out how the current reliance on mod_rewrite rules will translate for other webservers.

My FastCGI implementation just depends on asyncore, so no new dependencies were introduced with it.

Aside from the new deploy work, I also fixed lots of little bugs. I’ll probably start putting prerelease builds up on Sourceforge soon, as soon as I take the time to write up some preliminary install docs.

Eventually I’ll probably go ahead and make a deploy option where Singleshot runs as a standalone webserver, because I can. That will almost certainly depend on Twisted so I can get all the HTTP/1.1 support without having to write it myself.

Singleshot was still dependent on PIL for EXIF data. In Singleshot 1.0 I used EXIF.py, but EXIF.py was so slow that for 2.0 I made Singleshot use PIL’s exif parser if it was available. I profiled EXIF.py and tweaked a couple things to get some more speed (around 25%), but it still wasn’t enough. One of the reasons EXIF.py was slow is it was gathering much more data than I actually cared about — it was even decoding the JPEG thumbnails in the EXIF data.

So, using both EXIF.py and the EXIF 2.2 spec, I wrote my own EXIF parser that pulled out the tags I wanted.

Before doing this, pulling all the metadata out of my tree of test images took around 9 seconds with EXIF.py and 4 seconds with PIL’s EXIF parsing. Now it takes 0.6 seconds.

I also finally finished the work to enable more image types, adding PNG and GIF as supported image types when Singleshot is using PIL. I need to dig up how to get the size out of PNG and GIF from the file header before I can add PNG and GIF to the ImageMagick processor — I don’t want to have to fork off a process just to get image height and width.

I’ve been finishing up the rearranging of the Singleshot files and preparing to package it with some instructions for a release.

I took a few hours to experiment with running Singleshot under mod_python, though, to see how easy it would be. My proof-of-concept is clearly prototype-grade, but it was encouraging: Singleshot is really, really snappy when it’s not saddled with the overhead of CGI.