notes from Ken

Adobe products are rare in that I’ve never regretted upgrading — each upgrade has brought changes I’ve liked enough to think the upgrade was worthwhile. Contrast this with Quicken, where I’ve regretted every upgrade as it made the product slower and less reliable (until finally I’ve said “no more” and will not upgrade any further).

The only thing I don’t like about newer Adobe products is the activation stuff. I don’t pirate software and I don’t appreciate things that are there to ensure that I don’t, thus taking valuable time and energy (both mine and my computer’s) for something that benefits me not one bit. Adobe’s form of this seems to be much less intrusive and inconvenient than most companies’ implementations that I have experienced.

Microsoft product upgrades are usually a mixed bag — some real benefits and some negative changes (the exceptions usually being 1.0 -> 2.0 and possibly 2.0 -> 3.0 where they figure out what the product really is and thus those upgrades are usually clearly worth it).

I wonder how much of this problem where upgrades aren’t really *upgrades* comes from areas where the products are basically “done” yet the company has built a significant source of revenue model from upgrades and thus needs to push out a release every x months no matter what. It may be that switching to a “monthly fee” for software that one uses might actually be preferable to an upgrade treadmill where the changes don’t improve things and companies are motivated to try and force upgrades.

Photo and video processing are still “new” enough that significant upgrades are still possible in both user interface streamlining and just new features (including more speed).

I just spent a bit reviewing a list of the feeds that hadn’t had any new posts for a while and finding a bunch that had moved. Instead of serving up a nice HTTP 301 with the new feed they instead had a post saying “Here’s my new blog, come check it out!”. That makes me crazy. It makes me even more crazy when the “new” blog is also out of date.

Of course, very few blogging tools make it easy to serve a redirect in place of the feed. Technical people still have no excuse.

It also makes me crazy when people just break all their old post references instead of serving a redirect to the new site.

It’s so much nicer when someone moving to a new site makes the effort to ensure no feed readers watching the old site get broken and nobody coming across links into the old site get disappointed by a 404.

It’s All Text! is a pretty handy Firefox extension. It lets you set up an external editor and puts a little ‘edit’ button in the corner of every textarea that pops open an external editor of your choosing. Then it watches the file the external editor is editing and refreshes the textarea whenever it changes.
Particularly handy given all the things nowadays that encourage editing of document-type text in a browser.

I’ve recently started moving my linux box to a new x86_64 machine running Ubuntu 7.10. I searched for references to this bug but didn’t find anything with Google or launchpad, so I wanted to make a note of it so future victims can see what is going on and to remind me to report it.

The symptom is a crash when your cdb object is deallocated usually with a “double-free” memory corruption error message. Assuming a .cdb file named “foo.cdb”, the following script will repro the bug:

#!/usr/bin/python
import cdb
c = cdb.init(’foo.cdb’)
del c

with the following message:

fox@hercules:~$ python cdbrepro.py
*** glibc detected *** python: free(): invalid pointer: 0×00002b56c25bd750 ***
======= Backtrace: =========
/lib/libc.so.6[0×2b56c300ab0a]
/lib/libc.so.6(cfree+0×8c)[0×2b56c300e6fc]
python(PyDict_DelItem+0xfa)[0×44370a]
python(PyEval_EvalFrameEx+0×2e40)[0×485140]
python(PyEval_EvalCodeEx+0×830)[0×489d60]
python(PyEval_EvalCode+0×32)[0×489da2]
python(PyRun_FileExFlags+0×10e)[0×4ab4fe]
[…]
Aborted (core dumped)

Some other searching suggests that python-cdb’s use of PyMem_DEL is no longer recommended. I haven’t verified that this doesn’t cause other problems, but replacing cdbmodule.c’s use of PyMem_DEL with PyObject_Del (and the PyObject_NEW with _New, to use consistent naming).

As soon as Ubuntu’s bug tracker (launchpad) works again for me I’ll report the bug. Launchpad is timing out with an error message for me now.

I’ve previous posted about using PixVue to edit image metadata directly from Windows Explorer.   I just stumbled across Microsoft Photo Info while going to download Microsoft RAW Image Viewer so I could see thumbnails for CR2 files (Canon RAW images) in Explorer.

Microsoft Photo Info enables easy editing of “metadata” for digital photographs from inside Windows Explorer.

  Photo Info is not quite as snappy as PixVue used to be, but has the bonus of the company that created it still being in business. I can no longer find any download sites for PixVue that I trust.

Photo Info seems be to nice enough and gets the job done.

I’ve been playing with Adobe Flex which I’ll probably have something to say about soon.

Flex uses ActionScript which bears a remarkable similarity to JavaScript. It’s not quite the same but it’s close enough that I haven’t stumbled too badly with it working from JavaScript knowledge.  As with any "new" language the learning curve isn’t the language but the environment and libraries.

The experience with Flex and also the Greasemonkey hacking I’ve been doing lately has made me rather fonder of JavaScript than I’ve been in the past.  JavaScript is particularly nice when used in an environment that is safe from worrying about cross-browser API woes — such as a Firefox-only Greasemonkey user script or an environment like Flex. It’s not ideal but it has a lot of what I like in a language, including garbage collection, closures, reasonable typing and widespread deployment.

I’m playing with Adobe Contribute CS3.   I haven’t had much to say here lately, but maybe a clever client will remind me to post more.  I’ve been posting the occasional photo. I figure at this point that folks interested in my photos have probably subscribed to the feed there so I don’t usually cross-post them to here.

So far Contribute seems to be nice enough.  I haven’t tried to use it to edit anything other than WordPress blog entries so I don’t know how well it works with websites via FTP, WebDAV, SFTP, or local filesystem.  I was pleased to note that it supported SFTP — seeing folks use FTP with cleartext passwords makes me unhappy.

I miss the real-time spellchecking I get in Firefox textareas (or in Word, but I use Firefox more than Word). Seeing the little red squiggle is a lot less intrusive than remembering to run a modal spell-checking UI. Right-clicking on red-squiggled words and picking “Ignore”, “Add”, or correcting the spelling is nicer than using the modal spell-checking UI. I’ve also noticed that the real-time feedback is more likely to make me remember the correct spelling of a word in the future.

Well, enough of this post now that I’ve thought of something to talk about which isn’t related to Contribute…

Hm, I can’t tell if the “Allow Comments” button is on or off by default.  I guess I’ll find out.

Update: Seeing if editing an entry works… That’s kind of random, editing the entry apparently made it disappear. I had to edit it again to make it come back. Also it’s kind of slow, but that may be Dreamhost’s hosting of my WordPress install rather than Contribute.

I picked up a Logitech MX Revolution wireless laser mouse a week or so ago. It replaced my 3 year old Logitech wireless laser mouse. Who doesn’t like wireless lasers?

The new mouse is shaped a little differently than the old mouse and has a couple more buttons and one new wheel on the side. The new buttons are moderately useful but the key new feature over my old mouse is that the mouse wheel will spin like a flywheel if it’s pushed with enough force.

I hadn’t noticed how much I liked that for zooming around in a document until I had to use my wired, non-laser, flywheelless mouse at work.

A tool and a library I’ve been using or at least trying out:

Launchy

Free, open-source Windows app that indexes Program Files and any other directories you tell it to. Then alt-space pops up a command line box and it autocompletes as you type. I installed it a while ago and meant to mention it but it kind of faded into the background and I don’t think about using it anymore. It’s just there. No more do I need to concern myself with the program menu’s lengthy “organization” by vendor nor do I keep needing to add paths to my PATH. I used to do that so I could use Windows-Run to run things rather than rummaging in Program Files but Launchy is a much better solution. I’ve read that it resembles Quicksilver for the Mac.

SQLAlchemy

“SQLAlchemy is the Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.” — mostly it handles some tedium without getting in the way. ORMs still make me nervous since they tend to have a lot of “magic” in them but SQLAlchemy makes it pretty easy to use the convenient parts and override the “magic” if/when it’s necessary for performance. Which probably won’t happen too much but knowing it’s easy makes it easier to rely on SQLAlchemy in the meantime. I don’t know if I’ll keep using it, because like every other time I use a library like this I find myself spending more time trying to figure out how SQLAlchemy expresses something and I already know how to use SQL.

Live Free or Die Hard is everything I hoped it would be.

Magic Ink: Information Software and the Graphical Interface is pretty interesting.

The ubiquity of frustrating, unhelpful software interfaces has motivated decades of research into “Human-Computer Interaction.” In this paper, I suggest that the long-standing focus on “interaction” may be misguided. For a majority subset of software, called “information software,” I argue that interactivity is actually a curse for users and a crutch for designers, and users’ goals can be better satisfied through other means.

Information software design can be seen as the design of context-sensitive information graphics. I demonstrate the crucial role of information graphic design, and present three approaches to context-sensitivity, of which interactivity is the last resort. After discussing the cultural changes necessary for these design ideas to take root, I address their implementation. I outline a tool which may allow designers to create data-dependent graphics with no engineering assistance, and also outline a platform which may allow an unprecedented level of implicit context-sharing between independent programs. I conclude by asserting that the principles of information software design will become critical as technology improves.

Although this paper presents a number of concrete design and engineering ideas, the larger intent is to introduce a “unified theory” of information software design, and provide inspiration and direction for progressive designers who suspect that the world of software isn’t as flat as they’ve been told.

Ad-Supported services like many (most?) web sites and broadcast television are clearly a valid business model. I just wish more of them provided a simple way for me to pay the producers of the content (and a suitable fee to the entity distributing it to me, if they aren’t the same) to give me an Ad-Free version, including liberating all of my personal data from the service-in-question’s data aggregation mechanisms (not the summary data, necessarily, just what can be linked back to me).

I don’t see any benefit to me in some third party building a better model for my behavior than I even know of myself.

Last weekend I ordered a Canon SD800IS from Dell. Dell shipped it on Monday. So far so good.

Dell used DHL to ship it. I’m sure DHL is cheaper than UPS or Fedex because otherwise I don’t understand why anyone uses DHL. I got a call on Thursday from DHL saying they needed my apartment number to deliver. Of course, my apartment number was in the shipping address I gave Dell.

I called them back and the nice DHL rep updated the shipping address of my order and she said it’d be delivered today.

DHL never shows up with my new camera. Eventually, I check the tracking status and it says “Returned to shipper.” I call DHL again, wait on hold for a while, and talk to another customer service rep. Apparently, my package record was updated but Dell refused to allow the redelivery to a different (??) address and the package was instead returned to Dell.

So I called Dell and waited on hold for much longer than at DHL in order to talk to a customer service rep. I explained the story so far and gave the rep my order number. He put me on hold and called DHL. Shortly thereafter he came back and said that DHL’s business office was closed. He put me on hold again and called some other department.

When he came back, he said he could treat it like an exchange (.. but I never got it!) and they’d process it and ship it out again in eight business days (!!!). Unsurprisingly, I canceled the order. The web site still reports the order as open so I’ll be watching carefully to make sure my credit card is credited.

This is not the first time DHL fumbled a delivery for me but it is the first time the shipper apparently compounded the error. I would think that DHL just can’t manage to deliver to an apartment except the other time they fumbled a delivery to me it was to a detached single family residence.

This is most disappointing. I’m used to have good experiences ordering from Dell but this time I feel that they really blew it. They wanted to treat their (or their shipping company’s) error in shipping as an “exchange” and expected me to wait a week and a half for them to even ship it out again. I didn’t get clarification to learn if the rep meant eight business days from now or eight business days from when they received the bounced shipment because I canceled the order.

By the end of all this, Dell lost the order, DHL carried a camera thousands of miles in a circle, and I still don’t have my camera. Everybody loses.

Suppose your wxPython app is is using an embedded Internet Explorer window to display HTML. You want links clicked in that window to open the user’s default browser rather than following the link within the embedded Internet Explorer.

The obvious solution is to hook OnBeforeNavigate2 like so:

        self.Bind(iewin.EVT_BeforeNavigate2, self.OnBeforeNavigate2, self.ie)
...
    def OnBeforeNavigate2(self, evt):
        wx.LaunchDefaultBrowser(evt.URL)
        evt.Cancel = True

but if the link the user clicked is set to open a new window, the emebdded IE will just open a new window. So you hook NewWindow2. Oops, NewWindow2 doesn’t tell you what the URL is!

Here’s what I did, since I don’t care if my reader works on pre-SP2 Windows XP:

Define an Event to permit me to hook NewWindow3, which is a new event that includes the URL being opened, then hook it appropriately:

wxEVT_NewWindow3 = wx.activex.RegisterActiveXEvent('NewWindow3')
EVT_NewWindow3 = wx.PyEventBinder(wxEVT_NewWindow3, 1)
...
        self.Bind(EVT_NewWindow3, self.OnNewWindow3, self.ie)
...
    def OnNewWindow3(self, evt):
        self.logEvt(evt)
        # Veto the new window.  Cancel is defined as an "out" param
        # for this event.  See iewin.py
        evt.Cancel = True
        wx.LaunchDefaultBrowser(evt.bstrUrl)

The problem with working on a tool like a feed reader with live data is: As soon as the reader is good enough to use it is much too easy to get distracted reading feeds and forget to get back to work on the reader.

Ongoing small irritations finally piled high enough to push me to evolve my feed reader further. I finally created a category for the RSS aggregator-related posts: Readshot.

A brief history of this project:

• January 2005 — A proof-of-concept toy that used an IMAP store and copied posts from a single RSS feed to an IMAP server. I was still using Thunderbird and rawdog to do my feed reading at this point.
• October 2005 — began using my new aggregator. It had two components — a fetcher which ran from cron, used sqlite to keep track of state, and emitted new article messages to a Spread group and an IMAP storer which read from the Spread group and copied articles into an IMAP folder. I read the IMAP folder using Thunderbird.
• March 2006 — I collapsed the components into a single fetcher which ran from cron, used a SQLite data store, and put messages into an IMAP folder. I also improved the feed scheduling to reduce the number of fetches on relatively inactive feeds.
• January 2007 — moved from the SQLite-based aggregator to a new aggregator based on PostgreSQL. The new reader implements its own IMAP server using Twisted. It’s much faster than the old SQLite solution and keeps all of the state in the database rather than having “feed state” in a SQLite database and “user state” (flagging, read state) in an IMAP folder.

I made a very basic web UI with the SQLite-based aggreagtor which allowed me to subscribe to new feeds with a bookmarklet. I migrated that UI to the new aggregator. It was always inadequate and I supplemented it with manual SQL commands to manage feeds. I’d sit down to enhance the interface but keep getting distracted playing with Ajax toolkits because the UI I had in mind didn’t fit very well into the web browser.

This weekend I began working on a desktop client. It’ll be backed by the same database so the IMAP server will continue to work in case I want to read via IMAP for some reason such as to do offline reading. I have a very crude GUI which has already let me do some management of the feeds which was awkward to do before but the desktop client still exists more as doodles in a notebook than code.

The first version connects directly to the PostgreSQL database. Eventually I plan to split the database access out into its own tier. I don’t yet know enough of what I want that protocol to look like to do that now and want to get a better feel for what the client will need before taking that step. The plan to split the storage layer into its own tier guides the design of that layer.

There’s always been a bit of a mismatch between my needs in a mail client and my needs for a feed reader but until recently I put up with all the little cuts that resulted from using a mail client for feed reading because the benefits of a desktop experience and shared server state outweighed the pain of using a mail client (and IMAP) for feed reading.

I’m using wxPython to develop the client so at least in theory it should work on Windows, Linux, and Mac OS. I only plan to test and develop using Windows for the foreseeable future, though.

No project of mine advances as quickly as those that scratch an itch or alleviate a pain for me.

I still don’t know if I’ll ever release any of this. It would probably be academic since I imagine the demand for this kind of thing is very low given how excited people get about “no install” web readers. I think the the thought process that guides design decisions in the reader is more interesting to other folks than the actual code. I suppose the RSS fetcher part of the system might be interesting to other folks as its own component.

I posted a photo to photos.xythian.com.

It didn’t work. Some digging indicated that Singleshot’s JPEG height/width detection was failing on the new JPEG.

Singleshot’s JPEG header parsing code was only looking for a SOF0 marker and thus didn’t work on progressive JPEGs (which used SOF2). This is why it’s important to read the spec for file formats you’re reading rather than just cribbing information like “to get the height/width from a JPEG, look for the marker 0xC0 and then pull height and width from that block” from random code

I’m converting my RSS aggregator to use PostgreSQL as its back-end instead of SQLite.  It has served me well for quite a while but it’s time to move on from SQLite so I can add some features such as a feed management UI and an IMAP server.

I’m working on an IMAP server for my RSS aggregator.  I want to capture the read status and message flagging from my reading habits in my database.  I’ve been using message flagging to indicate articles I want to review later or remember for some reason. The new aggregator will have its own IMAP server to serve articles directly from the database rather than copying articles into another IMAP server.  This will also permit me to make a more dynamic folder view of the data rather than freezing the state of the IMAP view into a bunch of Maildirs.  This part has been  going well.  I’ll have more to say about this, too, but the short version is “Twisted is still pretty cool and is handling all most of the grotty IMAP4rev1 protocol bits and letting me just implement a few interfaces to present a view of my data as IMAP folders and messages.”

That isn’t why I’m posting now, though.  I’m going to tell you a little story starring one of my favorite kinds of bugs with my favorite mysterious symptoms.

My project was proceeding along nicely.  Then I did another import and now mutt is segfaulting when it tries to load the folder view.  I chase down a rabbit hole convinced that the reason is I’m sending more articles than mutt expects because my folder size counting code doesn’t agree with my FETCH processing code. Deceptively, I discover a case where these can be mismatched. But that isn’t what is happening.

Eventually I set this issue aside and resume using Thunderbird to test.  Deceptively, I’ve rebuilt the database in the meantime and am no longer using the same messages. The problem no longer appears and I thus blame mutt’s shoddy IMAP implementation.

Everything is great again and my IMAP server is really shaping up. I rebuild the database to use a larger percentage of my real dataset and work through a number of issues in my article processing code. 

Now Thunderbird starts hanging.  I recheck mutt. It’s segfaulting again. Curses! Both the Windows and Linux versions of Thunderbird appear to connect and work for a while and then they start hanging, spinning, and generally losing badly.  I need to figure this out to proceed.

After some printf debugging suggests everything is fine, it’s time to see what Thunderbird is actually seeing. I bring out Ethereal Wireshark.  Wireshark rapidly shows that some of my IMAP server’s command responses look like this:

2a 00 00 00 20 00 00 00 31 00 00 00 36 00 00 00
20 00 00 00 46 00 00 00 45 00 00 00 54 00 00 00
…

Most of the responses look fine but some of them have three nulls between each real character.  Uh oh.  I peek with ptrace and confirm the process really is send()ing three nulls between each character.  Why are my characters four-byt… oh. Before jumping to the conclusion, I refine my printf debugging to also print the type of the string being added to the outgoing buffer.

WRITE <type ’str’> : * 18 FETCH (
WRITE <type ’str’> : UID 48319
WRITE <type ’str’> :
WRITE <type ’str’> : RFC822.SIZE 865
WRITE <type ’str’> :
WRITE <type ’str’> : FLAGS (\Unseen)
WRITE <type ’str’> :
WRITE <type ‘unicode’> : BODY[HEADER.FIELDS (From To Cc Subject Date Message-Id Priority X-Priority References Newsgroups In-Reply-To Content-Type)] {333}

‘unicode’!  It is as I suspected.   It is idiomatic in Python when buffering writes to assemble a list of strings and then call .write(”".join(list)).  Twisted does this.

Python doesn’t have a “raw byte buffer” type. It has unicode strings (type ‘unicode’) and “raw” strings (type ’str’). I’m using “unicode” strings for virtually all of my data (and in the database) and raw strings to represent byte buffers.  I thought I had caught everywhere I was outputting and called an appropriate .encode() on the string.   Unfortunately, concatenating a ’str’ and a ‘unicode’ results in a ‘unicode’ instead of a TypeError.  Python got this wrong for ‘int’ and ‘float’ so it is no wonder that it is wrong for ’str’ and ‘unicode’. A single unicode leak will result in the entire write being unicode.  Python, it turns out, will also cheerfully write out the raw bytes of a unicode string.

Java has different types for “byte” and “char”. You just can’t pass a directly to anything that’s going to do i/o without casting or encoding. Java characters are all unicode, of course. Most of the time this is just another hoop to jump through when dealing with i/o in Java. Right now I really appreciate it. I wish Python i/o primitives threw an exception if you tried to write an unencoded ‘unicode’.  Even encoding as ‘utf-8′ would be better than writing out the raw bytes.

It’s not very robust of mutt and Thunderbird to hang, spin, or crash when they encounter unexpected nulls in the result from a network server. I pity the poor end user using one of these clients to connect to a shoddy server.

This weekend has been much warmer than last weekend. I went for a walk.

I stumbled upon Programmer’s Bill of Rights.  It has the expected collection of things a programmer should have at the workplace, including fast computers, nice monitors, good working environment, etc.  One bit in the post and one of the comments got me thinking.

In the post:

In college, I ran a painting business. Every painter I hired had to buy their own brushes. […] The “company” brushes were quickly neglected and degenerated into a state of disrepair. But painters who bought their own brushes took care of them. Painters who bought their own brushes learned to appreciate the difference between the professional $20 brush they owned and cheap disposable dollar store brushes. Having their own brush engendered a sense of enduring responsibility and craftsmanship.

and in the comments:

for consistency sake, you should apply this analogy to the entire programmer’s toolbox you have listed above, including chair. this is a standard way of doing business for men who make their living as mechanics. when you come to the job, you bring your own tools, and when you leave, you take them with you. the same should go for programmers.

Proton Soup on August 26, 2006 02:36 PM

Being able to use a machine I owned for work development at work would be nice.  Never again would I have to put up with slow hardware and lousy monitors at work because of some pinheaded finance person’s idea of a “standard computer”.

Some of the companies I’ve worked for outfitted developers with excellent hardware.  Sadly, more frequently I’ve had much better gear at home than at work.  If not for IT policy I would have been happy to supply my own gear because a good workstation is that important to me.