Why open formats matter

[mdlbear]

Here's an interesting exchange of blog posts on open formats and why they matter. I found it through a reference in this post on O'Reilly Radar, which is interesting in its own right; it's about the fact that Flikr and Picasa are opening their APIs to allow users to transfer their pictures between them.

That leads to "When the bough breaks" by long-time Mac guru Mark Pilgrim, explaining why he's just switched to Linux on a new Lenovo ThinkCentre after 22 years on Macs.

In fact, I spend the vast majority of my time using these and other open source applications (Carbon Emacs, Colloquy, Audacity, Seashore, Python, and a variety of command-line tools). Why keep running them on an operating system that costs money and restricts my rights and my usage?

...

I'm creating things now that I want to be able to read, hear, watch, search, and filter 50 years from now. Despite all their emphasis on content creators, Apple has made it clear that they do not share this goal. Openness is not a cargo cult. Some get it, some don't. Apple doesn't.

It's all about lock-in due to closed, undocumented, proprietary, binary file formats. As we'll see later, he was bitten one too many times. The next post is And Oranges by John Gruber, which is mostly an excellent discussion of the difference between the high-quality Mac user interface and the others, which are "good, but not great" -- and how that wasn't the main factor in Mark's decision.

Telling Pilgrim that he's making a mistake because Ubuntu doesn't have as refined or cohesive a UI as Mac OS X is like telling someone who is switching from a Chevy Tahoe to a Toyota Prius that he's not going to have as much cargo room. He knows it.

... but then he goes on to say that

Admitting that he has a point, or several points, or that he may well be correct that he's going to be more satisfied with Ubuntu than he was with Mac OS X, does not imply that Mac users are wrong or stupid or foolish.

And the truth is I'm not entirely sure he's making the right decision, even for himself. Forget all the niggling details he cites, and focus only on his central beef -- that Apple is a company that does not "get" openness, and that this deficiency is going to hinder Pilgrim's long-term access to the data he's creating. But if that's the case, and Pilgrim has been using Apple computers for 22 years, why hasn't it happened already?

This is what Mark takes issue with in the last post of our series, Juggling Oranges:

It has happened already, John. Over and over again.

He then goes on to list the occasions, dating back to the years he spent between 1983 and 1989, writing software for the Apple //e, "a platform that doesn't exist and can only be emulated with the help of ROMs which are illegal to redistribute."

And then came Tiger, and Mail.app 2.0. In Mac OS X 10.4, Apple deliberately changed Mail.app to use their proprietary .emlx data format, apparently to work around the limitations of Spotlight. Mail.app 2.0 helpfully auto-converted all my wonderful mbox files into Apple's shitty undocumented format. I'm now in the process of undoing the damage. I tried an emlx-to-mbox converter program, but it has bugs that ruin certain mail messages and corrupt the resulting mbox file. (Specifically, mail messages that contain a line that starts with the word "from".) Perhaps JWZ's emlx.pl script will fare better. JWZ knows mail.

He ends with:

Which brings us back to John Gruber's oranges. His counter-argument -- that lock-in hasn't been a problem for me yet, so why all the fuss now -- could not be further from the truth. It's been a constant problem for 22 years. Much of the data I've spent my life creating has been lost or seriously degraded through a series of proprietary formats and forced migrations. This is why I felt so betrayed, in particular, by Mail.app "upgrading" me away from mbox format. It took a lot of forethought on my part, not to mention actual time and effort, to convert all my disparate mail archives from all those different mail programs. I finally got everything into a single archive in an open, stable format... and just 3 short years later, Apple found a way to screw me one last time. It'll be the last time they get the chance.

And that, children, is why open formats are important, and why neither Microsoft nor Apple -- nor, ultimately, any proprietary software -- can be trusted to use them: it's all about lock-in. The greatest superiority of open-source software is that by construction it uses open formats. I have email and Usenet archives dating back to the mid-1980's that I can still read and search. My filksong lyrics date back to 1982, with only a single format transition (from plain text to LaTeX). I have programs and Makefile's that I wrote for long-obsolete Apollo workstations and the original 8086 PC that I can simply recompile and run two decades later. I have 15-year-old configuration files for X and Emacs that I'm still using.

I'm a proud, happy user of open-source software and open data formats, and when they pry my keyboard from under my cold, dead fingers, my children will still be able to read my files.

Comments

[mdlbear]

Good point, although the line gets fuzzy at times. Is the Linux kernel's internal device driver API open? Most people would say that it is, but in fact it changes in incompatible ways fairly often -- it's safe to say that, if you're maintaining a device driver, you'd better be keeping up with the kernel developers' mailing list. The same is true of many user-level APIs and file formats, although they change more slowly and there's usually a clear upgrade path.

And with any open source program, you always have the option of going back, picking the code apart, and using that as a library on which to build a file-conversion utility. You don't have that option at all with a proprietary format. That's really what I was getting at.