The Mandelbear's Musings

I started this DreamWidth account twelve years ago today in 2010. (I started my LJ account in June of 2002, so there will be another blogiversary coming up in a couple of months.) I believe I started cross-posting around then, although for a while I continued doing quick posts on LJ for things like birthdays. Eventually I imported all of my LJ posts, so DreamWidth is the authoritative copy now.

By 2010 I had already been using ljupdate, an Emacs-based LJ client, for a couple of years; I started working on my current emacs/make/git system in late 2015 when DW switched to HTTPS and ljupdate stopped working. For all I know it might have been fixed by now, but I like my hack better. (You can find it in MakeStuff/blogging/ on GitHub; it's acquired a few features since I wrote the README. I should fix that. Real Soon Now(TM).)

Another thing I need to do RSN is fix the links in imported posts that still point back to other posts in LJ, and add "imported-from" links to the posts that don't have "crossposted-to" links.

I will refrain from boring you with statistics. In part that's because I don't have everything archived in one place. Yet. That's another thing I've been meaning to do for several years now. (The reason for things being scattered is another thing I won't bore you with. In this post.) And besides, I just found a problem with my word-counting program.

Today the computer curmudgeon talks about how he blogs.

I think it's widely known that I update this journal (and by that I mean both my Dreamwidth journal, which you are probably reading now, and my Computer Curmudgeon website), using a rickety combination of shell scripts and makefiles called MakeStuff. There are several good reasons for that. (Whether "because I can" is a good reason or not is debatable. There are others.)

A little over a year ago I made a planning post, and the "Where I am now" section remains a pretty good, albeit sketchy, description of the process. There's also an even sketchier one in the README file for MakeStuff/blogging. It had a list of what I wanted to do next, but essentially the only thing I've actually done is posting in either HTML or Markdown.

I have, however, reorganized things a bit, so that all of the relevant scripts are in MakeStuff/blogging -- the last part was moving in the script, now called charm-wrapper, that takes the post's metadata out of its email-like header and turns it into a command line for charm, a livejournal/dreamwidth posting client written in Python. It isn't a very good solution, but it works.

And since I'm running out of time to make a post today, I'm going to start this series with the other utilities in MakeStuff/blogging. And then go add this list to the README.

• check-html is a simple wrapper for html-tidy, a popular HTML syntax checker. Since it's not actually being used to fix syntax, it can play fast-and-loose with things like the header (it's just text) and blog-specific tags like <cut> and <user>. It handles those by putting a space after the "<" character. It would be trivial to add this to the make recipe for post.
• last-post is a site-scraper that returns the URL of your most recent post. It's useful, because charm doesn't return it. (I eventually put that functionality into charm-wrapper, but it's still useful. Eventually it wants to take a date on the command line.
• word-count is what I use to generate the NaBloPoMo statistics at the bottom of posts in November. (In other months, you just get a straight list; you can also get a listing for a whole year.)

NaBloPoMo stats:
     43 2019/11/01--rabbit-rabbit-rabbit.html
   1731 2019/11/02--s4s-memorials.html
   1465 2019/11/03--done-since-1027.html
    145 2019/11/04--i-ought-to-post-something.html
    430 2019/11/05--how-to-makestuff.html
-------
   3814 words in 5 posts this month (average 762/post)
    430 words in 1 post today

Another fine post from The Computer Curmudgeon (also at computer-curmudgeon.com).
Donation buttons in profile.

It looks like the basic idea works, but I was using slashes as the delimiter for the sed command that puts the post url into the archived post. Since the url contains slashes, that failed.

Following up on mdlbear | Welcome, tumblr refugees: this might otherwise have just been a longish section of next Sunday's "done" post, but the Tumblr apocalypse (tumbling-down?) is happening now and I wanted to get tumblr_backup.py out there. (It's a tumblr backup script, via this tumblr post by greywash, who notes that the original post by Greymask has disappeared). I think some of my readers will find it useful.

It's also worth noting greywash | State of the Migration: On fannish archival catastrophes, and what happens next (by way of ysabetwordsmith; I saw this someplace else last week, but apparently didn't log it.)

More meta stuff:

• a remark by apricops: "It’s quite likely no... coincidence that that most ‘mismanaged’ and least profitable social media site is also the one that turned out to be most amenable to the formation of actual communities"
• pangodillo | What Dreamwidth lacks is the ability to use tags as "whisperspace".

Last month was National Blog Posting Month, for those of us with smaller goals than writing a novel.

I did not succeed in writing a post every day, however I did have enough days with two posts to bring the count back up to 30. It's been a rough month -- I'm not complaining.

NaBloPoMo stats:
    489 2018/11/01--nablopomo.html
     89 2018/11/01--thankful-thursday.html
    617 2018/11/02--learn-enough-to-be-dangerous.html
    340 2018/11/03--s4s-ripple.html
   1880 2018/11/04--done-since-1028.html
    479 2018/11/05--random-words.html
     73 2018/11/06--signal-boost-poetry-fishbowl.html
    246 2018/11/07--music-with-my-family.html
    117 2018/11/08--thankful-thursday.html
    399 2018/11/09--meta-sticky-post.html
    346 2018/11/10--s4s-mary-omeara.html
   1522 2018/11/11--done-since-1104.html
   2042 2018/11/12--multitasking.html
    814 2018/11/13--taking-turns.html
    179 2018/11/14--more-randomness.html
     91 2018/11/15--thankful-thursday.html
    844 2018/11/16--weighty-matters.html
    143 2018/11/17--power-balance.html
    302 2018/11/17--s4s--mary-ellen-carter.html
   1445 2018/11/18--done-since-1111.html
    125 2018/11/19--cat-desk.html
    194 2018/11/20--moms-cranberry-relish.html
    177 2018/11/21--update-on-colleen.html
    353 2018/11/22--thankful-thursday.html
    207 2018/11/24--a-random-haiku.html
    348 2018/11/24--s4s-band-practice.html
   1486 2018/11/25--done-since-1118.html
    134 2018/11/26--family-health-update.html
     85 2018/11/28--winterfaire-2018.html
     51 2018/11/29--thankful-thursday.html
-------
  15617 words in 30 posts this month (average 520/post)
      3 days with no posts

So there's that. There's also a useful word-counting script, as you can see above. It's in MakeStuff/blogging/word-count, but hasn't been made into a make target yet. Maybe next year?

In comments on "Done Since 2018-07-15" we started having a discussion of mirroring and cross-posting DW blog entries, and in particular what my plans are for implementing personal blog sites that mirror all or some of a -- this -- Dreamwidth journal.

Non-techie readers might conceivably want to skip this post.

Where I am now:

Right now, my blog posting process is, well, let's just say idiosyncratic. Up until sometime late last year, I was posting using an Emacs major mode called lj-update-mode; it was pretty good. It had only two significant problems:

1. It could only create one post at a time, and there was no good way to save a draft and come back to it later. I could live with that.
2. It stopped working when DW switched to all HTTPS. It was using an obsolete http library, and noone was maintaining either of them.

My current system is much better.

1. I run a command, either make draft or, if I'm pretty sure I'm going to post immediately, make entry. I pass the filename, without the yyyy/mm/dd prefix, along with an optional title. If I don't pass the title I can add it later. The draft gets checked in with git; I can find out when I started by using git log.
2. I edit the draft. It can sit around for days or months; doesn't matter. It' an ordinary html file except that it has an email-like header with the metadata in it.
3. When I'm done, I make post. Done. If I'm posting a draft I have to pass the filename again to tell it which draft; make entry makes a symlink to the entry, which is already in a file called yyyy/mm/dd-filename.html. It gets posted, and committed in git with a suitable commit message.

You can see the code in MakeStuff/blogging on GitHub. It depends on a Python client called charm, which I forked to add the Location: header and some sane defaults like not auto-formatting. Charm is mostly useless -- it does almost everything using a terminal-based text editor. Really? But it does have a "quick-post" mode that takes metadata on the command line, and a "sync" mode that you can use to sync your journal with an archive. Posts in the archive are almost, but not quite, in the same format as the MakeStuff archive; the main difference is that the filenames look like yyyy/mm/dd_HHMM. Close, but not quite there.

There's another advantage that isn't apparent in the code: you can add custom make targets that set up your draft using a template. For example, my "Done since ..." posts are started with make done, and my "Computer Curmudgeon" posts are started with make curmudgeon. There are other shortcuts for River and S4S posts. I also have multiple directories for drafts, separated roughly by subject, but all posting into the same archive.

Where I want to go:

Here's what I want next:

• The ability to post in either HTML or markdown -- markdown has a great toolchain, including the ability to syntax-color your code blocks.
• The ability to edit posts by editing the archived post and uploading it. Right now it's a real pain to keep them in sync.
• A unified archive, with actual URLs in the metadata rather than just the date and time in the filename.
• The ability to put all or part of my blog on different sites. I really want the computer-related posts to go on Stephen.Savitzky.net (usually shortened to S.S.net in my notes), and a complete mirror on steve.savitzky.net (s.s.net).
• Cross-links in both directions between my sites and DW.

How to get there:

Here's a very brief sketch of what needs to be done. It's only vaguely in sequence, and I've undoubtedly left parts out. But it's a start.

Posting, editing, and archiving

• Posting in HTML or markdown is a pretty easy one; I can do that just by modifying the makefiles and (probably) changing the final extension from .html to .posted so that make can apply its usual dependency-inference magic.
• Editing and a unified archive will both require a new command-line client. There aren't any. There are libraries, in Ruby, Haskell, and Javascript, that I can wrap a program around. (The Python code in charm doesn't look worth saving.) I wanted to learn Ruby anyway.
• The unified archive will also require a program that can go back in time and match up archived posts with the right URLs, reconcile the two file naming conventions, and remove the duplicates that are due to archiving posts both in charm and MakeStuff. Not too hard, and it only has to be done once.
• It would be nice to be able to archive comments, too. The old ljbackup program can do it, so it's feasible. It's in Perl, so it might be a good place to start.

Mirror, mirror, on the server...

This is a separate section because it's mostly orthogonal to the posting, archiving, etc.

• The only part of the posting section that really needs to be done first is the first one, changing the extension of archived posts to .posted. (That's because make uses extensions to figure out what rules to apply to get from one to another. Remind me to post about make some time.)
• The post archive may want to have its own git repository.
• Templating and styling. My websites are starting to show their age; there's nothing really wrong with a retro look, but they also aren't responsive (to different screen sizes -- that's important when most people are reading websites on their phones), or accessible (screen-reader friendly and navigable by keyboard; having different font sizes helps here, too). Any respectable static site generator can do it -- you may remember this post on The Joy of Static Sites -- but the way I'm formatting my metadata will require some custom work. Blosxom and nanoblogger are probably the closest, but they're ancient. I probably ought to resist the temptation to roll my own.

Yeah. Right.

Another fine post from The Computer Curmudgeon.

Took the car in for its 80Kmile intermediate service. Under $300. Except, of course, that it needed front brakes, front tires, wheel alignment, and a battery. Ouch! Well, I was told last time about the brakes and tires; I had the rear done at 70K. Should have remembered, not that it made any difference.

Spent too much of the workday working on some personal writing; it's taken up altogether too much of my time and attention the last couple of days. When I'd gotten it to what felt like a conclusion I felt drained and depressed for a while. On the other hand, I got some major insights out of it which I will share if I can ever get them sufficiently disentangled from the specific situation.

Fortunately I have the kind of job where I can sit at my desk ssh'ed in to home, put myself up for half a day's vacation, and still manage to finish up a technical report and get involved in what will probably turn out to be a patentable idea in the field of secure communication. So I ended the afternoon on a high note.

An hour's worth of together time in bed is always good [grin].

... and having concluded that nearly all of today's writing session amounted to personal therapy and possibly future reference material, I can happily get back to being a computer geek. Having a private, personal pseudo-blog hanging off one's home directory turns out to be remarkably useful, if only as a safe repository for the occasional rant, whinge, or self-indulgent personal essay.

It's easy, and I recommend it: make a locked (mode 700, for the Unix geeks. Make subdirectories called, e.g., 2008/06 or even 20078/06/19. Write. Occasionally copy drafts into LJ or your website, and move useless junk into the attic.

My darling flower_cat has a tendency to read my flist rather than her own, and she hardly ever posts, so she tends to be off peoples' radar somewhat. And she forgets how to add friends, so every so often I point her at my profile and walk her through adding a couple of people. We just did that tonight.

You're still more likely to get a phone call than a comment from her, though.

I don't think a real LJ exodus has started yet, but I've seen several people moving or preparing to move. I now have accounts on greatestjournal and insanejournal (where I paid $35 to become "permanently insane". So far I don't have any posts at either of them.

At some point, my plan is to post to an entirely local and personal blog on my home system, push it out to steve.savitzky.net, and push it from to the *journals, possibly with some filtering by tag. I expect that posts will be files with RFC822-style headers and HTML bodies -- this will give the same editing experience as the ljupdate (emacs) client, at least in its current hacked-up state, and an extensible place to put metadata.

The closest things I've seen are nanoblogger (which uses 822 headers), blosxom, and ikiwiki, but I'll probably end up stealing a little code from each of them, and doing the multi-blog stuff myself.

Google has announced OpenSocial, a new open API for social networks. The new standard will allow developers to create Facebook-like apps on any social network site that implements it with the same calls.

How does working offline, and moving data to the edge, apply to the idea of an "online community"?

What if the idea of membership was not defined by whether a server somewhere has your email address and password on it, but by whether the other members of the group know to pull from you, using a distributed revision control system?

To join a group, you would get the URL for someone else's repository, clone it, edit, add a user page that contains a URL for your own repository, and ask someone to pull. Groups that wanted to automate joining could run "pull bots", and groups that didn't could require out-of-group introductions, or require one user to "sponsor" another by linking to the new user's repository URL from his or her user page.

Then, to participate in conversations, first pull from the other members of the group and merge. At the simplest, use Ikiwiki to build an RSS view of the latest messages. Or use software that gives you a newsreader-like interface. Write your own messages, commit, and push where you're allowed to or wait for others to pull. Repository URLs could be stored inside the community archives, as sponsorship links or bot-maintained pages. Some could be .onion URLs to let users participate pesudonymously and free them from having to get conventional hosting.

So I see that permanent memberships are going on sale again -- for a week, starting Thursday. I've given them as gifts in the past, but I won't be doing it this year. I don't trust LJ anymore -- they're more concerned about looking good to investors than they are about their users' freedom of speech. (I was originally thinking of giving one as a graduation present, but I have since thought of something more personal, and arguably more useful.)

Like technoshaman says in this post, I'm looking forward to a day when we have a completely distributed blogosphere, where everyone controls their own content and it isn't tied to anyone else's server.

It's closer than you think.

Woke at 5:30 in the middle of a strange dream about living in zero gravity (more like Larry Niven's Integral Trees than space colonies, though it was never explained what was holding the air in -- it seemed to bother me somehow), which somehow morphed into zero-gravity blogging and thinking about how to add blogging to my sprawling websites using make, WebDAV, and code stolen from blosxom and nanoblogger. Maybe it's just as well I don't normally remember my dreams.

The flower_cat and I went to two holiday parties yesterday: the Baycon staff party at Chef Chu's at 1:00 (with me going as the tag-along spouse), and the Ricoh party at the Flea Street Cafe in the evening (with roles reversed). The Ricoh dinner was followed by a fantastic chocolate tasting -- eight milk chocolates (I only sampled one), eight darks, four 80% or more (YUM!!) and three astoundingly different kinds of unprocessed nibs.

Both of us tried hard not to eat too much; the Cat skipped the chocolate entirely (poor thing!). She blew a number of people away with how good she's looking these days; a gorgeous green dress with a sparkly jacket helped, but most of the effect was the weight loss. Last year she weighed a lot more, and was in pretty bad health besides.

In between we went to Fry's for a little Christmas shopping. At least I did; the Cat stayed in the car to let her back recover from the restaurant seating. We took surface streets home (Foothill Expressway, Homestead, Newhall, and Park) to look at the lights.

[edits for clarity 12/11]

Reporters Without Borders has published a Handbook for Bloggers and Cyber-Dissidents. It's all available online, either in PDF or HTML. Includes chapters on How to blog anonymously, Technical ways to get round censorship, and Ensuring your e-mail is truly private.

Required reading for political bloggers and whistle-blowers; lots of good info, mostly introductory, for others.