Series./blog.sh v1.2

The first post on this site is about a photo I put on Instagram in 2012. The link still works. The photo doesn't.

That post was about one archive — mine — and one rule: media comes home. This one is about what happened when that rule was applied to every platform I could find an export for.

The roster is closed

Twenty-two sources import now, in three groups.

Social networks: Bluesky, Facebook, Instagram, Mastodon, Pixelfed, Threads, Twitter/X.

Blogs and publishing platforms: beehiiv, Blogger, Ghost, Jekyll/Hugo and any markdown folder, LiveJournal, Medium, Movable Type and TypePad, podcasts (any feed with audio), Squarespace, Substack, Tumblr, Wix, WordPress and any RSS or Atom feed.

And dead sites: the Wayback Machine, for blogs whose platform no longer exists at all.

That last group is the one I did not expect to build, and it is the reason this post exists.

What changed isn't the count

An engine with eight importers is a blog engine that can read your old blog. An engine with twenty-two, covering every social network you are likely to have posted to, is something else: it is a place to put the things you wrote on other people's websites.

So here is the claim, plainly. This is a blog. You can also use it as an archive of your social networks. Point it at an export, and what comes out the other side is a folder of JSON files and the actual image and video and audio files, on your disk, with the origin recorded as a tag and every post reachable at a permanent address.

You do not have to put any of it on the web. The import writes the files and then asks whether to rebuild and deploy; answer no and the archive stays on your disk. An archive that only you read is still an archive.

Why the exports and not the APIs

Every importer here reads a file you asked the platform for, not a live API. That was a decision, and it holds up better every year: an export is a legal right in most of the world, it does not expire, it does not get repriced, and it does not need a developer account that a policy change can revoke. The three sources that talk to a network at all — Bluesky, Tumblr, LiveJournal — do it because those networks hand a public archive to anyone who asks, no token required.

The Facebook importer is built on Meta's own "Download Your Information". The Threads one reads the same export. Instagram, Mastodon, Pixelfed, Twitter, all the same shape: a file, not a permission.

The uncomfortable arithmetic

On the reference Facebook export used to build that importer, 95 % of the posts were mirrored in from somewhere else — Twitter, Posterous, the whole crosspost era. They are recognized and skipped, with a count, because those platforms' own exports carry the originals and importing both would double your archive with worse copies.

That number stayed with me. Nineteen out of twenty things on that account were not really written there. Whatever you think you have on any one platform, a good deal of it is a copy of something you put somewhere else — and the somewhere else has usually closed.

The point, again

An archive is only yours if the files are yours. Fourteen years was long enough to watch a few landlords disappear; twenty-two importers is what it took to stop renting.

Every source, and what each export does and doesn't contain, is in docs/importing.md.

./blog.sh 1.2 is out — the import release.

Eight import sources became twenty-two: every social network you are likely to have posted to, the blog platforms, and the Wayback Machine for blogs whose platform no longer exists.

Setting a site up is a conversation now. So is how it looks, with the candidate palette rendered on your own site before you keep it.

And a screen for walking an archive that got too big to scroll.

Until this release, installing ./blog.sh meant copying two files and editing 277 lines of commented YAML before the first build would run. Everything was documented. That is not the same as being easy.

./setup.sh asks instead, and checks every answer as it arrives.

The questions that were quietly dangerous

Three settings could be typed wrong in a way nothing would ever tell you about.

The timezone. A typo here does not fail — Ruby falls back to UTC and says nothing — it just dates every post two hours off, forever. The wizard offers the zone from the machine's own database, unless the machine sits on UTC, which is a fact about a datacenter rather than about the person answering; then it suggests from the language you are being asked in. And it refuses anything that isn't a real zone.

The address. It lives in two places, and env.sh overrides config/site.yml. The shipped example has it pointing at example.com, so filling in only the config leaves your site calling itself example.com in its feed, its sitemap and every share preview. The wizard writes both.

The Mastodon account id. The sidebar widget wants a numeric id and people fill in an @handle, after which it shows nothing at all, quietly. The wizard verifies your token against the instance on the spot — and the id comes back out of that same call, so it can simply be offered, already correct.

Choosing one comments network switches the other off, because a config with both is one the build refuses to load.

Nothing is written until the end

Answers are collected, then both files' diffs are shown with secrets masked, and one confirmation covers the lot. Ctrl-C anywhere leaves an existing install exactly as it was.

Every question can be skipped with Enter. Re-running the wizard is how you change any of this later — and editing the files by hand keeps working exactly as it did. The two are interchangeable in both directions, because a config file you are not allowed to open is not a config file.

The diff shown is a proper LCS diff, incidentally. Comparing line for line failed in a way that mattered: adding one entry to a list shifts every line below it, so a four-line change read as everything from here to the end of the file — precisely the impression a tool asking permission to edit your config must not give.

And your comments survive

Both wizards write through a text-level editor that substitutes values into the documented template and leaves every other byte alone. Loading the YAML and dumping it back would have thrown away the two hundred lines of explanation, the commented-out blocks you uncomment when you want a widget, and the folded scalars real sites keep HTML in.

Every write is read back and verified, and restored from its backup if the file does not say what it was asked to say.

The four-line start

git clone https://github.com/DanielSnor/blog.sh.git
cd blog.sh && ./setup.sh
./blog.sh add
./blog.sh preview

The second line used to be two file copies and an editor. That is the whole change, and it is the one I should have made first.

Every importer here reads an export file. That works right up until the platform is gone — and then there is no export to ask for, no admin to log into, and no account to recover. Posterous shut down in 2013. Blog.cz is gone. Your host is next, eventually.

For those blogs the twentieth import source is the Internet Archive.

Feed mode: what the crawler kept

The Archive did something useful without meaning to. For years it crawled blogs' RSS feeds — over and over, every few weeks, whenever a crawler happened past. Each capture is a snapshot of the last ten or twenty posts at that moment. Line them up oldest first and the overlaps reconstruct a history nobody saved.

That is what the importer does: reads the CDX index for the feed's address, deduplicates by digest so a hundred identical captures cost one fetch, pulls the original bytes through the id_ endpoint so the Archive's own toolbar never lands in your posts, and merges the overlapping windows the same way a re-import merges anything else.

Images come back through the same time machine.

Verified by rescuing posts out of a Posterous blog that has been dead for thirteen years.

Page mode: when there was never a feed

Plenty of blogs the Archive crawled had their feed missed entirely. Those fall through to page mode: every archived post page, newest capture of each.

The hard part is that an archived page is just HTML from an era with no conventions. So platform packs say which paths are posts and how that platform's markup spells a title, a date and a body. Blog.cz ships built in — /YYMM/slug addresses, the article div, Czech long-form dates, and the windows-1250 encodings of the period converted on the way in. B2evolution ships too, built from a real 2008 skin rather than from documentation: the stock bText template that skins almost never replaced, h3.bTitle titles, tags behind their localized label, and the two-digit y/m/d dates read in the order the template source says — which no rendered page could have told you.

Without a pack, POST_PATTERN covers the shape by hand. With neither, the run refuses and prints sample archived paths to build a pattern from, rather than guessing.

What it will not pretend

This is the part I care about most, because a rescue tool that quietly invents things is worse than none.

What the crawler never met stays lost, and the run says so. Unreadable captures are counted. Pages that can only be dated by when they were captured are counted separately, not silently backdated. And a missing image — which the Archive serves as an HTML error page with a cheerful 200 — is caught by failing to measure as an image, counted as lost, rather than saved as a broken file that looks fine in a listing.

A busy Archive is waited out rather than read as a blog that was never there.

What you get back

Posts, with their dates, at permanent addresses, with whatever images survived, tagged with their origin, in a folder you own. Not a screenshot, not a PDF of a page — the same JSON files every other post on your site is made of, editable, searchable, exportable again.

The Archive kept a copy of your blog because it keeps a copy of everything. This just reads it back into something you can use.

An earlier post here explains that every colour on this site comes from seven keys in a config file. That part hasn't changed. What has changed is that you no longer have to imagine what those seven values will look like before you commit to them.

Choosing between fourteen hex values is exactly as blind in a wizard as it is in YAML. So ./style.sh shows you.

Seven palettes, one keystroke

Whole palettes now ship in config/palettes.yml, light and dark both:

• Default blue, warm, monochrome and high contrast — the four from the gallery on this site, whose light modes are exactly what that page showed. Their dark modes are new; the gallery only ever had light homepages.
• Sunflower, cream and gold with olive links — the golden yellow stays on the tag pills, because yellow text on cream cannot be read and the accent here is above all a text colour.
• Garden, greens and khaki.
• Ocean, steel blue over navy.

Those last three come from the TangerineUI Classic family this engine's own palette grew out of — its bluebird is the default palette here, to the last hex value.

Adding your own is adding an entry to that file. The wizard lists whatever it finds, and a palette you add needs no translation to show up.

Then it shows you the site

Pick one — or finish the fourteen-value custom route — and you are offered a preview: your own built site in the candidate colours, light and dark side by side. On a fresh install with nothing to render yet, a bundled sample post goes through the real builder instead.

On a deployed site the preview travels the way a draft preview does. It is uploaded to your own /palette-preview.html, and the wizard answers with the full address and a QR code — so a palette chosen at an SSH prompt can be judged on the phone in your pocket, on the screen people will actually read it on.

Locally it lands in tmp/palette-preview.html and opens in a browser where there is one.

The colours go through the same code the build uses. That is not a detail: lib/colors_css.rb was extracted for exactly this, so a preview cannot drift from what a rebuild would produce. A preview that lies is worse than no preview.

Nothing is written until the usual confirmation.

The banner section measures for you

Give it the path to an image and it copies the file into place and measures it. banner.width and banner.height exist to reserve layout space before the image loads; they have always been copied by hand, and a stale pair makes every page jump as it loads.

Why this is its own wizard

setup.sh and style.sh both write config/site.yml. They are split by lifecycle, not by file: setup asks the things you answer once, this is everything you come back and fiddle with — palette, banner, about text, footer, social icons, sidebar widgets, fonts, analytics.

So it is a menu you dip into, not a form you complete. Which is the honest shape for the part of a site nobody ever finishes adjusting.

The Facebook importer was built against a real "Download Your Information" export. The first useful thing it told me was a number I did not expect.

On that export, 95 % of the posts were not written on Facebook. They were mirrored in — from Twitter, from Posterous, from the whole crosspost era when every service politely offered to repost you everywhere else.

They are skipped, and counted

Those posts do not import. Not because they are worthless, but because the platforms they actually came from have their own exports, and those carry the originals: the full text, the real timestamps, the media at the size it was uploaded. A Facebook mirror of a tweet is a worse copy of something you can import properly from the Twitter archive sitting in the same folder.

Import both and your archive doubles, with the lesser version winning half the time. So they are recognized and skipped with a count in the summary, and FACEBOOK_CROSSPOSTS=1 overrides it if you disagree. Wordless check-ins and app stories are counted skips too.

That leaves your own posts, with the photos and videos from the archive itself.

Both formats read the same

Meta lets you choose JSON or HTML on the way out, and people pick without knowing it matters. Here it doesn't: both are read, and they come out epoch-identical, down to the ids minted for re-import matching.

Getting there was the fiddly part. The JSON carries plain timestamps. The HTML prints a wall clock — with seconds, in the account's own timezone, daylight saving observed — which had to be measured against the JSON of the same account to be sure. And the HTML's dates arrive localized, so Czech and English are both understood, and a language the importer does not recognize skips what it cannot date and says so rather than guessing a month name.

Threads, the twenty-second

Threads closes the social-network roster, and its export has the strangest shape of the family: every post is a media list even when there is no media, with the text riding in a title field. The HTML page is read back into that same odd shape so both formats walk one mapping.

Ask for JSON where you get the choice. Only the JSON marks replies — an HTML run ends by saying so every time, because it cannot know whether there was anything to miss — and only the JSON carries seconds. The HTML page prints minutes, in Meta's fixed no-daylight-saving Pacific clock, which converts back exact to the minute against the same account's JSON.

One flag in that export is a trap worth naming: cross_post_source looks like the crosspost marker, and on real exports it also marks posts written directly in the Threads app. It records where a post was shared to. Skipping on it would empty your archive.

The encoding, finally

Meta's exports have mangled non-ASCII text for years — the bytes are UTF-8 read as Latin-1 and re-encoded, so every accented letter arrives as gibberish. The Instagram importer proved out a repair for it; it now lives in a shared module the whole Meta family uses.

Which means Czech, German, French and everyone else gets their own words back, instead of an archive of ÄŤ and é.

The part that stayed with me

Nineteen out of twenty posts on that account were copies of things written somewhere else — somewhere that has usually since closed. The crosspost era felt like insurance at the time. It turned out to be the opposite: the copies outlived the originals, in a worse form, on a platform that will not hand them back in one piece.

Ask for the export. From all of them.

A post here recently argued that the wizard menu got shorter because operations on a post belong in the post, not in a menu. That still holds. This release adds a screen anyway, and the reason is worth stating: finding a post is a different job from doing something to one.

./blog.sh browse shows the same posts list does, as a screen you stay in.

What it is

Arrows and Page Up/Down through the whole archive. Filters by type, state and tag, with the count next to each. And a search that filters as you type.

The search is the site's own. Words are ANDed, "a quoted phrase" counts as one, -word excludes, and diacritics never decide a match. The query language and the folding live in lib/search_query.rb, next to a note that it and assets/js/search.js change together — because a query that means one thing in your browser and another in your terminal is worse than having no search in the terminal at all.

It searches the full text of every post, so under the selected row there is a line of that post's own text showing you why it matched.

Space opens a read-only preview — the same markdown edit would hand you, with media lines shortened to their filenames. Enter opens it for editing and comes back to the same row, with the same filter still applied. z clears the filters.

Two small decisions that took the longest

The keys are deliberately none of the letters that mean an action elsewhere. p is "publish" in three dialogs and x is "delete" in two, so neither could be the preview key here — muscle memory does not check which screen you are on. The preview is the space bar, the way every file manager has done it for thirty years.

The rows lead with the title, not the slug. list still prints the slug first, because down a pipe the slug is the thing you copy into the next command — and browse falls back to exactly that when it is not talking to a terminal. But on screen, a title is what you are scanning for. On an imported archive that matters more than it sounds: over half the posts arrive with no title at all, so the row falls back to the slug for those, which is precisely the set where the slug is all there is.

Who this is actually for

Twenty-two importers mean archives that are thousands of posts long, not dozens. list was written when this site had fourteen. Scrolling a pipe is fine for fourteen.

For four thousand imported posts, from six platforms, half of them untitled, you need to be able to look — filter to one origin tag, search for a word you half remember, read the matching line, open the one you meant. That is the whole feature, and it only became necessary because the importing worked.

Every abort in this engine is correct where it stands. Each one also reports exactly one problem — the first one, from wherever in the code it happened to be noticed. Fix it, run again, meet the next one. Repeat until the thing builds.

./blog.sh doctor reads what is on disk and tells you the lot in one pass, in whole sentences, each with a fix line written for somebody who does not know which file that setting lives in.

It goes after the silent failures

A loud error at least announces itself. The problems doctor concentrates on are the ones that never say anything:

• An unknown timezone. Ruby falls back to UTC without a word, and every post is dated wrong from then on.
• A banner whose declared size no longer matches the file. Every page jumps as it loads, and nothing anywhere is broken enough to complain.
• A widget that can never show anything — configured with the wrong kind of identifier, rendering an empty box forever.
• A font named in the config but missing from assets/fonts/. The browser silently falls back and the site is simply not the site you designed.
• A deploy backend configured half way.
• The example's text still sitting where visitors would read it — the about box that still describes a blog engine's sample site rather than yours.

--online additionally asks whether the feeds, the analytics script and the access token still answer. Tokens expire quietly; that is their nature.

It runs when nothing else will

This is the part that took the most care. Doctor works on configurations too broken for anything else to load — including one whose YAML will not parse, which is exactly the moment you want it.

Related, and shipped alongside: a YAML syntax error in config/site.yml is now a sentence rather than a Psych backtrace from whichever entry point happened to read the file first. It names the line, the column, the three usual causes — a tab where spaces belong, a missing quote, a colon inside an unquoted value — and points at doctor.

Errors and advice are different things

Exit status is non-zero for errors only. Warnings are advice, and advice that fails your build is not advice.

That distinction is why doctor is safe to put in a cron or a pre-deploy check: it will stop you when something is actually wrong, and merely talk when something is merely worth knowing.

Why bother, on a tool this small

Because "it doesn't work and I don't know why" is where people give up, and the engine's own failure modes were the ones most likely to produce that sentence. The abort messages were all accurate. Accuracy one problem at a time is still a maze.