Content portability — import and export

Getting content in from another system, and out of this one (#487).

Four mix tasks:

All four take --actor EMAIL and --org SLUG, and every importer takes --dry-run.

From a release

An OTP release has no Mix, so the three importers also exist as functions on KilnCMS.Release. Run them through bin/kiln_cms rpc, against the node that is serving the site:

docker cp wordpress.xml <container>:/tmp/wordpress.xml
docker exec -it <container> /app/bin/kiln_cms rpc 'KilnCMS.Release.import_wordpress("/tmp/wordpress.xml", dry_run: true)'
docker exec -it <container> /app/bin/kiln_cms rpc 'KilnCMS.Release.import_ghost("/tmp/ghost.json", site_url: "https://blog.example.com", dry_run: true)'
docker exec -it <container> /app/bin/kiln_cms rpc 'KilnCMS.Release.import_content("/tmp/content.json", dry_run: true)'

The report is the same one the task prints, and it appears in the rpc terminal. Three differences from the tasks:

• Use rpc, not eval. The import enqueues media jobs, fetches images through SafeFetch and clears the content caches, so it needs the running application. bin/kiln_cms eval starts a separate VM with none of that, and the functions refuse there rather than fail halfway through.
• The path is read inside the container, so copy the export in first (docker cp, as above). On Coolify, run docker cp on the host over SSH, then run the rpc command from the host or from the application's terminal in the Coolify dashboard.
• An unknown option is refused. A keyword list has no option parser, so without this check a misspelt dry_run: would run a real import.

Each call returns {:ok, report} or {:error, message}. If the connection drops during a long import, run the same command again: what already landed is skipped. KilnCMS.Portability.Commands holds the code that both the tasks and the release run.

Always dry-run first

mix kiln.import.wordpress export.xml --dry-run

A dry run performs every read — resolving slugs, matching taxonomy, deciding what each record becomes — and no writes, then prints the same report a real run prints. It shares the collision check with the real run, so the two cannot disagree about what already exists.

It is a plan, not a validation. No changeset is built, so a record the create action would ultimately reject is still counted as importable — a dry run can over-report. Media is the reverse: it fetches nothing, so it reports what it would_import rather than what is reachable.

"Imported, but not as the source had it"

A record can land and still not match its source: the actor was not allowed to publish it (an editor importing live content into a site where editors cannot publish), a state could not be restored, a source date or byline would not apply. The record counts as created — it exists — and the report lists it under this heading with the reason, so the difference is something you read rather than something you find later in the editor. These were previously only written to the server log.

Migrating from WordPress

A step-by-step walkthrough is in Migrating from WordPress; this section is the reference.

# 1. In WordPress: Tools → Export → All content. You get a .xml (WXR) file.
# 2. Look before you leap.
mix kiln.import.wordpress wordpress.xml --dry-run

# 3. Try a slice of it for real.
mix kiln.import.wordpress wordpress.xml --limit 20

# 4. The rest. Re-running is safe: what already exists is skipped.
mix kiln.import.wordpress wordpress.xml

What maps to what

Deliberate decisions worth knowing

A future (scheduled) post lands as a draft, with its date intact. Importing it as published would put it live before the author intended, and this importer has no scheduling story — the one case where guessing costs more than asking.

Only referenced attachments are imported. A WordPress media library is usually far larger than the content that survives a migration; pulling all of it turns a ten-minute import into an hours-long one for assets nobody asked for. Body images and featured images are fetched; orphans are not.

An unreachable image does not cost you the post. The block keeps the source URL (so it renders, hotlinked) and the failure is listed in the report for you to re-upload.

Multiple categories collapse to one. WordPress allows many, Kiln has one. The first is kept.

Redirects are built from the permalink's path, not the whole URL — the old host is by definition not this one. An old home page (/) is skipped: pointing the new site's root at one imported post would break it.

Publication dates are restored. published_at is not an authored field — the :publish transition stamps utc_now — so the importer puts the source date back afterwards. Without that, a decade of archives all land in one arbitrary-ordered second and feeds and sitemaps inherit it. The restore mints one extra version per record, which is an honest record that the import set it.

Media variants are generated asynchronously, by a running app node. The importer enqueues VariantWorker jobs; it does not wait for them. If you run the import in a one-off container with nothing else consuming the media queue, the jobs sit available and imported images render full-size until a node picks them up. Run the import where the app is running, or drain the queue afterwards.

What is not imported

Comments, users, widgets, menus, theme settings, plugin data and post revisions. WXR carries some of them; none map onto anything here without an editorial decision a mix task should not be making silently. The imported record is the current version, and its history starts at the import.

Authors

Every author in the file is listed in the report — login, display name and email — with a marker showing whether it resolved to a Kiln user:

Authors (1 mapped, 1 unmapped):
  -> jo "Jo Example" <[email protected]>
   ~ sam "Sam" <[email protected]>

Resolution order:

1. --author-map login=kiln@email (repeatable, and it also accepts the source email on the left)
2. otherwise the source author's own email, matched against a Kiln user — right far more often than not when both systems served the same people

An author that resolves to nobody is not an error: the record is attributed to --actor, exactly as before. It is reported so you can decide whether that matters before the content is live.

The create always runs under --actor, so an import can never mint content a mapped author was not allowed to create; only the attribution moves afterwards, through a narrow action that fires no webhooks.

Migrating from Ghost

mix kiln.import.ghost ghost-export.json --site-url https://blog.example.com --dry-run

KilnCMS.Portability.Ghost reads Ghost's JSON export (Settings → Advanced → Import/Export) into the same source-neutral shape the WordPress parser produces, so everything above applies: the dry run, the re-run rules, media sideloading, redirects, author mapping and the report. The Ghost-specific decisions:

• --site-url is required when the export contains __GHOST_URL__, the placeholder Ghost 4+ writes in place of its own address, or root-relative /content/images/… paths (older Ghost). Without it every image would be unfetchable, so the task refuses to start rather than import broken images.
• The rendered html column is the body, not Ghost's lexical or mobiledoc editor source. A very old post stored with no rendered HTML is listed as unreadable, not imported empty.
• Members-only, paid and tier-restricted posts stay gated. Members-only posts import into the member audience (or the first gated audience configured); paid and tier posts into :paid when configured, otherwise the same audience with a printed note. With no gated audience they import as drafts. Dropping visibility would publish them to the open web.
• scheduled and email-only posts import as drafts. The task prints each one, because the report cannot show a decision the parser made.
• Internal #tags are skipped. Public tags become tags; no category is set.
• Redirects come from /{slug}/, Ghost's default permalink. A custom routes.yaml structure needs its redirects added by hand.
• The file is read whole, and files over 128 MB are refused up front.

Members, comments, tiers, offers and code injection are not imported; see Migrating from Ghost for what to do about each.

The portable JSON envelope

mix kiln.export.content --out content.json
mix kiln.export.content --type post --state published --out published-posts.json

With no --state, the export holds every record: draft, in review, published and archived. Name one or more --states to narrow it; an unknown --state is refused rather than exporting nothing.

Each record imports in the state it was exported in. Published records go through the ordinary publish transition, so they version, fire and enter delivery like any publish. Records in review or archived are put back into that state quietly, by an import-only action. A restore is not new editorial activity, so it sends no review emails and no webhooks. A state the importer does not recognise (an older or hand-written envelope) imports as a draft.

{
  "kiln_export": { "version": 1, "exported_at": "...", "types": ["post", "page"] },
  "records": [
    {
      "type": "post", "title": "...", "slug": "...", "locale": "en",
      "state": "published", "blocks": [ ... ], "category": "news",
      "tags": ["guides"], "seo_title": "..."
    }
  ],
  "media": [ { "id": "...", "url": "...", "filename": "...", "alt": "..." } ]
}

References travel by slug, not by uuid. A uuid is meaningless in the target database, so an envelope full of them would import as content with every relationship dropped. Media is the exception — it travels by manifest entry, which the importer resolves to a URL and sideloads.

blocks is the typed-block tree in its storage shape, which is what a create action accepts on the way back in. An export the importer cannot load is a report, not a backup.

What an envelope does not carry

Workflow history, versions, anchors, view counters, comments and form submissions. Those describe what happened to the content on this site; they are not the content, and replaying an anchor into another system as though it had happened there would be actively wrong. See mix kiln.audit.checkpoint and the governance exports for that ground.

Media bytes are not embedded either. An envelope with base64 images is unusable at any real site's scale, and the importer sideloads from the manifest — which does mean the source must still be reachable when you import, or --skip-media will keep the blocks pointing at it.

Scale

The WXR parser reads the whole file into memory and hands it to xmerl, which expands it to a charlist first — roughly 16 bytes per source byte before the document tree is built on top. Files past 64 MB are refused up front, with the size and the remedy, rather than OOM-killing the VM with no error and no partial progress:

export.xml is 210 MB; the parser's ceiling is 64 MB.
Use WordPress's own split export ... and run this task once per file —
re-running is safe, because what already landed is skipped.

Media sideloading runs concurrently, grouped so no single host sees more than one in-flight request. A WordPress export points overwhelmingly at one origin, so a flat concurrency pool would just be eight parallel requests at the site you are migrating away from.

Both phases print progress every 25 items:

media: 150/512
records: 400/4000

Without that, the entire multi-hour body of a run is silent and there is no way to tell "working" from "hung on a stalled fetch".

--drain-media runs the queued variant jobs before the task exits. Use it when importing in a one-off container where nothing else consumes the media queue — otherwise the jobs sit available and imported images render full size until a node picks them up.

--limit is still the way to try a slice first.

Uses

• Moving a site between Kiln instances
• Seeding a staging environment from production content
• A portable copy that does not depend on this database
• Org-level content portability for the managed offering (#334)

CSV, for flat types

The JSON envelope is lossless and unreadable; CSV is readable and lossy. It is the format an editor can actually round-trip through a spreadsheet, and it is deliberately narrow so the lossiness is never silent.

mix kiln.export.content --format csv --type listing --out listings.csv
# edit in a spreadsheet
mix kiln.import.content listings.csv --type listing --dry-run

title,slug,locale,state,city,seats
Leeds site,leeds,en,published,Leeds,40

title, slug, locale, state, then one column per field in its declared order.

One type per file. A CSV has one header row, so --type is required in both directions.

Prose is refused, not flattened. A type whose records carry blocks errors out and names the offending slugs. Rendering a rich-text body to a cell loses its structure, and the editor who then re-imports that file has silently deleted their own formatting — --format json is the answer for those types.

An unknown column is an error. A header typo (Seats, or city with a trailing space) would otherwise import every row with that field silently empty, which is the likeliest spreadsheet mistake and the hardest to spot afterwards.

Spreadsheet-safe both ways. Values are RFC 4180 quoted, and a cell starting =, +, -, @ is prefixed with ' so no spreadsheet executes it as a formula. The reader undoes exactly that, so =SUM(A1:A9) survives a round trip unchanged while a genuine 'tis keeps its apostrophe. A UTF-8 BOM — which both Excel and Google Sheets write — is stripped, and a quoted field that never closes (a truncated download) is refused rather than half-imported.

Import goes through the same write path, conflict policy, dry run and report as every other import — so re-importing an unchanged file skips every row.

Re-running, and what happens on a collision

Matching is by (slug, locale) — the identity the database enforces.

The trashed case is called out because destroy is a soft delete: the row and its unique index survive while the ordinary read hides it. Without the check the importer would plan a create and the database would refuse it with a bare slug: has already been taken, which points at nothing. Restore the record or purge it, then re-run.

There is no overwrite mode. A second import silently replacing edits an author made after the first one is not recoverable through any UI. Content sync is a different problem from content migration, and conflating them is how people lose work.

Authorization

Every read and write runs under --actor's own policies. An export contains exactly what that user could have read through the UI, and an import can only create what they could have created by hand. Neither is a way around authorization — an export endpoint that bypassed policy would be the most efficient exfiltration primitive in the system.

Media sideloading fetches URLs that came from a file someone uploaded, so it goes through KilnCMS.SafeFetch — the same path as oEmbed resolution. That resolves the host once, checks the answer, and connects to that literal address with SNI pointed back at the real name, which is what stops a name being re-resolved to 169.254.169.254 between the check and the connection. Redirects are refused rather than followed (a followed redirect is a fresh resolution the pin never sees), and the body is capped.

Extending to another source

KilnCMS.Portability.Import takes a source-neutral shape. KilnCMS.Portability.WXR produces it from WordPress and KilnCMS.Portability.Ghost from Ghost; a Drupal importer only has to produce the same shape to get the dry run, the conflict policy, the media sideloading, the redirects and the report for free.

KilnCMS.Blocks.Html is the shared HTML → blocks adapter, and is deliberately the only place HTML is read back into structure.

Related

• media-pipeline.md — what happens to a sideloaded asset
• redirects — how an imported permalink resolves
• #472 — 404 capture, the other half of the migration story