Deploying Kiln

How to run Kiln in production, from nothing: the environment it needs, the image, what happens at boot, which health endpoint to point what at, how to get the first admin, and where backups and the optional infrastructure fit. This is the standing guide; the checklists that used to be the only deploy material are kept under Audits & release checklists: deploy-p2.md and deploy-p3.md are rehearsal notes for specific feature batches (history), and deploy-staging.md / deploy-write-visual-editing.md are still the operator checklists for turning those two features on — all four assume you already have a running Kiln, which is what this guide gets you to.

Kiln is one OTP release plus Postgres. It serves its own public site and the editor console, runs background jobs on Oban (Postgres-backed — no broker), and caches in-process — so a working deployment is a container and a database. Everything else in this guide is optional.

Where the other operations guides fit:

The short version

1. Build the image from the Dockerfile (or pull one you built in CI).
2. Provide Postgres 17 with the vector extension available — the pgvector/pgvector:pg17 image is a drop-in.
3. Set DATABASE_URL, SECRET_KEY_BASE, TOKEN_SIGNING_SECRET, PHX_HOST, and PHX_SERVER=true.
4. Start the container. It runs pending migrations, then serves on PORT (default 4000).
5. Put a TLS-terminating reverse proxy in front on 443, and set TRUSTED_PROXIES so the app sees real client addresses.
6. Register the first user at /register, then promote it to admin from a release shell (below).
7. Point your restart-triggering healthcheck at /live and your load balancer or uptime monitor at /up.
8. Schedule scripts/backup.sh before you have anything to lose.

docker-compose.prod.yml at the repository root does steps 1–4 as a reference; the rest of this document explains each step and the decisions behind it.

Required environment

Three variables raise on boot when missing; two more are effectively required because their defaults are wrong for any real deployment. Every variable is documented in full in environment-variables.md — this is the deploy-time summary.

Two more that a real deployment almost always wants:

• PORT (default 4000) — the port the app listens on. The public URL is hardcoded to https/443; the expected topology is a TLS-terminating proxy on 443 forwarding to PORT.
• TRUSTED_PROXIES — comma-separated CIDRs of the reverse proxy in front (Coolify, Traefik, nginx, a cloud load balancer). Unset behind a proxy, every request looks like it came from the proxy, so every rate-limit bucket collapses into one counter for the entire internet and the per-IP brute-force protection on /sign-in stops being per-IP. Nothing errors — the app logs a warning once. Leave it unset only when the app is internet-facing directly. Details and the private-range caveat are under server & networking in environment-variables.md. On Fly.io, Railway or DigitalOcean App Platform, set CLIENT_IP_HEADER instead; see deploy-platforms.md.

Generate the two secrets once, store them in your secret manager, and never regenerate SECRET_KEY_BASE on an existing deployment.

mix phx.gen.secret   # SECRET_KEY_BASE
mix phx.gen.secret   # TOKEN_SIGNING_SECRET

(No checkout to hand? openssl rand -base64 64 | tr -d '\n' produces an equivalent value.)

Database TLS

The Postgres connection uses ssl: true by default. Set DATABASE_SSL=false only for a provider that genuinely cannot offer TLS — a Postgres container on the same private compose network is the usual case, and the reference compose file does exactly that. Point DATABASE_SSL_CACERTFILE at the provider's CA bundle to verify the server certificate; without it the connection is encrypted but skips peer verification.

Building the image

The Dockerfile is a multi-stage build: an hexpm/elixir builder compiles the release (with mix assets.deploy — there is no separate front-end build to deploy; the app is the site), and a slim Debian runner carries only the release plus the native libraries it shells out to — libvips for image processing, qpdf for PDF metadata stripping, postgresql-client-17 so in-app backups can run pg_dump. Nothing in the runner is optional to the build, and the versions of Elixir/OTP restate .tool-versions (the source of truth; mix kiln.toolchain.check fails when they drift).

Build with the commit and date stamped, so a running instance can say what it is on /editor/system:

docker build \
  --build-arg GIT_SHA="$(git rev-parse HEAD)" \
  --build-arg BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  -t kiln:local .

Both build args are optional — an image without them boots and reports its version, it just can't name the commit. See releasing.md.

A PaaS that builds from the Dockerfile can't run $(git rev-parse HEAD). For Coolify, turn on the application's Include Source Commit in Build setting: Coolify then passes the deployed commit as the SOURCE_COMMIT build arg, and the Dockerfile uses it whenever GIT_SHA isn't set, so every Redeploy records its commit. The build date has no equivalent; leave it unset or set BUILD_DATE as a build variable yourself.

Two things about the build worth knowing before you size a build host:

• Peak RAM is mix deps.compile, not the app compile. The Ash ecosystem plus the Nx/Axon/Bumblebee ML stack compiles in one BEAM; the Dockerfile splits the ML deps into their own RUN and caps the build BEAM to two schedulers so a small host isn't OOM-killed. A 2 GB build host is marginal; 4 GB is comfortable.
• The runner is deliberately a newer Debian than the builder (trixie vs bookworm) because bookworm's qpdf predates the options PDF stripping needs. Building on the older suite and running on the newer is the safe direction; don't "fix" the mismatch.

The build is the same wherever it runs — a docker build on your laptop, a CI job, or a PaaS that builds from the Dockerfile (Coolify does exactly that on Redeploy). Nothing about the image is host-specific: every deployment difference is environment.

What happens at boot

The image's CMD is:

/app/bin/migrate && /app/bin/server

1. bin/migrate runs KilnCMS.Release.migrate/0 — every pending Ecto migration, up. It runs unconditionally on every boot, so there is no separate "run migrations" step to forget on a fresh target or after a failed build (a platform's pre-deploy hook typically only runs inside an already-running container, which is a no-op precisely when you need it). Ecto.Migrator serialises concurrent runs with a lock on schema_migrations (ecto_sql's default :table_lock), so this stays safe when several replicas start at once: one migrates, the others wait, then all serve. During a rolling deploy the old release keeps serving against the migrated schema; the expand/contract policy that keeps that safe, and what zero-downtime does and does not cover, are in releasing.md.
2. bin/server sets PHX_SERVER=true and starts the release. The Ash + Nx/Axon/Bumblebee stack is not fast to cold-boot; the image's healthcheck allows a generous start period for it (below).

A boot failure is loud. The three required variables raise with a message naming the variable; a :prod release with dev_routes enabled refuses to boot (it would expose /admin, LiveDashboard and the mailbox unauthenticated); S3_BUCKET without S3_PUBLIC_BASE_URL raises; and so on. If the container exits immediately, docker logs (or the platform's deploy log) has the reason on the last lines.

Rolling back a migration is a manual step and never automatic, and because every boot re-runs bin/migrate, the order matters:

1. Stop the app (docker compose -f docker-compose.prod.yml --env-file .env.prod stop app). A running or restarting container would re-apply the migration you are about to undo.

2. Run the rollback from the image that contains the migration — the new one, not the one you are rolling back to. Ecto.Migrator only sees migration files present in the image, so an older image would silently skip the newer migration and report success:

   docker compose -f docker-compose.prod.yml --env-file .env.prod \
     run --rm --no-deps app /app/bin/kiln_cms eval \
     'KilnCMS.Release.rollback(KilnCMS.Repo, <version>)'

   <version> is the timestamp prefix of the migration to undo (the file name in priv/repo/migrations/), and to: is inclusive: everything at or after that version is rolled back, so name the oldest one you want undone, not the one you want to return to.

3. Then switch the image tag back and start.

Read the release's ### Upgrading notes in CHANGELOG.md before an upgrade — that section says whether a migration rewrites or drops data, which is what decides whether rolling the image back is enough.

Health endpoints — /live vs /up

Kiln has three probes, and pointing the wrong one at the wrong consumer is the one deploy mistake that makes a bad day worse:

Why two: /up returning 503 on a database outage is exactly right for a load balancer (stop sending users to a node that can't serve them) and exactly wrong for a restart trigger — restarting the app on a database outage it cannot fix only restart-storms every replica at the moment the platform is already degraded (#816). /live is answered by an endpoint plug ahead of tenant resolution (which itself reads the database), so it stays a pure "is this process serving" signal even with Postgres down.

Two things the image already handles, and that you must preserve if you change either: the healthcheck probes http://127.0.0.1:${PORT}/live, and that works because config/prod.exs's force_ssl excludes host 127.0.0.1 — without the exclusion Plug.SSL answers a 301 to https, and curl -f treats a 3xx as success, so the check would pass without ever seeing a 200. And the healthcheck asserts the container is serving HTTP, not merely that the BEAM is alive: a container that booted without PHX_SERVER runs migrations, answers rpc, serves nothing, and used to report healthy forever (#647).

The HEALTHCHECK in the Dockerfile — interval=30s timeout=5s start-period=60s retries=3 — is what a plain docker run or Compose uses. Plain Docker and Compose only mark the container unhealthy; nothing restarts it — restart: policies react to the process exiting, not to health. Who does act on it: Swarm restarts a task whose container healthcheck fails, and an autoheal sidecar on Compose (e.g. willfarrell/autoheal) does the same by watching that status. Kubernetes ignores the image HEALTHCHECK entirely and only runs the probes you declare — point liveness at /live and readiness at /up yourself. Coolify uses the check to judge a deploy and to show status. Platforms with their own health-check UI ignore the Dockerfile values; set the same path and a start period of at least a minute there.

First admin — bootstrap

Self-registration always lands on the :viewer role, so sign-up can never escalate privileges, and there is no seed script in a production release.

The easy path (#1317): visit https://<PHX_HOST>/setup. While the instance has no admin, a three-step wizard creates one — email + password, an optional site name/colour/theme — and then sends you to sign-in. The page gates itself on "no admin exists" (policy-enforced and race-safe, not just hidden) and redirects home forever after, so it is safe to leave deployed.

The release-shell path below remains for headless bootstrap: promote a registered user with authorization bypassed — once, and only for the first one.

1. Register your own account at https://<PHX_HOST>/register. (Want invite-only sign-up afterwards? config :kiln_cms, :registration_enabled, false is compile-time — set it in config/config.exs or a project overlay and rebuild the image; there is no environment variable for it.)

2. Attach to the running node and promote it:

   docker exec -it <container> /app/bin/kiln_cms remote

   alias KilnCMS.Accounts
   
   user = Accounts.get_user_by_email!("[email protected]", authorize?: false)
   Accounts.manage_user_access!(user, %{role: :admin}, authorize?: false)

   Or one-shot, without a shell:

   docker exec -it <container> /app/bin/kiln_cms rpc 'KilnCMS.Accounts.manage_user_access!(KilnCMS.Accounts.get_user_by_email!("[email protected]", authorize?: false), %{role: :admin}, authorize?: false)'

   Both remote and rpc connect to the live, fully-started app — that is why they work. bin/kiln_cms eval boots a separate instance without the repo started, so don't use it for this. Use straight ASCII quotes.

3. Sign in; promote everyone else through the editor UI as that admin — no authorize?: false needed again.

The same section in the README covers the development seed script and the platform variants (fly ssh console and the like).

Reverse proxy and TLS

The app listens on plain HTTP on PORT and expects a TLS-terminating proxy in front on 443 — Kiln does not terminate TLS itself, and force_ssl redirects plain-http requests on any host other than localhost/127.0.0.1 to https. Whatever the proxy — Coolify's Traefik, nginx, Caddy, a cloud load balancer — it must:

• send X-Forwarded-Proto: https on every request. config/prod.exs runs force_ssl with rewrite_on: [:x_forwarded_proto], so a request the proxy forwards as plain HTTP without that header is answered with a 301 to https://… — which the proxy terminates and forwards again, forever (nginx does not add the header by default; Caddy and Traefik do);
• forward X-Forwarded-For, and name the proxy in TRUSTED_PROXIES (above), or rate limiting keys on the proxy's address;
• pass WebSocket upgrades on /live, /ws/* — the editor is LiveView, and collaboration, GraphQL subscriptions and the visual-editing bridge are channels;
• send the request Host through unchanged — PHX_HOST and the origin check are validated against it, and multi-tenant sites are resolved from it.

Two symptoms and their usual causes: the browser reports too many redirects on every page → the proxy is not sending X-Forwarded-Proto; the editor loads but every LiveView shows "disconnected" or the page reloads in a loop → PHX_HOST (wrong host, or a scheme baked in) or the proxy not upgrading WebSockets.

Backups

Do this before the first real content, not after the first incident. backups.md is the full runbook — what constitutes a deployment's state (the database, media if on the Local adapter, and the environment including SECRET_KEY_BASE), the nightly cron using scripts/backup.sh, retention, off-site copy, and the restore runbook from total loss to serving. Two things it insists on that are worth repeating here:

• SECRET_KEY_BASE is part of the backup. A database restored under a different key boots and serves, but every database-stored key (KilnCMS.Keys.Vault — the DKIM private key among them) is unrecoverable, and every session and token is invalidated.
• The client tools must match the server's major version. The image ships postgresql-client-17; the in-app Backups page (/editor/backups) uses the same tools and the same variables (BACKUP_DIR, MEDIA_DIR, …) as the cron script — two front doors to one backup directory. Bump the client pin in the Dockerfile alongside any Postgres major upgrade.

In the reference compose stack both doors open on one host directory: Postgres is published on the host loopback (127.0.0.1:5432) so the cron in backups.md runs on the host against postgres://kiln:…@127.0.0.1:5432/kiln_prod and writes /var/backups/kiln, and that same directory is bind-mounted into the app at BACKUP_DIR, so the in-app page reads cron's manifest and cron's off-site copy and pruning cover what the page writes. One uid has to own that directory and both writers have to be it — the app runs as nobody (uid 65534) and makes its subdirectories 0700, so a shared group does not help. Create it before the first up (Docker otherwise auto-creates a bind source root-owned; a later chown still fixes it) and run the backups.md cron as nobody (or as root) rather than as a third user:

sudo mkdir -p /var/backups/kiln && sudo chown 65534:65534 /var/backups/kiln
# crontab: 17 3 * * * nobody  DATABASE_URL=… BACKUP_DIR=/var/backups/kiln …/scripts/backup.sh all

On any other platform, give BACKUP_DIR a persistent mount writable by nobody (the image already owns the default path, so a fresh named volume mounted there works) or the in-app page will report available and every job will fail at run time.

Optional infrastructure

Nothing beyond Postgres is required. Each of these is opted into by setting its variables, and the reference compose file starts the matching service behind a profile so you can enable exactly what you turned on:

Multi-node itself needs no broker: Phoenix PubSub is native, Oban is Postgres-backed, and node discovery is DNS_CLUSTER_QUERY. Two things a cluster does need: a shared media backend (object storage — Local uploads on one node are invisible to the others), and long node names. The release ships no rel/env.sh.eex, so it starts with mix release's defaults (RELEASE_DISTRIBUTION=sname), and a short-named node cannot connect to the kiln_cms@<ip> names DNSCluster resolves — set RELEASE_DISTRIBUTION=name and RELEASE_NODE=kiln_cms@<this node's IP> in each node's environment (the snippet Phoenix's Fly guide generates), or the nodes stay silently disconnected and cache busts never cross them.

The auth budgets need nothing from the cluster at all. The sign-in, second factor, reset-mail and credential-form budgets count in Postgres (throttle_counters, #1619), so they hold across nodes whether or not the nodes are connected. The flood ceilings (:api, :delivery, …) stay per node by design, so an N-node cluster admits N times each of those.

The reference docker-compose.prod.yml

docker-compose.prod.yml runs the app and a pgvector Postgres, with MinIO, Meilisearch and Dragonfly behind the profiles above. It is a starting point, not a platform: it does not terminate TLS, does not schedule backups (it only makes them possible — above), and keeps Postgres on the compose network with TLS off because the traffic never leaves the host. It needs the Docker Compose v2 plugin (2.24 or newer — older releases evaluate the nested default that lets DATABASE_URL replace the bundled Postgres eagerly); the project is namespaced kiln-prod so it never shares volumes with the dev docker-compose.yml on the same machine (they do share the default published ports 5432/9000/9001 — set POSTGRES_PORT / MINIO_PORT if both stacks are up).

# .env.prod (at the repo root): SECRET_KEY_BASE, TOKEN_SIGNING_SECRET, PHX_HOST,
# POSTGRES_PASSWORD (URL-safe: `openssl rand -hex 32`), TRUSTED_PROXIES, plus
# any optional variable from environment-variables.md
docker compose -f docker-compose.prod.yml --env-file .env.prod up -d
docker compose -f docker-compose.prod.yml --env-file .env.prod --profile storage up -d   # + MinIO
docker compose -f docker-compose.prod.yml --env-file .env.prod logs -f app

Configuration is one file, .env.prod, used two ways: --env-file interpolates the compose file (image tag, ports, the required secrets — it fails fast, naming the variable, if SECRET_KEY_BASE, TOKEN_SIGNING_SECRET, PHX_HOST or POSTGRES_PASSWORD are unset), and the app service loads the same file as its env_file, so any optional variable you add there reaches the container. The compose file's header is the contract for that file — how Compose parses it ($ expands; $$ or single quotes for a literal), that it must sit at the repository root or KILN_ENV_FILE must name it, why optional variables are passed through rather than listed with empty defaults (several are presence-checked: S3_BUCKET= would switch the storage adapter to S3 with no bucket), and every variable the file itself reads. Two of its points matter before you write the file:

• POSTGRES_PASSWORD is spliced into DATABASE_URL, so it must be URL-safe — openssl rand -hex 32. Base64 output (mix phx.gen.secret, openssl rand -base64) contains / more often than not, and / # ? % break or silently corrupt the URL while Postgres itself accepts them.
• An external or managed Postgres is DATABASE_URL=… DATABASE_SSL=true in .env.prod (plus DATABASE_SSL_CACERTFILE if you have the CA) — like everything under environment: except PHX_SERVER, they take your .env.prod value ahead of the bundled default. Then delete the postgres service and the app's depends_on from the compose file: left in, it starts with no password, restart-loops, and every later plain up -d (upgrades included) fails on the dependency.

Every app variable in that header is a row in environment-variables.md; the header marks the handful the compose file alone reads (KILN_IMAGE, KILN_ENV_FILE, BACKUP_HOST_DIR, the *_PORT publish ports), which are not.

Put a proxy on the same host in front of 127.0.0.1:4000 — the app is published on loopback only, on purpose. Caddy is the least configuration:

cms.example.com {
    reverse_proxy 127.0.0.1:4000
}

Caddy sends X-Forwarded-For and X-Forwarded-Proto and upgrades WebSockets by default; then set TRUSTED_PROXIES to the compose network's range (docker network inspect kiln-prod_default shows it — the host's loopback connections arrive from that bridge). Do not re-publish the app on 0.0.0.0 to reach it from a proxy on another machine: a Docker-published port bypasses the host firewall, and with TRUSTED_PROXIES set the app honours X-Forwarded-For from whoever reaches it — bind a private interface instead, or run the proxy inside the compose network.

Platform notes

Coolify (the project's own production). Production for this repository is a single VPS running Coolify, deploying by a manual Redeploy that builds the Dockerfile and starts the container: migrations run on boot, assets build at image build, and the health check is /live. Everything above applies as is; environment goes in Coolify's application settings (turn on Include Source Commit in Build so the image records its commit — see Building the image), and Coolify's Traefik is the proxy — set TRUSTED_PROXIES to its docker network range. The per-release checklists under Audits & release checklists were written against this target and describe, per feature batch, what to verify after a Redeploy — nothing there changes how to deploy.

Kubernetes / any orchestrator. Liveness probe /live, readiness probe /up, and a startupProbe on /live with a generous failureThreshold × periodSeconds (minutes, not seconds) rather than a fixed initialDelaySeconds: /live only answers once bin/migrate has finished, so a long data-rewriting migration under a 60-second liveness window gets the pod killed mid-migration and never converges. Several replicas may start together; the migration lock handles it. Use object storage for media, and long node names for clustering (above).

Render / Railway / Fly.io / DigitalOcean — one-click. Render, Fly.io and DigitalOcean have a template in the repository, and Railway a recipe, that run the published image with a database and persistent media: see deploy-platforms.md.

Other Dockerfile PaaS. Build from the Dockerfile; set the required variables as secrets; internal port PORT (4000); health check path /up for routing (the platform is the load balancer here) — and if the platform restarts on health failure, /live. fly ssh console gets you a shell for the first-admin bin/kiln_cms remote.

Production hardening checklist

The README's checklist is the canonical list; in deploy order:

• dev_routes off — it is; a :prod release refuses to boot otherwise.
• Database TLS on (default), with DATABASE_SSL_CACERTFILE when the provider publishes a CA bundle.
• TRUSTED_PROXIES set behind a proxy.
• PHX_HOST correct; CHECK_ORIGINS for any additional hostnames.
• CORS_ORIGINS unset unless a separate front end reads the headless API — the write API and the visual-editing bridge are inert cross-origin until it is set (deploy-write-visual-editing.md is the checklist for turning that on).
• Registration closed once the first admin exists, unless the site is meant to be open — config :kiln_cms, :registration_enabled, false is compile-time config, so this is a rebuild, not an env change.
• Backups scheduled and a restore rehearsed (backups.md).
• Error tracking / tracing (SENTRY_DSN, OpenTelemetry) — env-gated no-ops until set (observability.md).
• Sensitive files kept out of public storage — see the warning below.

Public media is readable by anyone with its URL

Warning. Every public media file — every image, and every document or A/V file that is not gated to an audience — is served with no authentication at /uploads/<key> (Local adapter, Plug.Static) or at S3_PUBLIC_BASE_URL + <key> (S3 and its CDN). Being unpublished, or unlinked, does not hide it.

The keys are random UUIDs, so such a URL can't be guessed — it works as a capability, and whoever holds it can read the file. For anything that must stay private: set S3_PRIVATE_BUCKET if you run the S3 adapter (the Local adapter's private directory needs nothing), then give the document or A/V item a non-public audience, which moves it to private storage served only through the authorized download and stream routes. Images can't be gated, so don't upload sensitive ones. On S3, don't let the public bucket be listed. The full explanation, and what gating can't undo, is in media-pipeline.md; the threat-model entry is Media (`/uploads/)* in [threat-model.md`](threat-model.md).

Jobs interrupted by a deploy

Stopping a node (SIGTERM, a rolling deploy, a scale-down) gives its running Oban jobs 15 s (shutdown_grace_period) to finish. A job still running after that is killed, and its row is left in executing with nobody working it.

Kiln runs Oban.Lifeline to recover these. Once a minute the cluster's Oban leader looks for jobs that have been executing for longer than KILN_OBAN_RESCUE_AFTER_MINUTES (default 180). It moves each back to available to run again, or to discarded if it had already used its last attempt. Without the rescue, such a row would stay executing for ever. For a unique worker it would also block every later enqueue of the same job.

The rescue goes by time alone. It cannot tell a dead job from a slow one, so the window must be longer than any job legitimately runs. Otherwise a live job runs a second time. The longest job with a timeout is a backup (KilnCMS.Backups.Worker, 2 h), and a test fails if any worker's timeout/1 comes within 30 minutes of the window. Raise the variable if you have added a longer worker. Lower it to get stranded work back sooner, but only if you know your longest real job. false switches the rescue off, for a deployment that runs its own rescuer (a Lifeline already in the Oban plugin list is left alone).

A rescued job starts again from the beginning. What that means for each kind of work:

• Idempotent, safe to repeat: firing and re-firing (artifacts upsert), search indexing and embeddings, media variants and the A/V metadata strip, CDN purges, scheduled publish/unpublish triggers, cron sweeps, the occurrence backfill, billing webhooks (claimed once per event).
• Delivered at least once: outbound webhooks, federation deliveries, web push, and single emails (transactional mail, form notifications, one newsletter recipient). A job killed after the remote side accepted the message but before Oban recorded success sends it again. This is the same guarantee an ordinary retry already gives.
• Newsletter fan-out (SendWorker) is safe to repeat. Each {send, subscriber} pair can be enqueued only once, so a repeat reaches only the recipients the first run missed.
• max_attempts: 1 workers are discarded, not re-run: backups, demo reset, slug regeneration, content-release go-live/rollback. A backup leaves a .partial file that is never mistaken for a finished one; take a new backup. A release left in publishing or rolling_back is freed with the admin-only abandon action in the release console.

Upgrading a deployment

For a running instance an upgrade is: read the release's ### Upgrading section in CHANGELOG.md, rebuild or pull the new image, restart (on the reference stack, docker compose -f docker-compose.prod.yml --env-file .env.prod up -d). Migrations run on boot; nothing else is automatic, and the changelog section is where a release says if it needs more (a new required variable, a reindex, a backfill). Downstream projects that pin Kiln as a submodule move the pin with mix kiln.update, which prints that section before it does — releasing.md.