Releasing Kiln
Downstream projects pin this repo as a submodule and update between tagged
releases (mix kiln.update). That only works if tags exist and carry honest
upgrade notes — this is the checklist for cutting one.
Why tag at all
main moves fast. A project that fast-forwards to whatever main is today
takes an unbounded, undocumented jump: new migrations against its live
database, possibly a changed overlay contract, and no place to have written
down "set this env var first". A tag is the unit that can carry that note.
Versioning
Semver, interpreted for a CMS core that projects overlay (the full definition
lives at the top of CHANGELOG.md):
| Bump | Means |
|---|---|
| major | the overlay contract broke — a projects/<name>/ subproject needs code changes to compile |
| minor | new capability, overlays keep compiling; may add migrations |
| patch | fixes only |
The major rule is the one to be strict about. mix kiln.update refuses a major
jump without --allow-major precisely because it's promised to mean "your
subproject needs work" — bumping major for a merely large release trains
people to pass the flag reflexively.
The plugin SDK (kiln_plugin)
The plugin contract ships in its own package,
sdk/kiln_plugin (#1909), released from its
own kiln_plugin-vX.Y.Z tags by release-clients.yml, not by a core release.
The core always builds against the copy in this repository, so a core release
needs nothing from Hex. Two things tie the two together:
- A core release that changed
sdk/kiln_pluginpublishes the package too: bump its@versionand CHANGELOG in the same release PR, and tagkiln_plugin-vX.Y.Zafter the core tag. Third-party plugins resolve the contract from Hex, andmix kiln.plugins.doctorchecks their requirement against the bundled version, so a core that bundles an unpublishedkiln_pluginversion can't be targeted by anyone else. - Publish
kiln_pluginbefore the core's docs are next published. The core's reference no longer contains the contract modules: everyKiln.Plugin/Kiln.Block/Kiln.FieldTypereference in it links to hexdocs.pm/kiln_plugin, which 404s until the first publish.
SemVer for the package mirrors the core's contract: a minor adds (a callback with a default, a DSL option), a major only alongside a core major.
Cutting a release
Confirm CI is green on
main. Theoverlay_driftjob is the one that matters most here: it builds the in-treeexampleoverlay against the core, so a green run is evidence the overlay contract still holds.It is not evidence the upgrade works. Run the Upgrade rehearsal workflow (Actions → Upgrade rehearsal → Run workflow; it also runs weekly) on
maintoo. For each of the last few releases it pins a scratch project at that tag, seeds it, and runs that release's ownmix kiln.updatetomain, tagged locally as the next-rc.0. Then it rebuilds with the project's unchanged overlay, migrates, runsmix kiln.blocks.backfilland reads every row back. It also checks that each old release prints the Upgrade notesupgrade_notes/3expects for its range. A release from an older major (0.12.1 → 1.x, picked with the workflow'stagsinput) is rehearsed with--allow-major, as a downstream would pass it once it has read the notes; the major-version guard itself is covered bykiln.update's own tests. To run one locally (it uses its ownkiln_cms_test_rehearse_*database and drops it afterwards):CANDIDATE_REF=origin/main scripts/upgrade_rehearsal/rehearse.sh v0.11.0If the release had candidates, the last one must also have passed its canary on kilncms.dev (step 5 of Cutting a release candidate). If
mainhas moved since that candidate in anything but the changelog, the version and the deploy templates, the change was never canaried: cut another candidate, or say in the release commit why it needs none.Write the changelog entry. Move
## [Unreleased]items into a new## [X.Y.Z]section, and renamedocs/changelog/unreleased.mdtodocs/changelog/vX.Y.Z.md— it already holds the long form of everything in that section.Add an
### Upgrade notessubsection if — and only if — moving to this release needs more than a rebuild:- a manual backfill or reindex step, and whether it can run after deploy;
- a migration that rewrites or drops data (say so, and say it's not reversible by rolling the pin back);
- anything else to do against a deployed instance.
Add a
### Breakingsubsection for anything that changes an observable contract or forces a change on the operator:- a new required env var or config key, or one whose default flipped;
- a response shape, status code or route that changed;
- anything a subproject must change to keep compiling.
Write both as imperative steps. Those two sections are the only thing
mix kiln.updateprints before it moves anyone's pin (#1325), so they are the last chance to warn an operator — and everything else you want to say belongs in the entry itself, which the next step files away.Condense and check.
mix kiln.changelog --condense mix kiln.changelog --verify HEAD--condenserewrites each entry inCHANGELOG.mddown to its own opening line plus a link to the long form underdocs/changelog/.--verifythen proves, paragraph by paragraph, that nothing was dropped on the way — run it before you commit, not after.Bump the version in
mix.exs. This is what a running instance reports (Kiln.Version) and what the update check compares against, so it must match the tag.Then regenerate the committed API specs, whose OpenAPI document states the version as
info.version:mix kiln.api.specsCI's documentation job fails the release PR if you forget.
Then bump the image tag in the one-click deploy templates, which pin an exact version so a stranger's first deploy is reproducible (#1529). Nothing gates these — a missed one silently keeps handing new users the previous release:
render.yamlfly.toml.do/app.yaml- the Railway recipe in
docs/deploy-platforms.md
and the
Stable (vX.Y.Z)line (and the exact-pin example below it) inREADME.md. The:latestreferences inREADME.mdanddocs/getting-started.mdfloat on purpose — leave them.git grep -nE '(^|[^0-9.])v?0\.9\.0([^0-9.]|$)' # the version you just leftCommit, tag, push.
git commit -am "chore: release vX.Y.Z" git tag vX.Y.Z git push origin main --tagsWatch the release image publish. Pushing the tag starts
.github/workflows/release.yml, which builds the release image and pushes it toghcr.io/the-verscienta/kiln_cmsasX.Y.Zandlatest, and from 1.0.0 also as the floating majorX(1). It is stamped with the commit and build date. A release candidate gets only its exact tag (see below), and so does a patch on an older line (see Patch releases and backports). Nothing to run by hand; it authenticates asGITHUB_TOKEN.The version bump in
mix.exsinvalidates the dep layer, but the build readsmain's cache for the rest: v0.9.0's took about nine minutes. Check the run before announcing the release.Then check that a stranger can pull it. A private image fails
docker pullwith a 403 for everyone but the maintainer, while the workflow reports success, because it pushed fine. This asks the registry as an anonymous client, with nothing to log out of first:tok=$(curl -s "https://ghcr.io/token?scope=repository:the-verscienta/kiln_cms:pull" \ | python3 -c 'import sys, json; print(json.load(sys.stdin).get("token", ""))') curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $tok" \ -H "Accept: application/vnd.oci.image.index.v1+json" \ https://ghcr.io/v2/the-verscienta/kiln_cms/manifests/X.Y.Z200means it is public. If it is401or403, open the package (repository → Packages →kiln_cms) → Package settings → Change visibility → Public.GitHub's documentation says a new package starts private and does not take its repository's visibility. The first publish here did not work that way: v0.9.0's image, pushed by this workflow from this public repository, answered
200straight after the build, with nobody having changed its settings. Run the check anyway rather than rely on either behaviour.Only
vX.Y.Ztags trigger it. A scratch or rescue tag publishes nothing, which is the same rulemix kiln.updateapplies to tags it cannot parse.Publish a GitHub release for the tag. This is not optional:
mix kiln.updatereads git tags, but the admin update page (Kiln.Updates) reads the releases API, because a running container has no checkout. A tag with no release leaves every deployed instance reporting "up to date" while a newer version exists. (The page asks kilncms.dev's release feed first and the releases API only as a fallback (#1877), but the feed is itself published from the GitHub release, so the release is still the step that makes a version visible.)Paste the changelog section in as the release body.
Publishing it also runs
.github/workflows/publish-releases.yml, which posts the release's notes (docs/changelog/vX.Y.Z.md) to kilncms.dev at/releases/<version>and relists/releases(#1870). Publish the release yourself, withghor the web UI: a release created by a workflow'sGITHUB_TOKENstarts no other workflow, so this one would not run. The job readsmain, so the tag must be on it. A pre-release is skipped. If it fails, fix the cause and re-run it from the Actions tab with the version as input.Verify from a project checkout:
cd <your kiln checkout> && git fetch --tags && mix kiln.update --check
Cutting a release candidate
A release candidate lets downstreams rehearse a release — the 1.0 upgrade in particular — before anyone's default update lands on it. Nothing floats onto a candidate by accident (#1541): each place that picks "the newest release" skips pre-releases, and each one has an explicit way in.
Tag it
vX.Y.Z-rc.N— the version it is a candidate for, then-rc.1,-rc.2, …. By semver1.0.0-rc.1sorts below1.0.0and above every earlier release, which is exactly why the tools below must be told about it. The-is what every one of them looks for, so do not spell a candidate any other way (v1.0.0rc1does not parse as a version at all).Leave the changelog under
## [Unreleased]. Do not condense, do not renamedocs/changelog/unreleased.md, do not add a## [X.Y.Z-rc.N]section — the final release does all of that, once.mix kiln.updatereads[Unreleased]as the candidate's own notes when the target is a pre-release, so its Breaking and Upgrade notes still print before a pin moves. Do bumpmix.exstoX.Y.Z-rc.N, so an instance on the candidate reports what it is running.The task that runs is the one in the checkout being moved, though, and only 0.12 and later read
[Unreleased]that way. A project on 0.11 or older moving to a 0.12 candidate is shown no notes for it: its task prints the earlier releases' notes, and then moves the pin anyway. The upgrade rehearsal (#1540) caught this for every release from 0.5.0 to 0.11.0. So for a 0.12 candidate, paste its### Upgrade notesand### Breakinginto the pre-release's notes (step 4) and say so where the candidate is announced. From 0.12 on, the notes print as described.Tag and push as in step 5, with the candidate's name.
release.ymlpublishes the image asX.Y.Z-rc.Nonly:lateststays on the previous final release, sodocker pull …:latestand the:latestreferences in the README are unaffected.Publish the GitHub release as a pre-release:
gh release create vX.Y.Z-rc.N --prerelease --title "vX.Y.Z-rc.N" --notes "…"--prereleaseis what keeps it off every deployed instance's admin update page:Kiln.Updatesasksreleases/latest, which GitHub defines as the newest release that is not a pre-release. A candidate published without the flag would become "latest"; the update check then refuses it ({:error, :prerelease}) rather than telling anyone to install it, but the page can no longer report on the final release either — fix it withgh release edit vX.Y.Z-rc.N --prerelease.Canary it on kilncms.dev (#1869). kilncms.dev is the one Kiln instance the project runs with real editors and real traffic, so it is where a candidate first meets a live database. Beta testers exercise the features. The canary catches what only an upgrade shows: an expand/contract mistake, a new boot warning, the cache under real load.
First check the migrations over the whole range, from the release kilncms.dev runs now:
mix kiln.migrations.check --base vA.B.C # the release kilncms.dev runsCI judged each pull request's migrations on their own. This judges them together, and it must pass, because it is what makes the rollback below safe. Then take a backup (
scripts/backup.sh), point the kilncms.dev app atghcr.io/the-verscienta/kiln_cms:X.Y.Z-rc.N(or, where the platform builds from source, at thevX.Y.Z-rc.Ntag), and redeploy. Migrations run on boot.Let it soak for about 48 hours, including a working day of editing and a docs publish (
publish-docs.ymlwrites every guide through the API). Then compare it with the previous release over the same hours of the week:- the boot log:
bin/migratefinished, and no boot warning is new (they go to stderr only, so read the container log); - Sentry, when
SENTRY_DSNis set, and the log's[error]lines; - headless API and editor save p95, from the panels in
observability.md(which needKILN_METRICS_ENABLED); - Oban failures.
What blocks the final:
- a migration that fails on boot, or has to be finished by hand;
- a new error class: an exception, Sentry issue or
[error]line the previous release did not produce, however rare; - a p95 regression on the headless API or editor save against the
previous release, and any p95 over its target in
performance.md.
Fix it on
main, cut-rc.N+1, and canary that one from the start. A blocker found on hour 47 restarts the 48.Rolling back is pointing the app at the previous release's image and redeploying. Do not run the down migrations. The range passed
mix kiln.migrations.check, so the previous release runs against the migrated schema, as the old nodes of a rolling deploy do (see Migrations: expand, migrate, contract). Two things do not go back with the image: rows a migration rewrote (the candidate's Upgrade notes name them), and Oban jobs queued for a worker only the candidate has, which fail on the previous release until their attempts run out.demo.kilncms.devdoes not take candidates. It resets to a golden snapshot (demo-mode.md) that would need recapturing for each one, so it moves to the final release once that is published.- the boot log:
How a downstream opts in. A plain mix kiln.update never targets a
candidate. To try one:
mix kiln.update --to vX.Y.Z-rc.N # exactly that candidate
mix kiln.update --pre # the newest release, candidates includedThe major-version guard still applies — 0.12.0 → 1.0.0-rc.1 needs
--allow-major like the final would. A pin left on a candidate stays there
under a plain update until the final release overtakes it (the task reports
the pin as ahead rather than downgrading it), and then moves to the final as
usual.
The client SDKs follow the same rule: a client-js-vX.Y.Z-rc.N tag publishes
to npm under the next dist-tag rather than latest, and Hex never resolves
a kiln_client pre-release unless a requirement names one.
Patch releases and backports
.github/SECURITY.md
sets the policy: the latest minor gets every fix, and the previous minor gets
security fixes for 90 days from the release date of the minor that replaced
it. The window applies within a major, from 1.0.0 on: 0.x gets nothing
after 1.0.0's release date, so there is no 0.12.x backport once 1.0.0 is
out. There are no long-lived maintenance branches. Each patch is cut from a
short-lived branch off the line's newest tag, and the branch is deleted
afterwards. v0.9.1 was the first patch cut this way.
When main will do. A patch carries fixes only. If everything on main
since the line's newest tag is a fix, cut the patch from main as in
Cutting a release and stop here. Otherwise, and always
for the previous minor, use a branch:
Fix it on
mainfirst, through a normal PR with its changelog entry. A backport is a copy of a fix that has already been reviewed, never the first place it lands.Branch from the line's newest tag. Name the branch after the version you are cutting:
git fetch --tags origin git checkout -b release/v1.0.3 v1.0.2 git cherry-pick -x <fix commit> # -m 1 if it is a merge commit-xrecords the source commit in the message. Expect conflicts inCHANGELOG.mdanddocs/changelog/unreleased.mdonly: take the tag's version of each, then add just this patch's entries and long-form blocks.Get CI on the exact tree.
ci.ymlruns on pushes tomainand on pull requests to any base, so push a base branch at the tag and open the release branch as a PR against it. Title it "don't merge": it exists only for the CI run.git push origin v1.0.2:refs/heads/release/v1.0.x git push -u origin release/v1.0.3 gh pr create --base release/v1.0.x --title "v1.0.3 (CI only, don't merge)" --body "…"Cut it on the branch: steps 2 to 4 of Cutting a release, with a
## [1.0.3]section,--condense, and themix.exsand API spec bump. For a patch on the previous minor, leave the deploy templates and the README's version line alone: they follow the latest line, andmainowns them.Tag the branch and push only the tag.
git tag v1.0.3 git push origin v1.0.3release.ymlpublishes the image as1.0.3.latestand the major tag1move only if this is the highest final release overall (forlatest) or on its major (for1), so1.0.3pushed after1.1.0moves neither. The run log's "Decide which floating tags move" step says why.scripts/release/floating_tags.shmakes the call, andtest/scripts/release_floating_tags_test.exscovers it.Publish the GitHub release without making it "latest" if a newer line exists:
gh release create v1.0.3 --latest=false --title "v1.0.3" --notes "…"Say
--latest=falseexplicitly. The releases API's default for a new release is to make it latest, andgh's own default is an automatic choice that weighs creation date as well as version. Neither is safe to rely on for an older line. Every deployed instance's update page (Kiln.Updates) asksreleases/latest, so a 1.1 instance would compare itself against1.0.3, read itself as ahead, and report "up to date" while a1.1.xpatch existed. If that happens, rungh release edit v1.1.1 --lateston the highest release.Its notes reach kilncms.dev only once they are on
main: after step 7, run "Publish release notes" from the Actions tab withv1.0.3as the version.Merge the branch back into
mainwith a merge commit. Openrelease/v1.0.3 → mainand merge it with Create a merge commit, never squash or rebase.mix kiln.updaterefuses to move a pin to a release that is missing commits the pin already has (it reads them as local patches to the core). The cherry-picked commit has its own SHA, so until the patch's tag is an ancestor of the newer release, a site onv1.0.3is refused an update to1.1.x. On the merge-back, move the patch's changelog entries into a## [1.0.3]section below the newer releases.If the latest minor needs the same fix as a patch and
mainis not releasable, cutrelease/v1.1.1fromv1.1.0in the same way, and mergerelease/v1.0.3into it before you tagv1.1.1. Then mergerelease/v1.1.1back intomain.v1.0.3is then an ancestor ofv1.1.1, and a site onv1.0.3can move straight to it.Delete the branches,
release/v1.0.3and the CI baserelease/v1.0.x. The tag keeps the history reachable.
A security fix's notice. Fill in the private advisory's affected and
patched versions per line (>= 1.0.0, < 1.0.3 and 1.0.3; >= 1.1.0, < 1.1.1 and 1.1.1), and publish it before you publish the GitHub release.
Publishing a release runs .github/workflows/publish-security.yml, which
turns every published advisory into a notice at
kilncms.dev/security-notices/<ghsa-id>, lists it on /security, and puts
it in the notices feed (#1873). An advisory published after the release
reaches the site on the workflow's daily run, or straight away from "Publish
security page" in the Actions tab. The workflow never unpublishes a notice:
withdraw a withdrawn advisory's notice in the editor too.
What mix kiln.update does with a patch tag. A plain update targets the
highest final release, not the newest tag and not the newest on the pin's own
line. A project pinned at v1.0.2 after v1.1.1 and v1.0.3 exist is
offered v1.1.1. A minor move needs no flag, and the Breaking and Upgrade
notes print for every release in between. To take the backport and stay on
the line:
mix kiln.update --to v1.0.3A pin at v1.0.3 is refused a move to v1.1.0, if v1.1.0 was released
before the fix: it does not have the fix, and the task reports the patch's
commits as ones the target is missing. Move to the 1.1.x patch that has
the fix instead. --force skips the check, but only use it if the release
notes say 1.1 was never affected.
Updating a project to a release
From inside the project's pinned Kiln checkout — kiln/upstream, upstream/,
wherever that project puts it (see
projects/README.md
for the overlay layout). The task refuses to run outside a Kiln checkout, so
pointing it at the project repo itself is an error, not a wrong answer:
cd <your kiln checkout>
mix kiln.update --check # what's new, new migrations, upgrade notes
mix kiln.update # move the pin to the newest releaseThen commit the moved pin, rebuild the image, and redeploy. Migrations run on
boot (see the CMD in the Dockerfile), so deploying applies
them — take a backup first (scripts/backup.sh) if the report listed any.
Useful flags: --to vX.Y.Z to land on a specific release rather than the
newest, --pre to let a release candidate count as the newest, --ref main
to deliberately track bleeding edge, --allow-major after
reading the upgrade notes, and --check --exit-code to fail a CI job when a
project has drifted behind upstream.
Migrations: expand, migrate, contract
A rolling deploy runs two releases at once. The first new node migrates the database on boot, and the old release's nodes keep serving against the new schema until the load balancer has moved every request over. A migration is only safe if the release before it keeps working against it. From 1.0 every schema change follows this policy (#1716):
- Expand. Add tables and columns. A new column is nullable or has a default, so the old release's inserts, which do not know it exists, still succeed. Keep the old table or column in place.
- Migrate. Backfill in an Oban job or a release task, not in the migration. From this release on, the code writes both shapes (or the new one only) and reads the new one.
- Contract. Drop, rename or tighten only in a later release, once no release that could still be running reads the old shape. That is at least the release after the one that stopped reading it.
The cases that come up:
- Renaming a column or table takes two releases: add the new one and
backfill it (release N, which stops reading the old one), then drop the old
one (release N+1). A plain
renameis a drop as far as the old release is concerned. - Changing a column's type is a rename in disguise. Add a column of the new type, backfill it, and drop the old one a release later.
NOT NULLneeds either a default, which Postgres 11+ applies without rewriting the table as long as the default is not volatile, or a backfill release first. Tighten a column tonull: falsein the release after the one whose code always sets it. The old release may still insertNULLuntil then.- Dropping a column waits until the release that stopped reading it has
shipped. Ash selects every attribute, so the old release breaks the moment
the column is gone (
20260919191545_drop_webhook_plaintext_secretis the example that prompted this policy). Remove the attribute from the resource in release N, then runmix ash.codegenfor the drop in release N+1. - An index on a large table is built concurrently. A plain
CREATE INDEXtakes a lock that blocks writes to the table for the whole build. In a resource, usecustom_indexes do index [...], concurrently: true end. For the unique index behind an identity, runmix ash.codegen <name> --concurrent-indexes. Either way codegen puts the index in its own migration with@disable_ddl_transaction trueand@disable_migration_lock true, because Postgres refusesCREATE INDEX CONCURRENTLYinside a transaction. An index on a table the same migration creates needs none of this.
The check
mix kiln.migrations.check runs on every pull request (the build job in
ci.yml). It reads each migration the PR adds, under
priv/repo/migrations and every overlay's projects/*/priv/repo/migrations,
and only its forward direction (up/change). It fails on:
| Flagged | Why |
|---|---|
drop table, rename table (or a column) | the old release still reads the table or column |
remove :col | the old release still selects the column |
modify that changes the type | the old release reads the old type. The previous type comes from from: or the migration history. If it is unknown, the change is flagged too. |
modify ... null: false, or add ... null: false without default: on an existing table | the old release may still write NULL, or omit the column |
execute SQL containing DROP TABLE/COLUMN/VIEW/SCHEMA/TYPE, RENAME, ALTER COLUMN ... TYPE or SET NOT NULL | a heuristic that flags the statement for review. Dropping and re-creating a function or trigger is not flagged. |
a non-concurrent index on a table listed in Mix.Tasks.Kiln.Migrations.Check.large_tables/0 | the build blocks writes |
concurrently: true without @disable_ddl_transaction true | the migration would fail at boot |
The history is exempt. Only the diff against the base is judged, so
mix kiln.migrations.check --all over today's history reports what the
policy would have caught. On 0.12.0 that was 106 findings: 65 non-concurrent
indexes on large tables, 34 SET NOT NULLs (most of them the #336
multi-tenancy org_id backfill-then-tighten migrations), 4 dropped tables,
2 NOT NULL columns added without a default, and the dropped webhook
secret. Run it locally before pushing. It is stdlib-only, so it runs in a
checkout with no deps/:
mix kiln.migrations.check # vs origin/main
mix kiln.migrations.check --base v0.12.0 # vs another refWhen the contract step is the point, say so in the migration, on the line above the statement (or at the end of its first line):
# kiln:contract-ok since v0.12.0 — 0.12.0 stopped reading webhook_endpoints.secret
alter table(:webhook_endpoints) do
remove :secret
endThe version names the shipped release that stopped reading the old
shape, so it may not be newer than the version in mix.exs. The reason is
required, and -- works in place of the em dash. The marker covers the one
statement it sits on. An alter table block counts as one statement, so a
marker above it covers every op inside. For an index build that is safe for
another reason, such as a table that is small everywhere, use
# kiln:lock-ok — <reason>. A malformed marker is itself a failure. So is
a marker that names an unshipped release, or one that excuses nothing.
What zero-downtime does and does not cover
The policy makes the schema safe for two releases at once. Kiln's boot
and probes already handle the rest of an ordinary rolling deploy
(deploy.md, "What happens at boot" and "Health endpoints"):
- Migrate on boot, serialised. Every node runs
bin/migratebefore it serves.Ecto.Migratorholds a lock onschema_migrations, so the first new node migrates and the rest wait. The lock does not block the old release's queries. - Readiness.
/liveonly answers oncebin/migratehas finished and the endpoint is up, and/upalso requires the database. Pointing the load balancer's readiness check at/upkeeps a new node out of rotation until it can serve. - Graceful drain on
SIGTERM. The release stops its listener and gives in-flight HTTP requests up to 15 s to finish (Bandit/Thousand Island'sshutdown_timeoutdefault). Running Oban jobs get Oban's 15 sshutdown_grace_period.
It does not cover:
- LiveView and WebSocket sessions. They are disconnected when their node
stops, and the client reconnects to a new node, which remounts the view.
LiveView's form recovery re-sends a form that has
phx-change, and an open GraphQL subscription must resubscribe. - A job that outlives the grace period, until it is rescued. Oban kills
it and leaves the row in
executing.Oban.Lifelinemoves it back toavailable(ordiscardedonce its attempts are spent) afterKILN_OBAN_RESCUE_AFTER_MINUTES, 180 by default, so the job runs again from the start, not where it stopped. Which jobs are safe to re-run is in Jobs interrupted by a deploy. - Readiness does not flip before shutdown.
/upkeeps answering 200 until the listener closes, so the load balancer should stop routing on its own deregistration (Kubernetes removes the pod from the Service endpoints on termination, but in parallel with theSIGTERM). A shortpreStopsleep closes that race. - Lock waits. Kiln sets no
lock_timeout, so a migration'sALTER TABLEwaits behind a long-running query, and every query on that table queues behind the waiting migration. Deploy outside a heavy export or report. - A release that skips one. Expand/contract assumes each release is deployed in turn. Jumping from N-1 to N+1 runs N's expand and N+1's contract in one boot while N-1 nodes are still serving. Deploy each minor in sequence, or accept a short outage and stop the old nodes first.
- Single-instance deployments. One container on one host (the Compose reference deployment, or a platform with a disk attached) has no second node to serve while the new one boots, so it drops traffic for the length of the boot however compatible the migration is.
Build stamping
The release workflow already does this for the published core image — this is
the recipe for building one yourself: locally, or for an overlay (which is the
only way to get a PROJECT= image, since the published one is deliberately
project-agnostic).
Build with the commit and date recorded, so a deployed instance can say exactly
what it is on /editor/system:
docker build \
--build-arg GIT_SHA="$(git rev-parse HEAD)" \
--build-arg BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
-t kiln:vX.Y.Z .Without them the image still boots and reports its version; it just can't name
the commit, which is the first thing you want when a deploy misbehaves. When
GIT_SHA is unset the Dockerfile falls back to a SOURCE_COMMIT build arg,
which Coolify passes when its Include Source Commit in Build setting is on.