CWA 2 is a prerelease (2.0.0-alpha), heading for a beta. It runs production sites; configuration can still change between releases, always with upgrade notes. View versions and support
Deployment & Operations

CI/CD

The template ships full CI/CD for both GitLab CI and GitHub Actions — Docker Buildx builds, tests, per-branch review apps, and staged Kubernetes deployments via Helm.

The template ships complete CI/CD for both GitLab CI and GitHub Actions. Every step delegates to shell functions in bin/devops/ — readable, modifiable scripts you own.

Choose your CI/CD provider when you run pnpm create cwa — the CLI wires up the correct pipeline files automatically. You can switch later by copying the relevant files from the template.

GitHub Actions

Three workflows drive your app, in .github/workflows/:

WorkflowTriggerWhat it does
ci.ymlEvery push, except one that changes only Markdown files at the repository rootBuild + test; deploy a review environment for non-main branches, or staging on main
production.ymlManualDeploy canary or full production, show production's pending migrations, or back up production's database (see Migrations and Backups)
cleanup.ymlPR closedTear down the review environment

The template repo also has publish-create-cwa.yml, which publishes the scaffolding CLI. create-cwa leaves it out of new projects, and a project that has it can delete it.

Images push to GHCR (ghcr.io/${{ github.repository }}). The workflows call the same bin/devops/ shell functions as GitLab CI. Required secrets and variables are declared in the workflow files themselves, and listed on CI/CD Variables.

On its own, GitHub Actions only gives a job the variables its env: block names. The template closes that gap: every job that checks out the repository runs bin/devops/github-export-vars.sh straight after the checkout, which exports every GitHub variable (repository, organisation and environment) to the job's later steps, as GitLab does. To set a variable from CI/CD Variables, add it under Settings → Secrets and variables → Actions → Variables. You don't need to change a workflow.

Secrets still need an env: mapping, except NUXT_* and CWA_API_* ones. Secrets aren't in vars, so the script can't export them, and mapping them explicitly keeps them masked in logs. A name the job already sets in env: is left alone, so the workflow's mapping wins over a variable of the same name. The template's deploy jobs map KUBE_CONTEXT, DATABASE_URL, MAILER_DSN, GCLOUD_JSON, APP_SECRET, the JWT and Mercure keys and ADMIN_PASSWORD from secrets, so set those as secrets: a variable with one of those names is ignored. KUBECONFIG and KUBE_NAMESPACE_PRODUCTION are secrets too. Secrets named NUXT_* or CWA_API_* need no mapping: the deploy jobs export them for you (see Site settings).
The workflows run on every push by default. Set the GitHub Actions variableCWA_CI_DISABLED=true (under Settings → Secrets and variables → Actions → Variables) to suppress them. This is used on the template repo itself, which deploys via GitLab CI and only mirrors to GitHub.

GitLab CI

The .gitlab-ci.yml has seven stages: build, test, review, staging, canary, production and cleanup. There is no GitLab Auto DevOps magic here.

GitLab runs branch pipelines only. The workflow creates no merge request pipelines, because every job is limited to branches. A merge request shows its branch pipeline's result, so Pipelines must succeed works as expected.

A push that changes only Markdown files at the repository root runs no pipeline, such as a push that only edits README.md or CHANGELOG.md. A Markdown file inside a directory still runs one. GitLab does this in the workflow:rules of .gitlab-ci.yml, and GitHub with paths-ignore: '*.md' in ci.yml. On GitLab a branch's first push always runs, and schedules and pipelines you start by hand aren't affected. If the previous pipeline failed, a Markdown-only push doesn't run it again: retry it by hand.

On GitLab an empty commit (git commit --allow-empty) runs no pipeline either, because the rule's changes matches no file. To run CI again without a change, use Build › Pipelines › Run pipeline or the pipelines API (POST /projects/:id/pipeline).

Pipeline Overview

On every branch: build then test then spin up a review app on Kubernetes. With browser tests turned on, they run beside the PHPUnit tests, and every deploy waits for both. When a branch merges to main: build and test run again, then the pipeline auto-deploys to staging, from where you manually promote through canary to production.

Stages

Build

Two parallel jobs — build api and build app — each run docker buildx and push to the GitLab Container Registry. Layer caching is handled with --cache-to / --cache-from against a dedicated cache image tag, so incremental builds are fast. The cache is written with mode=max, so it keeps every stage, including the app's pnpm fetch and base setup, not only the final image's layers. A branch's first build also reads the default branch's cache, so it doesn't start cold.

$CI_REGISTRY_IMAGE/php:$CI_COMMIT_REF_SLUG    ← FrankenPHP image (--target frankenphp_prod)
$CI_REGISTRY_IMAGE/app:$CI_COMMIT_REF_SLUG    ← Nuxt image       (--target prod)

The PHP image build uses api/Dockerfile; the Nuxt image uses app/Dockerfile. Build logic lives in build_api() and build_app() in bin/devops/k8s.sh. On GitLab the build jobs install only git, when the image lacks it (install_build_dependencies), never Helm or kubectl, which they don't use.

An unchanged image isn't built again

Each build labels its image (rocks.cwa.ci.inputs) with a hash of what the build depends on:

  • the git tree of its build context, api/ or app/;
  • the build options, such as the php base and the build target;
  • the ISO week.

Before it builds, the job reads that label on the branch's image, then on the default branch's. If one matches, the job skips the build and takes seconds. It reuses that image with the same digest, tags a default-branch image for the branch, and records the digest as a build would. So a push that doesn't touch api/ builds no new php image, and its deploy doesn't restart the php pods (see Kubernetes & Helm › When a deploy restarts pods).

  • The week is in the hash, so the first push of each week builds both images again. Base images that move, such as Node, or FrankenPHP in a from-source php build, reach your images at least weekly.
  • The first build always builds, because no image has the label yet.
  • A failed build pushes nothing, so the next push compares with the last good image and builds.
  • CWA_CI_BUILD_FORCE="true" always builds (see CI/CD Variables › Pipeline flags).

The job log says which happened: ♻️ <image>:<tag> was built from these inputs (<hash>): not building PHP again., or Building PHP: no image from these inputs (<hash>). (APP for the Nuxt image).

The php image builds on a base that holds the patched FrankenPHP, the Debian packages and the PHP extensions. build_api picks it: CWA_CI_PHP_IMAGE_FROM_SOURCE="true" builds it from source, CWA_CI_PHP_BASE_IMAGE names a published cwa-php-base tag or digest, and otherwise the Dockerfile's default applies: your template release's own cwa-php-base image, which the build pulls. See Docker › Where the base comes from and CI/CD Variables › php image.

Each build pushes only its branch tag, and records the digest it pushed: CWA_CI_PHP_IMAGE_DIGEST or CWA_CI_APP_IMAGE_DIGEST, in the dotenv artifact image_digest.env on GitLab and as a job output on GitHub. The deploys run exactly that build. See Which build a deploy runs.

Composer downloads about 180 packages from github.com per install. Anonymously, GitHub allows 60 requests an hour per IP, so on a shared runner an install can fail part way through with a 429. Set the optional GITHUB_TOKEN variable to authenticate them. build_api passes it to the image build as the composer_auth build secret, never a build arg, so the token stays out of the image history. The build log's github rate limit for this build: line prints 60 when the token didn't reach the build and 5000 when it did.

Test

Two parallel jobs against the image built in the previous stage:

JobWhat runs
unit testsvendor/bin/phpunit tests/Unit — fast, no database. It also runs the template's Souin and Caddy checks: bin/test/souin-purge-isolation.sh, bin/test/souin-flush-storage.sh (which checks that otter, badger and nuts still cache after a full flush), bin/test/souin-coalesced-5xx.sh (which checks that requests Souin coalesces onto an upstream 5xx all get that 5xx, not an empty 200), bin/test/souin-php-flush.sh (which checks that a PHP flush() behind the cache logs no not a flusher warning), bin/test/caddy-validate.sh (which checks that the Caddyfile loads for both Helm's and compose's SERVER_NAME) bin/test/cache-encoding.sh (which runs the real Caddyfile and checks that pages are cached compressed, one entry for each normalised Accept-Encoding), bin/test/souin-backend-timeout.sh (which checks that a 12-second render is served and cached once, not turned into a 504, and that the store is otter) and bin/test/storage-prefix-parity.sh (which checks that the deploy and php tidy GCLOUD_PATH_PREFIX the same way) and bin/test/site-lib.sh (which runs the sitemap warm from bin/lib/site.sh under plain sh and checks that k8s.sh still provides it). It also runs bin/test/php-base-version.sh, which checks that api/Dockerfile sets CWA_PHP_BASE before its first FROM, that the cwa_php_base stage holds none of the project's files, and how build_api picks the base. See Docker › The HTTP cache in Caddy
functional testsvendor/bin/phpunit tests/Functional with a live PostgreSQL service. The tests use API Platform's ApiTestCase, and the schema is rebuilt from the entity mapping (Doctrine SchemaTool, not migrations) before each test. For a large suite, see Faster Functional Tests

Both export JUnit XML for GitLab's test report UI. On GitHub the jobs are unit-tests and functional-tests.

A third job, browser tests, is opt-in. It runs this branch's production images in a throwaway stack and drives the site in Chromium with Playwright. Set CWA_CI_E2E_ENABLED="true" to turn it on; every deploy then waits for it. See Browser Tests.

Functional tests never send real email. Every CI variable reaches the test job, and a real environment variable beats api/.env.test, so a project's live MAILER_DSN would deliver any email a test sends, such as a contact form or a password reset. run_test_functional unsets MAILER_DSN and MAILER_EMAIL, and api/.env.test sets MAILER_DSN=null://null. Mail is still built, and Symfony's mailer assertions still see it, but nothing is delivered. Keep both: without the .env.test line, tests fall back to api/.env's smtp-relay host, which CI doesn't have, so a test that sends email errors. The job also generates a test JWT keypair (lexik:jwt:generate-keypair --skip-if-exists), so tests that sign in work in CI.

The CI JWT keys never reach the tests. run_test_functional also unsets JWT_SECRET_KEY, JWT_PUBLIC_KEY and JWT_PASSPHRASE. In CI these hold the keys' contents for the deploy, while the tests read api/.env's file paths and the keypair the job generates. With the CI values in place, the functional tests failed, and the job log printed the encrypted key.

Functional tests extend App\Tests\Functional\FunctionalTestCase. Its setUp() refuses to run unless the database name contains test, because it drops the schema. Entities a test persists directly need createdAt and modifiedAt set by hand, since the bundle sets them only on its own write path (see RoutesTest).

Review Apps

A branch (other than main) gets a review app, with its own Helm release, only once someone creates its Kubernetes namespace. The pipeline never creates namespaces. The URL follows the pattern https://$CI_COMMIT_REF_SLUG-review.$KUBE_INGRESS_BASE_DOMAIN.

Namespaces must be created by hand, with role bindings for the CI's identity. Letting one project create namespaces and role bindings would give it rights across the whole cluster, so the pipeline's permissions stop at the namespace. See the comment in ensure_namespace().

The review job checks for the namespace first (skip_review_without_namespace):

NamespaceGitLabGitHub Actions
Exists, and the CI can read itDeploysDeploys
Not found, or the CI is forbidden to read itThe job exits with code 3, which the job allows, so the pipeline is orange ("passed with warnings")A No review environment warning annotation. The deploy, fixtures and cache warm steps are skipped
Any other kubectl error, such as an unreachable cluster or a bad contextRedRed

A review app needs its own database. On GitLab, a review deploy whose external DATABASE_URL is production's database is refused, because it would migrate it, so scope production's DATABASE_URL and CWA_CI_POSTGRES_ENABLED="false" to the production environment, or leave review apps on the in-cluster PostgreSQL. See Migrations and Backups › A track on production's database.

The jobs that follow a review (the review fixture load, image variants and warm, performance audit and stop jobs) start with skip_unless_review_deployed. When the review job didn't deploy, they exit with code 3 and say why, instead of running against an environment that doesn't exist.

An orange pipeline on a branch just means that branch has no review app. A deploy that really fails is still red. Staging, canary and production don't make this check: a missing namespace there fails the job.

Creating a namespace for a branch

  1. Find the name. The job log prints it: No namespace '<name>', so this branch has no review environment to deploy to. It is KUBE_NAMESPACE, which defaults to <project name>-<environment slug>. On GitHub that is <repository name>-review-<branch slug>. On GitLab the slug is GitLab's CI_ENVIRONMENT_SLUG for review/<branch>, which GitLab shortens and may give a random suffix, so copy the name from the log.
  2. Create the namespace, and bind the identity your CI deploys with to a role in it. For example, with the built-in edit role:
    kubectl create namespace <name>
    kubectl create rolebinding ci-deploy --namespace <name> \
      --clusterrole=edit --serviceaccount=<ci-namespace>:<ci-service-account>
    

    Which identity and role you need depends on your cluster and on how the CI connects (the GitLab agent, or the KUBECONFIG secret on GitHub). Give it the same access as your other environments.
  3. Run the review job again.

A namespace that exists without the role bindings still shows orange, because the CI is forbidden to read it.

A review app is stopped by GitLab's stop review job, which GitLab runs when the branch is deleted (for example on a merge with Delete source branch) and which you can also start from the environment list. On GitHub, the Cleanup Review workflow stops it when the pull request closes. Stopping one uninstalls its Helm release and deletes its database volume, but leaves the namespace, so delete the namespace by hand once the branch is gone.

GitLab's stop review doesn't check out its branch (GIT_STRATEGY: none), because the branch may already be deleted. It fetches bin/devops and bin/lib from the default branch instead, with a sparse checkout, so a stop runs the default branch's k8s.sh and project.sh, not the branch's own. k8s.sh sources bin/lib/site.sh, so a project's own stop review job needs bin/lib in its sparse checkout too, or the job fails when it loads k8s.sh.

Review app storage

A review app stores its uploads under its own path of the bucket, <GCLOUD_PATH_PREFIX>review/<slug>/ (Storage path prefix). Stopping it deletes them. Before helm uninstall, delete review runs the bundle's delete-stored-files in the review app's API pod. This happens in GitLab's stop review job and in GitHub's Cleanup Review workflow, which runs when a pull request closes:

php bin/console silverback:api-components:delete-stored-files gcloud --all \
  --only-if-value="$GCLOUD_PATH_PREFIX" --matching='~(^|/)review/[^/]+/$~'

--all empties the filesystem, which is only the pod's prefix. The guard checks the prefix the pod really has, so the delete runs only when it ends in review/<slug>/. On a review app without that prefix, --all would empty the bucket, so the command refuses and deletes nothing.

A failure doesn't stop the job. When the pod can't be found, the guard refuses or the delete fails, the job prints a warning that names the app's real path and the lifecycle rule's prefix, and uninstalls the release anyway. With GCLOUD_PATH_PREFIX=site-a/:

WARNING: could not delete <release>'s stored files under site-a/review/feat-x/; a bucket lifecycle rule on site-a/review/ is the backstop.

Add the lifecycle rule for that prefix so those files are removed later (see CI/CD Variables › Review app paths).

Staging

Runs automatically on every merge to main, while CWA_CI_STAGING_ENABLED is "true". GitLab defaults it to "true"; on GitHub, ci.yml has no default, so set the variable to turn staging on. Calls deploy staging which runs helm upgrade --install for a separate -staging release. Staging runs under the production environment, so it deploys into production's namespace and gets production's variables.

If that gives staging production's database (an external DATABASE_URL), staging never migrates it. See Migrations and Backups › A track on production's database.

Canary

A manual job that runs deploy canary. Useful for routing a portion of production traffic to the new image before full rollout. Enable/disable via $CWA_CI_CANARY_ENABLED.

Like production, it waits for the builds and tests (see Production below).

Production

A manual job requiring explicit trigger from GitLab. Calls deploy (stable track), then delete canary and delete staging to clean up the intermediate environments. To undo a release, see Kubernetes & Helm › Rollback.

On GitLab, the staging and production jobs retry once only when CI itself failed: a runner or scheduler failure, a GitLab API failure, or the job's timeout. A deploy that failed is never run again by itself, so a failed maintenance window isn't backed up and migrated a second time.

Production and canary can't start until build api, build app, unit tests and functional tests have passed, and browser tests and staging too when they are enabled. Each of these needs is optional, so CWA_CI_BUILD_DISABLED and CWA_CI_TEST_DISABLED still work.

Beside it, the migrations plan production job shows production's pending migrations and their SQL, read-only, so you can read them before you start the deploy. A deploy can also back the database up first, or close the site for a breaking migration. See Migrations and Backups. The production deploy itself always plans again and runs pending migrations in a Job before the rollout, so a failing migration stops it while the old pods keep serving (see Migrations and Backups › Migrating in a Job before the rollout).

Which build a deploy runs

A deploy never runs the moving branch tag. It pins the image by digest, so every pod started later runs the build that was deployed.

  • Staging, canary and production deploy <track tag>@<digest>, such as production@sha256:…, with pullPolicy: IfNotPresent. Once the migrations and both rollouts have succeeded, the deploy copies the pipeline's own build to the track's tag, :staging, :canary or :production, through the registry API. Nothing is pulled or pushed again.
  • Review apps stay on their branch tag, pinned the same way: <branch slug>@<digest>.
  • With no build digest, the deploy pins the digest the branch tag has when the deploy runs. That's the case when CWA_CI_BUILD_DISABLED skipped the build, and always in GitHub's production.yml, which builds nothing, so it deploys whatever :main holds at that moment. If the registry can't be reached, the deploy fails before Helm.

So a restart, an eviction, a scale-up or the nightly orphan scan (Kubernetes & Helm › Daily Orphan Scan), which runs migrations when it starts, never runs a build that wasn't deployed against the live database. The API and Nuxt pods, the orphan scan and the image variants Jobs (Kubernetes & Helm › Image Variants Jobs) all run the deployed build. A new build changes the pod spec, so it always rolls out. A reused image keeps its digest, so its pods don't restart (see Kubernetes & Helm › When a deploy restarts pods).

The registry keeps a fixed set of tags: main, staging, canary, production, and one for each review branch. There are no tags per commit. A track's tag is created by its first deploy, and only a deploy that succeeded moves it. It also keeps the deployed build tagged, so a registry that cleans up untagged images doesn't remove it once :main moves on.

The deploy waits for the Nuxt rollout, then the API rollout, each for at most CWA_CI_DEPLOY_ROLLOUT_TIMEOUT seconds (900). A rollout that doesn't finish in time, such as a pod whose migration fails on start and never becomes ready, fails the deploy with THE cwa ROLLOUT DID NOT FINISH: the track tag still points at the last good build. (cwa-pwa for Nuxt), while the old pods keep serving. So a failed migration or rollout leaves the track's tag on the last build that deployed well. If the deploy succeeds but the registry refuses the tag copy three times, the deploy only warns, and the next deploy copies it.

On GitHub, the deploy jobs need packages: write to add the track tag in GHCR. The template's review, staging and production jobs ask for it, with contents: read. If you copy only part of a workflow, keep their permissions blocks.

To go back to an earlier build, see Kubernetes › Rollback.


The Devops Scripts

All CI logic lives in two scripts sourced at the top of every job, bin/devops/setup.sh and bin/devops/k8s.sh. k8s.sh sources a third, bin/lib/site.sh, from the repository root. Copy all three together.

bin/devops/setup.sh

Sets shared environment variables:

  • CI_APPLICATION_REPOSITORY / CI_APPLICATION_TAG — image repo and SHA tag
  • PHP_REPOSITORY / APP_REPOSITORY — full image paths
  • PHP_REPOSITORY_CACHE / APP_REPOSITORY_CACHE — layer cache image paths
  • DOMAIN — the host of CI_ENVIRONMENT_URL, unless it's already set by bin/devops/project.sh or a CI variable. With neither, it's empty, and check_kube_domain fails the job. There's no fallback host
  • CWA_CI_DEPLOYMENT_BRANCH — defaults to main
  • COMPOSER_AUTH — built from GITHUB_TOKEN when it's set, and left unset otherwise
  • COMPOSER_MAX_PARALLEL_HTTP=6 — half composer's default of 12, which is a big enough burst to be throttled even when authenticated
  • KUBERNETES_VERSION / HELM_VERSION — the kubectl and Helm versions install_dependencies installs, 1.31.0 and 3.19.0 unless a CI variable sets them (see CI/CD Variables › Kubernetes and hostnames)

bin/devops/k8s.sh

Contains all the functions the pipeline calls:

FunctionWhat it does
install_dependenciesInstalls Helm, kubectl, curl, and other tools on the Alpine CI runner
install_build_dependenciesThe GitLab build jobs' version: installs git if the image doesn't have it. The build jobs never install Helm or kubectl
setup_docker_environmentHandles Docker-in-Docker host config for Kubernetes runners
build_api / build_appdocker buildx build --push with registry layer caching, unless reuse_image finds an image built from the same inputs. build_api passes COMPOSER_AUTH as the composer_auth build secret. Both then call save_image_digest, which records the digest for the deploys. See An unchanged image isn't built again
image_inputs / reuse_imageCalled by the builds. image_inputs hashes the build context's git tree, the build options and the ISO week. reuse_image looks for that hash in the label of the branch's image, then the default branch's, and reuses a match unless CWA_CI_BUILD_FORCE is "true"
run_test_phpunitRuns PHPUnit unit tests; outputs JUnit XML
run_test_functionalRuns phpunit tests/Functional against the job's own database (set up by setup_test_db_environment); outputs JUnit XML to api/build/logs/phpunit/functional.xml. Unsets MAILER_DSN, MAILER_EMAIL and the JWT_* keys first and generates a test JWT keypair (see Test). See Faster Functional Tests once the suite is slow
run_browser_testsLogs in to the registry and runs e2e/run.sh against this ref's php and app images. Only runs when CWA_CI_E2E_ENABLED is "true". See Browser Tests
helm_initUpdates and builds Helm chart dependencies
review_namespace_statePrints present, missing (the namespace is not found, or the CI is forbidden to read it) or error (any other kubectl failure) for $KUBE_NAMESPACE. The GitHub review job calls it directly
skip_review_without_namespaceRuns first in the GitLab review job. Exits with code 3 when the namespace is missing, which the job allows, so the pipeline shows orange. See Review Apps
skip_unless_review_deployedRuns first in the jobs that follow a review. Exits with code 3 when the review job uploaded no environment_url.txt, meaning it didn't deploy
ensure_namespaceVerifies the K8s namespace exists and fails the job if not. Staging, canary and production rely on it
create_docker_pull_secretCreates or updates the imagePullSecret for the GitLab registry with kubectl apply, so releases sharing a namespace don't race with "already exists".
check_kube_domainFails the job when there is no DOMAIN to deploy to, or when it starts or ends with a dot. A dot there means the variable in the environment url was empty, as in https://staging. with no KUBE_INGRESS_BASE_DOMAIN
deploy [track]Generates values.tmp.yaml from CI variables and runs helm upgrade --install, with a project's project_values applied last. It runs check_cdn_config, check_cache_config, check_admin_password (a warning only, see CI/CD Variables › The admin password), check_php_memory_budget, check_api_scaling and gcloud_path_prefix before anything changes in the cluster, and deploy_secrets, pin_images and database_guard before Helm. After Helm it waits for the rollouts (wait_for_rollouts), then runs promote_images
check_api_scaling [track]Runs inside deploy. Fails the deploy on more than one API pod, or a redis cache store, without scaled mode, and on settings scaled mode can't use. See Scaled Mode › What the deploy refuses. In scaled mode it also prints the database connections the API pods may open (see Scaled Mode › Database connections)
migrations_plan [track]Shows the track's pending migrations and their SQL, read-only, in a Job with the new php image. GitLab's migrations plan production job and GitHub's migrations-plan action run it. See Migrations and Backups › The migrations plan
database_guard [track]Runs inside deploy, before Helm. On a track that shares production's database, shared_database_guard runs instead (see Migrations and Backups › A track on production's database). Otherwise, on production it looks for a window an earlier deploy left on, then always plans and, with migrations pending, runs the window, the backup and the migrate Job (CWA_CI_MIGRATIONS_JOB_DISABLED="true" turns the migrate Job off). Other tracks do nothing unless a backup or migrate Job switch is on. See Migrations and Backups
finish_maintenance_window [track]Runs after purge_rendered_html in the production deploy. Ends a maintenance window, this deploy's or one an earlier deploy left on (found by its <fullname>-maintenance-window ConfigMap): waits for the API rollout, flushes the HTTP cache, warms /_api, turns maintenance off and deletes the ConfigMap. Nothing to do without a window. See Migrations and Backups › When the window fails
review_database_checkRuns inside deploy for a review app with an external DATABASE_URL. Refuses the deploy when it's production's database, read from RELEASE_PRODUCTION in KUBE_NAMESPACE_PRODUCTION, and only warns when it can't read it. See Migrations and Backups › A track on production's database.
pin_images [track]Runs inside deploy, before the database step. Sets the image tags Helm gets to <track tag>@<digest>, and moves no tag. See Which build a deploy runs
promote_images [track]Runs at the end of deploy, once both rollouts finished. Copies the build to the track's tag (:staging, :canary or :production); review apps have none. See Which build a deploy runs
database_backup_now [track]A backup on demand: the read-only plan, for the server's version, then the backup. GitLab's database backup production job and GitHub's database-backup action run it. See Migrations and Backups › A backup on demand
check_cache_configRuns inside deploy. Fails the deploy when CADDY_CACHE_EXTRA_CONFIG names a store or a timeout block, or CWA_API_CADDY_CACHE_BACKEND_TIMEOUT isn't a Go duration. See CI/CD Variables › HTTP cache
gcloud_path_prefix [track]Runs inside deploy. Tidies GCLOUD_PATH_PREFIX as php does, fails on a ., .. or review segment, and adds review/<slug>/ for a review app. See Storage path prefix
check_php_memory_budgetRuns inside deploy. Fails the deploy when the php memory budget doesn't fit the API's memory limit
deploy_secrets [release]Runs inside deploy. Takes APP_SECRET, MERCURE_JWT_SECRET and the JWT key pair from a CI variable, else from the release's live Secret, else generates them. See CI/CD Variables › Secrets
ensure_jwt_keysCalled by deploy_secrets. Generates a JWT key pair when there's none, or derives JWT_PUBLIC_KEY from a supplied key, and fails on a key without its passphrase
ensure_tls_certificateRuns inside deploy before Helm. When production's hostnames change, it issues the new certificate first. See Changing a live site's hostnames
purge_rendered_html [track]Waits for the Nuxt and php rollouts to finish, then clears every cached page. With Cloudflare configured it also purges everything at Cloudflare with app:edge-cache:purge-everything, and falls back to a full flush followed by warm_api_cache if that fails. In scaled mode it flushes the whole HTTP cache, followed by warm_api_cache. Runs after every deploy. In scaled mode the warm goes through the API Service (see Scaled Mode › Every deploy flushes and warms), then the deploy checks that the cache is shared in Redis, which only warns (see Scaled Mode › Checking the cache is shared)
warm_api_cacheRuns warm-http-cache in the API pod, gzip then br, each pass for at most the track's warm time (CWA_CI_API_WARM_HTTP_CACHE_MAX_TIME_<TRACK>, see CI/CD Variables › php (API); none on staging and review apps by default), after a full flush: a deploy's, new image variants' or a fixture load's. A failure only warns
generate_image_variants [track]Waits for the API rollout, then creates every missing image variant in one-off Kubernetes Jobs, and purges the HTTP cache if it made any. See Generating image variants after a deploy
image_variants_and_warm [track]GitLab's post-deploy job: generate_image_variants, then warm_cache, each only when its switch is on. A failed half doesn't stop the other, and either one fails the job. On GitHub they're two steps of the deploy job
performance_audit [base_url]Runs a Lighthouse audit of a few pages. base_url defaults to https://$DOMAIN. Only runs when you start it. See Performance Audit
load_fixtures [track]Loads fixtures into the php pod with --append, so existing content is kept, unless CWA_CI_FIXTURES_PURGE asks for a purge. Warns first when the API pod's ADMIN_PASSWORD is unset or admin, and loads anyway. Then flushes the whole HTTP cache and runs warm_api_cache. Only runs when CWA_CI_FIXTURES_ENABLED is "true". See Fixtures on a deploy
persist_environment_urlWrites https://$DOMAIN to environment_url.txt, the artifact the later jobs check, and SITE_ENVIRONMENT_URL=https://$DOMAIN to environment_url.env
delete [track]Runs helm uninstall for review/canary/staging cleanup. For review, it first deletes the app's stored files. See Review app storage

bin/lib/site.sh

The site functions that don't need Kubernetes. k8s.sh sources this file, so the pipeline calls them like its own functions. It's plain POSIX sh. If you source k8s.sh from somewhere other than the repository root, set CWA_SITE_LIB to this file's path.

FunctionWhat it does
warm_cache [base_url]Refills the page cache after purge_rendered_html, and reports each page's status and time to first byte. base_url defaults to https://$DOMAIN. See Warming the cache after a deploy
sitemap_pagesReads /sitemap.xml, following one level of sitemap index, and lists its pages for warm_cache and performance_audit

Project hooks

Keep setup.sh, k8s.sh and the chart exactly as the template ships them, so each release is a plain file copy. A project's own deploy logic goes in an optional bin/devops/project.sh. A project without one deploys with the template's own settings.

setup.sh sources project.sh in every job, on GitLab and GitHub, after its own defaults and before it sets DOMAIN and the image paths. k8s.sh isn't loaded yet, so project.sh can't call its functions. In it you can:

  • Set DOMAIN, the host the deploy serves, for example to choose which of several sites a job deploys. setup.sh keeps a DOMAIN that's already set, and otherwise takes the host of the job's environment url. A DOMAIN CI variable works the same way. check_kube_domain, persist_environment_url, the ingress hosts, NUXT_SITE_URL, the host-based defaults (see CI/CD Variables › Minimum for a new site), and the cache warm and performance audit jobs all follow it.
  • Export site settings and other variables, such as KUBE_INGRESS_ALIAS_DOMAINS, or NUXT_* and CWA_API_* values for the site settings passthrough, such as each site's CWA_API_WEBSITE_NAME. Use export: the passthrough only sees exported variables.
  • Reserve more names with SITE_ENV_RESERVED_API_EXTRA and SITE_ENV_RESERVED_PWA_EXTRA, space-separated. Add the names your own chart values set on a container, so a site setting of the same name fails the deploy instead of being ignored. See Reserved names. They can be CI variables too.
  • Define project_values. deploy calls it with the track (stable for production, staging, canary or review) and passes what it prints to helm upgrade as the last -f values file, after the template's own. Use it for the values of a template you add to helm/cwa/templates, or to change one of the chart's values. If it exits non-zero, the deploy stops before helm. The five --set values (APP_SECRET, the JWT keys and the Mercure JWT keys) still win over it, as --set always does in helm.
bin/devops/project.sh
# Sourced by bin/devops/setup.sh in every job. POSIX sh only.

# One repository, two sites: SITE is a CI variable scoped to each environment.
if [ "$SITE" = "shop" ]; then
  export DOMAIN="shop.example.com"
  export NUXT_PUBLIC_SITE_NAME="Example Shop"
  export CWA_API_WEBSITE_NAME="Example Shop"
fi

# Chart values, applied after the template's.
project_values() {
  if [ "$1" = "stable" ]; then replicas=2; else replicas=1; fi
  cat <<EOF
redirects:
  replicaCount: $replicas
EOF
}
Write the hooks in POSIX sh. GitLab's jobs run on alpine, where the shell is busybox ash: no arrays and no declare. A hook that only works in bash passes on GitHub and fails on GitLab.

environment_url.env is for a job whose url is worked out in the job, such as a multi-site deploy. The template's own jobs don't read it, and their environment urls stay as they are. If your project.sh sets DOMAIN, you can point your GitLab deploy job's environment url at it: add the file as a dotenv report and use the variable as the url:

  environment:
    url: $SITE_ENVIRONMENT_URL
  artifacts:
    paths: [ environment_url.txt ]
    reports:
      dotenv: environment_url.env

The cache warm and performance audit jobs pass no url, so warm_cache and performance_audit use https://$DOMAIN, and a DOMAIN set here reaches them. The GitHub Performance audit workflow sources setup.sh too.


After each deploy

The deploy's commands in the API (the flush and warm, the fixture load, the image variants purge and the maintenance commands) run in a ready API pod that isn't shutting down, never through kubectl exec deploy/…. Straight after a rollout, deploy/… can pick the old pod while it terminates, and a flush there would empty the wrong pod's cache.

Why every deploy clears the page cache

Cached pages link to the JavaScript and CSS files of the build that rendered them. A new front-end build gives those files new names and deletes the old ones. Without a purge, the cache keeps serving pages that point at files that no longer exist, so visitors see an unstyled page that does not work until the cached copy expires, which can take up to an hour.

purge_rendered_html prevents this. It waits until the new Nuxt pods have fully replaced the old ones, because an old pod could otherwise put an old page back into the cache. Then it runs php bin/console silverback:api-components:purge-rendered-html in the php pod. That command clears only the rendered pages, which share the cwa-html cache tag, and leaves cached API responses alone. It uses the pipeline's existing kubectl access, so it needs no extra credentials. If a site has no HTTP cache configured, the command does nothing and still succeeds.

If Cloudflare or another CDN caches your pages, this purge reaches it only when Souin's CADDY_CACHE_CDN_CONFIG is set to purge it. See Purges must reach Cloudflare.

When CADDY_CACHE_CDN_CONFIG contains provider cloudflare, purge_rendered_html also purges everything at Cloudflare. Cloudflare can also hold API responses, through the optional API rule (see Cloudflare › API responses (optional)), and the cwa-html tag doesn't cover them, so a release that changes API output would otherwise leave stale responses at the edge. Staging uses the same variable, so a staging deploy also empties the edge for production; the cache warm and normal traffic refill it.

The deploy purges the rendered pages in Souin, then runs php bin/console app:edge-cache:purge-everything in the API pod. That template command sends one purge everything for the zone in CADDY_CACHE_CDN_CONFIG straight to Cloudflare's API, and leaves Souin alone. It tries up to three times when Cloudflare answers 429 or 5xx, or doesn't answer, and exits 1 on any other failure or when the variable has no Cloudflare zone. The php pod keeps its /_api cache. A new pod warmed it when it started (see Kubernetes & Helm › Warming the API cache on start), and a pod the deploy didn't restart still runs the same build. With one API pod, that cache holds only this build's responses.

If either command fails, purge_rendered_html prints The edge-only purge failed: flushing the HTTP cache instead, which also purges everything at Cloudflare... and falls back to the full flush. A project that hasn't copied the command into api/src/Command/ takes this path on every deploy. The flush reaches Cloudflare as a purge of everything in the zone, and it empties the pod's warm /_api cache, so purge_rendered_html warms /_api again in the API pod, the same two passes as the pod's start: warm-http-cache for the gzip entries server-side rendering uses, then --accept-encoding=br for the entries browsers and Cloudflare use. Each pass stops after the track's warm time (CWA_CI_API_WARM_HTTP_CACHE_MAX_TIME_<TRACK>, 60 seconds on production), and a failed pass only prints a warning. On one site with 111 routes, 60 seconds didn't cover one pass.

In scaled mode the deploy always flushes, because the cache in Redis outlives the pods, then warms through the API Service (see Scaled Mode › Every deploy flushes and warms).

Generating image variants after a deploy

When a filter is added or changed, every image is missing a variant, and each would otherwise be generated during a visitor's request. generate_image_variants makes them first, after a deploy and before the cache warm. On GitLab it's the first half of the image variants and warm <track> job. On GitHub it's a Generate image variants step, after the fixtures step. Only production runs it by default; set CWA_CI_IMAGE_VARIANTS_PRODUCTION="false" to turn it off there, and the other tracks opt in (see CI/CD Variables › Image variants).

It never runs in the API pod that serves the site, where generating many large images once ran php out of memory and the site returned 503s. Instead:

  1. It waits for the API rollout, because the Jobs use the release's image.
  2. It creates a Job from the release's <release>-image-variants CronJob (see Kubernetes › Image Variants Jobs). The Job runs imagine:generate with -v and --max-memory, under its own memory limit (CWA_CI_IMAGE_VARIANTS_MEMORY, 1536Mi by default).
  3. When a Job stops at --max-memory with variants still missing (exit code 3), it deletes that Job and starts another, up to CWA_CI_IMAGE_VARIANTS_MAX_RUNS Jobs (50). The log prints each Job's output, leaving out the variants that were already stored.
  4. If any variant was generated, it runs purge-http-cache in the API pod, so cached API responses list the new variants, then warms /_api again, gzip then br, as after the Cloudflare flush (see Why every deploy clears the page cache). Each variant is stored with the field's imagineCacheControl (see Cache-Control on stored files).

A failure never fails the deploy: the GitLab job is allow_failure: true, and the step is continue-on-error: true on GitHub, where it also raises an ::error annotation. The log ends with a banner that says why:

Exit codeBannerWhat to do
3IMAGE VARIANTS NOT FINISHEDVariants are still missing after CWA_CI_IMAGE_VARIANTS_MAX_RUNS Jobs. The next deploy carries on, and a missing variant is still generated on its first read
137IMAGE VARIANTS FAILED: the Job ran out of memoryThe Job was OOM-killed. Raise CWA_CI_IMAGE_VARIANTS_MEMORY, or lower CWA_CI_IMAGE_VARIANTS_MAX_MEMORY. Visitors aren't affected
124IMAGE VARIANTS FAILED: Job … did not finishThe Job wasn't scheduled, or passed its deadline. Run kubectl describe job on it
1 or 2IMAGE VARIANTS FAILED: imagine:generate exited …1: a variant failed, and the -v output above lists it. 2: an unknown --filter or --class

A failed Job is kept so you can read it with kubectl logs, and Kubernetes deletes it after a day. A release without the <release>-image-variants CronJob makes the job fail with IMAGE VARIANTS NOT GENERATED.

Warming the cache after a deploy

A purge leaves the cache empty, so the first visitor to each page would wait for a server render. warm_cache fills it again after a deploy. On GitLab it's the second half of the image variants and warm <track> job. On GitHub it's a Warm the page cache step, after the fixtures and image variants steps, so a new environment is warmed once its content exists. Only production is warmed by default; the other tracks opt in (see CI/CD Variables › Cache warm). The production deploy never waits for the warm. It reads /sitemap.xml from https://$DOMAIN, or the url passed to warm_cache (following one level of sitemap index), then requests every page anonymously, three at a time by default, and prints each page's status and time to first byte.

Any page that doesn't return 200 is listed under a CACHE WARM FAILED banner, and on GitHub it also raises an ::error annotation on the run summary. The warm fails, but it is allowed to (allow_failure: true on GitLab, continue-on-error: true on GitHub), so the pipeline shows a warning instead of a failed deploy. By then the new release is already live, and the deploy and its cleanup steps have already run. Read the job's log when it warns.

warm_cache lives in bin/lib/site.sh, which k8s.sh sources. The warm's switches (which tracks are warmed, how many pages at once, TLS verification, extra curl options) are listed in CI/CD Variables › Cache warm.

This warm requests pages, so it refills the rendered HTML and the API responses those renders fetch. Separately, a new php pod warms its own /_api cache before it takes traffic, in a postStart hook, and the deploy warms /_api again after a full flush. Those cover API responses only, never HTML, and purge_rendered_html still drops every page after the deploy, so this warm is still needed. See Kubernetes › Warming the API cache on start.

One job for variants and the warm

On GitLab, the image variants and the cache warm run in one job per track: image variants and warm review, image variants and warm staging, image variants and warm canary and image variants and warm production, each in the same stage as its deploy.

  • The job calls image_variants_and_warm, which generates the variants first and then warms. The warm must never come first: its renders would make the serving API pod generate every missing variant at once, and load shedding gave visitors 503s for about five minutes.
  • Each half keeps its own switch: CWA_CI_IMAGE_VARIANTS_<TRACK> and CWA_CI_WARM_CACHE_<TRACK>. Production's are on unless "false", and the other tracks' are off unless "true". The job is created when either switch is on, and skips the half that's off.
  • If one half fails, the other still runs. Either failure fails the job, which is allow_failure: true, so a live deploy is never marked failed.
  • The performance audit <track> jobs needs: the combined job.

On GitHub the two are steps that run one after the other in the deploy job.

Auditing performance after a deploy

warm_cache tells you how quickly the server answered. The manual performance_audit job runs Lighthouse against a few pages and tells you what a visitor experiences. It never fails a deploy. See Performance Audit.


Variables

Every variable the pipeline reads, with its default and the Helm value it sets, is on CI/CD Variables. It starts with the few a new site must set, and covers site settings: your own NUXT_* and CWA_API_* values, passed to the containers by prefix.


Fixtures on a deploy

Fixtures seed a new environment: the scaffold pages and the first admin user. They are off unless CWA_CI_FIXTURES_ENABLED is "true" (see Pipeline flags).

  • GitLab: a load fixtures job appears for review apps and for production, after the deploy job. It never runs automatically; trigger it from the pipeline UI. With CWA_CI_PRODUCTION_DISABLED="true", there is no production fixture job, because the job it waits for doesn't exist.
  • GitHub Actions: a Load fixtures step runs after each review deploy, and after a manual production run, while the variable is "true". GitHub has no manual jobs inside a push-triggered workflow, so set the variable for a new environment's first deploy, then remove it.

Staging has no fixture job on either. It runs under the production environment, so with an external database (DATABASE_URL) it uses production's database, and a fixture load there would write to production.

By default load_fixtures runs doctrine:fixtures:load --append, so it never deletes existing content. The admin user fixture can be run any number of times, and resets the admin's password to ADMIN_PASSWORD each time, so set that before the first load: with it unset or admin, the load warns and the site's super-admin login is admin / admin (see CI/CD Variables › The admin password). On a database that already has content, appending creates only what is missing and keeps existing content. See Appending to an Existing Site.

After a successful load, load_fixtures flushes the whole HTTP cache (silverback:api-components:purge-http-cache), API responses and pages alike. The load changes the database underneath what Souin has cached since the deploy, and nothing else would purge it, so production could otherwise serve the old content for up to a year.

The flush also empties the /_api cache, so load_fixtures then runs warm_api_cache, the same two passes (gzip, then br) a deploy runs after a full flush. Pages aren't warmed: the next visitor to each one waits for a render. With CWA_CI_API_WARM_HTTP_CACHE_MAX_TIME set to 0 the warm is skipped, and a failed pass only warns.

Purging before a load

CWA_CI_FIXTURES_PURGE is for an early project whose database should be rebuilt from a changed scaffold. It runs doctrine:fixtures:load without --append, which empties every table first.

TrackPurges withOtherwise
Review apps"true" or "force"Appends
Production"force" only"true" prints a notice and appends, so a project-wide "true" meant for review apps can't empty production

Only the fixture job reads it. A deploy or a pod restart never purges.

On GitHub the fixture step runs after every deploy while CWA_CI_FIXTURES_ENABLED is set. With CWA_CI_FIXTURES_PURGE set too, every review deploy empties the review database, and every manual production run with "force" empties production. Unset CWA_CI_FIXTURES_PURGE once the database has been rebuilt. Add CWA_CI_FIXTURES_PURGE to repository variables, not secrets: the workflows read vars.CWA_CI_FIXTURES_PURGE.

Changing a live site's hostnames

Production's certificate covers its main hostname and every entry in KUBE_INGRESS_ALIAS_DOMAINS. You change that list when you launch a site, add www, or retire the preview hostname. To do it, edit the variables and deploy:

  • Add a hostname: add it to KUBE_INGRESS_ALIAS_DOMAINS.
  • Retire a hostname, such as the preview host: remove it from KUBE_INGRESS_ALIAS_DOMAINS. It then goes offline. The template has no redirect feature, by design.
  • Change the main hostname at launch, for example new.example.org → example.org: the main hostname is the production environment URL, https://$KUBE_INGRESS_BASE_DOMAIN, so change KUBE_INGRESS_BASE_DOMAIN (or the DOMAIN your project.sh sets). Staging and review apps are subdomains of it, so they move too. Keep the old host in KUBE_INGRESS_ALIAS_DOMAINS if it should still work.

When the list differs from the live certificate's, the deploy creates a new cert-manager Certificate for the new list. It waits until the certificate is issued, for up to CWA_CI_TLS_CERTIFICATE_TIMEOUT (default 600s), and only then runs Helm. The ingress moves from one valid certificate to another, with no downtime. When the list is unchanged, which is almost every deploy, the deploy keeps the live certificate and nothing is reissued. The order and case of the hostnames don't matter.

Point a new hostname's DNS at the cluster's ingress before you deploy. Let's Encrypt's HTTP-01 challenge must reach it. If it can't, the certificate isn't issued and the deploy fails before anything changes. The live site keeps its current certificate, and the log lists the hostnames to check. Fix the DNS and run the deploy again.
Why the certificate is issued first. If the names on the live certificate change in place, cert-manager serves a temporary self-signed certificate while it gets the new one. That takes minutes, or up to an hour of back-off if a challenge fails. With HSTS, browsers refuse the connection outright, on every hostname, including the live one. Let's Encrypt also allows only 5 certificates for the same set of names per week, so don't switch the list back and forth.

Some details:

  • The deploy compares against the certificate on the chart's own ingress, which it finds by the release name. This is why changing the main hostname at launch is still safe.
  • Other ingresses in the namespace are ignored, even with the same chart labels. For example, a redirect ingress you added by hand for a retired hostname keeps its own certificate. The deploy doesn't compare against it or remove it.
  • New certificates are named <CWA_CI_TLS_SECRET_NAME>-stable-api-<hash>, from a hash of the hostname list. After a successful deploy, older ones created this way are deleted, with their secrets. The one just replaced is kept, so a rollback still has a valid certificate. You don't change CWA_CI_TLS_SECRET_NAME by hand.
  • Only production (the stable track) is covered. Review, staging and canary each serve one hostname that doesn't change. The first deploy of a release issues its certificate directly, because nothing is live yet.

Permissions

The CI service account needs access to certificates.cert-manager.io in the namespace. The deploy creates, patches, gets, watches, lists and deletes Certificates, and deletes the secrets of the ones it removes. It checks only create up front. Without it, the deploy prints a warning and falls back to changing the live certificate in place, which causes the downtime described above.

With CWA_CI_INGRESS_ENABLED set to "false" there is no ingress, so none of this runs. Nor does it with an empty CWA_CI_CLUSTER_ISSUER.

Changing hostnames by hand

Use this if the CI account can't manage Certificates, or your pipeline doesn't have ensure_tls_certificate.

  1. Choose a new value for CWA_CI_TLS_SECRET_NAME, for example letsencrypt-cert-2. The production ingress uses the secret <CWA_CI_TLS_SECRET_NAME>-stable-api.
  2. Apply a Certificate that writes that secret and covers the new hostnames:
    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: letsencrypt-cert-2-stable-api
    spec:
      secretName: letsencrypt-cert-2-stable-api
      issuerRef:
        group: cert-manager.io
        kind: ClusterIssuer
        name: letsencrypt-prod # your CWA_CI_CLUSTER_ISSUER
      dnsNames:
        - example.org
        - www.example.org
    
  3. Wait until it is ready: kubectl wait --for=condition=Ready certificate/letsencrypt-cert-2-stable-api -n <namespace> --timeout=600s.
  4. Set CWA_CI_TLS_SECRET_NAME to the new value, update the hostname variables, and deploy.
  5. Keep the old Certificate and secret until the site works on the new one. Then delete them.