CWA Logo
CWA 2 is a prerelease (2.0.0-alpha), heading for a beta. It runs production sites; configuration can still change between releases, always with upgrade notes. View versions and support
Deployment & Operations

Troubleshooting

Find a problem by its symptom or error message — local dev, single-server production, Caddy and Souin, Mercure, Nuxt, the database, Kubernetes deploys and Cloudflare — with a fix for each and a link to the full explanation.

Each problem below has its own heading, named after what you see: the symptom or the error text. Search the page for your error message. Each entry links back to the reference section that explains the part involved.

Troubleshooting for other parts of CWA is listed at the end, in Elsewhere in the docs.

Local development

create-cwa says a port is in use

The stack publishes ports 80 and 443 (Caddy in php), 3001 (the Nuxt dev server in app), 5432 (PostgreSQL in database) and 8025 (Mailpit's web UI in mailpit). create-cwa checks them before it downloads anything, and names what holds each one it finds in use:

  • A container, usually another CWA project's stack. The message gives the command, docker compose -p <project> stop, or docker stop <container> for a container outside Compose.
  • A process on your machine. For 80 and 443 that's often Laravel Valet (valet stop), Herd, nginx or Apache. For 5432 it's a local PostgreSQL (Postgres.app, or brew services stop postgresql). For 8025 it's a local Mailpit or MailHog.

For 3001, 5432 and 8025 you can move the stack instead: change published: 3001 under app:, published: 5432 under database:, or published: 8025 under mailpit:, in compose.override.yaml. bin/up prints Mailpit's address from the published port, so it follows the change. The CLI creates the project without starting the stack, so run bin/up in it once the port is free. A docker compose up that fails with port is already allocated or address already in use has the same cause. See Installation › Scaffold your project.

create-cwa says docker info failed

Docker is installed but its daemon isn't running. Start Docker Desktop, or the Docker service on Linux, wait until docker info answers, then run bin/up in the project the CLI created. bin/up checks the same thing first, and stops with bin/up: Docker isn't running: start Docker Desktop (or the Docker daemon) and run bin/up again. When docker isn't installed at all, the CLI doesn't offer to start the stack.

create-cwa warns that Docker has too little memory

Under about 3.5 GiB of memory for Docker, the CLI warns that the stack needs about 4 GiB, and still starts it. With too little, a container can run out of memory, for example while the app container runs pnpm install or a build runs. In Docker Desktop, raise the memory in Settings › Resources, apply, and run bin/up again.

bin/up fails at "Starting the stack"

bin/up runs docker compose up -d --wait, and stops if a container doesn't turn healthy within its time budget (see Installation › Let bin/up start the site). It names the services that aren't healthy and prints the last 40 log lines of each:

bin/up failed at "Starting the stack (docker compose up -d --wait)" (after 912s): not healthy: php
--- last 40 log lines of php ---
…
    more: docker compose logs -f php
Fix it and run bin/up again; finished phases are skipped quickly.

When no service is named, the message ends the images didn't build or the containers didn't start (see the output above): the error is in the build output above it.

bin/up fails at "Loading fixtures"

bin/up loads the fixtures with doctrine:fixtures:load --append and stops with timed out after <n>s when the load uses up what's left of the time budget, or with doctrine:fixtures:load exited <code> when it fails. Either way it prints the last 40 log lines of php and database, and the load's own error is just above them. Fix it and run bin/up again: --append only creates what's missing, so a second run doesn't duplicate anything. To start the stack without loading them, run bin/up --skip-fixtures. See Appending to an Existing Site for what a load keeps.

bin/up warns that the first visit may be slow

    warning: / still answered 503 after 120s; the first visit may be slow (docker compose logs app)

After the fixtures, bin/up requests / and /login from inside php so the Nuxt dev server compiles them, and gives each up to 120 seconds. This is only a warning: the site still starts, and the summary is still printed. Your first visit to the page compiles it instead, which can take a while. If the page doesn't load at all, read the dev server's log with docker compose logs app. A status of 000 means the request got no answer.

php stays unhealthy

docker compose ps shows php as unhealthy, or docker compose up -d --wait fails on it. Its health check requests /_api/_/health on a loopback-only site at port 2080 inside the container (see Docker › Waiting for the stack). Read the last checks, and request the route yourself:

docker inspect -f '{{json .State.Health}}' $(docker compose ps -q php)
docker compose exec php curl -sS http://127.0.0.1:2080/_api/_/health
  • 503 with "reason":"database": the database is down or unreachable. Check docker compose ps database and its logs.
  • The connection is refused: Caddy isn't running yet, because the entrypoint is still running composer install or the migrations (docker compose logs php). The check allows 300 seconds for that; a first composer install on a slow connection can take longer, and php turns healthy once it answers. If the start finished and it's still refused, the image's api/Dockerfile is newer than its Caddyfile, which has no port 2080 site: copy both, with app/Dockerfile, and run docker compose build php app.
  • 400: Symfony refused the Host. The check sends BROWSER_SERVER_NAME (default localhost), so it must match TRUSTED_HOSTS.

The database container exits because its data is from an older PostgreSQL

docker compose logs database shows:

Error: in 18+, these Docker images are configured to store database data in a
       format which is compatible with "pg_ctlcluster" …
       Counter to that, there appears to be PostgreSQL data in:
         /var/lib/postgresql

The database container runs PostgreSQL 18, and its database_data volume holds data from an older PostgreSQL, such as 16. PostgreSQL 18 won't start on another major version's data directory, so php never turns healthy either.

If you don't need the local content, remove the volumes and start again. This deletes the local database: every page, component and user in it. -v also removes the caddy_data volume, so the browser warns about the certificate again (see Docker › Volumes):

docker compose down -v
bin/up

bin/up creates an empty PostgreSQL 18 database and loads the fixtures. To keep the content, dump it with the old version first. A one-off PostgreSQL 16 container can read the volume (<project> is the Compose project name, by default the folder's name), and --clean lets the dump replace the tables bin/up migrates:

docker compose down
docker run -d --name old-pg -v <project>_database_data:/var/lib/postgresql/data postgres:16-alpine
docker exec old-pg pg_dump -U app --clean --if-exists app > local.sql
docker rm -f old-pg
docker compose down -v
bin/up --skip-fixtures
docker compose exec -T database psql -U app app < local.sql

Use your own POSTGRES_USER and POSTGRES_DB in place of app if you set them. On a single server the volume holds the live site, so never use down -v there without a dump.

Every page returns 502 or 503 after you run pnpm install on the host

In development, compose.override.yaml bind-mounts app/ and app/node_modules into the app container, and that container runs pnpm install and then pnpm dev. A pnpm install on your machine, such as for a dependency bump, rewrites the shared node_modules while the container's dev server is still running from it. The dev server then crash-loops with errors like Cannot find module 'typescript', a missing @nuxt/cli/dist/dev/index.mjs or Cannot resolve module "@nuxt/kit", and Caddy returns 502 or 503 for every page. Restart the container so it reinstalls from inside:

docker compose restart app

When you check its logs afterwards, only look at lines after the restart. The crash from before it is still in the log and looks like a new failure. See Docker › Development Workflow.

Every dev page takes about 30 seconds

This is almost always missing PostgreSQL statistics, not slow I/O. CWA's components use Doctrine JOINED inheritance, so loading a page emits a wide multi-table join. The component subclass tables hold only a handful of rows each — permanently below the threshold that triggers PostgreSQL's autoanalyze — so those tables are never analysed, the planner has no statistics, and it wildly over-estimates the join and throws parallel workers at a query that returns almost nothing. Fix it by analysing the database once:

docker compose exec database psql -U app -d app -c 'ANALYZE;'

The template's php entrypoint does this on every compose up, but a new small fixture table can still trigger it until the next restart. See Docker › Starting the Stack.

This is easy to misdiagnose because it's a pure query-planner problem with no I/O. A near-idle database, tiny tables, and a 99% cache-hit ratio are all true and completely consistent with it — none of them rule the database out. If dev is mysteriously slow, go straight to EXPLAIN (ANALYZE, BUFFERS) and check SELECT relname, reltuples FROM pg_class for tables reporting -1 (never analysed).

composer update fails with a raw.githubusercontent.com 404

If Composer dies downloading a Symfony Flex recipes index with an HTTP 404, the cause is usually a stale GitHub token in the container's /config/composer/auth.json, not a network or GitHub outage. GitHub returns 404, not 401, for a bad token on raw.githubusercontent.com — and a plain curl of the same URL returns 200 because it sends no token, which misleadingly "proves" the network is fine. The /config volume (caddy_config, see Docker › Volumes) persists across compose down/up, so the bad token survives restarts. Clear it:

docker compose exec php rm -f /config/composer/auth.json

Then re-run composer update (or set a valid GITHUB_TOKEN).

Composer or php reports a missing ffi extension

The project's Composer dependencies require ext-ffi for the libvips imagine driver (see Docker › Image processing). A php image built without them has no libvips or ffi. Rebuild the dev php image:

docker compose build php && docker compose up -d php

Caddy reports unexpected EOF after you edit the Caddyfile

If Caddy fails with unexpected EOF while api/frankenphp/Caddyfile looks valid on disk, it's a stale-config issue, not a syntax error. On macOS/Windows (Docker Desktop), a single-file bind mount caches the file's size, so an edit that makes the file longer — what most editors' atomic save does — is served to the container truncated. Edits that shrink or keep the length work, which makes it look random. Confirm by comparing byte counts:

wc -c < api/frankenphp/Caddyfile
docker compose exec php sh -c 'wc -c < /app/frankenphp/Caddyfile'

If they differ, recreate the container (restart isn't enough): docker compose up -d --force-recreate php. The CWA template avoids this by reading dev config through the ./api directory mount rather than mounting single files — you'll only hit it if you reintroduce a single-file mount in compose.override.yaml. Production is unaffected: the image bakes the Caddyfile in.

The dev server loads, but nothing on the page responds

On Nuxt 4.5, if pages render but never hydrate, so buttons do nothing and the login form reloads the page, check the browser console for a SyntaxError about __unhead_devtoolsPlugin being declared twice. @unhead/bundler 3.4.1 registers its DevTools runtime import once for each of Nuxt's client and server Vite servers. Turn off unhead's own DevTools panel in nuxt.config.ts:

export default defineNuxtConfig({
  unhead: {
    vite: {
      devtools: false,
    },
  },
})

This only affects development. Nuxt DevTools and useSeoMeta still work, and the production build is unchanged. The template sets this. Remove it once an unhead release fixes the duplicate.

The rate limit never triggers on the dev stack

The rate limit never applies to private addresses, so your own browser may never trigger it locally. To try it, send an X-Forwarded-For header with a public address: the request comes from a trusted private address, so Caddy believes the header. See Docker › Origin protection settings.

Single-server production

docker compose stops with required variable … is missing a value

compose.prod.yaml requires every secret and site value, and names the one that's missing, for example required variable SITE_HOST is missing a value: set SITE_HOST to the site's domain. Set it in the env file you pass with --env-file: .env.prod, if bin/server/init wrote it. Run Compose by hand as docker compose --env-file .env.prod … (see Deploy to One Server › Running commands by hand). See Docker › Production Docker Compose.

php won't start: SITE_HOST and SITE_ALIASES must be host names without a scheme or port

php's entrypoint builds TRUSTED_HOSTS and the CORS origins from SITE_HOST and SITE_ALIASES, and stops on a value that isn't a bare host, such as https://example.com or example.com:443. Set SITE_HOST=example.com, and separate SITE_ALIASES with spaces.

A comma between aliases stops it with SITE_ALIASES is space-separated, not comma-separated: '<value>'., because Compose puts SITE_ALIASES into Caddy's SERVER_NAME, which refuses a,b.

bin/server/init says .env.prod already exists

init never overwrites the site's settings, because they hold its secrets: a new JWT key pair signs every user out. Edit .env.prod instead. To start again on a new site, delete it first. See Deploy to One Server › Write the settings with init.

A script says No .env.prod: run bin/server/init first.

deploy.sh, backup.sh and restore.sh read the site's settings from .env.prod at the root of the checkout. Run bin/server/init, or, on a server taking over a site, copy the old server's .env.prod. CWA_SERVER_ENV_FILE names a different file (see CI/CD Variables › Single-server scripts).

deploy.sh stops with ❌ The migrations plan failed: nothing was deployed.

The new php image couldn't plan the pending migrations against the live database, and the log above shows why. The images are built, but nothing restarted, so the site still runs the previous deploy. Fix the migration, then deploy again. See Migrations and Backups › On one server.

A backup fails with ❌ The dump doesn't read back (pg_restore --list): no backup written.

pg_dump wrote a file that pg_restore can't read, for example because the disk filled up, so backup.sh deleted it. Free some space, or move backups/ elsewhere with CWA_SERVER_BACKUP_DIR. When deploy.sh took the backup before migrating, the deploy stopped there and nothing restarted.

deploy.sh ends with ❌ Deployed, but image variants failed

The site is deployed: only the image variants failed, and they run in their own containers. The line before it says why:

  • Image variants ran out of memory (exit 137): a container went over CWA_SERVER_IMAGE_VARIANTS_MEMORY (1536m). Raise it, or lower CWA_SERVER_IMAGE_VARIANTS_MAX_MEMORY (1G) so each container stops sooner and the next carries on.
  • imagine:generate exited 1: a variant failed; the log above lists which.

WARNING: image variants still missing after <n> runs isn't a failure: the next deploy carries on. See CI/CD Variables › Single-server scripts.

deploy.sh warns the page warm failed

The deploy finished, but some sitemap pages didn't return 200 when it requested them from the server itself, and the log above lists them. They render on their first visit instead. Open one to see its error. A sitemap with no pages fails the warm too.

The site stays in maintenance mode after restore.sh fails

restore.sh turns maintenance mode on before it replaces anything, and turns it off only at the end, so a restore that stops halfway leaves visitors on the maintenance page. Fix the error and run it again, or turn maintenance off by hand:

docker compose --env-file .env.prod exec php bin/console silverback:api-components:maintenance off

No complete backup in backups/. means it found no database dump and media archive with the same time. See Deploy to One Server › Restore a backup.

Caddy and Souin

Check every Caddyfile change with bin/test/caddy-validate.sh. frankenphp adapt only parses the Caddyfile, so errors raised while Caddy loads its modules, like several below, pass it. The script runs frankenphp validate, which loads every module, for both SERVER_NAME shapes the template uses: Helm's :80 and compose's several ports. It also checks each with php forwarding Mercure to a hub of its own (scaled mode), and the hub's own mercure-hub.Caddyfile. Run it locally with docker compose exec -T php sh -s < bin/test/caddy-validate.sh. It checks the Caddyfile at /etc/caddy/Caddyfile, which is the copy built into the image, so rebuild php after editing. The unit-tests job runs it in CI too.

How the template's Caddyfile caches is in Docker › The HTTP Cache in Caddy.

php won't start with unrecognized directive after you copy the template's Caddyfile

The Caddyfile uses two Caddy modules that a php image built without them doesn't have: query_allowlist, the template's own handler for the query allowlist (see Origin Protection › Query Parameters on Pages), and rate_limit, from mholt/caddy-ratelimit. Such an image refuses the file. In dev, the --watch reload keeps the old config running, and the next restart crash-loops. Rebuild the image, or let CI build it:

docker compose build php && docker compose up -d php

php won't start after you copy the template's Caddyfile, because an import isn't found

The Caddyfile's mercure block imports the hub's settings from mercure-hub.caddy, which sits beside it. Copy api/frankenphp/mercure-hub.caddy and mercure-hub.Caddyfile too, and the api/Dockerfile line that copies them to /etc/caddy/, then rebuild php. See Real-Time Updates (Mercure) › In the Template.

php keeps restarting with token recognition error at: '#'

Comments inside the @use_cache expression break Caddy, and frankenphp adapt does not warn you. The text between the backticks is a CEL expression, not Caddyfile syntax. CEL has no comments. A # line inside the backticks is a parse error, but frankenphp adapt does not check the text inside the backticks, so it reports no problem. The error only appears at runtime, and the php container keeps restarting. Put comments above the block. After changing it, run adapt, then recreate the container and check that it becomes healthy:

docker compose up -d --force-recreate php
docker compose ps php      # should reach "healthy", not "Restarting"

A cache matcher never matches when a cookie is absent

If you edit the Souin @use_cache expression in the Caddyfile, note that an absent cookie does not equal "" in Caddy matchers — unlike a missing header, which does. A clause like {http.request.cookie.api_component} == "" only matches when the cookie is present but empty, so requests with no cookie at all (SSR, curl, health checks) silently miss the cache. Match on the Cookie header instead, so an absent or empty cookie both count as "not authenticated":

!{http.request.header.Cookie}.matches("api_component=[^;]+")

Nothing is cached after a full flush

The php log shows BADGER-INSERTION-ERROR or DB Closed. Unpatched Souin v1.7.9 closes the badger and nuts storages on a full flush, and they stay closed until php restarts. A full flush is Purge all cached data, purge-http-cache, every fixture load, a deploy in scaled mode, and a deploy whose Cloudflare edge-only purge fails. The default in-memory otter storage is not affected. The template's build applies v1.7.9-flush-no-close.patch, which empties the store and leaves it open, so this means the php image was built without it. Rebuild the API image with the patch and its lines in api/Dockerfile. Until then, restart php after each full flush.

The first visitor after a flush or a rollout gets a 504

A page that renders cold in more than Souin's backend timeout gets a 504 from the cache, and Cloudflare passes it on. Souin's own default is 10 seconds, and a heavy CMS page took about 12. The Caddyfile sets 30 seconds; raise it with CWA_API_CADDY_CACHE_BACKEND_TIMEOUT. A timeout block in CADDY_CACHE_EXTRA_CONFIG replaces the Caddyfile's whole block and brings the 10 seconds back. See Docker › The cache store and backend timeout. To avoid cold renders in the first place, purge one page instead of everything: see Cloudflare › Purges must reach Cloudflare.

Requests that waited on a failed render get an empty 200

Souin sends only one request upstream for a URL that isn't cached, and the others for that URL wait for its response. In unpatched Souin v1.7.9, when that response is a 500, 502, 503 or 504, the waiting requests get a 200 with an empty body and no Cache-Status header, and an SSR render can cache a broken page. The template's build applies v1.7.9-singleflight-5xx.patch, so they get the same error. A 5xx that is private or sets a cookie isn't shared: each waiting request goes upstream itself. bin/test/souin-coalesced-5xx.sh, in the unit-tests job, fails on an unpatched binary. If you see empty 200s, rebuild the API image with the patch and its COPY and patch lines in api/Dockerfile.

php is OOMKilled while serving a large file

Souin reads the whole body of a cacheable response into memory before it decides anything, so a large file served through the cache can take php past its memory limit. Keep large files out of @use_cache: see Docker › What the page cache stores.

PURGE or BAN sent to the site returns 405

That's intended: they never purged anything. See Docker › PURGE and BAN.

Mercure

The module's own checks for live updates are in Real-time Updates › Troubleshooting.

php won't start with a Mercure issuer or compatibility-mode error

FrankenPHP 1.13 ships Mercure 1.0, which treats unknown directives as errors and has no demo or ui. If MERCURE_EXTRA_DIRECTIVES sets either, use debugger instead, in development only. The template's Caddyfile uses an issuer block, which a 0.x Mercure doesn't know, so the Caddyfile and the php image must come from the same template. Two errors mean the Caddyfile and the binary don't match:

  • the "publisher_jwt", "subscriber_jwt", … directives work only in compatibility mode: the Caddyfile has top-level publisher_jwt and subscriber_jwt directives. Move the keys into an issuer block, as described in Hub Caddyfile.
  • an issuer identifier is required in modern mode: the Caddyfile is still set up for compatibility mode (protocol_version_compatibility with top-level publisher_jwt and subscriber_jwt), but the binary was built without the deprecated_topic and deprecated_claim tags, so the hub runs in modern mode anyway. Remove the directive and move the keys into an issuer block.

caddy adapt reports neither error. Only a running hub does, so check php's logs after it starts.

php won't start with only one unnamed Mercure hub is allowed per configuration

An API image built with Mercure 1.0.3 or later refuses the compose stack's Caddyfile unless the mercure block has a name. Compose's SERVER_NAME has several ports, so Caddy runs one server per port, each with its own hub. The healthcheck never passes, and the error only shows in the php container's log. The template sets name cwa in the mercure block, in mercure-hub.caddy: add it back if it's missing, then recreate php. See Hub Caddyfile.

Admins get no private updates locally

With a Mercure 1.0 hub, the cookie is __Secure-mercure_access_token, and browsers only accept it over HTTPS. The template's dev stack serves the hub at https://localhost/.well-known/mercure, so this works out of the box. mercure.yaml pins the name as cookie_name, because mercure-bundle 0.6 drops the __Secure- prefix when kernel.debug is on, as it is in dev, and the hub then wouldn't read the cookie. If you change CADDY_MERCURE_PUBLIC_URL to a plain http:// URL, change that cookie_name to a prefix-less name, and set the same name as the hub's cookie_name, in mercure-hub.caddy (see The subscriber cookie).

If the API and the front end are on different registrable domains, the cookie needs SameSite=None: see API Authentication › The cookie's SameSite.

Nuxt

The sitemap is empty, or the Nuxt log shows UND_ERR_HEADERS_OVERFLOW

Node refuses a response whose headers pass 16 KB by default. A large collection's Surrogate-Key passes that, so Caddy removes the header from /_api responses on server-side rendering's internal hostname; if a response still carries it there, check the Caddyfile has that rule. The app image also raises Node's limit with NODE_OPTIONS=--max-http-header-size=65536; a deploy that sets its own NODE_OPTIONS replaces that. See Cloudflare › Purges must reach Cloudflare.

Database

The in-cluster PostgreSQL restarts empty, or the API can't connect after a credentials change

Changing the user, password or database name restarts the PostgreSQL pod. Without persistence it starts empty; with persistence it keeps the old values, so the API can't connect until you change them in PostgreSQL. See CI/CD Variables › Database.

Kubernetes deploys

The php image build hangs on apt-get, install-php-extensions or the Go build

An uncached php build fetches packages from Debian, which can stall on a slow runner. apt uses HTTPS, and each fetching step has a time limit and up to three attempts, logged as retry-fetch: attempt <n> failed or passed <s>s: …. The builder stage's Go module fetches and xcaddy build get the same limits and retries, and Go fetches over HTTP/1.1; a sum.golang.org stream error (stream error … INTERNAL_ERROR) is the Go-side symptom. The Dockerfile pulls the prebuilt cwa-php-base, so a build runs neither the Go build nor the extension installs unless you build the base from source. See Docker › Patches in the build and Docker › Where the base comes from.

The review job is orange and says No namespace '<name>'

The branch has no Kubernetes namespace, so it has no review app. Create the namespace and its role bindings, then run the job again. See CI/CD › Review Apps.

The deploy fails before Helm runs

The deploy checks several things before it changes anything in the cluster, and its log says which one failed and what to change:

The deploy stops with MIGRATIONS PLAN FAILED, DATABASE BACKUP FAILED or MIGRATIONS FAILED

These come from the database step before the rollout, and none of them changes the live site's code. Every production deploy runs the plan, and the migrate Job when migrations are pending. The plan needs PostgreSQL, the only database the template supports. A plan fails when the dry run of the migrations fails. A migration that writes data from up() or postUp() instead of with addSql() doesn't fail it: the plan lists it without SQL and the deploy warns (see Migrations and Backups › The migrations plan). A backup fails on a missing, public or media CWA_CI_DATABASE_BACKUP_BUCKET, or a missing or media CWA_CI_DATABASE_BACKUP_GCLOUD_JSON, and a dump bigger than CWA_CI_DATABASE_BACKUP_STORAGE (5Gi) gets its Job evicted. On GitLab, a Protected key variable never reaches a branch that isn't protected (see Migrations and Backups › The backup bucket). If the log says MAINTENANCE MODE IS ON, a maintenance window was open: its last lines say how to restore the backup and turn maintenance off. The next production deploy ends the window, with or without CWA_CI_MAINTENANCE_DURING_MIGRATIONS (see Migrations and Backups › When the window fails). See Migrations and Backups.

A review deploy stops with REVIEW APP NOT DEPLOYED

The review app's external DATABASE_URL names production's database (the same host, port and database), which a review app would migrate. On GitLab this usually means production's DATABASE_URL and CWA_CI_POSTGRES_ENABLED="false" are scoped to All environments. Scope them to production, or give review apps a database of their own. See Migrations and Backups › A track on production's database.

If the log instead warns could not read production's database URL, the check couldn't find production: set RELEASE_PRODUCTION or KUBE_NAMESPACE_PRODUCTION if production's release or namespace has another name, or give the review job read access to production's Secret.

A scaled deploy prints THE API CACHE IS NOT SHARED IN REDIS

!!!! SCALED MODE: THE API CACHE IS NOT SHARED IN REDIS !!!!

After a scaled deploy's flush and warm, three requests through the API Service got no cache hit from Redis (on GitHub, also an ::error annotation, "API cache not shared"). The deploy still succeeds, because the release is already live. Until it's fixed, a purge on one php pod can leave the others' copies stale, so an edit may show on some refreshes and not others. Read the Redis pod's log (kubectl logs deploy/<fullname>-redis, as the message prints), look for Redis errors in the php pods' logs, then press Purge all cached data once Redis answers. See Scaled Mode › Checking the cache is shared.

A scaled deploy warns up to … database connections

The php pods at their peak, times CWA_CI_API_WORKERS, plus 5, is more than the in-cluster PostgreSQL's 97 usable connections. The deploy goes ahead. Lower CWA_CI_API_AUTOSCALE_MAX, CWA_CI_API_WARM_SURGE or CWA_CI_API_WORKERS, or raise the database's max_connections. See Scaled Mode › Database connections.

A deploy didn't restart the pods

A deploy restarts a Deployment's pods only when its image digest or settings change. An image built from unchanged inputs is reused with the same digest, so a push that didn't touch api/ or app/ leaves those pods running. To restart them, run kubectl rollout restart deploy/<fullname> (or <fullname>-pwa), or set CWA_CI_BUILD_FORCE="true" and run the pipeline. See Kubernetes & Helm › When a deploy restarts pods.

Every deploy job fails on its first line, or stop review can't load k8s.sh

bin/devops/k8s.sh sources bin/lib/site.sh from the repository root. Keep bin/lib/ beside bin/devops/, and add bin/lib to the sparse checkout of a project's own stop review job. From another directory, set CWA_SITE_LIB to the file's path.

The deploy fails with No digest for …, or warns Couldn't tag … as production

No digest for <image>:<tag> stops the deploy before Helm runs. It means the pipeline had no build digest and the branch tag isn't in the registry, for example after CWA_CI_BUILD_DISABLED on a branch that was never built. See CI/CD › Which build a deploy runs.

Couldn't tag <image>@<digest> as production comes after a deploy that succeeded, when the registry refused to copy the build to the track's tag (:staging, :canary or :production) three times. The deploy only warns, the pods run the new build, and the next deploy copies the tag. On GitHub, check that the deploy job's permissions include packages: write.

The deploy fails with THE cwa ROLLOUT DID NOT FINISH

The new API pods (or Nuxt pods, for cwa-pwa) didn't become ready within CWA_CI_DEPLOY_ROLLOUT_TIMEOUT seconds (900). The usual cause is a migration that fails when the pod starts: on staging, canary or review apps, or on production's first deploy, since production's deploy otherwise runs its migrations in a Job before Helm and stops there when they fail. The old pods keep serving and the track's image tag stays on the last good build. The log prints the kubectl describe and kubectl logs commands to find out why. Fix forward and deploy again, or roll back with Kubernetes & Helm › Rollback.

A new php pod takes longer to become ready

Its postStart hook warms the /_api cache first, in two passes (gzip, then br) of up to the track's warm time each (60 seconds on production and canary; staging and review apps don't warm by default, see CI/CD Variables › php (API)), and Kubernetes starts no probe until the hook returns. The old pod keeps serving meanwhile. The warm's summary is in the pod's log. See Kubernetes › Warming the API cache on start.

var/log/prod.log is missing, or the API's log is empty

Production logs to stderr as JSON: read it with kubectl logs or your cluster's log explorer. A request's lines are written only when it logs an error. See Kubernetes › Logs.

No error mail arrives

CWA_API_ERROR_MAIL_TO needs a working MAILER_DSN and a MAILER_EMAIL to send from. A failed send is dropped without a log line, only a 5xx sends one, and each unique error sends at most one an hour per pod. See Kubernetes › Error mail.

The API pod stays unready, and every probe times out

The php probes request /_api/_/site_config_parameters.jsonld, a path that would pass through the Souin cache. Souin combines concurrent requests for the same cache key into one request to php. In unpatched Souin 1.7.9, a request that is given up while php is still working keeps hold of that key, so every later probe waits for the 10-second backend timeout and fails, and the pod stays unready for good. The template prevents this three ways:

  • The template's Souin build applies the fix from darkweak/souin#850 (api/frankenphp/souin/v1.7.9-singleflight-fix.patch), so a request that times out releases the key. The patch can be removed once a Souin release includes it.
  • The Caddyfile's @use_cache skips the probe path for the kubelet's kube-probe/ user agent, so probes always reach php.
  • Liveness uses the same request as readiness, so a pod whose php has stopped answering is restarted.

Also keep timeoutSeconds: 5 on both probes. See Kubernetes › Health Checks.

Visitors get 502 while a deploy rolls out

A pod that closes its listener as soon as it's told to stop still gets requests for a few seconds, until the ingress and php's proxy drop its endpoint. The chart's preStop hooks hold both pods open for 5 seconds first. See Kubernetes › Stopping a pod without 502s.

The image variants and warm job ends with ran out of memory (exit 137)

The Job generating variants went over its own memory limit. The site isn't affected, because the Job doesn't serve requests. Raise CWA_CI_IMAGE_VARIANTS_MEMORY, or lower CWA_CI_IMAGE_VARIANTS_MAX_MEMORY so each Job stops sooner and the next one carries on. See CI/CD › Generating image variants after a deploy.

Visitors get 503 under load

php gives up on a request that waits too long for a free worker, rather than letting it queue into a 504. See Origin Protection › Load shedding.

The per-visitor rate limit never applies

Every visitor looks internal when the ingress controller's Service uses externalTrafficPolicy: Cluster. Set it to Local, and check visitor_ip in php's access log. See Origin Protection › Ingress rate limits.

Browser tests

The browser tests fail with API writes the API refused as a mismatched request

An API write got a 405, 406 or 415: the front end sent a method, Accept or Content-Type the API doesn't take for that resource. The message lists each request. See Browser Tests › What they check.

The browser tests fail with The API answered with a server error during the run

Every test passed, but the API sent an error mail, so a request got a 5xx that the UI didn't show. Find it in the php log the job prints, or keep the stack with E2E_KEEP_STACK=true and read its log. See Running them locally.

The browser tests fail with Not every test ran and passed

A test was skipped, or --grep or test.only left one out. Every listed test must run in CI. See Browser Tests.

The browser tests fail on pages in the sitemap, a second page in the sitemap or og:image

The template's tests take their pages from the sitemap. The sitemap must list at least two pages, and the first one must have an og:image tag. Run the tests with E2E_KEEP_STACK=true, read the sitemap with docker exec <stack>-php-1 curl -sk https://localhost/sitemap.xml, and make sure your fixtures publish the pages it should list. See Browser Tests › What they check.

The browser tests fail with cache check(s) failed

Before the browser tests, e2e/cache-checks.sh checks the cache on the production configuration, and the line starting FAILED: says which check and what it got:

  • first view or after purge-rendered-html wanted 200 stored, or second view wanted 200 hit: that sitemap page isn't cached for a visitor. The got part is the status and the page's Cache-Status header, or no-cache-status. Every page the sitemap lists must be cacheable.
  • no Surrogate-Key on <first page> that <second page> doesn't also have: the check purges a tag only the first sitemap page has, and found none. Each page has its own route tag, so this means the first page's response carried no Surrogate-Key of its own: check that it's a CWA page served from a route.
  • the other page stays cached wanted 200 hit: purging one page's tag also dropped the other page.
  • over HTTPS or http://php.local…: the API didn't answer on that route. Check the php log the job prints.

Port 2019 has its own message: Caddy's admin API (port 2019) answered from the app container. means Caddy's admin API isn't bound to php's loopback only. See Browser Tests › The test stack.

The browser tests can't start the images on an ARM Mac

Images from your CI registry are amd64 only. Set DOCKER_DEFAULT_PLATFORM=linux/amd64, or build the images locally. See Running them locally.

Cloudflare

Cloudflare returns 526 for a new hostname or after a staging deploy

The origin's certificate isn't valid yet. See Cloudflare › DNS and TLS and Each deploy track.

A certificate can never be issued or renewed behind Cloudflare

Always Use HTTPS redirects the HTTP-01 challenge. Turn it off. See Each deploy track.

An edit doesn't show at the edge

See Cloudflare › Checking purges for the log messages and how to check that purges arrive.

Pages stay DYNAMIC under the Cache Rule

Cloudflare doesn't cache a response that carries Set-Cookie. See The cache rules must match the template's exclusions.

Pages built from page data all show one page's content after client-side navigation

The API rule is missing the component positions exclusion. See Cloudflare › API responses.

Browsing quickly gets a 429 from Cloudflare

A rate limiting rule is set too low. See Free protection worth turning on.

Elsewhere in the docs

Faster Functional Tests

When a project's functional test suite slows CI down — measure where the time goes, reset the database without rebuilding the schema, and shard the job by test.