Scaled Mode

Scaled mode runs a site's API (the php pods) on two or more pods instead of one. It's off by default, and a project turns it on for each deploy track with one switch.

The goal is redundancy. By default one php pod serves the whole API, so every node upgrade, eviction or out-of-memory restart is an API outage until the pod comes back. With two or more pods, losing one is not an outage. Uncached ("cold") load also spreads across the pods, and pods can be added for a deploy's cache warm and then removed again.

Terms used on this page

• Pod: one running copy of a server. A replica is one of several identical pods. A node is the machine pods run on. A node upgrade or eviction moves pods off a node by stopping them.
• Souin: the HTTP cache inside each php pod. A purge removes the cached copies of one thing (a tag). A flush removes everything. A warm requests API responses so they're cached before visitors ask for them.
• Redis: a small in-memory database. In scaled mode it holds Souin's cache, so every php pod shares one copy.
• Mercure: the hub that pushes live updates to open browsers.
• HPA (autoscaler): adds and removes pods by CPU load, between a minimum and a maximum. PDB (PodDisruptionBudget): stops Kubernetes taking down more than a set number of pods at once.
• Track: where a deploy goes: review, staging, canary or production.

Do you need it?

One API pod is enough for most sites. Most requests are cache hits, and one pod serves several hundred cached responses a second. During a short API outage, Cloudflare and browsers keep serving pages they have already cached. For most brochure and CMS sites, a few minutes of API downtime during node maintenance is acceptable.

Turn it on when:

• the site can't accept a few minutes of API downtime during a node upgrade, an eviction or a crash;
• the API pod keeps restarting because it runs out of memory;
• uncached traffic (signed-in users, a busy user area, heavy editing, cold pages after a flush) is more than one pod can render, after you've tried a bigger pod first (see Kubernetes & Helm › Resource Requests and Limits).

What it doesn't protect:

• The database. There is still one. If it's down, cached pages keep working, but logins, edits and uncached requests fail in either mode.
• Redis and the Mercure hub are one pod each. Losing either slows the site or pauses live updates, but doesn't take the site down (see What can go wrong).

Why a second pod needs a shared cache

Without scaled mode, two things live inside the php pod that break with a second pod:

1. The cache. Souin keeps it in the pod's memory. A write handled by pod B purges only pod B's cache, so pod A keeps serving the old response. With production's one-year cache lifetime, that copy stays stale.
2. Mercure's queue. Visitors connected to pod A never get updates published on pod B, and no error is shown.

That's why the template refuses more than one API pod unless scaled mode is on (see Kubernetes & Helm › php stays at one replica). Scaled mode moves the cache into one Redis and Mercure into one hub, which every php pod shares. The symptoms of several pods without a shared cache are an edit that shows on some refreshes and not others, and live updates that reach some people and not others.

The third problem, several pods running migrations at once, is already solved in every mode: the entrypoint takes a database lock, so one pod migrates and the others wait (see Kubernetes & Helm › Running Migrations).

Turning it on

Set the switch for the track and deploy. Nothing else needs setting by hand.

Then read the deploy log. It prints the mode before it touches the cluster:

Scaled mode: 2 to 3 API pods sharing this release's Redis (256Mi).

What the deploy refuses

The deploy checks these before it changes anything in the cluster, and fails with The API's scaling settings can't be deployed: followed by each problem:

• More than one php pod without scaled mode: CWA_CI_API_AUTOSCALE_MIN or CWA_CI_API_AUTOSCALE_MAX above 1 while the autoscaler is on (the default), or CWA_CI_API_REPLICA_COUNT above 1 with CWA_CI_API_AUTOSCALE=false. Like the chart, the check ignores the setting that has no effect, so a CWA_CI_API_REPLICA_COUNT of 2 beside the autoscaler is allowed: the chart doesn't use it. The message names the switch to set. The chart refuses more than one pod too, for a helm install without the pipeline. Since template 2.0.0-alpha.43 (32e9199); before, any of the three above 1 was refused.
• A minimum above the maximum, or below 1.
• CWA_CI_API_AUTOSCALE set to anything but true in scaled mode, which runs the API under its autoscaler.
• CWA_CI_RESET_DATABASE=true in scaled mode, which would drop the schema each time an API pod starts.
• A cache store set by hand. In scaled mode CWA_API_CADDY_CACHE_STORAGE must be unset, because the store is the release's own Redis. Without scaled mode, a store that names redis is refused.
• A CWA_CI_API_REDIS_MEMORY not written in Mi or Gi, such as 256M.
• A CWA_CI_API_WARM_SURGE or CWA_CI_API_WARM_CONCURRENCY that isn't a whole number, or a CWA_CI_API_WARM_CONCURRENCY of 0: it's requests per pod, so at least 1.

CADDY_CACHE_EXTRA_CONFIG can't name a store in any mode (see CI/CD Variables › HTTP cache).

What runs in scaled mode

The shared cache

The chart starts one Redis for each release (templates/redis.yaml, 62bfeb9). Helm creates and removes it with the release, so staging, production and each review app get their own.

• It's only a cache. It keeps nothing on disk, so a restart starts empty. Nothing in it ever needs keeping.
• Nothing else may use it. A flush deletes every key in it.
• It never evicts Souin's tag index. Redis uses volatile-lru: when it's full it drops cached responses, which expire, and never the tag index, which doesn't. Without the index, a purge couldn't find the pages it should remove.
• Its memory is CWA_CI_API_REDIS_MEMORY. The container gets about 1.4 times that, plus 16Mi, for Redis's own overhead: 374Mi for the default 256Mi.
• It has a password: REDIS_PASSWORD if you set one, else the one already deployed, else a new one. A NetworkPolicy lets only the release's php pods and its Redis watch reach it, on clusters that enforce NetworkPolicy. The password protects it either way.

The chart points every php pod's Souin at it through CADDY_CACHE_STORAGE. Its timeouts are 100ms, with no retries, so a Redis outage makes requests uncached rather than slow. While Redis is down, responses carry a Cache-Status with REDIS-INSERTION-ERROR, and caching starts again by itself once Redis is back. The image's Redis store is patched so that pods sharing one Redis all drop every page under a purged tag (see Docker › PHP Dockerfile stages).

The Mercure hub

Mercure gets its own pod (templates/mercure-hub.yaml, d51851c, eca4d2b). It runs the php image's own frankenphp with mercure-hub.Caddyfile, so its Mercure version always matches php's, and it has no database and never migrates.

• php publishes to the hub's pod, and stops running a hub of its own.
• Browsers still subscribe at /.well-known/mercure on the API's hostname. php forwards those requests to the hub's pod, so the ingress and TLS don't change.
• The hub's settings are php's. Both read them from api/frankenphp/mercure-hub.caddy. If you change a hub setting, change it there, never in either Caddyfile.
• It stops cleanly. A 5-second grace period ends open subscriptions when the pod stops, and browsers reconnect (05090d5).

It's the free, open-source hub, which runs as one copy. When it restarts, live updates pause and browsers reconnect.

The API pods

In scaled mode the php pods change in four ways (9959bf3):

• A PodDisruptionBudget with maxUnavailable: 1. A node drain moves the pods one at a time, so one keeps serving. With one pod a budget would block every drain, which is why the default has none (see Kubernetes & Helm › Surviving node scale-down). The single pod's safe-to-evict: "false" annotation goes, because the budget replaces it.
• Pods spread across nodes where possible, so losing a node never takes them all.
• An HPA on CPU only, between 2 and 3 pods by default. It adds pods at once and removes them only after ten minutes. The memory target is dropped, because php rarely hands memory back and a memory target would hold the HPA at its maximum.
• No startup warm. A new pod shares a cache that's already warm, so it skips the warm a single pod runs before its probes start (see Kubernetes & Helm › Warming the API cache on start).

The Caddy rate limit stays per pod and isn't divided: each pod limits the traffic that reaches it, which is what protects it.

Every deploy flushes and warms

Redis outlives the php pods, so a rollout no longer empties the cache. Every scaled deploy flushes the whole cache, then warms /_api again. The deploy that turns scaled mode on flushes too.

The deploy waits for the rollout, and then for the old php pods to be gone (up to 120 seconds), before it flushes. Straight after a rollout, the API Service still sends some connections to pods that are stopping, and warm requests on those fail (4d2ad50). Commands run in a php pod that is ready and not stopping (ba19fb4).

The warm goes through the API Service, so every pod renders a share and each response is stored once in Redis (a4bd818). It sends CWA_CI_API_WARM_CONCURRENCY requests at a time for each pod (2 by default), times the HPA's minimum plus any surge: 4 at the default minimum of 2. Keep the total within what the database can take. Each pass still stops at CWA_CI_API_WARM_HTTP_CACHE_MAX_TIME seconds. Fixture loads and the image variants job warm through the Service the same way after their flushes.

The warm opens a new connection for every request (--requests-per-connection=1, see Console Commands › Spreading a warm across pods), so the Service picks a pod for each one. The Service balances connections, not requests, and a warm that kept its connections open split unevenly: on preview one pod served 252 requests and the other 38. A warm on a single pod, through http://localhost, keeps its connections open as before. Since template 2.0.0-alpha.43 (4779237), which pins bundle 2.0.0-alpha.34: the option needs it.

A failed warm only warns. It never fails a live deploy.

More pods for the warm

With CWA_CI_API_WARM_SURGE set (say 2), a scaled deploy:

1. raises the HPA's minimum by that many pods before the flush (2 + 2 = 4), and its maximum too when it would be below the new minimum;
2. waits for the extra pods to be ready, for up to CWA_CI_API_WARM_SURGE_TIMEOUT seconds, because the cluster may need a new node. If they aren't all ready in time, it warms on the pods there are and warns;
3. warms with CWA_CI_API_WARM_CONCURRENCY requests per pod, counting the extra pods: (2 + 2) × 2 = 8 by default;
4. puts the minimum and maximum back. The HPA removes the extra pods ten minutes later.

If the deploy job dies during the surge, the higher minimum stays until the next deploy resets it.

A surge is only useful in scaled mode. With a cache in each pod, every extra pod would warm its own copy.

The Redis watch

While php can't reach Redis, purges are lost but still answer 204. An edit made then could leave old copies cached, for up to a year in production. The Redis watch notices, flushes once Redis is back, and emails (0d9f9d5).

It's a CronJob in the release (templates/redis-watch-cronjob.yaml) that runs the template's app:redis-watch command every 2 minutes. It's on in scaled mode unless CWA_CI_API_REDIS_WATCH="false", and CWA_CI_API_REDIS_WATCH_SCHEDULE changes the schedule. Each run:

1. Checks Redis three times, 2 seconds apart. Redis counts as down only if every check fails. A check fails when the job can't connect to Redis itself, or when php can't: the job requests /_api/_/site_config_parameters.jsonld through the API Service and reads its Cache-Status. php's view is what matters, because its 100ms timeouts can fail while the job's own 2-second connection works.
2. When Redis is down, emails once that the cache store is unreachable. Later runs don't repeat the email. A failed send is tried again next run.
3. When Redis is reachable again after an outage, flushes it once, and emails that the cache store is back and was flushed. A failed flush keeps the outage open, so the next run tries again. A failed send of the "back" email is kept and sent, with the same text, by the next run that reaches Redis, without flushing again (since template 2.0.0-alpha.43, e0e30c8).
4. With provider cloudflare in CADDY_CACHE_CDN_CONFIG, also purges everything in the Cloudflare zone, because the edge may have cached a stale copy from Redis. A failed Cloudflare purge is tried again next run, without flushing Redis again.
5. When Redis restarted between two runs, only emails. A restart starts empty, so nothing is stale. A restart caused by a deploy sends no email: the job knows the Helm release's revision, and a deploy that changes Redis's spec recreates it under a new revision, so only a restart within one revision emails (230869d).

The email goes to MAILER_EMAIL, the same address as the daily orphan scan. With MAILER_EMAIL empty, no email is sent. The subject starts with the site's name.

Its state is kept in the database, because it must outlive Redis: one row in redis_watch_state, created by the template's migrations Version20261020162222, Version20261020171138 (the release revision) and Version20261020172842 (an unsent recovery email). A project that copies the watch needs all three migrations too. A run that starts before the release has migrated exits cleanly with The database is not migrated yet; checking next run.

If the job can't reach Redis but php can, the run fails as the job's own fault (its password, or a NetworkPolicy that leaves it out) and isn't treated as an outage.

When Redis is unreachable

The site stays up, uncached and a little slower: about 0.2 seconds a request in the template's tests. The watch usually handles it. If it lasts, or you want to check:

1. Notice: the watch's email; responses whose Cache-Status shows REDIS-INSERTION-ERROR; or Redis connection errors in php's log.
2. Confirm: check the Redis pod's status and log with kubectl get pods and kubectl logs for <release>-redis. An OOMKilled reason in kubectl describe pod means it was full.
3. Fix: restart Redis, or raise CWA_CI_API_REDIS_MEMORY and deploy if it was full.
4. Clean up: if the watch hasn't flushed, press Purge all cached data in the site settings (see Page Caching › Purging All Cached Data), or run purge-http-cache in a php pod.
5. Check: Cache-Status no longer shows the error, and an edit shows on the live page.

What can go wrong

The database is the next ceiling. Each php pod runs 4 workers (php.workers), and each worker can hold a database connection, so pods × 4 must stay well under the database's connection limit. Past that, the database needs more resources or a connection pooler.

The HPA is too slow for a sudden spike. It takes up to a minute to see the CPU, then about half a minute for each pod, and minutes if the cluster needs a new node. For a launch you know about, raise CWA_CI_API_AUTOSCALE_MIN and deploy beforehand.

Identical misses can render once per pod. Souin merges identical requests for the same uncached item only within one pod, so with 3 pods the same item can render up to 3 times at once.

What it costs

On top of the default single pod, for each scaled release, at the minimum of 2 pods:

The HPA can add a third php pod at peak, and a warm surge adds pods for about ten minutes per deploy. The Redis watch requests 50m and 128Mi for a few seconds every 2 minutes.

What changes for your site's code

Most of this already matters with one pod, because pods restart:

• cache.app and your own cache pools are per pod. For data every pod must see, use the database (Symfony's DoctrineDbalAdapter). Never use the cache's Redis: a flush deletes everything in it.
• Local files (var/, /tmp, a local Flysystem adapter) are per pod. Use Cloud Storage.
• Sessions: the template's firewalls are stateless. A stateful firewall or flash messages need a session handler in the database.
• Rate limiters and login throttling need a store the pods share, such as the database.
• Scheduled or one-off work belongs in a CronJob or a Job, never in the entrypoint, postStart or a timer inside php.
• Anything you added to the entrypoint after migrations runs on every pod. Make it safe to run more than once at the same time.
• kubectl exec deploy/… in your project.sh reaches one pod. That's fine for the shared cache or the database, and wrong for anything held in one pod's memory.

Testing it

The browser tests can run against scaled mode (dd7ffe9). E2E_SCALED=true sh e2e/run.sh adds e2e/compose.scaled.yaml: two php replicas behind one php service, sharing a Redis and the hub, with the chart's store settings. Docker's DNS spreads the browser's and Nuxt's connections across the replicas, and the two replicas starting together also exercise the migration lock. In CI, set CWA_CI_E2E_SCALED="true". This is how to reproduce something that works on one pod and breaks on two. See Browser Tests › Running them locally.

Local development stays on one php pod.

What has been checked so far

The rehearsal on preview.cwa.rocks, 2026-10-20, with CWA_CI_API_SCALED_PRODUCTION=true: two to three API pods on different nodes, Redis at 256Mi and the hub pod. The full results, with the commands, are in components-web-app #85 › Rehearsal on preview.cwa.rocks.

From the template's notes and changelog:

• Purges across pods: bin/test/souin-redis-shared-purge.sh runs in the unit-tests job. Two instances on one Redis store 200 pages under a shared tag at the same time, and a purge through one leaves none cached.
• Redis down (locally): requests are served uncached, and caching resumes by itself when Redis is back.
• The Redis watch (locally, with the Redis container paused): an entry cached before the pause was still a hit afterwards; the watch flushed, and the next request was stored afresh.
• Mercure (locally): an admin's draft update published through php reached a subscriber connected through php. With the hub stopped, saves still returned 200.
• Deploys on preview.cwa.rocks: the first scaled deploys found three problems, all fixed on main. The flush ran in the old, stopping pod instead of Redis; 8 of 27 warm requests failed on stopping pods; and Kubernetes refused a surge minimum (4) above the maximum (3).

Turning it off

Set the switch to "false", or delete it, and deploy. Also unset any CWA_CI_API_AUTOSCALE_MIN or _MAX above 1, or the deploy refuses. Helm removes Redis, the hub, the PDB and the Redis watch. php goes back to one pod with its own in-memory cache and its own hub, and that pod starts with an empty cache and warms it before its probes start. Nothing in Redis needs keeping.