Browser Tests

The template has opt-in browser tests, in e2e/. They take the php and app images this branch just built, start them in a throwaway stack, and use the site in Chromium the way visitors and editors do. When they're on, every deploy waits for them, so a build that breaks rendering a page or signing in never reaches a site.

The template's e2e/README.md is the short version of this page.

What they check

The template's tests assume only the admin user and the pages its sitemap lists. They don't use the template's sample content, so a project can keep e2e/ unchanged and the tests pass on its own site. The media checks run only when the project has the template's Image component.

They cover:

• every page in the sitemap renders and hydrates;
• the first page in the sitemap is served from the cache on its second request;
• the loading bar shows while a page loads after a client-side navigation, from the first sitemap page to the second;
• the first sitemap page's OG image renders;
• an admin can log in and out;
• a password reset email links to a working reset page.

So the sitemap needs at least two pages, and its first page needs an og:image tag, or those tests fail.

Editing and publishing content isn't tested here: the module and the bundle test that.

A run also fails when:

• an API write gets a 405, 406 or 415, which means the front end and the API disagree about a method or a content type;
• the page throws an uncaught error;
• a cache or media check fails (see The test stack);
• the API sent an error mail (Kubernetes & Helm › Error mail), meaning it answered a request with a 5xx that the UI may have hidden;
• not every listed test ran and passed (check-all-ran.mjs), so a skipped or filtered-out test can't pass a run.

Every timeout is bounded: 90 seconds a test, 30 seconds a navigation, 15 seconds an action and 10 seconds an assertion.

Turning them on

Set the CI variable CWA_CI_E2E_ENABLED to "true" (see CI/CD Variables › Pipeline flags). Nothing runs until you do. CWA_CI_TEST_DISABLED="true" skips them along with the other tests.

• GitLab: a browser tests job in the test stage, after build api and build app. The review, staging, canary and production jobs needs: it, so none of them runs when it fails. It runs Docker in Docker (docker:27-dind).
• GitHub: the reusable workflow .github/workflows/browser-tests.yml. ci.yml calls it after the builds, before review and staging, and production.yml calls it on main's images before the deploy. A failed run stops those deploys. It has a 30-minute limit.

The Playwright report, with traces, screenshots and videos of failed tests, is kept as a job artifact for 14 days. GitLab also reads its JUnit file, so the results show in the pipeline's test report.

The test stack

e2e/run.sh boots the single-server production configuration: the project's compose.yaml and compose.prod.yaml (see Docker › Production Docker Compose), with e2e/compose.yaml on top adding only what a test run needs:

• the branch's php and app production images (frankenphp_prod and prod) in place of builds, so a run tests the images CI built and never builds its own;
• PostgreSQL on tmpfs, so every run starts empty;
• a Cloud Storage emulator, reached through php's GCLOUD_API_ENDPOINT, used only when you ask for it (see Media storage in the tests);
• Mailpit, which catches every email, such as the password reset;
• the Playwright runner, built from e2e/Dockerfile.

Every production setting comes from compose.prod.yaml, so a change there that breaks the site fails the tests too. run.sh sets the values it requires, with SITE_HOST=localhost, a new set of secrets and a new JWT key pair each run, and unsets every other variable the compose files read, so nothing from your shell, the CI job or a local .env reaches the stack.

It waits for the stack to be healthy (migrations run on boot), loads the fixtures, flushes the HTTP cache and waits for the site. Then, before any browser test, it checks the cache on the production configuration. e2e/cache-checks.sh runs in the php container and prints one line for each check:

• the API answers over HTTPS, and over the plain-HTTP http://php.local route the Nuxt server renders with, without a redirect;
• every page in the sitemap is a 200, stored on its first view and a hit on its second;
• when the sitemap lists two or more pages, purging one cache tag (Surrogate-Key) of the first page that the second doesn't have, a component's own tag when there is one, renders the first page again, while the second page stays a hit. Each page has its own route tag, so there is always one. A one-page sitemap skips this check;
• purge-rendered-html renders every page again, while the API stays cached.

Then run.sh checks that Caddy's admin API (port 2019) can't be reached from the app container, and, when the project has the Image component (api/src/Entity/Image.php), runs the media checks in e2e/media-checks.sh:

• the admin signs in and uploads a PNG to the Image component;
• the original is served as image/png, and each image variant as image/webp;
• run.sh recreates the php container, and every one of those files is still served.

Then it flushes the cache again so the browser tests start cold. A failed check fails the run, with FAILED: and what it wanted on the check's line.

So every page in the sitemap must be cacheable for a visitor: a page that is never stored fails the run. The checks need at least one page in the sitemap. Without the Image component, run.sh prints skipped: media checks (no Image component in this project) and goes on.

run.sh relies on the project's own compose.prod.yaml, so copy e2e/ together with compose.prod.yaml, api/frankenphp/docker-entrypoint.sh, api/frankenphp/site-hosts.sh, api/frankenphp/Caddyfile, api/config/services.php, api/src/Flysystem/ and api/Dockerfile when you take the tests into a project.

After the tests, it stops and deletes the stack.

The stack publishes no ports, so it can run beside your dev stack.

Each checkout gets its own stack, named cwa-e2e-<directory> after the checkout's directory, and its own tests container. So two projects can run their tests on one machine at once, and one run's clean-up never removes another's containers or volumes. COMPOSE_PROJECT_NAME sets a different name.

Media storage in the tests

By default the stack stores media on the server, as compose.prod.yaml does (MEDIA_STORAGE=local, in its media_data volume; see Docker › Media on one server). That's why recreating php must keep the uploaded files.

To run the same tests with media in a bucket, as Kubernetes deploys store it, set CWA_CI_E2E_STORAGE="gcloud" in CI, or E2E_STORAGE=gcloud locally. php then stores media in the Cloud Storage emulator. GCLOUD_API_ENDPOINT is the only change this needs in the API: GoogleCloudStorageFactory uses an emulator, with no credentials, when it's set. Leave it unset on every deploy, where empty means Google.

The stack also sets a review app's storage path prefix, GCLOUD_PATH_PREFIX: review/e2e/. After the tests, run.sh checks that every stored file is under it, in the bucket or in public/uploads, then runs the same guarded delete-stored-files a stopped review app runs and checks nothing is left. Either failure fails the run. The check covers the media checks' upload and whatever your fixtures and your own tests store.

Running them locally

You need Docker and openssl. Build the two production images, then run the tests against them:

docker build --target frankenphp_prod -t cwa-e2e-php:local api
docker build --target prod -t cwa-e2e-app:local app
E2E_PHP_IMAGE=cwa-e2e-php:local E2E_APP_IMAGE=cwa-e2e-app:local sh e2e/run.sh

The reports land in e2e/playwright-report/ and e2e/test-results/. When a run fails, the script prints the last 80 lines of the php and app logs.

Writing your own tests

The template's tests check only what every CWA site has. Your project's own specs are where its pages and content get tested: add them under e2e/tests/, as *.spec.ts files.

import { expect, logIn, test } from './fixtures'

test('an admin sees the contact page, and so does a visitor', async ({ page, visitor }) => {
  await logIn(page)
  await page.goto('/contact')
  await expect(page.getByRole('button', { name: 'Edit' })).toBeVisible()

  await visitor.goto('/contact')
  await expect(visitor.getByRole('heading', { name: 'Contact us', exact: true })).toBeVisible()
})

• Import test and expect from tests/fixtures.ts, never from @playwright/test. Its page fixture is what fails a test on a refused API write or an uncaught error. Importing types from @playwright/test is fine.
• logIn(page) signs in as the admin. The password is made for each run and loaded with the fixtures. It waits for /login to hydrate before it fills the form, because before that the form submits natively and reloads /login.
• Use the visitor fixture to see a page as an anonymous visitor, beside the signed-in page. It has its own browser context, is checked the same way as page, and is closed after the test.
• Tests share one database and run one at a time. A test that edits content should use its own unique text, such as one with Date.now() in it, so it can't match another test's.

Type checks, lint and conventions

The tests are type-checked and linted before they run, and every rule is an error, so a type or lint error fails the browser tests job. Playwright only strips types, so nothing else would catch them.

• tsc, with the strict e2e/tsconfig.json;
• ESLint, with e2e/eslint.config.mjs: @nuxt/eslint-config's stylistic rules (the module's style), eslint-plugin-playwright, and typescript-eslint's no-floating-promises, which also catches a missing await on your own helpers.

Run them yourself in e2e/. npm ci also gives your editor the types:

cd e2e
npm ci
npm run typecheck
npm run lint
npm run lint:fix

The rules come with these conventions, from the template's e2e/README.md:

• Await every Playwright call and every helper that returns a promise.
• Use retrying assertions: await expect(locator).toBeVisible(), not expect(await locator.isVisible()).
• No fixed waits and no forced clicks. waitForTimeout and force: true are errors. Wait for the response, URL or element the test is about.
• Find elements the way a person does: getByRole, getByLabel or getByText, with exact names. Use a CSS selector only where the markup gives nothing better, with a comment that says why.