Browser Tests

The template has opt-in browser tests, in e2e/. They take the php and app images this branch just built, start them in a throwaway stack, and use the site in Chromium the way visitors and editors do. When they're on, every deploy waits for them, so a build that breaks logging in or publishing never reaches a site (#157, 3fd2c01, 8c36f8c). Since template 2.0.0-alpha.34 · release notes

The template's e2e/README.md is the short version of this page.

What they check

The template's tests cover:

• every page in the sitemap renders and hydrates;
• a public page is served from the cache on its second request;
• a redirecting route answers with a 308;
• the loading bar shows while a page loads after a client-side navigation;
• a page's OG image renders;
• an admin can log in and out;
• an admin can edit and publish a text component, and an anonymous visitor then sees it;
• an admin can upload and publish an image, and a visitor then sees it;
• a password reset email links to a working reset page.

A run also fails when:

• an API write gets a 405, 406 or 415, which means the front end and the API disagree about a method or a content type;
• the page throws an uncaught error;
• the API sent an error mail, meaning it answered a request with a 5xx that the UI may have hidden;
• not every listed test ran and passed (check-all-ran.mjs), so a skipped or filtered-out test can't pass a run.

Every timeout is bounded: 90 seconds a test, 30 seconds a navigation, 15 seconds an action and 10 seconds an assertion.

Turning them on

Set the CI variable CWA_CI_E2E_ENABLED to "true" (see CI/CD Variables → Pipeline flags). Nothing runs until you do. CWA_CI_TEST_DISABLED="true" skips them along with the other tests.

• GitLab: a browser tests job in the test stage, after build api and build app. The review, staging, canary and production jobs needs: it, so none of them runs when it fails. It runs Docker in Docker (docker:27-dind).
• GitHub: the reusable workflow .github/workflows/browser-tests.yml. ci.yml calls it after the builds, before review and staging, and production.yml calls it on main's images before the deploy. A failed run stops those deploys. It has a 30-minute limit.

The Playwright report, with traces, screenshots and videos of failed tests, is kept as a job artifact for 14 days. GitLab also reads its JUnit file, so the results show in the pipeline's test report.

The test stack

e2e/run.sh starts e2e/compose.yaml, which has:

• the branch's php and app production images (frankenphp_prod and prod);
• PostgreSQL on tmpfs, so every run starts empty;
• a Cloud Storage emulator for uploads, reached through php's GCLOUD_API_ENDPOINT;
• Mailpit, which catches every email, such as the password reset;
• the Playwright runner, built from e2e/Dockerfile.

It makes new secrets and a new JWT key pair for each run, waits for the API (migrations run on boot), loads the fixtures, flushes the HTTP cache and waits for the site. Then it runs the tests one at a time, and stops and deletes the stack.

The stack publishes no ports, so it can run beside your dev stack.

GCLOUD_API_ENDPOINT is the only change the tests need in the API: GoogleCloudStorageFactory uses an emulator, with no credentials, when it's set. Leave it unset on every deploy, where empty means Google.

Running them locally

You need Docker and openssl. Build the two production images, then run the tests against them:

docker build --target frankenphp_prod -t cwa-e2e-php:local api
docker build --target prod -t cwa-e2e-app:local app
E2E_PHP_IMAGE=cwa-e2e-php:local E2E_APP_IMAGE=cwa-e2e-app:local sh e2e/run.sh

The reports land in e2e/playwright-report/ and e2e/test-results/. When a run fails, the script prints the last 80 lines of the php and app logs.

Writing your own tests

Add your project's own flows under e2e/tests/, as *.spec.ts files.

import { expect, logIn, test } from './fixtures'

test('an admin sees the contact page, and so does a visitor', async ({ page, visitor }) => {
  await logIn(page)
  await page.goto('/contact')
  await expect(page.getByRole('button', { name: 'Edit' })).toBeVisible()

  await visitor.goto('/contact')
  await expect(visitor.getByRole('heading', { name: 'Contact us', exact: true })).toBeVisible()
})

• Import test and expect from tests/fixtures.ts, never from @playwright/test. Its page fixture is what fails a test on a refused API write or an uncaught error. Importing types from @playwright/test is fine.
• logIn(page) signs in as the admin. The password is made for each run and loaded with the fixtures.
• Use the visitor fixture to see a page as an anonymous visitor, beside the signed-in page. It has its own browser context, is checked the same way as page, and is closed after the test. On main after template 2.0.0-alpha.35, not yet released (43ece66)
• Tests share one database and run one at a time. A test that edits content should use its own unique text, such as one with Date.now() in it, so it can't match another test's.

The template's editing test publishes straight after the last keystroke. Since @cwa/nuxt 2.0.0-alpha.16, Publish sends the waiting save first (cwa-nuxt-module#394), so the test checks that this still works.

Type checks, lint and conventions

The tests are type-checked and linted before they run, and every rule is an error, so a type or lint error fails the browser tests job. Playwright only strips types, so nothing else would catch them. On main after template 2.0.0-alpha.35, not yet released (43ece66, 08b40db)

• tsc, with the strict e2e/tsconfig.json;
• ESLint, with e2e/eslint.config.mjs: @nuxt/eslint-config's stylistic rules (the module's style), eslint-plugin-playwright, and typescript-eslint's no-floating-promises, which also catches a missing await on your own helpers.

Run them yourself in e2e/. npm ci also gives your editor the types:

cd e2e
npm ci
npm run typecheck
npm run lint
npm run lint:fix

The rules come with these conventions, from the template's e2e/README.md:

• Await every Playwright call and every helper that returns a promise.
• Use retrying assertions: await expect(locator).toBeVisible(), not expect(await locator.isVisible()).
• No fixed waits and no forced clicks. waitForTimeout and force: true are errors. Wait for the response, URL or element the test is about.
• Find elements the way a person does: getByRole, getByLabel or getByText, with exact names. Use a CSS selector only where the markup gives nothing better, with a comment that says why.