Faster Functional Tests

The template's functional tests job runs two tests. Almost all of its time is spent starting the container and Postgres, so there is nothing here to speed up. This page is for a project whose own suite has grown to hundreds of tests.

It is guidance, not a template default. Sharding a small suite only adds runner cost, and GitLab's parallel: count is fixed in the YAML, so it can't size itself to the suite.

Measure first

Two numbers tell you which fix you need:

• The reset's share of the run. Time FunctionalTestCase::setUp() in api/tests/Functional/FunctionalTestCase.php and compare it with the whole suite. In that project, rebuilding the schema was 66% of the run time, spread evenly across scenarios.
• The per-class distribution. run_test_functional writes JUnit XML to api/build/logs/phpunit/functional.xml. Each test class is a <testsuite> and each test a <testcase>, both with a time attribute. If a few classes take most of the time, fix those classes instead.

Reset the database without rebuilding the schema

The template's FunctionalTestCase drops and recreates the whole schema before every test. That project measured four ways to reset:

Truncate mode is a trap: the purger runs one statement per table, and each is its own transaction. Delete mode deletes in Doctrine's commit order, so foreign keys need no special handling. ORMPurger comes from doctrine/data-fixtures, which the template already installs through doctrine/doctrine-fixtures-bundle.

Build the schema once, for the first test, then purge. Keep the template's check that the database name contains test:

<?php

declare(strict_types=1);

namespace App\Tests\Functional;

use ApiPlatform\Test\ApiTestCase;
use Doctrine\Common\DataFixtures\Purger\ORMPurger;
use Doctrine\ORM\EntityManagerInterface;
use Doctrine\ORM\Tools\SchemaTool;

abstract class FunctionalTestCase extends ApiTestCase
{
    private static bool $schemaCreated = false;
    protected function setUp(): void
    {
        parent::setUp();
        self::bootKernel();

        $manager = self::getContainer()->get('doctrine')->getManager();
        \assert($manager instanceof EntityManagerInterface);
        $connection = $manager->getConnection();
        $database = $connection->getParams()['dbname'] ?? null;
        if (!\is_string($database) || !str_contains($database, 'test')) {
            throw new \RuntimeException(\sprintf('Functional tests drop the schema. Refusing to run against "%s": point DATABASE_URL at a database whose name contains "test".', $database));
        }

        if (!self::$schemaCreated) {            $connection->executeStatement('CREATE EXTENSION IF NOT EXISTS citext');
            $metadata = $manager->getMetadataFactory()->getAllMetadata();
            $schemaTool = new SchemaTool($manager);
            $schemaTool->dropSchema($metadata);
            $schemaTool->createSchema($metadata);
            self::$schemaCreated = true;
        } else {
            (new ORMPurger($manager))->purge(); // PURGE_MODE_DELETE is the default            $this->restartSequences($manager);        }
        $manager->clear();
    }

    private function restartSequences(EntityManagerInterface $manager): void
    {
        $connection = $manager->getConnection();
        $sequences = $connection->fetchFirstColumn(
            "SELECT quote_ident(sequence_schema) || '.' || quote_ident(sequence_name)
             FROM information_schema.sequences WHERE sequence_schema = current_schema()"
        );
        foreach ($sequences as $sequence) {
            $connection->executeStatement('ALTER SEQUENCE '.$sequence.' RESTART');
        }
    }
}

The static flag lives for the whole PHPUnit process, so each run (and each shard) builds the schema exactly once. Don't turn on PHPUnit's backupStaticProperties, which would reset it.

That project also tried DAMA\DoctrineTestBundle, which wraps each test in a rolled-back transaction. It was a new dependency for about 3% more, and it doesn't roll back sequences.

The sequence trap

A delete leaves sequences where they were, so a purged database hands out new ids. Any test that asserts an IRI such as /_api/…/1 then fails, depending on which tests ran before it.

• A sequence that Doctrine creates for the SEQUENCE id strategy is a standalone sequence that no column owns. TRUNCATE … RESTART IDENTITY only restarts sequences owned by the truncated tables' columns, so it leaves these alone and the ids drift. That project hit exactly this.
• Only an explicit ALTER SEQUENCE … RESTART for each sequence, as in restartSequences() above, keeps ids stable with either reset.

CWA's own entities use UUIDs, and the template's entities do too, so a project with no integer ids has no sequences to restart.

Shard the job by test

Once the reset is cheap, the fixed cost of each job dominates. At that point, split the suite across parallel jobs with GitLab's parallel: N. Each copy of the job gets its own postgres service, and CI_NODE_INDEX (from 1) and CI_NODE_TOTAL tell it which share to run:

functional tests:
  parallel: 4  script:
    - setup_test_db_environment
    - run_test_functional

PHPUnit has no built-in sharding, but it can list the suite and run part of it. vendor/bin/phpunit tests/Functional --list-test-ids prints one ID per test, and --test-id-filter-file <file> runs only the IDs in a file, one per line. Your shard script writes this job's share to a file, and run_test_functional passes it to phpunit.

• Shard by test, not by class. In that project one feature file held 100 of the 211 scenarios, so no split by file could balance it. Weight each test by its time in an earlier JUnit report (a data provider counts each data set), and assign the heaviest first, each to the lightest shard so far.
• An empty shard must exit 0 without starting PHPUnit. Then a shard that got nothing can't run the whole suite by mistake:

  [ -s "$SHARD_FILE" ] || { echo "No tests for shard $CI_NODE_INDEX"; exit 0; }
  

• Choose N from the fixed cost per job. It was about 81s there, so extra shards soon cost more runner minutes than they save. N=4 was the sweet spot. N=8 doubled the runner minutes to save about 55s.
• Share one vendor cache across shards. The template's functional tests job runs composer install and has no cache. If you add one, key it on api/composer.lock. A key that includes the job name or branch makes every shard of a new branch start cold.

Give each shard its own JUnit file name. GitLab merges the reports of parallel jobs in the test report.

Prove the order doesn't matter

A reset that leaves anything behind makes a test depend on the ones before it. Sharding changes that order, so check it before you trust the shards. Run the full suite:

• in its default order,
• with --order-by reverse,
• with --order-by random under several --random-order-seed values.

Results in that project

The runner minutes roughly halved as well.

See CI/CD for the rest of the pipeline.