Testing Strategy

Threat model: untested and mis-tested code

Every merge without adequate tests is a gamble. Regressions slip through review; authorization bugs hide behind happy-path E2E; performance cliffs surface only under production load. Security teams cannot sign off on artifacts whose behavior was never verified. The pyramid exists to allocate finite CI minutes and developer attention where they catch the most defects per dollar.

The inverted pyramid anti-pattern

Teams under schedule pressure often pile UI-driven Selenium suites on top of a shallow unit base—the ice-cream cone or inverted pyramid. Every release waits on a two-hour E2E job; failures are ambiguous (was it the UI, the API, the DB seed, or a flaky network?); developers re-run until green or merge with [skip ci]. Security regressions in business logic never get E2E coverage because combinatorics explode.

Swiss cheese model of defense

No single test layer catches everything—think Swiss cheese: each layer has holes; stacked layers reduce the chance a defect reaches production. Unit tests hole-punch on mocked I/O; integration tests miss UI routing; E2E misses rare branch logic. Add SAST/SCA (Security Scanning) as another slice. Pipeline design maps layers to stages so a failure at any slice blocks promotion.

flowchart TB
  subgraph pyramid["Test pyramid → CI stages"]
    U["Unit tests\n(every commit, < 3 min)"]
    I["Integration\n(PR + main, Testcontainers)"]
    E["E2E smoke\n(post-deploy staging)"]
    P["Performance budget\n(nightly / release)"]
  end
  U --> I --> E --> P
  subgraph gates["Pipeline gates"]
    G1["Coverage ≥ 80%"]
    G2["Pact verify"]
    G3["Playwright smoke"]
    G4["k6 p95 < 300ms"]
  end
  U -.-> G1
  I -.-> G2
  E -.-> G3
  P -.-> G4

Pipeline integration: layer → stage mapping

Align pyramid layers with CI pipeline stages. Unit tests run on every push with concurrency cancel; integration runs after build artifact exists; E2E runs against deployed staging with the same image digest that passed earlier gates—not a rebuild.

name: Test Pyramid
on: [pull_request, push]
jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run test:unit -- --maxWorkers=4
  integration:
    needs: unit
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env: { POSTGRES_PASSWORD: test }
        ports: ['5432:5432']
    steps:
      - uses: actions/checkout@v4
      - run: npm run test:integration
  e2e-smoke:
    needs: integration
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - run: npx playwright test --grep @smoke

stages: [unit, integration, e2e]

unit-test:
  stage: unit
  script: [npm ci, npm run test:unit]
  artifacts:
    reports:
      junit: junit-unit.xml

integration-test:
  stage: integration
  services: [postgres:16]
  script: [npm run test:integration]
  needs: [unit-test]

e2e-smoke:
  stage: e2e
  script: [npx playwright test --grep @smoke]
  needs: [integration-test]
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Threat → solution

Without unit tests, every change is validated only by compilation and manual click-through. Input sanitization regressions, off-by-one authorization checks, and broken HMAC verification ship silently. The solution is a dense unit suite that runs in under three minutes on every commit, with coverage and mutation gates that prove tests actually assert behavior—not just execute lines.

Fast tests: rules of thumb

• No real network, DB, or sleep—inject clocks and fake repos.
• Parallelize: pytest -n auto, Jest --maxWorkers, Surefire parallel.
• Keep fixtures in memory; use factories (factory_boy, @testFactory) not 10 MB JSON blobs.
• Fail on slow tests: pytest --durations=20 in CI to surface offenders > 500 ms.

Mocks: when they help and when they lie

Mocks stub collaborators so you test one class in isolation. Over-mocking creates tests that pass while production fails— the mock returns success but the real HTTP client times out. Prefer fakes (in-memory repo) over mocks for persistence; reserve mocks for external SDKs you do not own.

# tests/unit/test_token_service.py
from unittest.mock import Mock
from app.auth.token_service import TokenService

def test_verify_rejects_expired_token():
    clock = Mock()
    clock.now.return_value = 1_700_000_000
    svc = TokenService(clock=clock, secret="test-secret")
    assert svc.verify("expired.jwt.here") is False

Coverage thresholds

Line coverage alone is a vanity metric—100% coverage with empty assertions proves nothing. Still, a minimum threshold (typically 80% line / 70% branch) prevents large untested dumps from merging. Enforce differential coverage on changed files for monorepos: only the diff must meet the bar.

Mutation testing: Pitest & Stryker

Mutation frameworks inject small bugs (flip > to >=, delete a guard) and check if tests fail. High line coverage with low mutation score means tests are hollow. Run mutation nightly—not every PR—it's CPU-heavy.

Pipeline integration: coverage gate

name: Unit + Coverage Gate
on: pull_request
jobs:
  unit-java:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { distribution: temurin, java-version: 21 }
      - run: mvn -B test jacoco:report
      - name: Enforce JaCoCo minimum
        run: |
          mvn jacoco:check -Djacoco.haltOnFailure=true \
            -Djacoco.rules.coverage=0.80
      - uses: actions/upload-artifact@v4
        with:
          name: jacoco-xml
          path: target/site/jacoco/jacoco.xml

  unit-js:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run test:unit -- --coverage
      - name: Istanbul threshold
        run: |
          node -e "
            const s=require('./coverage/coverage-summary.json').total;
            if(s.lines.pct<80) process.exit(1);
          "

unit-test:
  stage: test
  image: eclipse-temurin:21
  script:
    - mvn -B test jacoco:report jacoco:check
  coverage: '/Total.*?([0-9]{1,3})%/'
  artifacts:
    reports:
      junit: target/surefire-reports/TEST-*.xml
      coverage_report:
        coverage_format: jacoco
        path: target/site/jacoco/jacoco.xml

unit-js:
  stage: test
  image: node:20
  script:
    - npm ci
    - npm run test:unit -- --coverage --coverageReporters=cobertura
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

$ pytest tests/unit -q --cov=app --cov-fail-under=80
$ mvn test -Dtest="*UnitTest"
$ npm run test:unit -- --watchAll=false
$ mvn org.pitest:pitest-maven:mutationCoverage -DtargetClasses=com.acme.auth.*

Threat model

Unit tests with mocked repositories never validate SQL migrations, connection pool exhaustion, or JSON field renames across services. Deploying untested integration paths causes silent data corruption, 500s on first real traffic, and cross-team blame when "your service changed the schema." Consumer-driven contracts prevent the classic microservices trap: provider ships breaking API; consumer discovers it in production Friday night.

Testcontainers pattern

Testcontainers spins real container images per test class or suite, maps random ports, and tears down after JVM exit. Use singleton containers for speed (one Postgres for entire suite) with careful state reset between tests.

@Testcontainers
@SpringBootTest
class OrderRepositoryIT {
  @Container
  static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:16")
      .withDatabaseName("orders_test");

  @DynamicPropertySource
  static void props(DynamicPropertyRegistry r) {
    r.add("spring.datasource.url", postgres::getJdbcUrl);
    r.add("spring.datasource.username", postgres::getUsername);
    r.add("spring.datasource.password", postgres::getPassword);
  }

  @Autowired OrderRepository repo;

  @Test void persistsOrderWithIdempotencyKey() {
    var order = repo.save(new Order("idem-abc", BigDecimal.TEN));
    assertThat(repo.findByIdempotencyKey("idem-abc")).contains(order);
  }
}

WireMock for outbound dependencies

When you cannot run the real payment gateway in CI, WireMock simulates latency, error codes, and payload shapes. Record production traffic once (sanitized), commit stubs to VCS, verify stubs match OpenAPI spec in lint stage.

Pact: consumer-driven contracts

The consumer writes a pact file describing expected request/response. CI publishes to the Pact Broker. The provider runs verify pact against deployed or in-process API—breaking changes fail before merge. Enable can-i-deploy to block deploy when consumer/provider versions are incompatible.

sequenceDiagram
  participant C as Consumer CI
  participant B as Pact Broker
  participant P as Provider CI
  participant CD as Deploy gate
  C->>C: Run consumer tests → pact.json
  C->>B: Publish pact (v1.2.3+abc123)
  P->>B: Fetch pending pacts
  P->>P: Verify against provider API
  P->>B: Record verification result
  CD->>B: can-i-deploy? consumer + provider + env
  B-->>CD: OK / BLOCK

Pipeline integration

name: Integration + Pact
on: [pull_request, push]
jobs:
  integration:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { distribution: temurin, java-version: 21 }
      - run: docker pull postgres:16
      - run: mvn -B verify -Pintegration
      - uses: actions/upload-artifact@v4
        with:
          name: junit-integration
          path: target/failsafe-reports/*.xml

  pact-consumer:
    needs: integration
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm run test:pact
      - name: Publish to Pact Broker
        run: |
          npx pact-broker publish ./pacts \
            --consumer-app-version=${{ github.sha }} \
            --broker-base-url=${{ secrets.PACT_BROKER_URL }} \
            --broker-token=${{ secrets.PACT_BROKER_TOKEN }}

  pact-provider-verify:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - run: |
          npx pact-broker verify \
            --provider payments-api \
            --broker-base-url=$PACT_BROKER_URL \
            --provider-base-url=https://staging.internal/api

integration-test:
  stage: test
  image: docker:24
  services: [docker:24-dind]
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - docker pull postgres:16
    - mvn -B verify -Pintegration
  artifacts:
    reports:
      junit: target/failsafe-reports/TEST-*.xml

pact-publish:
  stage: test
  image: node:20
  script:
    - npm run test:pact
    - npx pact-broker publish ./pacts
        --consumer-app-version=$CI_COMMIT_SHA
        --broker-base-url=$PACT_BROKER_URL
        --broker-token=$PACT_BROKER_TOKEN
  needs: [integration-test]

pact-verify:
  stage: test
  script:
    - pact-broker verify --provider payments-api
        --broker-base-url=$PACT_BROKER_URL
        --provider-base-url=$STAGING_API_URL
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Threat model

Integration tests prove services talk correctly; they do not prove the login button works after a CSS refactor or that TLS termination routes /api to the right backend. E2E gaps manifest as revenue-impacting outages— checkout broken, MFA loop, admin panel exposed without auth. Conversely, a bloated E2E suite blocks releases and trains teams to ignore red builds.

Smoke tests post-deploy

After staging deploy, run a @smoke tagged subset: login, create resource, read resource, logout—under two minutes. Gate production promotion on smoke green. Use the same container digest tested in CI, deployed to staging via GitOps.

// e2e/smoke/auth.spec.ts
import { test, expect } from '@playwright/test';

test.describe('@smoke', () => {
  test('user can sign in and reach dashboard', async ({ page }) => {
    await page.goto('/login');
    await page.getByLabel('Email').fill(process.env.E2E_USER!);
    await page.getByLabel('Password').fill(process.env.E2E_PASS!);
    await page.getByRole('button', { name: 'Sign in' }).click();
    await expect(page).toHaveURL(/dashboard/);
    await expect(page.getByRole('heading', { name: 'Overview' })).toBeVisible();
  });
});

Synthetic monitoring

Production synthetics (Checkly, Datadog Synthetics, Grafana k6 cloud) hit critical paths every few minutes from multiple regions. They are E2E tests with SLAs—alert on-call when p95 exceeds budget, not when a developer's laptop sleep broke local WebDriver. Store credentials in vault; rotate synthetic user passwords like any service account.

Visual regression

Percy, Chromatic, or Playwright snapshot compare screenshots against baselines—catch unintended UI drift. Run on design-system PRs and marketing pages; avoid snapshotting dynamic charts with live data (false positives). Store baselines in Git LFS or vendor cloud; review diffs in PR checks.

name: E2E Smoke (post-deploy)
on:
  workflow_dispatch:
    inputs:
      staging_url: { required: true }
  deployment_status:
jobs:
  smoke:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm ci && npx playwright install --with-deps chromium
      - name: Run smoke against staging
        env:
          BASE_URL: ${{ github.event.deployment_status.environment_url }}
          E2E_USER: ${{ secrets.E2E_USER }}
          E2E_PASS: ${{ secrets.E2E_PASS }}
        run: npx playwright test --grep @smoke --retries=0
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-traces
          path: test-results/

e2e-smoke:
  stage: deploy-verify
  image: mcr.microsoft.com/playwright:v1.42.0-jammy
  variables:
    BASE_URL: https://staging.example.com
  script:
    - npm ci
    - npx playwright test --grep @smoke --retries=0
  artifacts:
    when: on_failure
    paths: [test-results/, playwright-report/]
  needs: [deploy-staging]
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Threat model

A green functional suite hides N+1 queries, lock contention, and GC pauses that only appear at scale. One ORM change can double database round-trips; one unbounded cache can OOM the pod under Black Friday traffic. Without automated performance gates, regressions reach production and become emergency tuning sprints—while security patches wait in queue.

Performance budgets

Define budgets per endpoint or user journey: p95 latency < 300 ms, error rate < 0.1%, throughput ≥ 500 RPS at 50 VUs. Store baselines in Git; fail CI when regression exceeds 10% without approved exception. Align budgets with SLOs—CI thresholds should be stricter than prod alerts to catch drift early.

// perf/smoke.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 20,
  duration: '60s',
  thresholds: {
    http_req_duration: ['p(95)<300'],
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/api/health`);
  check(res, { 'status 200': (r) => r.status === 200 });
  sleep(0.3);
}

JMH for JVM hot paths

JMH benchmarks serialization, hashing, and regex engines in isolated JVM forks. Run only when relevant paths change— full JMH suite can take 30+ minutes. Compare against stored baseline JSON; flag >5% regression.

Pipeline integration: threshold gates

name: Performance Gate
on:
  pull_request:
    paths: ['src/**', 'perf/**']
  schedule:
    - cron: '0 3 * * *'
jobs:
  k6-smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: grafana/[email protected]
        with:
          filename: perf/smoke.js
        env:
          BASE_URL: https://staging.example.com
          K6_CLOUD_TOKEN: ${{ secrets.K6_CLOUD_TOKEN }}
      - name: Assert thresholds (k6 exits non-zero on breach)
        run: echo "k6-action fails job if thresholds exceeded"

  gatling-nightly:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { distribution: temurin, java-version: 21 }
      - run: mvn -B gatling:test
      - uses: actions/upload-artifact@v4
        with:
          name: gatling-report
          path: target/gatling/**/index.html

stages: [test, perf]

k6-smoke:
  stage: perf
  image: grafana/k6:latest
  script:
    - k6 run perf/smoke.js
  variables:
    BASE_URL: https://staging.example.com
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_PIPELINE_SOURCE == "schedule"

gatling-load:
  stage: perf
  image: eclipse-temurin:21
  script:
    - mvn -B gatling:test
  artifacts:
    paths: [target/gatling/]
    expire_in: 7 days
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

jmh-benchmark:
  stage: perf
  script:
    - mvn -Pjmh package
    - java -jar target/benchmarks.jar -rf json -rff jmh-results.json
  artifacts:
    paths: [jmh-results.json]
  rules:
    - changes: [src/main/java/com/acme/crypto/**]

Threat model

When test output lives only in ephemeral CI logs, developers skip reading them; managers assume green checkbox means tested; auditors cannot prove regression suites ran before a SOC2 sampling period. Missing reports also hide skipped security tests— a job that exits 0 because zero tests executed still looks successful.

JUnit XML: the lingua franca

pytest (--junitxml), Jest (jest-junit), Go (-json → converter), and Maven Surefire all emit JUnit-compatible XML. GitLab ingests natively into MR Test widget; GitHub uses third-party or dorny/test-reporter for PR annotations.

GitHub annotations

Publish test failures inline on PR diffs—developers fix without opening CI logs. Pair with actions/upload-artifact for Playwright traces and Allure ZIP on failure.

name: Test Report
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          npm ci
          npm run test:ci -- --reporters=default --reporters=jest-junit
        env:
          JEST_JUNIT_OUTPUT_DIR: ./reports
          JEST_JUNIT_OUTPUT_NAME: junit.xml
      - name: Publish test report
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: Unit Tests
          path: reports/junit.xml
          reporter: jest-junit
      - name: Upload Allure
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: allure-results
          path: allure-results/
      - name: JaCoCo to Codecov
        uses: codecov/codecov-action@v4
        with:
          files: ./coverage/jacoco.xml
          token: ${{ secrets.CODECOV_TOKEN }}

test:
  stage: test
  script:
    - pytest --junitxml=report.xml --cov=app --cov-report=xml
    - generate-allure-report || true
  coverage: '/TOTAL.*\s+(\d+%)$/'
  artifacts:
    when: always
    expire_in: 14 days
    paths:
      - report.xml
      - coverage.xml
      - allure-report/
    reports:
      junit: report.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

Allure for narrative debugging

Allure attaches screenshots, HTTP logs, and step hierarchies—essential for E2E failure triage. Generate on every run; publish HTML to GitLab Pages or S3 on main. Do not store Allure on every PR forever— expire_in: 7 days on artifacts.

JaCoCo & Istanbul in merge gates

Upload coverage to SonarQube or Codecov for diff coverage comments. Block merge when overall or new-code coverage drops. JaCoCo XML path: target/site/jacoco/jacoco.xml; Istanbul: coverage/lcov.info or Cobertura export for GitLab native widget.

$ pytest --junitxml=report.xml --cov=app --cov-report=html
$ allure serve allure-results/
$ open target/site/jacoco/index.html
$ npx nyc report --reporter=lcov

Threat model: trust erosion

Flaky tests are a supply-chain vulnerability for process integrity. The attack does not require a hacker—teams internalize "CI is broken again" and merge with admin override, skip failing jobs, or disable checks in branch protection. A real SQL injection finding in SAST gets lumped with another random E2E timeout and ignored. Measuring flake rate (failures that pass on retry without code change) is as important as coverage percentage.

Quarantine workflow

1. Detect — track retry-success rate per test ID in CI analytics (Buildkite Test Engine, GitLab Flaky Test report).
2. Quarantine — move to @quarantine tag; exclude from merge-blocking suite; file ticket with owner.
3. SLA — fix within 5 business days or delete the test (no infinite quarantine).
4. Verify — run quarantined tests 50× in isolation before returning to blocking suite.
5. Postmortem — if flake rate > 2% of runs, pause feature work for stability sprint.

flowchart TD
  F[Flaky failure detected] --> R{Retry passed?}
  R -->|Yes| Q[Quarantine test\nnon-blocking]
  R -->|No| B[Blocking failure\nfix normally]
  Q --> T[Jira ticket + owner]
  T --> FIX[Root cause fix]
  FIX --> SOAK[50x soak run]
  SOAK -->|pass| UNQ[Restore to blocking]
  SOAK -->|fail| Q
  Q --> SLA{SLA exceeded?}
  SLA -->|Yes| DEL[Delete or rewrite test]

Retry anti-pattern

CI-level --retries=3 on every test hides flakiness and multiplies pipeline duration. Acceptable only for known infra blips (container pull) with metrics exported—never as permanent policy. Jest flaky: true and Playwright retries should be zero on merge-blocking smoke; use separate quarantine job with retries for investigation.

Pipeline integration: detect without normalizing

name: Flake Detection
on:
  schedule:
    - cron: '0 4 * * 1-5'
jobs:
  soak-unit:
    runs-on: ubuntu-latest
    strategy:
      matrix: { run: [1,2,3,4,5,6,7,8,9,10] }
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run test:unit -- --retries=0

  blocking-smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright test --grep @smoke --grep-invert @quarantine --retries=0
      # No retries — failures are real or quarantine candidate

test-blocking:
  stage: test
  script:
    - pytest -m "not quarantine" --retries=0
  artifacts:
    reports:
      junit: report.xml

test-quarantine:
  stage: test
  allow_failure: true
  script:
    - pytest -m quarantine -v
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

# GitLab 16+ flaky test detection surfaces in MR widget automatically
# when junit history shows inconsistent results