Skip to content

Tests

Tests in Qaack allow you to automatically validate your environment during the build process. Internally, Qaacks spawns a container with Playwright and other needed services to later collect the results.

For a full reference of how to define tests in your configuration file, see the Qaackfile Reference.

Test Types

Screenshots

Captures visual snapshots of specific pages in your environment.

  • Useful settings:
    • Define paths to capture (defaults to /).
  • Use case: Quick visual verification that the layout isn't broken.

Info

Settings are declared in the Qaackfile

Playwright

Runs end-to-end tests using the Playwright framework. This feature is designed to allow the developers use it with a minimal setup. Qaack officially support a set of versions that are pre-downloaded, so the there is no installation time nor any extra storage use.

Info

We invite you to adjust your Playwright code and dependencies to the oficially supported versions and we do not consider a good practice to install other versions.

  • Useful settings:
    • dir: The directory containing your package.json and Playwright tests (default: test/playwright).
    • prepare: The name of the prepare script to run before running the tests (default: prepare.sh).
  • Dependencies:
    • By default, Qaack runs npm install in the specified directory.
    • If a prepare.sh (or configured prepare script) exists in the directory, it will be executed instead of the default installation command. Use this for custom setup steps.
  • .npmrc:
    • To handle private registries or authentication, create a file-type variable named QCK_TESTER_PLAYWRIGHT_NPMRC in your repository or group settings. Qaack will automatically inject this file into the test directory.
  • Reporting:
    • Qaack automatically configures Playwright to use json and html reporters.
    • The HTML report is captured and available as a downloadable archive in the panel.
  • Playwright versions available:
    • 1.55
    • 1.56
    • 1.57
  • Node environment: The tester runs on Node.js 24 (LTS), ensuring compatibility with modern tools like Lighthouse 12.

Lighthouse

Performs a Lighthouse audit to measure performance, accessibility, best practices, and SEO.

  • Useful settings: You can set a threshold (default: 80). If any category score falls below this value, the test execution will be marked as Failed.

Visual Regression

Compares screenshots of a Merge Request (MR) or Pull Request (PR) environment against its source branch (baseline).

  • How it works:
    1. Captures screenshots in the baseline environment.
    2. Captures screenshots in the MR/PR environment.
    3. Performs a pixel-by-pixel comparison.
  • Threshold: A similarity percentage (e.g., 99) can be set. If the similarity is lower than the threshold, the test fails.
  • Heatmaps: Comparison results include heatmaps highlighting the differences between the images.

Important

For Visual Regression to work, the baseline environment (usually the source branch like develop or main) must have already been built and the corresponding tests executed so that "baseline" images are available for comparison.

Caveats and Pitfalls

  • Base URL: You don't need to hardcode the environment URL in your tests. Qaack automatically updates the baseUrl in your playwright.config.ts (if found) to match the dynamic environment URL.
  • Ephemeral Storage: Any files created by your tests inside the tester service will be lost once the service is deleted. Always use the built-in reporting mechanisms.
  • Memory Limits: Heavy end-to-end test suites may require more resources. if you experience crashes, consider splitting tests or optimizing your suite.