Skip to main content

Running in CI

Run the small owned-browser suite on pull requests. It requires no external browser server: Hosanna global setup starts the debugger, hst run, Vite, and Chromium and waits for actual port/readiness signals.

The hosanna-ui-samples-public workflow is the source example for the commands below. Keep CI on the same package scripts developers use locally.

GitHub Actions example​

name: Regression tests

on:
  pull_request:
  push:
    branches: [main]

jobs:
  regression-web:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    permissions:
      contents: read
    env:
      HSC_LICENSE_KEY: ${{ secrets.HSC_LICENSE_KEY }}

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version-file: .nvmrc
          cache: npm

      - name: Install dependencies
        run: npm ci --no-audit --no-fund

      - name: Restore build-config secret overlays
        run: npx hst build-config:restore-secrets --quiet
        env:
          BUILD_CONFIG_SECRETS_DEV_BASE64: ${{ secrets.BUILD_CONFIG_SECRETS_DEV_BASE64 }}
          BUILD_CONFIG_SECRETS_PROD_BASE64: ${{ secrets.BUILD_CONFIG_SECRETS_PROD_BASE64 }}

      - name: Restore the pinned Hosanna SDK
        run: |
          if [ -z "${HSC_LICENSE_KEY}" ]; then
            echo "::error::HSC_LICENSE_KEY is required for the compiler and licensed SDK"
            exit 1
          fi
          npx hst sdk:install
          npx hst config source

      - name: Install Chromium and system dependencies
        run: npm run test:ui:install -- --with-deps

      - name: Run browser regression suite
        run: npm run test:ui:ci

      - name: Upload regression report and diagnostics
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: regression-web-${{ github.run_id }}
          path: test-results/
          if-no-files-found: warn
          retention-days: 14

Before copying this into another application, replace npm run test:ui:ci with its checked-in CI regression script and restore only the SDK and build-config overlays that application actually needs. The sample repository workflow may combine this job with lint/build/unit checks; its checked-in YAML is authoritative.

Runtime and dependency setup​

  • Read the Node version from .nvmrc or the repository's supported equivalent.
  • Use npm ci, not a floating install.
  • Cache npm's download cache through setup-node; do not cache generated build config containing secrets.
  • Install the Chromium revision matched by the locked Playwright dependency.
  • Let global setup start and stop the application. Do not add an arbitrary sleep or a second manually started Vite server.

Parallelism, retries, and timeouts​

A Hosanna Vitest project owns one application runtime and one fixed port set. Keep fileParallelism: false, maxWorkers: 1, and non-concurrent sequencing inside that project.

If CI sharding is required, give every shard a distinct management/app/Vite/browser port set and artifact root. The shared launcher does not allocate independent shard ports automatically.

Vitest owns test retry policy. Prefer no local retries and at most a small CI-only retry count after tests are deterministic; a retry must not replace state reset or readiness waits. Keep launch timeout (HS_TEST_TIMEOUT_MS/config.timeoutMs), Vitest hookTimeout, and testTimeout aligned with the slowest supported target.

The sample config uses one retry when CI is set and no retries locally. npm run test:ui:ci first runs the dedicated integration TypeScript check, then the desktop web suite.

Artifacts and failure status​

Upload the whole artifact directory with if: always() so reports survive both passing and failing test steps. A failed Vitest assertion still returns a non-zero command status; artifact upload must not mask it.

The artifact can include:

  • report.html and report.json;
  • named proof screenshots and state;
  • automatic failure screenshot, hierarchy, focus, debugger context, logs, and commands;
  • actual and diff PNGs from visual comparison;
  • full-run WebM, MP4, or MOV when enabled for a supported target.

After downloading the artifact, open <testRunId>/report.html, not the redirect if your artifact viewer rewrites relative paths.

Environment variables and secrets​

The deterministic hosanna-ui-samples-public browser suite should not require an external service or user credential. If an application does:

  • map the value with envSecret(...);
  • expose it only to the test step;
  • never echo it or put it in build-config source;
  • avoid video for a flow that renders the secret;
  • do not run secret-dependent jobs for untrusted fork pull requests.

Roku developer passwords and compiler/source access keys belong in CI secrets. Physical-device jobs also require an attached, reachable device and are usually scheduled or manually dispatched rather than required on every pull request.

Require the fast browser regression job for changes that affect runtime navigation, input, rendering, or sample rigs. Keep physical-device matrices separate and visible, but do not mislabel browser platform expressions as hardware certification.

For investigation steps, see Reports and Failure Investigation.

Reference workflow: hosanna-ui-samples-public/.github/workflows/ci.yml.

Talk to us