Skip to main content

Troubleshooting and Best Practices

Start with the semantic failure and generated diagnostics. Most flaky application tests come from an unobserved transition, shared state, changing content, or a target assumption—not from a Vitest assertion.

Flaky waits and timing​

Symptom: a focus, screen, or control wait fails intermittently.

  • Assert the result of the previous action before issuing the next action.
  • Wait on hs.wait.forScreen, forFocus, forControl, forProperty, forData, forImage, or forIdle.
  • Add stableMs when state may briefly pass through the expected value.
  • Give every complex wait a description; it appears in timeout diagnostics.
  • Use hs.pause(ms, reason) only for an external system with no observable state.
  • Do not increase the entire suite timeout to hide one missing readiness condition.

Unstable or ambiguous selectors​

Symptom: no control matches, or multiple controls match the same ID.

  • Add a stable application id or explicit testId at the relevant control boundary.
  • Do not select by label text that changes with localization/editorial content.
  • Do not select a serialized child index or browser DOM path.
  • For collection content, use semantic row metadata and content IDs, then assert minimum counts rather than mutable order.
  • Treat an ambiguity error as an application test-contract issue; do not silently take the first result.

Animation and screenshot differences​

Symptom: state assertions pass but PNG comparison changes.

  • Wait for application hierarchy/focus stability and image readiness before capture.
  • Disable or complete animation through an application-supported state.
  • Keep fonts, Chromium revision, operating system, dimensions, and assets consistent.
  • Use target-specific baselines.
  • Mask only an explicitly dynamic bounded control.
  • Prefer a report proof over pixel comparison for live editorial, clock, random, or network-driven content.
  • Inspect baseline, actual, and .diff.png before updating.

Port conflicts and stale processes​

Symptom: the launcher reports occupied ports or an existing runtime lock.

  • Stop the other test or Hosanna dev run using that port set.
  • Check for a still-running Vitest/hst process after an interrupted debug session.
  • Configure a complete unique port set for concurrent suites or shards.
  • Do not delete a live lock to force two runs onto the same debugger port.

The launcher waits for actual Vite and application readiness; a long startup sleep will not fix a port collision or mismatched test-run handshake.

Missing browser or system libraries​

Symptom: Chromium cannot launch.

npx playwright install chromium

On supported Linux workers:

npx playwright install --with-deps chromium

Use the Playwright version in the lockfile. Reinstalling a different global browser version can create rendering and baseline differences.

Wrong application or startup timeout​

Symptom: startup waits for the expected app/test-run ID and times out.

  • Run the normal hst run command to verify the application builds.
  • Restore required ignored build-config secret overlays.
  • Verify appName and generated application ID.
  • Turn on HOSANNA_TEST_VERBOSE=1 and inspect the buffered launch/browser output.
  • Confirm the remote-debug and MCP test overlay is enabled for a non-production environment.
  • On physical Roku, verify LAN reachability, device IP, developer password, ECP, debug-log port, and HS_TEST_DEBUG_HOST if automatic host selection is wrong.

Capability mismatch​

Symptom: the client says a target does not support touch, text, deep links, or screenshots.

Use createHosannaTestTarget and test.runIf(...) for a truly target-specific scenario. Do not catch and ignore capability errors. See the current matrix in Devices, Browsers, and Input Methods.

Test independence​

  • Apply a default named precondition before every test.
  • Clear or seed only application-owned test state, then relaunch and wait.
  • Keep one worker/file sequence per owned runtime.
  • Do not rely on another test to create a selected item or navigation history.
  • Keep credentials and mutable remote accounts out of the default suite.
  • Clean host-side data in precondition/teardown helpers if the scenario creates it.
  • Verify that a single file and a single -t filtered test both pass alone.

Decide what to cover​

Good default-suite candidates are startup, primary navigation, focus restoration, back/history, one remote path, one supported touch path, key rendering/focus states, and a few valuable failure/edge states.

Keep these elsewhere:

  • exhaustive function/data branches: unit tests;
  • all service response permutations: service/contract tests;
  • rapidly changing editorial copy/order: structural minimum/readiness assertions;
  • native rendering or hardware input claims: physical-device validation;
  • low-value screenshots of every screen: omit them.

Keep the default suite fast​

  • Launch the app once per Vitest run.
  • Reset through lightweight named preconditions.
  • Capture proofs only at decision points.
  • Keep visual assertions opt-in or limited to high-value stable states.
  • Disable video during rapid local authoring when the project supports it.
  • Run a small browser suite on pull requests and broader/device matrices on schedule.
  • Remove redundant UI cases already protected by lower-level tests.

Return to Regression Testing Overview or review Writing and Running Tests.

Talk to us