🧪 Test Hub
testing: Demo target v3.0.0 · built-in demo site

DEMO-017 Freight: visual check of the home page

Edit Load test Duplicate

Screenshot comparison (OpenCV + scikit-image) against a blessed baseline: tiny changes pass, a redesign fails with a red-marked image of what moved. Delete DEMO-017__baseline.png in the tests folder to re-bless after an intended change.

demofreightimagevisual version: 1.0 DEMO-017__freight_visual_check.py group: Freight regression

Runs
9
Pass rate
89%
8 passed / 1 failed
Avg duration
2s
p95 3s
Estimate
2s
avg of last 10 judged runs
Flakiness
0.25
0 stable · 1 alternates
Current streak
6
passed

Last 9 results newest on the right — click a square to open that run

Duration per run point color = outcome; dashed = 7-run average; click a point to open the run

Step timing trends the same step compared across the last 9 runs — slowest steps first, one line each

How long it takes distribution of 9 runs — two humps mean two different behaviours hiding behind one average

fastest 1s · slowest 3s

Why this test failed grouped by message, last 90 days

timesmessagelast
1 AssertionError: the home page looks different: similarity 0.6117 (min… Sep 24 open

Outcomes by target version

What this test does plain language, derived from the code

  1. page.goto(ctx.base_url + "freight/")
  2. page.wait_for_load_state("networkidle") # the map is an image: let it land
  3. shot = ctx.artifacts_dir / "screenshots" / "current.png"
  4. shot.parent.mkdir(parents=True, exist_ok=True)
  5. page.screenshot(path=str(shot))
  6. baseline = Path(__file__).with_name(f"{ctx.test_id}__baseline.png")
  7. if not baseline.exists():
  8. baseline.write_bytes(shot.read_bytes())
  9. ctx.log(f"no baseline yet -- saved this run as the baseline ({baseline.name}). "
  10. f"Later runs compare against it.")
  11. return
  12. current, expected = cv2.imread(str(shot)), cv2.imread(str(baseline))
  13. assert current is not None and expected is not None, "could not read the images"
  14. if current.shape != expected.shape:
  15. raise AssertionError(
  16. f"the page size changed: baseline {expected.shape[1]}x{expected.shape[0]}, "
  17. f"now {current.shape[1]}x{current.shape[0]}")
  18. score, diff = ssim(cv2.cvtColor(expected, cv2.COLOR_BGR2GRAY),
  19. cv2.cvtColor(current, cv2.COLOR_BGR2GRAY), full=True)
  20. changed = float(np.mean((diff < 0.9).astype(np.float32)))
  21. ctx.log(f"similarity {score:.4f}; {changed * 100:.2f}% of pixels differ")
  22. marked = current.copy()
  23. marked[(255 - (diff * 255)).astype("uint8") > 60] = (0, 0, 255)
  24. cv2.imwrite(str(shot.with_name("02-what-changed.png")), marked)
  25. assert score >= SSIM_MIN and changed <= DIFF_MAX, (
  26. f"the home page looks different: similarity {score:.4f} (min {SSIM_MIN}), "
  27. f"{changed * 100:.1f}% of pixels changed (max {DIFF_MAX * 100:.0f}%). "
  28. f"02-what-changed.png in this run's screenshots marks it in red. If the "
  29. f"change is intended, delete {baseline.name} to re-bless the new look.")
Show the code
"""Does Acme Freight still LOOK right? Screenshot vs. a blessed baseline.

The first run saves a baseline next to this test (DEMO-017__baseline.png);
every later run compares the home page against it with OpenCV +
scikit-image and fails when too much moved -- leaving a copy of the page
with the changed areas painted red, so a person sees at a glance WHAT moved.

A small change (the version number in the header) stays under the
threshold; a redesign does not. After an INTENDED redesign, delete the
baseline file to re-bless the new look (the next run saves a fresh one).
"""
from pathlib import Path

SSIM_MIN = 0.97      # 1.0 = identical
DIFF_MAX = 0.02      # fraction of pixels allowed to differ


def run(page, ctx):
    page.goto(ctx.base_url + "freight/")
    page.wait_for_load_state("networkidle")        # the map is an image: let it land
    shot = ctx.artifacts_dir / "screenshots" / "current.png"
    shot.parent.mkdir(parents=True, exist_ok=True)
    page.screenshot(path=str(shot))

    baseline = Path(__file__).with_name(f"{ctx.test_id}__baseline.png")
    if not baseline.exists():
        baseline.write_bytes(shot.read_bytes())
        ctx.log(f"no baseline yet -- saved this run as the baseline ({baseline.name}). "
                f"Later runs compare against it.")
        return

    import cv2
    import numpy as np
    from skimage.metrics import structural_similarity as ssim

    current, expected = cv2.imread(str(shot)), cv2.imread(str(baseline))
    assert current is not None and expected is not None, "could not read the images"
    if current.shape != expected.shape:
        raise AssertionError(
            f"the page size changed: baseline {expected.shape[1]}x{expected.shape[0]}, "
            f"now {current.shape[1]}x{current.shape[0]}")
    score, diff = ssim(cv2.cvtColor(expected, cv2.COLOR_BGR2GRAY),
                       cv2.cvtColor(current, cv2.COLOR_BGR2GRAY), full=True)
    changed = float(np.mean((diff < 0.9).astype(np.float32)))
    ctx.log(f"similarity {score:.4f}; {changed * 100:.2f}% of pixels differ")

    marked = current.copy()
    marked[(255 - (diff * 255)).astype("uint8") > 60] = (0, 0, 255)
    cv2.imwrite(str(shot.with_name("02-what-changed.png")), marked)

    assert score >= SSIM_MIN and changed <= DIFF_MAX, (
        f"the home page looks different: similarity {score:.4f} (min {SSIM_MIN}), "
        f"{changed * 100:.1f}% of pixels changed (max {DIFF_MAX * 100:.0f}%). "
        f"02-what-changed.png in this run's screenshots marks it in red. If the "
        f"change is intended, delete {baseline.name} to re-bless the new look.")

All runs

RunStatusQueuedDuration VersionTriggerBatch
#492 passed 2026-09-25 07:00:10 2s 3.0.0 schedule schedule: Freight regression …
#372 passed 2026-09-24 22:05:21 2s 3.0.0 group group: Freight regression
#350 passed 2026-09-24 20:41:32 2s 3.0.0 cli-adopted DEMO-017
#314 passed 2026-09-24 18:59:58 2s 3.0.0 cli-adopted 8 tests (terminal)
#306 passed 2026-09-24 18:57:59 2s 2.1.0 cli-adopted 8 tests (terminal)
#298 passed 2026-09-24 18:57:54 1s 2.0.0 cli-adopted DEMO-017
#294 failed 2026-09-24 18:55:40 2s 2.0.0 cli-adopted 8 tests (terminal)
#155 passed 2026-09-24 18:52:00 3s 1.1.0 cli-adopted 8 tests (terminal)
#17 passed 2026-09-24 18:48:19 2s 1.0.0 cli-adopted 8 tests (terminal)