🧪 Test Hub
testing: Demo target v3.0.0 · built-in demo site

DEMO-020 Freight: public API contract, auth and limits

Edit Load test Duplicate

API testing without a browser page: key checks (401), the shipment JSON contract, exact quote money, and the trial key's rate limit (429 with Retry-After). Reports every broken promise in one run.

apidemofreightsecurity version: 1.0 DEMO-020__freight_api_contract.py group: Freight platformgroup: Python and TypeScript, side by side

Runs
8
Pass rate
75%
6 passed / 2 failed
Avg duration
1s
p95 2s
Estimate
1s
avg of last 10 judged runs
Flakiness
0.29
0 stable · 1 alternates
Current streak
4
passed

Last 8 results newest on the right — click a square to open that run

Duration per run point color = outcome; dashed = 7-run average; click a point to open the run

Step timing trends the same step compared across the last 8 runs — slowest steps first, one line each

How long it takes distribution of 8 runs — two humps mean two different behaviours hiding behind one average

fastest 1s · slowest 2s

Why this test failed grouped by message, last 90 days

timesmessagelast
2 AssertionError: the API broke its promises: Sep 24 open

Outcomes by target version

What this test does plain language, derived from the code

  1. api = Freight(page, ctx).url("api/v1/")
  2. key = {"X-API-Key": ctx.secret("freight_api_key", DEMO_KEY)}
  3. problems = []
  4. ⏱ Timed phase: auth: no key
  5. r = page.request.get(api + "shipments/AF-100001")
  6. if r.status != 401:
  7. problems.append(f"auth: a request with no key should get 401, got {r.status}")
  8. ⏱ Timed phase: GET a shipment
  9. r = page.request.get(api + "shipments/AF-100001", headers=key)
  10. if r.status != 200:
  11. problems.append(f"GET shipments/AF-100001 answered {r.status}: {r.text()[:200]}")
  12. else:
  13. body = r.json()
  14. for field, kind in CONTRACT.items():
  15. if field not in body:
  16. problems.append(f"contract: field '{field}' is missing "
  17. f"(the response has: {', '.join(sorted(body))})")
  18. elif not isinstance(body[field], kind):
  19. problems.append(f"contract: '{field}' should be {kind}, got {body[field]!r}")
  20. ⏱ Timed phase: POST a quote with SPRING10
  21. r = page.request.post(api + "quotes", headers=key, data={
  22. "origin": "San Francisco", "destination": "San Diego",
  23. "weight_kg": 120, "service": "standard", "promo": "SPRING10"})
  24. if r.status != 200:
  25. problems.append(f"POST quotes answered {r.status}: {r.text()[:200]}")
  26. else:
  27. quote = r.json()
  28. subtotal, discount = float(quote["subtotal"]), float(quote["discount"])
  29. if abs(discount - round(subtotal * 0.10, 2)) > 0.011:
  30. problems.append(f"money: SPRING10 took ${discount:.2f} off a ${subtotal:.2f} "
  31. f"subtotal -- 10% is ${subtotal * 0.10:.2f}")
  32. ⏱ Timed phase: rate limit: 6 calls on the trial key
  33. answers = [page.request.get(api + "shipments/AF-100002",
  34. headers={"X-API-Key": TRIAL_KEY}) for _ in range(6)]
  35. codes = [a.status for a in answers]
  36. if codes[-1] != 429:
  37. problems.append(f"limits: the trial key allows 5 calls a minute, but 6 in a row "
  38. f"answered {codes}")
  39. elif not answers[-1].headers.get("retry-after"):
  40. problems.append("limits: a 429 must say when to retry (Retry-After header)")
  41. ctx.log(f"{len(problems)} broken promise(s)")
  42. assert not problems, "the API broke its promises:\n - " + "\n - ".join(problems)
Show the code
"""API testing, beside the browser tests: no page to look at, just requests
and JSON -- the promises a partner integration depends on.

  auth      no key -> 401; a valid key -> 200
  contract  a shipment carries the fields clients read (id, status, eta ...)
  money     an API quote with SPRING10 takes exactly 10% off
  limits    the trial key allows 5 calls a minute; the 6th gets 429 with a
            Retry-After header

It checks EVERYTHING and reports every broken promise together (soft
assertions): when a release breaks the API three ways, you want to read all
three in one run, not fix-rerun-fix-rerun. page.request is Playwright's HTTP
client -- same cookies and proxy as the browser, no page load needed.

The key comes from Settings -> Credentials ("freight_api_key") when set, so
a real key never has to be written into a test file.
"""
from _lib.freight import Freight

DEMO_KEY = "acme-demo-7f3a91c2"          # published on the demo's developers page
TRIAL_KEY = "acme-trial-2b91e4d0"        # 5 requests per minute
CONTRACT = {"id": str, "status": str, "origin": str, "destination": str,
            "service": str, "weight_kg": (int, float), "eta": str, "events": list}


def run(page, ctx):
    api = Freight(page, ctx).url("api/v1/")
    key = {"X-API-Key": ctx.secret("freight_api_key", DEMO_KEY)}
    problems = []

    with ctx.timed("auth: no key"):
        r = page.request.get(api + "shipments/AF-100001")
    if r.status != 401:
        problems.append(f"auth: a request with no key should get 401, got {r.status}")

    with ctx.timed("GET a shipment"):
        r = page.request.get(api + "shipments/AF-100001", headers=key)
    if r.status != 200:
        problems.append(f"GET shipments/AF-100001 answered {r.status}: {r.text()[:200]}")
    else:
        body = r.json()
        for field, kind in CONTRACT.items():
            if field not in body:
                problems.append(f"contract: field '{field}' is missing "
                                f"(the response has: {', '.join(sorted(body))})")
            elif not isinstance(body[field], kind):
                problems.append(f"contract: '{field}' should be {kind}, got {body[field]!r}")

    with ctx.timed("POST a quote with SPRING10"):
        r = page.request.post(api + "quotes", headers=key, data={
            "origin": "San Francisco", "destination": "San Diego",
            "weight_kg": 120, "service": "standard", "promo": "SPRING10"})
    if r.status != 200:
        problems.append(f"POST quotes answered {r.status}: {r.text()[:200]}")
    else:
        quote = r.json()
        subtotal, discount = float(quote["subtotal"]), float(quote["discount"])
        if abs(discount - round(subtotal * 0.10, 2)) > 0.011:
            problems.append(f"money: SPRING10 took ${discount:.2f} off a ${subtotal:.2f} "
                            f"subtotal -- 10% is ${subtotal * 0.10:.2f}")

    with ctx.timed("rate limit: 6 calls on the trial key"):
        answers = [page.request.get(api + "shipments/AF-100002",
                                    headers={"X-API-Key": TRIAL_KEY}) for _ in range(6)]
    codes = [a.status for a in answers]
    if codes[-1] != 429:
        problems.append(f"limits: the trial key allows 5 calls a minute, but 6 in a row "
                        f"answered {codes}")
    elif not answers[-1].headers.get("retry-after"):
        problems.append("limits: a 429 must say when to retry (Retry-After header)")

    ctx.log(f"{len(problems)} broken promise(s)")
    assert not problems, "the API broke its promises:\n  - " + "\n  - ".join(problems)

All runs

RunStatusQueuedDuration VersionTriggerBatch
#471 passed 2026-09-25 00:30:11 1s 3.0.0 group group: Python and TypeScript,…
#397 passed 2026-09-24 23:34:49 1s 3.0.0 group group: Freight platform
#360 passed 2026-09-24 22:04:44 1s 3.0.0 group group: Freight platform
#329 passed 2026-09-24 19:47:27 1s 3.0.0 cli-adopted 3 tests (terminal)
#326 failed 2026-09-24 19:46:19 2s 2.1.0 cli-adopted 3 tests (terminal)
#323 failed 2026-09-24 19:45:49 1s 2.0.0 cli-adopted 3 tests (terminal)
#320 passed 2026-09-24 19:45:20 1s 1.1.0 cli-adopted 3 tests (terminal)
#317 passed 2026-09-24 19:44:24 2s 1.0.0 cli-adopted 3 tests (terminal)