API testing without a browser page: key checks (401), the shipment JSON contract, exact quote money, and the trial key's rate limit (429 with Retry-After). Reports every broken promise in one run.
apidemofreightsecurity version: 1.0 DEMO-020__freight_api_contract.py group: Freight platformgroup: Python and TypeScript, side by side
Runs
8
Pass rate
75%
6 passed / 2 failed
Avg duration
1s
p95 2s
Estimate
1s
avg of last 10 judged runs
Flakiness
0.29
0 stable · 1 alternates
Current streak
4
passed
Duration per run point color = outcome; dashed = 7-run average; click a point to open the run
Step timing trends the same step compared across the last 8 runs — slowest steps first, one line each
How long it takes distribution of 8 runs — two humps mean two different behaviours hiding behind one average
fastest 1s · slowest 2s
Why this test failed grouped by message, last 90 days
| times | message | last | |
|---|---|---|---|
| 2 | AssertionError: the API broke its promises: | Sep 24 | open |
Outcomes by target version
What this test does plain language, derived from the code
- api = Freight(page, ctx).url("api/v1/")
- key = {"X-API-Key": ctx.secret("freight_api_key", DEMO_KEY)}
- problems = []
- ⏱ Timed phase: auth: no key
- r = page.request.get(api + "shipments/AF-100001")
- if r.status != 401:
- problems.append(f"auth: a request with no key should get 401, got {r.status}")
- ⏱ Timed phase: GET a shipment
- r = page.request.get(api + "shipments/AF-100001", headers=key)
- if r.status != 200:
- problems.append(f"GET shipments/AF-100001 answered {r.status}: {r.text()[:200]}")
- else:
- body = r.json()
- for field, kind in CONTRACT.items():
- if field not in body:
- problems.append(f"contract: field '{field}' is missing "
- f"(the response has: {', '.join(sorted(body))})")
- elif not isinstance(body[field], kind):
- problems.append(f"contract: '{field}' should be {kind}, got {body[field]!r}")
- ⏱ Timed phase: POST a quote with SPRING10
- r = page.request.post(api + "quotes", headers=key, data={
- "origin": "San Francisco", "destination": "San Diego",
- "weight_kg": 120, "service": "standard", "promo": "SPRING10"})
- if r.status != 200:
- problems.append(f"POST quotes answered {r.status}: {r.text()[:200]}")
- else:
- quote = r.json()
- subtotal, discount = float(quote["subtotal"]), float(quote["discount"])
- if abs(discount - round(subtotal * 0.10, 2)) > 0.011:
- problems.append(f"money: SPRING10 took ${discount:.2f} off a ${subtotal:.2f} "
- f"subtotal -- 10% is ${subtotal * 0.10:.2f}")
- ⏱ Timed phase: rate limit: 6 calls on the trial key
- answers = [page.request.get(api + "shipments/AF-100002",
- headers={"X-API-Key": TRIAL_KEY}) for _ in range(6)]
- codes = [a.status for a in answers]
- if codes[-1] != 429:
- problems.append(f"limits: the trial key allows 5 calls a minute, but 6 in a row "
- f"answered {codes}")
- elif not answers[-1].headers.get("retry-after"):
- problems.append("limits: a 429 must say when to retry (Retry-After header)")
- ctx.log(f"{len(problems)} broken promise(s)")
- assert not problems, "the API broke its promises:\n - " + "\n - ".join(problems)
Show the code
"""API testing, beside the browser tests: no page to look at, just requests
and JSON -- the promises a partner integration depends on.
auth no key -> 401; a valid key -> 200
contract a shipment carries the fields clients read (id, status, eta ...)
money an API quote with SPRING10 takes exactly 10% off
limits the trial key allows 5 calls a minute; the 6th gets 429 with a
Retry-After header
It checks EVERYTHING and reports every broken promise together (soft
assertions): when a release breaks the API three ways, you want to read all
three in one run, not fix-rerun-fix-rerun. page.request is Playwright's HTTP
client -- same cookies and proxy as the browser, no page load needed.
The key comes from Settings -> Credentials ("freight_api_key") when set, so
a real key never has to be written into a test file.
"""
from _lib.freight import Freight
DEMO_KEY = "acme-demo-7f3a91c2" # published on the demo's developers page
TRIAL_KEY = "acme-trial-2b91e4d0" # 5 requests per minute
CONTRACT = {"id": str, "status": str, "origin": str, "destination": str,
"service": str, "weight_kg": (int, float), "eta": str, "events": list}
def run(page, ctx):
api = Freight(page, ctx).url("api/v1/")
key = {"X-API-Key": ctx.secret("freight_api_key", DEMO_KEY)}
problems = []
with ctx.timed("auth: no key"):
r = page.request.get(api + "shipments/AF-100001")
if r.status != 401:
problems.append(f"auth: a request with no key should get 401, got {r.status}")
with ctx.timed("GET a shipment"):
r = page.request.get(api + "shipments/AF-100001", headers=key)
if r.status != 200:
problems.append(f"GET shipments/AF-100001 answered {r.status}: {r.text()[:200]}")
else:
body = r.json()
for field, kind in CONTRACT.items():
if field not in body:
problems.append(f"contract: field '{field}' is missing "
f"(the response has: {', '.join(sorted(body))})")
elif not isinstance(body[field], kind):
problems.append(f"contract: '{field}' should be {kind}, got {body[field]!r}")
with ctx.timed("POST a quote with SPRING10"):
r = page.request.post(api + "quotes", headers=key, data={
"origin": "San Francisco", "destination": "San Diego",
"weight_kg": 120, "service": "standard", "promo": "SPRING10"})
if r.status != 200:
problems.append(f"POST quotes answered {r.status}: {r.text()[:200]}")
else:
quote = r.json()
subtotal, discount = float(quote["subtotal"]), float(quote["discount"])
if abs(discount - round(subtotal * 0.10, 2)) > 0.011:
problems.append(f"money: SPRING10 took ${discount:.2f} off a ${subtotal:.2f} "
f"subtotal -- 10% is ${subtotal * 0.10:.2f}")
with ctx.timed("rate limit: 6 calls on the trial key"):
answers = [page.request.get(api + "shipments/AF-100002",
headers={"X-API-Key": TRIAL_KEY}) for _ in range(6)]
codes = [a.status for a in answers]
if codes[-1] != 429:
problems.append(f"limits: the trial key allows 5 calls a minute, but 6 in a row "
f"answered {codes}")
elif not answers[-1].headers.get("retry-after"):
problems.append("limits: a 429 must say when to retry (Retry-After header)")
ctx.log(f"{len(problems)} broken promise(s)")
assert not problems, "the API broke its promises:\n - " + "\n - ".join(problems)
All runs
| Run | Status | Queued | Duration | Version | Trigger | Batch |
|---|---|---|---|---|---|---|
| #471 | passed | 2026-09-25 00:30:11 | 1s | 3.0.0 | group | group: Python and TypeScript,… |
| #397 | passed | 2026-09-24 23:34:49 | 1s | 3.0.0 | group | group: Freight platform |
| #360 | passed | 2026-09-24 22:04:44 | 1s | 3.0.0 | group | group: Freight platform |
| #329 | passed | 2026-09-24 19:47:27 | 1s | 3.0.0 | cli-adopted | 3 tests (terminal) |
| #326 | failed | 2026-09-24 19:46:19 | 2s | 2.1.0 | cli-adopted | 3 tests (terminal) |
| #323 | failed | 2026-09-24 19:45:49 | 1s | 2.0.0 | cli-adopted | 3 tests (terminal) |
| #320 | passed | 2026-09-24 19:45:20 | 1s | 1.1.0 | cli-adopted | 3 tests (terminal) |
| #317 | passed | 2026-09-24 19:44:24 | 2s | 1.0.0 | cli-adopted | 3 tests (terminal) |