Test an OpenRTB bidder with fixtures in CI

What the test checks

The OpenRTB validation in CI guide covers checking request fixtures: install the CLI, read its exit codes, pin a spec snapshot. That protects whoever builds requests. A bidder needs a different test, one aimed at what it sends back. For each recorded request, this one asserts four things:

1. The fixture itself is a valid bid request, so the test input is trustworthy.
2. The bidder answers with the status code the fixture expects.
3. A bid passes validation against the request that produced it.
4. A bid has no warnings either, because the bidder controls every field in it.

Lay out the fixtures

Put each fixture in a folder named for the outcome you expect. The folder is the assertion, so adding a test case means dropping a file in the right place:

.github/workflows/bidder-fixtures.yml
scripts/test-bidder.sh
fixtures/
  bid/
    banner-300x250.json
    ctv-video-15s.json
  no-bid/
    blocked-adomain.json
    floor-above-cap.json
    tmax-too-short.json
  invalid/
    missing-imp.json
  malformed/
    truncated.json
server.mjs
package.json
package-lock.json

• bid/: valid requests the bidder must bid on. Expect 200 and a clean BidResponse.
• no-bid/: valid requests the bidder must pass on. Expect an empty 204. The three samples cover a floor above the price cap, an advertiser domain in BidRequest.badv, and a BidRequest.tmax shorter than the network reserve.
• invalid/: requests that fail validation. Expect 200 with BidResponse.nbr 2 (Invalid Request).
• malformed/: bodies that are not JSON. Expect 400.

The expectations match the Node.js bidder tutorial. If your bidder answers invalid requests with 400 instead of nbr, change that one rule in the script. Download the files:

• bidder-fixtures.yml: the GitHub Actions workflow (save it as .github/workflows/bidder-fixtures.yml)
• scripts/test-bidder.sh: the test script
• Fixtures: bid/banner-300x250.json, bid/ctv-video-15s.json, no-bid/blocked-adomain.json, no-bid/floor-above-cap.json, no-bid/tmax-too-short.json, invalid/missing-imp.json, malformed/truncated.json
• README.md: layout and local run instructions

Record fixtures from real traffic

Hand-written requests test the cases you thought of. Captured requests test the ones exchanges actually send: odd sizes, missing optional objects, extensions you never read. Log a small sample of raw request bodies in production, then pick the interesting ones.

Strip personal data before anything reaches the repository. The bidder does not need IP addresses, device advertising ids, precise location, or user ids to decide its outcome on a fixture. This jq filter removes them, and --summary then checks the whole capture in one pass:

$ jq -c 'del(.device.ip, .device.ipv6, .device.ifa, .device.geo.lat, .device.geo.lon,
          .user.id, .user.buyeruid, .user.eids, .user.geo)' captured/*.json > captured.ndjson
$ rtblint validate --summary captured.ndjson
6 payload(s): 5 valid, 1 invalid
1 error(s), 0 warning(s)

Rule frequencies:
       1  error     openrtb.field.required
$ echo $?
1

The summary shows how often each rule fired across the capture. Requests that pass become candidates for bid/ or no-bid/. A request that fails is a candidate for invalid/, and also a finding to raise with the exchange that sent it. After scrubbing, run each chosen fixture through the bidder once by hand to confirm the folder you picked matches what it should do.

Validate a response against its request

--type response tells the CLI the payload is a BidResponse, and --request supplies the bid request it answers. With both, RTBlint checks the response on its own and then cross-checks it: every Bid.impid must name an impression in the request, Bid.mtype and the markup in Bid.adm must match a format that impression offers, the response id must echo the request id, and deal ids, seats, and currency must be ones the request allowed.

$ rtblint validate --type response --request fixtures/bid/banner-300x250.json out/banner-300x250.response.json
OK (OpenRTB 2.6-202606 bid response): no issues found.
$ echo $?
0

$ jq '.seatbid[0].bid[0].mtype = 2' out/banner-300x250.response.json > bad.json
$ rtblint validate --type response --request fixtures/bid/banner-300x250.json bad.json
FAILED (OpenRTB 2.6-202606 bid response): 1 error(s), 1 warning(s).
- [warning] seatbid[0].bid[0].adm: mtype 2 declares video markup (VAST XML), but adm has no VAST or DAAST document root. (openrtb.bid.adm.vast_root_missing) · spec 4.2.3
- [error] seatbid[0].bid[0].mtype: mtype 2 declares video markup, but imp "1" does not offer a video subtype. (openrtb.bid.mtype_not_offered) · spec 4.2.3
$ echo $?
1

$ rtblint validate --type response --request fixtures/bid/banner-300x250.json missing.json
Failed to read missing.json: No such file or directory (os error 2)
$ echo $?
2

The exit codes are the contract the script builds on: 0 when the payload is valid (warnings allowed), 1 on spec errors, and 2 when the CLI cannot run at all, such as a missing file or an unknown --version. Exit 2 must never count as a pass or as an ordinary failure. It means the test itself is broken.

Why warnings fail a bidder build

A request fixture with warnings came from someone else. A response with warnings came from your code, and each warning names something you can fix. Here is a bid whose Bid.nurl lost the price macro:

$ rtblint validate --type response --request fixtures/bid/banner-300x250.json nomacro.json
OK with warnings (OpenRTB 2.6-202606 bid response): 1 warning(s).
- [warning] seatbid[0].bid[0].nurl: nurl or burl is present without ${AUCTION_PRICE}; a billing notice cannot report the clearing price. (openrtb.bid.price_macro_missing) · spec OpenRTB 2.6 §4.4
$ echo $?
0

The CLI exits 0, so an exit-code gate would ship it. The bidder would then receive win notices without a clearing price. The openrtb.bid.price_macro_missing rule exists for exactly this. The script reads the JSON report and fails on any issue, not only on errors.

The test script

#!/usr/bin/env bash
# Replays recorded bid requests against a running bidder and checks each
# answer. The folder a fixture sits in says what the bidder must do:
#   fixtures/bid/        valid request, expect 200 and a clean BidResponse
#   fixtures/no-bid/     valid request, expect 204
#   fixtures/invalid/    request that fails validation, expect 200 with nbr 2
#   fixtures/malformed/  body that is not JSON, expect 400
# Usage: BIDDER_URL=http://localhost:8080/bid bash scripts/test-bidder.sh [fixtures]
set -euo pipefail

FIXTURES="${1:-fixtures}"
BIDDER_URL="${BIDDER_URL:-http://localhost:8080/bid}"
SPEC="${SPEC:-2.6-202606}"
OUT="${OUT:-out}"
mkdir -p "$OUT"
shopt -s nullglob
failures=0

pass() { echo "ok    $1"; }
fail() { echo "FAIL  $1: $2"; failures=$((failures + 1)); }

# rtblint exits 0 when valid, 1 on spec errors, 2 when it cannot run.
# Exit 2 stops the script: a broken check must not look like a pass.
lint() {
  local status=0
  rtblint validate --version "$SPEC" --format json "$@" >"$OUT/report.json" || status=$?
  if [ "$status" -eq 2 ]; then
    echo "rtblint could not run: rtblint validate $*" >&2
    exit 2
  fi
  return "$status"
}

# Posts a fixture, writes the body to $2, prints the HTTP status.
post() {
  curl -s -o "$2" -w '%{http_code}' -X POST "$BIDDER_URL" \
    -H 'content-type: application/json' -H 'x-openrtb-version: 2.6' \
    --data-binary "@$1"
}

for req in "$FIXTURES"/bid/*.json; do
  name="bid/$(basename "$req")"
  resp="$OUT/$(basename "$req" .json).response.json"
  lint "$req" || { fail "$name" "fixture is not a valid request"; continue; }
  status="$(post "$req" "$resp")"
  [ "$status" = 200 ] || { fail "$name" "expected 200, got $status"; continue; }
  if ! lint --type response --request "$req" "$resp"; then
    fail "$name" "response fails validation against its request"
    jq -c '.issues[]' "$OUT/report.json"
    continue
  fi
  # Your own responses should be clean, so warnings fail the build too.
  if ! jq -e '.issues == []' "$OUT/report.json" >/dev/null; then
    fail "$name" "response has warnings"
    jq -c '.issues[]' "$OUT/report.json"
    continue
  fi
  pass "$name"
done

for req in "$FIXTURES"/no-bid/*.json; do
  name="no-bid/$(basename "$req")"
  lint "$req" || { fail "$name" "fixture is not a valid request"; continue; }
  status="$(post "$req" "$OUT/body")"
  if [ "$status" = 204 ] && [ ! -s "$OUT/body" ]; then
    pass "$name"
  else
    fail "$name" "expected an empty 204, got $status"
  fi
done

for req in "$FIXTURES"/invalid/*.json; do
  name="invalid/$(basename "$req")"
  if lint "$req"; then
    fail "$name" "fixture passes validation, so it tests nothing"
    continue
  fi
  status="$(post "$req" "$OUT/body")"
  if [ "$status" = 200 ] && jq -e '.nbr == 2' "$OUT/body" >/dev/null 2>&1 &&
    lint --type response "$OUT/body"; then
    pass "$name"
  else
    fail "$name" "expected 200 with a valid nbr 2 response, got $status"
  fi
done

for req in "$FIXTURES"/malformed/*; do
  name="malformed/$(basename "$req")"
  status="$(post "$req" "$OUT/body")"
  if [ "$status" = 400 ]; then
    pass "$name"
  else
    fail "$name" "expected 400, got $status"
  fi
done

if [ "$failures" -gt 0 ]; then
  echo "$failures fixture(s) failed"
  exit 1
fi
echo "all fixtures passed"

lint wraps rtblint validate with a pinned spec snapshot and JSON output. It returns the CLI's 0 or 1 to the caller and stops the whole script on 2. post sends a fixture the way an exchange would, with the x-openrtb-version header, and prints only the status code. The response body lands in out/, so a failed run leaves evidence to inspect.

Two checks guard the fixtures themselves. A file in bid/ or no-bid/ must be a valid request, or the test would be measuring a bad input. A file in invalid/ must fail validation, or it no longer tests the invalid path. The nbr answer to an invalid request is validated too, as a response on its own, since there is no valid request to pair it with.

Run it locally

Install the CLI with cargo install rtblint --version 0.14.7, or download a release binary as the workflow below does. Start the bidder, then run the script:

$ bash scripts/test-bidder.sh fixtures
ok    bid/banner-300x250.json
ok    bid/ctv-video-15s.json
ok    no-bid/blocked-adomain.json
ok    no-bid/floor-above-cap.json
ok    no-bid/tmax-too-short.json
ok    invalid/missing-imp.json
ok    malformed/truncated.json
all fixtures passed
$ echo $?
0

The same run against a build whose nurl lost ${AUCTION_PRICE}:

$ bash scripts/test-bidder.sh fixtures
FAIL  bid/banner-300x250.json: response has warnings
{"id":"openrtb.bid.price_macro_missing","severity":"warning","message":"nurl or burl is present without ${AUCTION_PRICE}; a billing notice cannot report the clearing price.","path":"seatbid[0].bid[0].nurl","section":"OpenRTB 2.6 §4.4"}
FAIL  bid/ctv-video-15s.json: response has warnings
{"id":"openrtb.bid.price_macro_missing","severity":"warning","message":"nurl or burl is present without ${AUCTION_PRICE}; a billing notice cannot report the clearing price.","path":"seatbid[0].bid[0].nurl","section":"OpenRTB 2.6 §4.4"}
ok    no-bid/blocked-adomain.json
ok    no-bid/floor-above-cap.json
ok    no-bid/tmax-too-short.json
ok    invalid/missing-imp.json
ok    malformed/truncated.json
2 fixture(s) failed
$ echo $?
1

And the first failure from a build that put a fresh UUID in BidResponse.id instead of echoing the request id:

FAIL  bid/banner-300x250.json: response fails validation against its request
{"id":"openrtb.response.request_id_mismatch","severity":"error","message":"Response id \"b758b416-5c9e-4740-a76d-66d3c864b498\" does not echo the bid request id \"8f2b6c1e-4d7a-4a51-9b3e-2c0d5f7a9e10\"; BidResponse.id must be the id of the request it answers.","path":"id","section":"4.2.1"}

Each failure names the fixture, the rule id, and the JSON path. The no-bid, invalid, and malformed cases still pass in both broken builds, which narrows the bug to the bid path.

The GitHub Actions job

# Replays recorded bid requests against the bidder on every push and pull
# request, and fails when a response breaks the spec or the expected outcome.
name: Bidder fixtures
on: [push, pull_request]
permissions:
  contents: read
jobs:
  bidder-fixtures:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    env:
      RTBLINT_VERSION: v0.14.7
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 22
          cache: npm
      - name: Install bidder dependencies
        run: npm ci
      - name: Install the RTBlint CLI
        run: |
          mkdir -p "$RUNNER_TEMP/rtblint"
          curl -fsSL "https://github.com/aleksUIX/rtblint/releases/download/$RTBLINT_VERSION/rtblint-linux-x86_64.tar.gz" \
            | tar -xz -C "$RUNNER_TEMP/rtblint"
          echo "$RUNNER_TEMP/rtblint" >> "$GITHUB_PATH"
      - name: Start the bidder
        run: |
          node server.mjs > bidder.log 2>&1 &
          for _ in $(seq 1 50); do
            if curl -s -o /dev/null http://localhost:8080/; then exit 0; fi
            sleep 0.2
          done
          echo "The bidder did not start." >&2
          exit 1
      - name: Replay fixtures and validate responses
        run: bash scripts/test-bidder.sh fixtures
      - name: Show the bidder log
        if: always()
        run: cat bidder.log
      - name: Keep responses for review
        if: failure()
        uses: actions/upload-artifact@v7
        with:
          name: bidder-responses
          path: out/

The job installs the bidder's dependencies with npm ci, so commit package-lock.json. It downloads the RTBlint CLI from the pinned GitHub release rather than compiling it, so the job needs no Rust toolchain. The bidder starts in the background and the next step polls until it answers. Any HTTP status counts as up, since GET / returns 404 on this bidder.

The bidder log prints on every run, and the responses are kept as an artifact when the job fails. The workflow passes actionlint, and its steps were run locally against the Node.js bidder with the macOS build of the same release. The RTBlint GitHub Action shown in the CI guide also accepts type and request inputs, which suits checking stored pairs. This job calls the CLI directly because it needs a running bidder between the checks.

Testing the Rust bidder

The Rust bidder answers the same fixtures with the same status codes, so only the build and start steps change. Its unit test already validates a bid against its request with the rtblint-core crate, so run it first:

    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
      - name: Unit tests
        run: cargo test --release
      # Install the RTBlint CLI exactly as in the Node.js job.
      - name: Start the bidder
        run: |
          cargo build --release
          ./target/release/openrtb-bidder-rust > bidder.log 2>&1 &
          for _ in $(seq 1 50); do
            if curl -s -o /dev/null http://localhost:8080/; then exit 0; fi
            sleep 0.2
          done
          echo "The bidder did not start." >&2
          exit 1
      # Then the same replay, log, and artifact steps.

GitHub's hosted Ubuntu images include rustup and a stable toolchain, so the job needs no setup action. The longer timeout leaves room for a release build without a cache. Run against the Rust binary, the fixture script prints the same seven ok lines as the Node.js run, unchanged.

Keep the fixtures honest

• When you change bidder behavior on purpose, move the affected fixtures between folders in the same pull request. The diff then shows the behavior change to reviewers.
• Turn every production incident into a fixture before fixing it. The fixture fails first, then passes with the fix.
• Bump RTBLINT_VERSION and SPEC deliberately, one at a time, and read the new findings in that pull request.
• Keep the capture-and-scrub step in a script, so new fixtures never skip the scrub.

To look at a single failing response by hand, paste it with its request into the tester, which runs the same pair check in the browser.

Questions people ask