Backfill historical data

A backfill is the one workload that can exhaust a monthly quota in an afternoon. Plan it before you start it.

Estimate first

# One request tells you roughly how big the job is.
page = get("/v1/cs2/matches?status=completed&date_from=2025-01-01&date_to=2025-12-31&limit=500")
print(page["meta"]["count"], "in the first page")

Then the arithmetic:

requests ≈ (matches / 500)                       # walking the list
         + matches × layers_per_match            # detail calls

The list is cheap; the per-match detail calls dominate. 2,000 matches × 3 layers = 6,000 requests — 30% of a month — before you've fetched a single odds series.

Only fetch layers that exist

Filter at the list level so you never pull a match you'd discard:

curl -H "Authorization: Bearer $ESPORTSODDS_API_KEY" \
  "https://api.esportsodds.gg/v1/cs2/matches?status=completed&has=rounds,depth&date_from=2026-01-01&limit=500"

and gate each call on data_available rather than requesting blind. A request that returns [] still costs you one.

Make it resumable

Backfills get interrupted — by a quota, a deploy, a laptop lid. Checkpoint the cursor, not the offset:

def backfill(db, date_from, date_to):
    cursor = db.get_checkpoint("matches") or None
    while True:
        params = {"status": "completed", "date_from": date_from, "date_to": date_to, "limit": 500}
        if cursor:
            params["cursor"] = cursor
        page = get("/v1/cs2/matches", params=params)

        for match in page["data"]:
            if db.has_match(match["id"]):
                continue                      # already done — no request spent
            ingest_match(db, match)

        cursor = page["meta"]["next_cursor"]
        db.set_checkpoint("matches", cursor)
        if not cursor:
            return

Two properties worth having: skipping already-ingested matches makes a rerun nearly free, and checkpointing the cursor means a restart resumes rather than re-walks.

Pace it

Nothing forces a backfill to finish today. Spending a fixed budget per day turns a job that would blow your quota into one that fits alongside normal traffic:

DAILY_BUDGET = 400

spent = 0
for match in resumable_matches():
    if spent >= DAILY_BUDGET:
        break
    spent += ingest_match(db, match)     # returns requests used

Watch X-Quota-Remaining and stop early if normal traffic needs the headroom.

Odds history is the expensive part

/odds?match=<id> is per match and cursor-paginated, so a match with a long price series is several requests on its own. If you only need the outcome rather than the path, fetch the closing lines: one page per match, taking the rows flagged is_closing.

Rate limit vs quota

You will hit the monthly quota long before the per-second rate limit (20 rps sustained). Don't bother throttling to a slow crawl — just handle 429 rate_limited with Retry-After and stop entirely on quota_exceeded, which retrying cannot fix. See errors and retries.