Say you want to track prices for every event at Madison Square Garden. In November 2026 that was 447 events on up to 9 marketplaces, 1,954 event pages in all, and the list keeps changing as shows are added and rescheduled.
Monitoring that many doesn’t take a complex system. It’s two small Python jobs and one Postgres database: one job keeps the schedule in sync with the event catalog, and one captures whatever is due. The rules live in SQL, and Postgres does the queueing, so scaling up is starting more copies of the same worker.
Job 1: keep the schedule in sync
You don’t have to find each event on each marketplace yourself. The free event catalog (GET /v1/events) already matches events across Ticketmaster, StubHub, SeatGeek, TickPick, Vivid Seats, Gametime, Viagogo, GoTickets, Paciolan, Telecharge, Broadway Direct and Cirque du Soleil, and keeps their URLs up to date. Search it for the venue:
import os
import requests
response = requests.get(
"https://api.tickets.dev/v1/events",
params={"query": "madison square garden"},
headers={"x-api-key": os.environ["TICKETS_DEV_KEY"]},
)
for event in response.json()["events"]:
for source in event["sources"]:
print(event["name"], source["marketplace"], source["url"])The first job turns that into a schedule. It pages through every match, keeps the events at the Garden itself, and adds each marketplace page to a scheduletable with the event’s start time. It also creates the tables from schema.sql, shown below, and marks pages whose event is over. Run it from cron once an hour: new events start due right away, and a rescheduled event picks up its new date.
import os
import psycopg
import requests
API_KEY = os.environ["TICKETS_DEV_KEY"]
DATABASE_URL = os.environ["DATABASE_URL"]
QUERY = "madison square garden"
VENUE = "Madison Square Garden" # the query also matches its parking lots
# A page has ended two days past the start, or once it has been gone for a
# day with nothing captured. The row stays, so you can see how it ended.
END_PAGES = """
UPDATE schedule s SET status = 'ended', status_at = now()
WHERE s.status NOT IN ('ended', 'dropped')
AND (
s.starts_at < now() - interval '2 days'
OR (s.status = 'gone' AND NOT EXISTS (
SELECT 1 FROM captures c
WHERE c.url = s.url AND c.status IN ('ok', 'empty')
AND c.captured_at > now() - interval '1 day'))
)"""
def catalog(query):
"""Every event the free catalog matches, page by page."""
page = 1
while True:
response = requests.get(
"https://api.tickets.dev/v1/events",
params={"query": query, "page": page, "pageSize": 100},
headers={"x-api-key": API_KEY},
timeout=30,
)
response.raise_for_status()
body = response.json()
yield from body["events"]
if not body["hasMore"]:
return
page += 1
with psycopg.connect(DATABASE_URL, autocommit=True) as db:
db.execute(open("schema.sql").read())
events = pages = 0
for event in catalog(QUERY):
# Skip parking, other rooms, and passes without a date.
if event["venue"]["name"] != VENUE or not event["eventDateUtc"]:
continue
events += 1
for source in event["sources"]:
# New pages start due now. A rescheduled event keeps its status
# and gets the new date.
db.execute(
"INSERT INTO schedule (url, starts_at) VALUES (%s, %s) "
"ON CONFLICT (url) DO UPDATE SET starts_at = EXCLUDED.starts_at",
(source["url"], event["eventDateUtc"]),
)
pages += 1
ended = db.execute(END_PAGES).rowcount
print(f"{events} events, {pages} marketplace pages, {ended} newly ended")
The filter matters. The search matched 875 events, and 411of them were MSG parking lots or the smaller Infosys Theater in the same building. Another 17, mostly season tickets and multi-day passes, have no date, so they can’t be scheduled. For your own venue, check the venue names the search returns before you trust the count.
Check more often as the event gets close
Resale prices and listings move most in the final days before an event. Months out, a snapshot looks a lot like the one before it. So each page gets a rate by how far away its event is:
| Time to the event | Checks a day | One check every |
|---|---|---|
| Final 24 hours | 72 | 20 min |
| 1 to 7 days out | 24 | 1 h |
| 8 to 30 days out | 6 | 4 h |
| More than 30 days out | 2 | 12 h |
Treat these as starting values. If you trade or report on an event, raise the final-day rate; if you only need the trend, lower the far-out one. The rates live in one SQL function, check_every, so changing one is a one-line edit.
Three tables and a queue view
Everything the system knows lives in three tables:
| Table | One row per | What it holds |
|---|---|---|
schedule | Marketplace page | The event's start time, the page's status and when that status was set |
captures | Capture attempt | When it ran, how it went, the error code, the listing count and the get-in price |
snapshots | Successful capture | The full snapshot as JSON, keyed by the capture |
A page’s whole state is its status and status_at. capturesis the point of record: it keeps every attempt, failed ones included, so you can answer “what did we see on this page and when” without opening a snapshot. The bodies live apart in snapshots, so they can move to cheaper storage later without touching the record.
The scheduling rules are SQL too. check_every holds the rates above, and the queue view works out when each active page is next due from its status:
| Status | Means | Due again |
|---|---|---|
new | Just added | Now |
ok | Captured, with listings | After its rate |
empty | Captured, nothing on sale | After its rate |
processing | A worker has it | In 10 minutes, if the worker died |
failed | Capture didn't complete | In 10 minutes |
gone | Event page not found | In an hour |
ended | Event is over | Never |
dropped | Bad URL or request | Never |
-- One row per marketplace page. status and status_at are its whole state.
CREATE TABLE IF NOT EXISTS schedule (
url text PRIMARY KEY,
starts_at timestamptz NOT NULL,
-- new, processing, ok, empty, failed, gone, ended or dropped
status text NOT NULL DEFAULT 'new',
status_at timestamptz NOT NULL DEFAULT now()
);
-- The record: one row per capture attempt, failed ones included.
CREATE TABLE IF NOT EXISTS captures (
id bigserial PRIMARY KEY,
url text NOT NULL,
captured_at timestamptz NOT NULL DEFAULT now(),
status text NOT NULL,
error_code text,
listing_count int,
get_in_price numeric
);
CREATE INDEX IF NOT EXISTS captures_by_url ON captures (url, captured_at);
-- The full snapshot of each successful capture.
CREATE TABLE IF NOT EXISTS snapshots (
capture_id bigint PRIMARY KEY REFERENCES captures (id),
snapshot jsonb NOT NULL
);
-- How often to check a page, by time to its event.
CREATE OR REPLACE FUNCTION check_every(starts_at timestamptz) RETURNS interval
LANGUAGE sql STABLE AS $$
SELECT CASE
WHEN starts_at - now() <= interval '24 hours' THEN interval '20 minutes'
WHEN starts_at - now() <= interval '168 hours' THEN interval '60 minutes'
WHEN starts_at - now() <= interval '720 hours' THEN interval '240 minutes'
ELSE interval '720 minutes'
END
$$;
-- Every active page and when it's next due. Each status waits its own way.
CREATE OR REPLACE VIEW queue AS
SELECT url, starts_at, status,
status_at + CASE status
WHEN 'new' THEN interval '0'
WHEN 'processing' THEN interval '10 minutes' -- a worker died mid-capture
WHEN 'failed' THEN interval '10 minutes'
WHEN 'gone' THEN interval '1 hour' -- recheck in case it comes back
ELSE check_every(starts_at) -- ok, empty
END AS due_at
FROM schedule
WHERE status NOT IN ('ended', 'dropped')
AND starts_at > now() - interval '2 days'; -- a safety cap on the endJob 2: capture what's due
The second job is one loop. It claims the most overdue page, captures it, and writes the result to all three tables in one transaction. The rest is SQL.
import os, time
import psycopg
import requests
from psycopg.types.json import Jsonb
API_KEY = os.environ["TICKETS_DEV_KEY"]
DATABASE_URL = os.environ["DATABASE_URL"]
# Take the most overdue page and mark it processing. SKIP LOCKED hands every
# worker a different page; if a worker dies, the page is due again in 10 minutes.
CLAIM = """
WITH next AS (
SELECT url FROM queue
WHERE due_at <= now()
ORDER BY due_at
LIMIT 1
FOR UPDATE SKIP LOCKED
)
UPDATE schedule s SET status = 'processing', status_at = now()
FROM next
WHERE s.url = next.url
RETURNING s.url"""
def capture(url):
"""One attempt. Returns (status, snapshot or error code)."""
try:
response = requests.get(
"https://api.tickets.dev/v1/capture",
params={"url": url},
headers={"x-api-key": API_KEY},
timeout=120,
)
except requests.RequestException as error:
return "failed", type(error).__name__
if response.ok:
snapshot = response.json()
# Empty means nothing on sale right now, with the marketplace's note
# (sold out, canceled) in snapshot["note"] when it gives one.
return ("ok" if snapshot["stats"]["listingCount"] else "empty"), snapshot
try:
error = response.json().get("error", {})
except ValueError: # a proxy error page, not our JSON
error = {"retryable": response.status_code >= 500}
code = error.get("code", f"HTTP {response.status_code}")
if code == "event_not_found":
return "gone", code
if code == "quota_exceeded" or response.status_code == 401:
raise SystemExit(f"stopping: {code}") # top up or fix the key, then restart
return ("failed" if error.get("retryable") else "dropped"), code
def record(db, url, status, detail):
snapshot = detail if status in ("ok", "empty") else None
stats = snapshot["stats"] if snapshot else {}
with db.transaction():
capture_id = db.execute(
"INSERT INTO captures (url, status, error_code, listing_count, get_in_price) "
"VALUES (%s, %s, %s, %s, %s) RETURNING id",
(url, status, None if snapshot else detail,
stats.get("listingCount"), stats.get("getInPrice")),
).fetchone()[0]
if snapshot:
db.execute("INSERT INTO snapshots (capture_id, snapshot) VALUES (%s, %s)",
(capture_id, Jsonb(snapshot)))
db.execute("UPDATE schedule SET status = %s, status_at = now() WHERE url = %s",
(status, url))
print(f"{url}: {status} {stats.get('listingCount', '') if snapshot else detail}")
with psycopg.connect(DATABASE_URL, autocommit=True) as db:
while True:
row = db.execute(CLAIM).fetchone()
if row is None:
time.sleep(10)
continue
status, detail = capture(row[0])
record(db, row[0], status, detail)
Run seed.py first, since it creates the tables, then start the worker. The catalog needs your live key and is free. While you test the worker, give it the sandbox key tk_test_sandbox instead: it gets fixtures and is never billed, so you can watch it work through the schedule before spending captures.
The worker has no threads. To capture more at once, start more copies of it, on one machine or several.
Postgres is the queue
You don’t need Redis, SQS or a job framework to share work between workers. The queue view is the queue, and one Postgres clause makes it safe: FOR UPDATE SKIP LOCKED. A worker locks the row it picks, and any other worker running the same query at the same moment skips that row and takes the next one. Nobody waits, and no page is captured twice.
A claim marks the page processing in the same statement, so a worker that dies mid-capture needs no cleanup: after ten minutes the view counts the page as due again. If you extend the claim to join another table, say an events table for the event’s name, write FOR UPDATE OF s SKIP LOCKED so only the schedule rows are locked and the joined rows stay free for everyone else.
Errors, retries and when to stop
Every error has a code and a retryable flag, and the worker branches on those. The HTTP status alone can mislead you: quota_exceeded and rate_limited are both 429, and only one is worth retrying. Failed captures are never billed.
| Code | What it means | What the worker does |
|---|---|---|
event_not_found | The event ended, moved or the URL is wrong | Marks it gone and rechecks hourly; ended after a day with nothing captured |
quota_exceeded | Your captures are used up | Stops; top up and restart it |
rate_limited | Capacity is momentarily full | Marks it failed; it's due again in 10 minutes |
capture_timeout, capture_failed, capture_unavailable | The capture didn't complete this time | Marks it failed; it's due again in 10 minutes |
Marketplaces sometimes take an event page down and put it back, for a postponed show or a reissued date, so the worker doesn’t drop an event on the first event_not_found. A day of them means the event is gone. That is also how it knows an event is over: the listed date says when it starts, not when it ends, so the worker keeps checking until the marketplace takes the page down. The seeder marks a page ended after a day of not found, or two days past the start either way.
A capture that times out usually fails after about a minute, and the worker waits out that minute. The ten-minute wait after a failure keeps a page that keeps failing from taking a worker every few minutes. The API docs list every code.
What it costs to watch the Garden
One capture is one event page at one moment. On the day we pulled the catalog, the Garden’s 1,954 pages split like this:
| Time to the event | Pages | Captures a day |
|---|---|---|
| Final 24 hours | 12 | 864 |
| 1 to 7 days out | 79 | 1,896 |
| 8 to 30 days out | 257 | 1,542 |
| More than 30 days out | 1,606 | 3,212 |
| Total | 1,954 | 7,514 |
That’s about 7,500 captures a day, or 313an hour. Most pages are months out and cost two captures a day; the total climbs as more events reach their final week. Failed captures aren’t billed and don’t count. Plans and per-capture prices are on the pricing page.
How long a capture takes
A capture loads the marketplace’s live event page, so it takes seconds. Across successful customer captures from October 10 to November 9, 2026, the median took 6.1 seconds and 90% finished within 12.4.
| Marketplace | Median | 90th pct. | Median listings |
|---|---|---|---|
| Ticketmaster | 6.3 s | 10.3 s | 403 |
| SeatGeek | 5.2 s | 10.4 s | 842 |
| StubHub | 5.7 s | 18.3 s | 181 |
| TickPick | 10.2 s | 18 s | 1,411 |
| Vivid Seats | 4.2 s | 17.7 s | 414 |
| Gametime | 2.2 s | 5 s | 449 |
| Viagogo | 4.3 s | 15.2 s | 190 |
| GoTickets | 1.6 s | 2.9 s | 216 |
TickPick, which has the most listings per event, is also the slowest. Plan around the 90th percentile, because a StubHub capture at 18 seconds holds its slot three times as long as the median one.
How many workers to run
How many captures you need running at once follows from the captures a day and the average capture time of 7.3 seconds:
captures a day × 7.3 s ÷ 86,400 = captures in flight
For the Garden, 7,514 captures a day need about 0.6in flight on average. Each worker runs one capture at a time, so that’s one worker. Run a few more anyway: they leave room for slow captures, failures and the weeks when many events are close at once.
A dashboard for queue health
Because the queue and the record are plain tables, you can build your own dashboard on them in Metabase, Grafana or whatever you already use. Five queries cover it:
-- Captures a day: how much work got done, and on how many pages.
SELECT captured_at::date AS day,
count(DISTINCT url) AS pages,
count(*) AS captures,
count(*) FILTER (WHERE status IN ('ok', 'empty')) AS succeeded
FROM captures
WHERE captured_at >= now() - interval '30 days'
GROUP BY day
ORDER BY day;
-- Captures an hour over the last two days. Flat bars mean it keeps up.
SELECT date_trunc('hour', captured_at) AS hour,
count(*) AS captures
FROM captures
WHERE captured_at >= now() - interval '48 hours'
GROUP BY hour
ORDER BY hour;
-- Queue size, from the same view the workers claim from.
SELECT count(*) AS active,
count(*) FILTER (WHERE due_at <= now()) AS due_now,
count(*) FILTER (WHERE due_at < now() - interval '10 minutes') AS overdue
FROM queue;
-- Pages by status.
SELECT status, count(*) AS pages
FROM schedule
GROUP BY status
ORDER BY pages DESC;
-- Expected captures a day at the current rates, active pages only.
SELECT round(sum(86400 / extract(epoch FROM check_every(starts_at)))) AS expected_per_day
FROM queue;The queue size reads the same queueview the workers claim from, so it can’t disagree with what they pick. Ended and dropped pages aren’t in the view, so they never count as behind.
The hourly chart is the one to watch. When the workers keep up, the bars are flat, because pages come due at a steady rate. A few overdue pages during a burst is normal. If due_nowkeeps growing or more than about a fifth of the active pages stay overdue, the workers can’t keep up: start another one, lower the far-out rates, or check the status counts for failing captures. Daily captures should track expected_per_day; once they do and the backlog stays near zero, you have room to add another venue.
Leave ended pages out of any “expected” number, as the queries above do. They pile up fast, since every show that ends stays in the table, and if you count them as due the system looks several times further behind than it is.
Optional: move the history to a warehouse
The schedule and the record stay small enough for Postgres. The snapshots are what grow. The Garden’s 7,514 captures a day, at a median of 181 to 1,411 listings per capture depending on the marketplace, come to over 1 million listing rows a day. To query across all of them, for example price per section across every show in a season, move the listings to a warehouse built for it: ClickHouse, Snowflake or Databricks all work.
The setup that holds up: have the worker also write each snapshot as a JSON file to object storage such as S3, and let the warehouse load the files on its own, with ClickPipes in ClickHouse Cloud, Snowpipe in Snowflake or Auto Loader in Databricks. Postgres keeps the schedule and the record, with a path column in captures pointing at each file, and the snapshots table can go.
In ClickHouse, keep one row per capture in the snapshot’s own shape, with the listings as an array. This is close to the table Tickets.dev keeps its own captures in:
CREATE TABLE captures
(
source LowCardinality(String),
eventId String,
eventName String,
eventDateUtc String,
sourceUrl String,
capturedAt DateTime64(3, 'UTC'),
currency LowCardinality(String),
stats Tuple(
listingCount UInt32,
ticketCount UInt32,
getInPrice Float64,
medianPrice Float64
),
listings Array(Tuple(
listingId String,
inventoryType LowCardinality(String),
section String,
row String,
quantity UInt32,
ticketPrice Float64,
fee Float64,
totalPrice Float64
))
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(capturedAt)
ORDER BY (source, eventId, capturedAt);Sorting by marketplace, event and time keeps one event’s history together on disk, and monthly partitions make old months cheap to drop. Per-listing questions unpack the array with ARRAY JOIN:
-- Lowest and median price per section, from the last day of captures.
SELECT eventName,
source,
l.section AS section,
min(l.totalPrice) AS lowest,
median(l.totalPrice) AS median
FROM captures
ARRAY JOIN listings AS l
WHERE capturedAt > now() - INTERVAL 1 DAY
GROUP BY eventName, source, section
ORDER BY eventName, lowest;Snowflake and Databricks work the same way: a VARIANT column with FLATTEN in Snowflake, an array of structs with explode in Databricks. Whichever you use, precompute what your dashboards read, such as get-in and median price per section per event, every few minutes, so a page loads a small rollup instead of scanning millions of rows.
You don’t need this step to start: Postgres alone handles a workload the size of the Garden’s. For what to do with the data either way, see tracking prices over time and price and inventory alerts.



