August 30, 2026·8 min read

How to Scrape ChatGPT Responses (and Their Citations) at Scale

If you want to know how to scrape ChatGPT responses, the short answer is: you can't get them from the OpenAI API, so you either automate the ChatGPT web interface yourself with a headless browser (and fight auth walls, streaming responses, and anti-bot systems), or you use a scraping API that runs prompts against the real interface and returns structured JSON — answer text, cited sources, the web searches ChatGPT ran, and the entities it mentioned.

That second sentence hides a lot of pain, so this guide walks through the whole problem honestly: what the web ChatGPT response actually contains, how to build a DIY scraper and where it breaks, and what a structured-API approach looks like in practice — including real request and response shapes.

Why the OpenAI API doesn't give you ChatGPT responses

This trips up almost everyone at first. api.openai.com and chatgpt.com are different products that happen to share models.

When a user types a commercial query into the ChatGPT web app, the product frequently runs live web searches, synthesizes an answer from what it finds, attaches citations to specific sources, and — for shopping-style queries — can render product cards. All of that behaviour is driven by ChatGPT's own system prompts, tool orchestration, and search stack.

Call the OpenAI API with the same text and you get a raw model completion: no product system prompt, different (often configurable) model defaults, no guarantee of web search, and none of the interface-level structures like citation pills or shopping cards. If you're building brand monitoring, GEO tooling, or citation research, the API answer is simply the wrong dataset — it's not what any real user sees.

So "scraping ChatGPT" specifically means capturing the web interface's output. There are two ways to do that.

Route 1: DIY scraping with a headless browser

The honest starting point. You spin up Playwright or Puppeteer, log into ChatGPT, type prompts, and read the DOM. Here's a minimal sketch of the idea:

# DIY approach — works in a demo, degrades in production
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://chatgpt.com")
    # 1. You need an authenticated session (cookies from a real login)
    # 2. Type the prompt and submit
    page.fill("#prompt-textarea", "best project management tools for agencies")
    page.keyboard.press("Enter")
    # 3. The answer streams in over SSE — you must wait for it to finish
    page.wait_for_timeout(30000)
    # 4. Now parse whatever the DOM looks like this week
    answer = page.inner_text("[data-message-author-role='assistant']")
    print(answer)

This runs. The problems start when you try to do it 10,000 times a day:

Auth walls. ChatGPT's useful behaviour (search, citations) is tied to sessions. Sessions expire, accounts get flagged for automation, and you end up managing a pool of accounts — which is both fragile and squarely against OpenAI's terms of use. Factor that into your risk assessment honestly.
Streaming responses. Answers arrive as a server-sent-event stream, token by token. Naive scrapers grab half-finished answers; robust ones need logic to detect stream completion, including for long research-style answers.
Selector churn. ChatGPT's frontend changes constantly. The data-* attributes and DOM structure your parser depends on can break without notice, and citation markup is especially unstable because it's rendered inline.
Anti-bot systems and IP blocks. Datacenter IPs get challenged or blocked. You'll need residential proxies, browser fingerprint management, and retry logic — an entire infrastructure discipline of its own.
Regional variance. ChatGPT answers differ by region and session. If your users are in five countries, scraping from one VPS in Virginia tells you almost nothing about four of those markets.
Cost. A real browser per request means real CPU and RAM per request. A headless fleet plus proxies plus maintenance engineering is routinely the most expensive line item in a scraping stack.

None of this is impossible — it's just a permanent operational tax. Some teams should pay it (if scraping infrastructure is your product, for instance). Most teams building GEO or brand-monitoring tools shouldn't.

Route 2: a structured API for ChatGPT responses

The alternative is an API that maintains that browser fleet for you and returns parsed JSON. This is what llmdata's ChatGPT endpoint does: you send a prompt and a country code, llmdata runs it against the real ChatGPT web interface — including its live web search — and returns the full answer plus the structures around it.

The request:

curl -X POST https://api.llmdata.dev/v1/monitor/chatgpt \
  -H "Authorization: Bearer $LLMDATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "best project management tools for agencies",
    "country": "US",
    "include": { "markdown": true, "sources": true }
  }'

The response:

{
  "success": true,
  "result": {
    "text": "For agencies, the most recommended tools are...",
    "markdown": "For agencies, the most recommended tools are...",
    "sources": [
      { "title": "...", "url": "https://...", "position": 1 }
    ],
    "searchQueries": ["best agency project management 2026"],
    "entities": ["Asana", "Monday.com", "ClickUp"]
  }
}

Four fields here do the work that DIY parsers struggle with most:

Field	What it gives you	Why it's hard to extract yourself
`text` / `markdown`	The complete answer, post-stream	Requires SSE completion detection
`sources`	Every cited link with title, URL and position	Citation markup is inline and changes often
`searchQueries`	The web searches ChatGPT ran to build the answer	Not visible in the final DOM at all without inspecting the network layer
`entities`	Brands and products mentioned	Requires NER on top of raw text

searchQueries deserves a highlight: knowing what ChatGPT searched for tells you which fan-out queries stand between a user's prompt and your content — often more actionable than the answer itself. There's also a shoppingCards field for the queries where ChatGPT returns product cards, which matters if you're tracking commerce-intent prompts.

Extracting citations at scale: a working pattern

Citations are usually the point. Whoever ChatGPT cites for "best CRM for small business" is winning that query. Here's a compact pattern for building a citation report across a prompt set:

import requests, collections

PROMPTS = [
    "best project management tools for agencies",
    "asana vs monday for client work",
    "how do agencies track billable hours",
]

cited = collections.Counter()

for prompt in PROMPTS:
    r = requests.post(
        "https://api.llmdata.dev/v1/monitor/chatgpt",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"prompt": prompt, "country": "US",
              "include": {"sources": True}},
    )
    data = r.json()
    if data.get("success"):
        for s in data["result"].get("sources", []):
            domain = s["url"].split("/")[2]
            cited[domain] += 1

for domain, count in cited.most_common(20):
    print(f"{count:3d}  {domain}")

Store the raw responses too (they're small), so you can diff answers over time and detect the day a competitor displaces you.

Handle non-determinism or your data lies to you

One run per prompt is an anecdote, not a measurement. ChatGPT's answers vary between sessions, regions and hours — the same prompt can name your brand at 9am and omit it at noon. Practical rules:

Repeat runs. Run each prompt multiple times per measurement window and report a mention rate (e.g. "cited in 7 of 10 runs"), not a binary.
Separate by country. Answers differ by market; a country parameter per request makes this trivial. Never average US and DE runs into one number.
Diff on structure, not text. Compare the sources and entities arrays between runs rather than raw text — wording varies far more than the underlying citations do.

This is exactly why scraping the real interface matters more than API sampling: the variance you're measuring is the variance your customers experience.

Legal and ToS considerations (the honest paragraph)

Automating the ChatGPT interface is against OpenAI's terms of use, and that's true regardless of whether you do it yourself or through a vendor. The broader legal picture around scraping publicly rendered web content is still evolving and differs by jurisdiction. What you should actually do: understand that ToS violation and illegality are not the same thing, avoid collecting any personal data, keep request volumes reasonable, and — if this data is load-bearing for your business — talk to a lawyer rather than a blog post. We'd rather say that plainly than pretend the question doesn't exist.

DIY vs API: the summary table

	DIY (Playwright + proxies)	Structured API (llmdata)
Setup time	Days to weeks	Minutes
Maintenance	Ongoing (selectors, auth, blocks)	None on your side
Citations parsed	You write and maintain the parser	`sources` array with positions
Search queries captured	Network-layer reverse engineering	`searchQueries` field
Country targeting	Your proxy pool	`country` parameter
Best for	Teams whose product is scraping infra	Teams building on top of the data

Where to go next

ChatGPT is one surface. The same brand-visibility questions apply to Perplexity — the most citation-dense engine, covered in our guide to getting Perplexity sources via API — to Google AI Overviews, and to Grok, which we cover in how to scrape Grok answers. Tracking all of them through one response schema is the entire reason llmdata exists.

llmdata is currently in early access. If you're building a GEO tool, an AI rank tracker, or just want to know what ChatGPT tells your customers, join the early access.