ScrapeNest API

Returns metadata for the API key making the request. Use it to sanity-check auth and read your live rate limits.

curl https://scrapenest.dev/v1/me \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY"

import os, httpx

r = httpx.get(
    "https://scrapenest.dev/v1/me",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
)
r.raise_for_status()
print(r.json())

const r = await fetch("https://scrapenest.dev/v1/me", {
  headers: { "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}` },
});
console.log(await r.json());

Response

{
  "prefix": "sn_AbCdEfGhi",
  "label": "production",
  "is_active": true,
  "is_internal": false,
  "rate_limit_per_minute": 60,
  "rate_limit_per_day": 10000,
  "credit_balance": 48230,
  "created_at": "2026-05-01T12:00:00Z",
  "last_used_at": "2026-05-14T15:24:11Z"
}

Fetches a single URL and returns clean JSON. Use this when no dedicated endpoint exists for the source.

Request body

curl -X POST https://scrapenest.dev/v1/scrape/url \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "render": "auto",
    "extract": true
  }'

import os, httpx

r = httpx.post(
    "https://scrapenest.dev/v1/scrape/url",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
    json={
        "url": "https://example.com/article",
        "render": "auto",
        "extract": True,
    },
    timeout=60.0,
)
r.raise_for_status()
data = r.json()
print(data["title"], "-", data["credits_charged"], "credits")

const r = await fetch("https://scrapenest.dev/v1/scrape/url", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com/article",
    render: "auto",
    extract: true,
  }),
});
const data = await r.json();
console.log(data.title, "-", data.credits_charged, "credits");

Response

{
  "success": true,
  "url": "https://example.com/article",
  "final_url": "https://example.com/article",
  "status_code": 200,
  "method": "fetch",
  "proxy_used": "datacenter",
  "elapsed_ms": 312,
  "title": "Example Article",
  "description": "Lead summary...",
  "text": "Cleaned article body...",
  "structured": { "@type": "Article", "headline": "..." },
  "cache_hit": false,
  "escalated": false,
  "credits_charged": 1
}

Credit cost depends on the resolved tier: 1 direct, 5 browser-only, 10 residential-only, 40 residential+browser, 1 on a cache hit, plus 5 when ai_query runs. Blocked and empty fetches are free.

Response fields to know. text is the readability extraction, except that when it comes back short (under 1,500 chars) and is a small fraction of the page’s detected main content, the main content is returned instead. title is absent when the document has no <title>. structured.product appears when the page declares a product (schema.org Product JSON-LD, microdata, or og:/product: meta) with name, price, currency, rating, review_count, brand, availability as declared (each only when present) and source ("jsonld", "microdata" or "meta"). cache_hit: true also covers a page our fetch service answered from its own page cache: it bills the cache-hit rate and is never escalated, while method and proxy_used name the tier that originally fetched it. exit_country is the exit’s country when country was requested and the fetch succeeded. With markdown_mode: "main", language-selector link lists and link lists or control widgets ahead of the first real content block are dropped too.

Long pages. Set max_chars to bound the response. Whenever text is returned the response carries text_total_chars; when more text follows the window, next_offset says where, and sending it back as offset reads on. With focus, the window is the passages that best match it, in page order, joined by [... N chars skipped ...] markers. The same applies to markdown, reported as markdown_total_chars and markdown_next_offset. Windowing is free and works on cached pages.

Auto-escalation. When an auto or browser fetch is refused by the target’s anti-bot, we automatically retry once in stealth mode — you don’t need to resend anything. If that retry succeeds, escalated is true and credits_charged reflects the stealth rate; if it stays blocked, the request is free. Two things switch it off besides auto_escalate: false: render: "direct" is a hard tier pin and never escalates, and escalation needs at least 20 seconds left on the request budget — so a tight timeout_seconds (say 20) disables it in practice. The same flag gates two cheaper fallbacks: a page still refused after that gets one retry as a plain (non-browser) HTTP client unless you set your own User-Agent, and on render: "auto" a page that comes back as an empty JavaScript shell gets one browser render (billed at the browser tier only if it delivers content).

When success is false, error_code is one of: anti_bot_block (anti-bot refused — already retried in stealth unless you disabled it), timeout (exceeded the time budget — raise timeout_seconds), not_found (target returned 404/410), empty_response (no usable content), rate_limited (the target answered 429 without a challenge — slow down), connection_failed (render: "direct" could not connect; not transient at that tier, so switch to "auto" or "stealth"), or upstream_failed (transient fetch error, or a bare 503 from the target — retry). With render: "direct", a fetch that got no page at all is reported as anti_bot_block or connection_failed, with method: "fetch". Two failures come back as a real error envelope instead: 422 country_unavailable (no exit for country; not retryable) and 502 upstream_failed when the fetch service itself can’t run the request. Neither is billed.

Fetches up to 10 URLs in a single call. Each URL is fetched independently, so one URL failing never fails the rest. Fetches run concurrently five at a time, so a full 10-URL batch completes in roughly two waves rather than in the time of a single fetch. The render options apply to every URL in the batch.

Auto-escalation and latency. Batch runs the same auto-escalation as /v1/scrape/url, controlled by the same auto_escalate flag (default true); it accepts no timeout_seconds. A URL refused by anti-bot on auto or browser is retried once in stealth and billed at the stealth rate only if that retry succeeds; render: "direct" never escalates. Each URL’s retries get at most 45 s, inside one ~100 s budget measured from the start of the batch, and a stealth retry only starts while at least 20 s of that remains, so a URL blocked late in the call comes back failed instead of escalating. A batch holding a hostile URL can take tens of seconds up to ~100 s, and longer in the worst case (each URL’s first fetch has its own ~100 s ceiling and only five run at once), which can cross the ~100 s edge timeout and fail with 524. Keep hostile URLs to a few per batch, or fetch them through /v1/scrape/url with a timeout_seconds you control.

Request body

curl -X POST https://scrapenest.dev/v1/scrape/batch \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/a",
      "https://example.com/b",
      "https://example.com/c"
    ],
    "render": "auto",
    "extract": true
  }'

import os, httpx

r = httpx.post(
    "https://scrapenest.dev/v1/scrape/batch",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
    json={
        "urls": [
            "https://example.com/a",
            "https://example.com/b",
            "https://example.com/c",
        ],
        "render": "auto",
        "extract": True,
    },
    timeout=120.0,
)
r.raise_for_status()
body = r.json()
print(body["succeeded"], "/", body["requested"], "-", body["credits_charged"], "credits")
for item in body["results"]:
    print(item["url"], item["success"], item.get("title"))

const r = await fetch("https://scrapenest.dev/v1/scrape/batch", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    urls: [
      "https://example.com/a",
      "https://example.com/b",
      "https://example.com/c",
    ],
    render: "auto",
    extract: true,
  }),
});
const body = await r.json();
console.log(`${body.succeeded}/${body.requested} - ${body.credits_charged} credits`);
for (const item of body.results) console.log(item.url, item.success, item.title);

Response (abbreviated)

{
  "results": [
    {
      "url": "https://example.com/a",
      "final_url": "https://example.com/a",
      "status_code": 200,
      "method": "fetch",
      "proxy_used": "datacenter",
      "elapsed_ms": 284,
      "title": "Page A",
      "text": "Cleaned body...",
      "cache_hit": false,
      "escalated": false,
      "success": true,
      "ok": true,
      "credits_charged": 1
    },
    {
      "url": "https://example.com/blocked",
      "final_url": "https://example.com/blocked",
      "status_code": 403,
      "method": "fetch",
      "proxy_used": "datacenter",
      "elapsed_ms": 640,
      "cache_hit": false,
      "escalated": false,
      "success": false,
      "ok": false,
      "credits_charged": 0,
      "error": "example.com blocked the request (anti-bot protection detected). ...",
      "error_code": "anti_bot_block"
    }
  ],
  "requested": 3,
  "succeeded": 2,
  "credits_charged": 3
}

Each item carries its own success, credits_charged, and error. Branch on success; ok is emitted alongside it as a deprecated alias with the same value. succeeded counts the items that returned a usable page; credits_charged at the top level is the sum across all items. Items also carry escalated and cache_hit, and exit_country on success with country set; text, title and structured.product follow the /v1/scrape/url rules.

When include_usage=true, the usage.breakdown reports the batch total under a single fetch key. Before any fetch, a batch whose worst case (every URL billed at the highest tier its render could reach, stealth included) would push the account past the 5,000-credit overage floor is refused with 402 insufficient_credits.

AI-friendly multi-engine web search. Returns the SERP (titles, URLs, snippets, engines, scores). No page fetching.

Request body

The default engine mix follows depth. fast queries Bing alone (~0.5 s, lowest coverage); balanced — the default — queries Bing + DuckDuckGo + Brave; thorough and stealth query Bing + Google + DuckDuckGo + Brave and wait up to ~5 s for a cold Google render, within timeout_seconds; if Google isn’t ready by then, gserp_status says so and the other engines’ results come back (repeats are cached and fast). Work that doesn’t fit the budget is skipped rather than failing the call. If the budget runs out before any engine answers, the call still returns 200 with empty results, partial: true, error: "search_deadline_exceeded", a note and 0 credits (allow at least 1 s); an engine failure well inside the budget is a 502 upstream_failed, also unbilled. Setting time_range drops Bing from the default, whose scraper silently discards the date filter, leaving DuckDuckGo + Brave — or Google + DuckDuckGo + Brave on thorough/stealth. That substitution is silent; it is not reported in note.

An explicit engines list is honoured verbatim and overrides all of the above — no depth-based mix, no time_range swap. It accepts exactly the thirteen values in the table, "brave" included, on /v1/search, /v1/search/deep and /v1/search/answer. Brave is in the default mixes only while the operator’s Brave switch is on; when it is off they drop Brave silently (an explicit "brave" is still sent). The "bing news" and "duckduckgo news" values pull straight from the news vertical — the same lane topic="news" routes to. topic picks the vertical. Omitted (the default), the search covers the whole web and blends news in when the query looks like a current event, with news holding at most half of the results; "general" searches the whole web with that auto-routing turned off (an explicit engines list or categories, or depth: "fast", also turns it off); "news" pins that news vertical for every query; "research" searches academic literature (Crossref, OpenAlex, PubMed) and returns papers with DOIs rather than web pages; "code" adds GitHub, Stack Overflow, Hacker News, and MDN to the general web engines. All four work on /v1/search, /v1/search/deep, and /v1/search/answer. On /v1/search, depth sets the engine mix and how long a cold query waits (above), and thorough and stealth behave identically. On /v1/search/deep it additionally governs the per-page fetch budget and anti-bot effort, which is where the tiers diverge most. page is 1-based pagination on /v1/search: page=2 returns the next num_results results with no repeats from earlier pages. The engines give at most 50 results for one query, so a page past the last result comes back empty with a note and bills 0. Deep and answer ignore page.

Exclusions. A literal site:/-site: in the query is enforced like include_domains/exclude_domains. A -word or -"phrase" drops results whose title, snippet or URL words (host and path) contain it, including the word’s plural and possessive forms (or the singular of a negated plural); if every result matches, all are kept.

curl -X POST https://scrapenest.dev/v1/search \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"climate report 2026 ipcc","num_results":10}'

import os, httpx

r = httpx.post(
    "https://scrapenest.dev/v1/search",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
    json={"query": "climate report 2026 ipcc", "num_results": 10},
)
r.raise_for_status()
for item in r.json()["results"][:3]:
    print(item["position"], item["title"], "-", item["url"])

const r = await fetch("https://scrapenest.dev/v1/search", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ query: "climate report 2026 ipcc", num_results: 10 }),
});
const body = await r.json();
for (const it of body.results.slice(0, 3)) {
  console.log(it.position, it.title, "-", it.url);
}

Response fields

An empty results with partial: false means the engines answered and found nothing; with partial: true it means some of them did not answer.

Response (typical, abbreviated)

{
  "query": "climate report 2026 ipcc",
  "results": [
    {
      "position": 1,
      "url": "https://...",
      "title": "...",
      "snippet": "...",
      "engine": "bing",
      "engines": ["bing", "brave", "duckduckgo"],
      "score": 1.0,
      "published_at": "2026-04-12T14:30:00Z"
    }
  ],
  "took_ms": 318,
  "cache_hit": false,
  "credits_charged": 5,
  "unresponsive_engines": [],
  "engine_status": {
    "bing": { "status": "ok", "results": 10 },
    "duckduckgo": { "status": "ok", "results": 10 },
    "brave": { "status": "ok", "results": 9 }
  },
  "partial": false
}

Response (when extras are populated; results omitted here for brevity)

{
  "query": "weather in tokyo",
  "results": [],
  "images": [
    {
      "url": "https://example.com/tokyo.jpg",
      "thumbnail": "https://example.com/tokyo_t.jpg",
      "title": "Tokyo skyline",
      "source_url": "https://example.com/article"
    }
  ],
  "suggestions": ["weather in tokyo tomorrow", "weather in tokyo this week"],
  "answers": [
    { "text": "Tokyo is currently 22°C, partly cloudy.", "source": "weather.com" }
  ],
  "infoboxes": [
    { "title": "Tokyo", "subtitle": "Capital of Japan", "url": "https://en.wikipedia.org/wiki/Tokyo" }
  ],
  "took_ms": 462,
  "cache_hit": false,
  "credits_charged": 5
}

Search and fetch the top N pages in one round trip. Same SERP as /v1/search, plus cleaned text for each fetched page.

Request body

All fields from /v1/search (including depth and engines; page is accepted but ignored), plus:

Here depth also controls how hard we fetch each page. fast (~2.5 s/page) and balanced (default, ~3.5 s/page) fetch every page in parallel with no anti-bot escalation — lowest latency, and genuinely bot-walled pages simply fall back to their SERP snippet. thorough (~6 s/page) and stealth (~12 s/page) escalate bot-walled pages through a real-browser bypass; stealth has the best coverage on the hardest sites. thorough and stealth also wait up to ~4 s for a cold Google render before picking the pages, and only when the budget still leaves the page fetches their full time. An explicit render is honoured at every depth (a browser render gets at least ~6 s per page); left unset it is direct on fast/balanced. Easy pages still return in a couple seconds regardless; the larger budget only applies to pages that actually need the bypass. If the page-fetch phase fails or runs out of time, the call falls back to search results only (empty pages, a note says so, billed as a plain search) rather than erroring. With under ~2 s of budget left there is no page fetch at all: the call returns search results only with partial: true; if your timeout_seconds leaves too little for even that, it returns 200 with empty results, partial: true, error: "search_deadline_exceeded" and 0 credits. The response carries the same coverage diagnostics as /v1/search, and results is capped at num_results.

curl -X POST https://scrapenest.dev/v1/search/deep \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "how does retrieval-augmented generation work",
    "fetch_top": 3
  }'

import os, httpx

r = httpx.post(
    "https://scrapenest.dev/v1/search/deep",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
    json={"query": "how does retrieval-augmented generation work", "fetch_top": 3},
    timeout=90.0,
)
body = r.json()
for page in body["pages"]:
    print(page["url"], "-", len((page.get("text") or "")), "chars")

const r = await fetch("https://scrapenest.dev/v1/search/deep", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ query: "rag systems", fetch_top: 3 }),
});
const body = await r.json();
for (const p of body.pages) console.log(p.url, "-", (p.text || "").length);

Response (abbreviated; results carries the same SERP shape as /v1/search)

{
  "query": "how does retrieval-augmented generation work",
  "results": [],
  "pages": [
    {
      "url": "https://...",
      "final_url": "https://...",
      "status": 200,
      "method": "browser",
      "elapsed_ms": 640,
      "title": "...",
      "text": "..."
    }
  ],
  "took_ms": 2110,
  "fetched_count": 3,
  "cache_hit": false,
  "credits_charged": 20
}

Each page reports method — one of "fetch", "browser", "stealth", "cf_bypass", or "unknown" — plus status, final_url and elapsed_ms. A page that couldn’t be fetched carries an error string instead of text, and isn’t counted in fetched_count or billed.

Which pages count. A fetched page is counted and billed only when it is on the query’s subject and usable: at least 200 characters of text, and not a bot wall, consent gate, login redirect or not-found page. An unusable page comes back with its text removed and error set to blocked, not_found or no_content; an off-subject page is dropped from pages and counted in note. While the budget allows, each discarded page is replaced by fetching the next on-subject result. When the call delivers nothing at all — no SERP results, no fetched pages, no images — the base 5 credits are dropped too and the call is free.

RAG-style: searches the web, fetches the top sources, then asks an LLM to write a cited answer. Returns the answer plus [1]-style citations mapping to source URLs.

Request body

Accepts the same search fields as /v1/search (including depth and engines; page is accepted but ignored) plus render ("auto" | "direct" | "browser"; an explicit value is honoured at every depth, and left unset it is "direct" on fast and "auto" otherwise). Instead of fetch_top, the number of sources to read is set by:

Here depth sets the page-fetch effort for the sources: fast ~2.5 s/page with no escalation (a browser render gets at least ~6 s), balanced ~6 s/page, thorough ~9 s/page, stealth ~12 s/page, each within an overall fetch ceiling of roughly 6 / 14 / 20 / 28 s, before the answer model runs. thorough and stealth also wait up to ~4 s for a cold Google render before choosing the sources. Raise it when answers come back grounded in snippets. timeout_seconds is accepted (0.1–20) but has no effect on this endpoint. Responses carry note but not the search coverage diagnostics, and are cached only when at least one source page was fetched. Errors (502 no_sources, 502 llm_upstream_failed, 502 upstream_failed, 503 llm_not_configured) aren’t billed.

Sources that count. A bot wall, consent gate, login redirect, not-found page or page under 200 characters isn’t used as a fetched source: that source falls back to its search snippet and is neither counted nor billed, and while the budget allows the next on-subject result is fetched in its place. When none of the retrieved sources is relevant to the question, the model isn’t called: answer is the fixed text “No relevant sources were found for this question: none of the retrieved pages address it, so no answer was generated.”, citations is empty, and the call bills 5. An answer whose first sentence says the sources don’t address the question also bills a flat 5; a real answer that notes a gap later is billed normally.

curl -X POST https://scrapenest.dev/v1/search/answer \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"is the new gpt-5 multimodal","max_sources":4}'

import os, httpx

r = httpx.post(
    "https://scrapenest.dev/v1/search/answer",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
    json={"query": "is the new gpt-5 multimodal", "max_sources": 4},
    timeout=120.0,
)
body = r.json()
print(body["answer"])
for c in body["citations"]:
    print(f"[{c['index']}] {c['url']}")

const r = await fetch("https://scrapenest.dev/v1/search/answer", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ query: "is gpt-5 multimodal", max_sources: 4 }),
});
const body = await r.json();
console.log(body.answer);
for (const c of body.citations) console.log(`[${c.index}] ${c.url}`);

Response (abbreviated)

{
  "query": "is the new gpt-5 multimodal",
  "answer": "Yes. GPT-5 supports text, image, audio... [1][2]",
  "citations": [
    { "index": 1, "url": "https://openai.com/...", "title": "...", "snippet": "..." }
  ],
  "took_ms": 4200,
  "cache_hit": false,
  "credits_charged": 30
}

Translate text between languages.

Request body

BCP-47 region aliases are normalised server-side: pt-BR, zh-CN, en-GB, and underscore forms like en_US all map to the right target code. Pass whatever your stack hands you; no need to canonicalise. Supported: ar, cs, da, de, el, en, es, fi, fr, he, hi, hu, id, it, ja, ko, nb, nl, pl, pt, ro, ru, sv, th, tr, uk, vi, zh-hans, zh-hant.

Chinese. zh, zh-CN and zh-SG mean Simplified ("zh-hans"). zh-Hant, zh-TW, zh-HK and zh-MO mean Traditional and come back as target "zh-hant"; if the Traditional translator is unavailable the text is translated to Simplified and target and note both say "zh-hans", so the metadata always matches the script returned.

Errors. An unsupported target or source is a 422 validation_error whose details[] names the field; the message lists the supported codes. When source is "auto" and the detected language isn’t supported, the call is 400 unsupported_detected_source; pass source explicitly.

curl -X POST https://scrapenest.dev/v1/translate \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text":"Hello world","target":"ja"}'

import os, httpx

r = httpx.post(
    "https://scrapenest.dev/v1/translate",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
    json={"text": "Hello world", "target": "ja"},
)
print(r.json()["translated"])

const r = await fetch("https://scrapenest.dev/v1/translate", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ text: "Hello world", target: "ja" }),
});
console.log((await r.json()).translated);

Response

{
  "translated": "こんにちは世界",
  "detected_source": "en",
  "target": "ja",
  "credits_charged": 1
}

Response fields

• translated: the translated text.
• detected_source: the language we detected the source as (BCP-47).
• target: echoes the requested target language.
• note: only present when we adjusted behaviour (source equals target so text was returned unchanged, an alias was applied, or auto-detection fell back to a heuristic). Omitted from normal paid translations.
• credits_charged: 1 per call, or 0 when source equals (or auto-detects to) target.

Timestamped transcript of any audio source. Supports podcast episodes (Apple Podcasts or Overcast episode pages, or an episode’s audio file URL from an RSS feed; feed URLs are not accepted), YouTube (captions when available, else audio transcription), X Spaces, SoundCloud, Facebook video, TikTok, and direct MP3/M4A/WAV/OGG links.

Request body

curl -X POST https://scrapenest.dev/v1/audio/transcript \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://podcasts.apple.com/us/podcast/.../id..."}'

import os, httpx

r = httpx.post(
    "https://scrapenest.dev/v1/audio/transcript",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
    json={"url": "https://podcasts.apple.com/us/podcast/.../id..."},
    timeout=600.0,
)
body = r.json()
print(body["title"], "/", body["channel"])
print(body["text"][:400])

const r = await fetch("https://scrapenest.dev/v1/audio/transcript", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://podcasts.apple.com/us/podcast/.../id..." }),
});
const body = await r.json();
console.log(body.title, "/", body.channel);
console.log(body.text.slice(0, 400));

Response (abbreviated)

{
  "url": "...",
  "audio_url": "https://traffic.megaphone.fm/....mp3",
  "canonical_url": "https://podcasts.apple.com/us/podcast/.../id...?i=...",
  "title": "Episode title",
  "channel": "Show name",
  "thumbnail": "https://is1-ssl.mzstatic.com/.../artwork.jpg",
  "duration_seconds": 3618.2,
  "language": "en",
  "source": "apple_podcasts",
  "chunks": [
    { "start": 0.0, "end": 4.2, "text": "Welcome back to the show..." }
  ],
  "text": "Welcome back to the show... [full transcript]",
  "took_ms": 41200,
  "cache_hit": false,
  "credits_charged": 620
}

chunks is omitted when the upstream source doesn’t provide timestamps (some podcast feeds and short clips); the text field always contains the full transcript regardless. canonical_url and thumbnail are populated when the resolver can pin them; treat absence as “not available”, not an error.

If part of a long recording still can’t be transcribed after retries, the response returns the rest with partial: true and missing_ranges ([start, end] spans in seconds). Only the covered time is billed, and a partial result is never cached, so a retry runs fresh. Neither field is set on a complete transcript.

YouTube cookies (optional, for the long tail)

~95% of YouTube URLs work out of the box via the captions API. The remaining ~5% (YouTube Shorts that fall through the captions path, videos with auto-captions disabled, geo-restricted content, members-only, age-gated) require a logged-in YouTube session to download the audio. For those, pass a Netscape-format cookies.txt in the cookies field.

How to obtain cookies:

1. Use a burner Google account. Not your personal one, since every transcript request using these cookies adds to that account’s watch history.
2. Install a cookies-export browser extension (Chrome: Get cookies.txt LOCALLY; Firefox: cookies.txt).
3. Log into youtube.com in that browser, click the extension on any YouTube tab, choose Export. You’ll get a file starting with # Netscape HTTP Cookie File.
4. Send the entire file contents as the cookies field. We accept up to 128 KB; only the youtube.com lines actually matter.

Omit cookies for the typical case. Add it only when you’ve previously seen login_required on a video you specifically need.

Identical request shape to the sync endpoint. Returns immediately with a run_id (202 Accepted); poll GET /v1/scrape/runs/{run_id} for the completed transcript. Recommended for long-form audio (multi-hour podcasts, full conference talks) so you’re not holding an HTTP connection open for the duration of the transcription. Credit cost is identical and is charged when the job completes. Failed runs are not billed.

Initial response (202)

{
  "run_id": "f0e0c0a0-1234-5678-9abc-def012345678",
  "status": "queued",
  "status_url": "/v1/scrape/runs/f0e0c0a0-1234-5678-9abc-def012345678",
  "created_at": "2026-05-14T16:00:00Z",
  "expires_at": "2026-05-21T16:00:00Z"
}

Pulls Google Maps reviews for any place. Returns up to ~10,000 unique reviews, deduplicated and sorted newest-first.

Pricing. 100 credits per 50 reviews returned (rounded up to the next 50), with a 40-credit floor when 50 or fewer come back — so you only pay for what’s available. 200 reviews bills ~400 credits; the 10,000-review cap bills ~20,000. Before the scrape starts, a request whose worst case (max_reviews all returned) would push the account past the 5,000-credit overage floor is refused with a real 402 insufficient_credits status line; lower max_reviews or top up. When a capture stops short of max_reviews because one of our own paths failed (not because the place has fewer reviews), it bills 2 credits per delivered review whenever that is less than the tier price (e.g. 3 reviews of a requested 50 bill 6, not 40).

Realistic delivery by place size

The note field in the response is your authoritative count of what came back and why. The review_count field reflects what Google publishes for the place, which is often much higher than what’s actually accessible through any documented endpoint. This reflects Google’s spam-filter and pagination behavior, not a limitation of this API.

Request body

Finding a FID

Open the place on Google Maps. The URL contains a segment like !1s0x47e6…:0x8ddc…!8m2!…. The FID is the 0x…:0x… hex pair immediately after !1s. Copy it whole, separator included.

curl -X POST https://scrapenest.dev/v1/scrape/google-maps/reviews \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "place_id": "0x47e66e2964e34e2d:0x8ddca9ee380ef7e0",
    "max_reviews": 5000
  }'

import os, httpx

r = httpx.post(
    "https://scrapenest.dev/v1/scrape/google-maps/reviews",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
    json={
        "place_id": "0x47e66e2964e34e2d:0x8ddca9ee380ef7e0",
        "max_reviews": 5000,
    },
    timeout=300.0,
)
body = r.json()
print(len(body["reviews"]), "reviews -", body["credits_charged"], "credits")

const r = await fetch("https://scrapenest.dev/v1/scrape/google-maps/reviews", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    place_id: "0x47e66e2964e34e2d:0x8ddca9ee380ef7e0",
    max_reviews: 5000,
  }),
});
const body = await r.json();
console.log(body.reviews.length, "reviews -", body.credits_charged, "credits");

Response (abbreviated)

{
  "place_id": "0x47e66e2964e34e2d:0x8ddca9ee380ef7e0",
  "name": "Eiffel Tower",
  "address": "Av. Gustave Eiffel, 75007 Paris, France",
  "rating": 4.7,
  "review_count": 486214,
  "google_maps_url": "https://www.google.com/maps/place/...",
  "reviews": [
    {
      "review_id": "ChdDSUhNMG9nS0VJQ0FnSURBNS0z...",
      "review_url": "https://www.google.com/maps/contrib/.../reviews?hl=en",
      "author_name": "Jane D.",
      "author_id": "1234567890",
      "author_profile_url": "https://www.google.com/maps/contrib/1234567890/reviews?hl=en",
      "author_avatar_url": "https://lh3.googleusercontent.com/...",
      "rating": 5,
      "text": "Amazing view, worth the wait.",
      "language": "en",
      "relative_time": "2 weeks ago",
      "posted_at": 1778630400,
      "posted_at_iso": "2026-05-13T00:00:00Z",
      "helpful_count": 4
    }
  ],
  "star_histogram": [8214, 5190, 17920, 69450, 385440],
  "note": "Returned 5000 unique reviews after deduplication.",
  "took_ms": 197000,
  "cache_hit": false,
  "credits_charged": 10000
}

Heads up: name, address, rating, review_count, and star_histogram can be null when the FID resolves but the metadata block isn’t served. The reviews array is always the source of truth for what you get.

star_histogram is a flat array of 5 ints ordered 1★ → 5★ (so [8214, 5190, 17920, 69450, 385440] means 8,214 one-star reviews and 385,440 five-star reviews). When the whole review list was read (the place has fewer reviews than max_reviews), review_count and star_histogram are computed from that full set and are exact. photos is omitted from a review entry when the reviewer didn’t attach any; treat its absence as the empty case. Reviews recovered through the page-rendering fallback carry a posted_at/posted_at_iso approximated from Google’s relative label (“2 weeks ago”), not an exact timestamp.

Identical request body to the sync endpoint. Returns immediately with a run_id (202 Accepted); poll GET /v1/scrape/runs/{run_id} for the final result. Recommended for any large pull (it doesn’t hold a long-lived HTTP connection open, and you can fire-and-forget across many places). Credit cost is identical and is charged when the job completes. Failed runs are not billed. Async runs share the sync endpoint’s cache, so a repeat inside the TTL completes at once and bills the cache-hit rate.

Initial response (202)

{
  "run_id": "f0e0c0a0-1234-5678-9abc-def012345678",
  "status": "queued",
  "status_url": "/v1/scrape/runs/f0e0c0a0-1234-5678-9abc-def012345678",
  "created_at": "2026-05-14T15:00:00Z",
  "expires_at": "2026-05-21T15:00:00Z"
}

Returns nearby retail gas-station prices for a US location. Accepts a 5-digit ZIP, City, ST, or City, State. Backed by public station-finder listings.

Request body

curl -X POST https://scrapenest.dev/v1/scrape/gas-prices \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "location": "37122",
    "grade": "regular",
    "limit": 20
  }'

import os, httpx

r = httpx.post(
    "https://scrapenest.dev/v1/scrape/gas-prices",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
    json={"location": "Mount Juliet, TN", "grade": "regular", "limit": 20},
    timeout=60.0,
)
body = r.json()
for s in body["stations"][:5]:
    print(f"{s['brand']:>18}  {s.get('price')}  {s['address']}")

const r = await fetch("https://scrapenest.dev/v1/scrape/gas-prices", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ location: "37122", grade: "regular", limit: 20 }),
});
const body = await r.json();
for (const s of body.stations.slice(0, 5)) {
  console.log(`${s.brand}  ${s.price}  ${s.address}`);
}

Response (abbreviated)

{
  "location": {
    "query": "37122",
    "zip": "37122",
    "city": "Mount Juliet",
    "state": "TN"
  },
  "grade": "regular",
  "stations": [
    {
      "brand": "Kroger",
      "sub_brand": "Kroger Fuel Center",
      "station_id": "21847",
      "address": "401 S Mt Juliet Rd, Mount Juliet, TN",
      "city": "Mount Juliet",
      "state": "TN",
      "price": 2.789,
      "currency": "USD",
      "unit": "USD/gallon",
      "reporter": "anon_18234",
      "updated_relative": "12 minutes ago"
    }
  ],
  "region_stats": {
    "state": "TN",
    "region": "Tennessee",
    "region_type": "state",
    "lowest_price": 2.699,
    "average_price": 2.952
  },
  "source_url": "https://scrapenest.dev",
  "fetched_at": "2026-05-19T17:42:00Z",
  "took_ms": 1820,
  "cache_hit": false,
  "credits_charged": 10
}

Field rules:

• location.zip, location.city, location.state are populated when the resolver can identify them; location.query always echoes the verbatim input.
• region_stats is omitted when the source doesn’t render the regional summary block (typically rural ZIPs).
• region_stats.region_type is "state" when region is a US state name and state holds the USPS code; it’s "city" when the source returned a metro-level summary (state is then null).
• The source’s regional figures cover regular only. For any other grade, region_stats is computed from the returned stations of that grade (lowest and average, labelled with the searched city) and note says so; when no prices of that grade came back, region_stats is omitted with a note.
• note is set when stations were returned without per-station prices, when an ambiguous query was auto-resolved, or when other caveats apply.

Poll the status of a run created by one of the two async endpoints, /v1/scrape/google-maps/reviews/async and /v1/audio/transcript/async; no other endpoint has an async variant. Returns one of queued, running, completed, failed, cancelled. When completed, result holds the same body shape as the synchronous endpoint.

Polling does not consume credits, but it does count against your rate limit. Runs expire after 7 days.

The response also includes an optional progress object on long-running scrapes (currently large Maps reviews pulls). Shape: {"current": <int>, "target": <int>, "detail": "<string>"}. It advances as reviews are collected, through each fetch path the run tries. Use it to display a progress bar to your users; absent means the worker hasn't emitted a milestone yet.

To stop an in-flight run, hit POST /v1/scrape/runs/{run_id}/cancel. Runs bill only when they complete, so a queued or running run cancelled before completing is not billed (credits_charged: 0). A run that already completed, failed or was cancelled is returned unchanged, and a completed run stays billed. The response has the same shape as the GET:

curl -X POST https://scrapenest.dev/v1/scrape/runs/f0e0c0a0-1234-5678-9abc-def012345678/cancel \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY"

{
  "run_id": "f0e0c0a0-1234-5678-9abc-def012345678",
  "endpoint": "POST /v1/scrape/google-maps/reviews",
  "status": "cancelled",
  "created_at": "2026-05-14T15:00:00Z",
  "started_at": "2026-05-14T15:00:02Z",
  "completed_at": "2026-05-14T15:01:10Z",
  "cancelled_at": "2026-05-14T15:01:10Z",
  "expires_at": "2026-05-21T15:00:00Z",
  "credits_charged": 0,
  "progress": { "current": 1200, "target": 5000, "detail": "..." }
}

RUN=$(curl -s -X POST https://scrapenest.dev/v1/scrape/google-maps/reviews/async \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"place_id":"0x47e6...:0x8ddc...","max_reviews":5000}' \
  | jq -r .run_id)

while true; do
  STATUS=$(curl -s https://scrapenest.dev/v1/scrape/runs/$RUN \
    -H "Authorization: Bearer $SCRAPENEST_API_KEY")
  STATE=$(echo "$STATUS" | jq -r .status)
  case "$STATE" in completed|failed|cancelled) echo "$STATUS" | jq; break;; esac
  sleep 10
done

import os, time, httpx

API_KEY = os.environ["SCRAPENEST_API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}
BASE = "https://scrapenest.dev"

with httpx.Client(base_url=BASE, headers=HEADERS, timeout=30.0) as client:
    submit = client.post(
        "/v1/scrape/google-maps/reviews/async",
        json={"place_id": "0x47e6...:0x8ddc...", "max_reviews": 5000},
    )
    submit.raise_for_status()
    run_id = submit.json()["run_id"]

    while True:
        time.sleep(10)
        body = client.get(f"/v1/scrape/runs/{run_id}").json()
        if body["status"] in ("completed", "failed", "cancelled"):
            break

    if body["status"] == "completed":
        print(len(body["result"]["reviews"]), "reviews")
    else:
        print(body["status"], body.get("error_code"), body.get("error_message"))

const API_KEY = process.env.SCRAPENEST_API_KEY;
const BASE = "https://scrapenest.dev";
const headers = { "Authorization": `Bearer ${API_KEY}`, "Content-Type": "application/json" };

const submit = await fetch(`${BASE}/v1/scrape/google-maps/reviews/async`, {
  method: "POST",
  headers,
  body: JSON.stringify({
    place_id: "0x47e6...:0x8ddc...",
    max_reviews: 5000,
  }),
});
const { run_id } = await submit.json();

while (true) {
  await new Promise(r => setTimeout(r, 10000));
  const body = await (await fetch(`${BASE}/v1/scrape/runs/${run_id}`, { headers })).json();
  if (["completed", "failed", "cancelled"].includes(body.status)) {
    console.log(body.status, body.result?.reviews?.length ?? body.error_message);
    break;
  }
}

Response (result holds the same body as the synchronous endpoint)

{
  "run_id": "f0e0c0a0-1234-5678-9abc-def012345678",
  "endpoint": "POST /v1/scrape/google-maps/reviews",
  "status": "completed",
  "created_at": "2026-05-14T15:00:00Z",
  "started_at": "2026-05-14T15:00:02Z",
  "completed_at": "2026-05-14T15:03:15Z",
  "expires_at": "2026-05-21T15:00:00Z",
  "credits_charged": 10000,
  "result": {}
}

On a successful run, cancelled_at, error_code, error_message and progress are absent rather than null. A failed run carries error_code and error_message; an in-flight Maps run carries progress as {"current": int, "target": int, "detail": str} once the worker has emitted its first milestone.

Captures a PNG or JPEG screenshot of any public URL. Returns base64-encoded image bytes inline. Auto-falls-back to a stealth backend when the target site blocks the normal headless browser — same endpoint, same response shape. If you already know the target is hostile, skip the failed first attempt with render: "stealth".

Request body

curl -X POST https://scrapenest.dev/v1/scrape/screenshot \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com",
    "full_page": true,
    "format": "png"
  }' \
  | jq -r .image_base64 | base64 -d > screenshot.png

import base64, os, httpx

r = httpx.post(
    "https://scrapenest.dev/v1/scrape/screenshot",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
    json={
        "url": "https://news.ycombinator.com",
        "full_page": True,
        "format": "png",
    },
    timeout=120.0,
)
r.raise_for_status()
data = r.json()
with open("screenshot.png", "wb") as f:
    f.write(base64.b64decode(data["image_base64"]))
print(f"{data['width']}x{data['height']}, {data['bytes']} bytes via {data['method']}")

import { writeFileSync } from "node:fs";

const r = await fetch("https://scrapenest.dev/v1/scrape/screenshot", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://news.ycombinator.com",
    full_page: true,
    format: "jpeg",
    jpeg_quality: 80,
  }),
});
const data = await r.json();
writeFileSync("screenshot.jpg", Buffer.from(data.image_base64, "base64"));
console.log(`${data.width}x${data.height}, ${data.bytes} bytes via ${data.method}`);

Response

{
  "url": "https://news.ycombinator.com",
  "final_url": "https://news.ycombinator.com/",
  "status_code": 200,
  "method": "browser",
  "escalated": false,
  "elapsed_ms": 2470,
  "width": 1280,
  "height": 1179,
  "bytes": 253264,
  "format": "png",
  "image_base64": "iVBORw0KGgoAAAANSUhEUgAA...",
  "cache_hit": false,
  "credits_charged": 25
}

The method field tells you which engine handled the capture: "browser" is the normal headless path (~2–3 s, 15–25 credits), "stealth" is the anti-bot path for hostile sites (~8–12 s, 35 credits). You don't have to choose — we route per target — but render: "stealth" forces it when you already know the site blocks normal browsers. escalated is true when the requested tier was refused by anti-bot and the capture was taken in stealth (billed at the stealth rate); a selector miss never escalates. selector_applied appears only when selector was set: true when the image is cropped to the matched element, false when it is the full viewport.

Error delivery. A capture that resolves within ~20 seconds returns a real HTTP status line (422 selector_not_found, 422 target_error, 502 screenshot_failed…) with the plain {"error": {...}} envelope. Past that window the response has already begun streaming, so the status line is locked at 200 and the same error arrives in-body as {"error": {...}, "http_status": 502}. Branch on the transport status first, then check for an error key when it is 200. Failures and timeouts don’t bill either way.

Renders any URL to PDF. Returns base64-encoded bytes. Useful for archiving articles, generating invoice/receipt copies, or capturing legal evidence.

Request body

curl -X POST https://scrapenest.dev/v1/scrape/pdf \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "format": "A4"
  }' | jq -r .pdf_base64 | base64 -d > article.pdf

import os, base64, httpx

r = httpx.post(
    "https://scrapenest.dev/v1/scrape/pdf",
    headers={"Authorization": f"Bearer {os.environ['SCRAPENEST_API_KEY']}"},
    json={"url": "https://example.com/article", "format": "A4"},
    timeout=120,
)
data = r.json()
with open("article.pdf", "wb") as f:
    f.write(base64.b64decode(data["pdf_base64"]))
print(f"{data['bytes']} bytes -> article.pdf")

import fs from "node:fs/promises";

const r = await fetch("https://scrapenest.dev/v1/scrape/pdf", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPENEST_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com/article", format: "A4" }),
});
const data = await r.json();
await fs.writeFile("article.pdf", Buffer.from(data.pdf_base64, "base64"));
console.log(`${data.bytes} bytes -> article.pdf`);

Response

{
  "url": "https://example.com/article",
  "final_url": "https://example.com/article",
  "status_code": 200,
  "elapsed_ms": 2120,
  "format": "A4",
  "landscape": false,
  "bytes": 20925,
  "pdf_base64": "JVBERi0xLjQKJdPr6eEKMSAwIG9iago...",
  "cache_hit": false,
  "credits_charged": 25
}

Error delivery works exactly as on /v1/scrape/screenshot: a render that resolves within ~20 seconds returns a real status line (422 target_error, 502 pdf_failed…), while anything slower has already started streaming and arrives as {"error": {...}, "http_status": 502} inside a 200. Failures don’t bill.