# ScrapeNest API: guide for AI agents > Use this file to teach an LLM or coding agent how to call the ScrapeNest > public API. Everything below is current as of the latest deploy. If you need > a machine-readable spec, fetch https://scrapenest.dev/openapi.json (public, > unauthenticated, cacheable for 5 minutes). Official client libraries are > published for Python and Node/TypeScript (see section 6). --- ## 1. Quick context ScrapeNest is a paid, metered scraping-as-an-API. One Bearer key gives you: * A general "fetch any URL" endpoint with optional headless browser and optional AI extraction. * Site-specific endpoints that return cleaned JSON (Google Maps reviews, audio/podcast/YouTube transcripts). * AI-friendly web search with domain/date/topic filters, deep search (search + fetch), and RAG-style answer with citations. * Async job + poll workflow for long-running scrapes, so a single open HTTP connection isn't a bottleneck. Every response includes `credits_charged` and (where relevant) `cache_hit` so you can predict cost before you hit production scale. Cache hits are billed at 1 credit. Credits map roughly to $0.0002 USD each on the smallest plans and trend cheaper at higher tiers. Base URL: `https://scrapenest.dev` --- ## 2. Authentication Every `/v1/*` endpoint uses a Bearer token. Get one at https://scrapenest.dev/dashboard/keys. Keys start with `sn_` and are shown once at creation time. ```bash curl -X GET https://scrapenest.dev/v1/me \ -H "Authorization: Bearer sn_YOUR_API_KEY" ``` If you 401, your key is wrong, revoked, or you forgot the `Bearer ` prefix. If you 402, your monthly credits ran out. Top up at /dashboard/billing. --- ## 3. Endpoints All request bodies are JSON. All responses are JSON. All timestamps are ISO 8601 UTC unless noted. Most endpoints support a `cache_ttl_seconds` override on the request (see each endpoint for its accepted range and default). Request format notes that apply to every endpoint: * **Strict booleans.** Bool fields (`include_usage`, `force_proxy`, `block_assets`, `block_ads`, `simulate_behavior`, `extract`, `include_html`, `return_markdown`, `full_page`, `landscape`, etc.) reject string truthy values. Send real JSON booleans `true` / `false` only. `"yes"`, `"1"`, `0` will 422. * **Cache hits charge `credits_charged: 1`.** When a request matches a cached response, the response carries `cache_hit: true`, `credits_charged: 1`, and `usage.breakdown` `{"cache_hit": 1}` regardless of the original call's cost. * **Error envelope.** Every 4xx/5xx response is `{"error": {"code": "...", "message": "..."}}`. Validation errors also include a structured `details[]` array with `loc / type / msg` entries describing each invalid field. ### 3.1 GET /v1/me Returns information about the API key making the request. Free. Accepts GET or HEAD. Request: ```bash curl -X GET https://scrapenest.dev/v1/me \ -H "Authorization: Bearer sn_YOUR_API_KEY" ``` Response: ```json { "prefix": "sn_AbCdEfGhi", "label": "production", "is_active": true, "is_internal": false, "rate_limit_per_minute": 30, "rate_limit_per_day": 500, "credit_balance": 842, "created_at": "2026-05-01T12:00:00Z", "last_used_at": "2026-05-13T15:24:11Z" } ``` `rate_limit_*` reflect your current plan (see section 5). Free shown above. `credit_balance` is your account's remaining credits (null for internal keys). Credit cost: 0. --- ### 3.2 POST /v1/scrape/url Fetches a single URL and returns extracted JSON. Use this when no dedicated endpoint exists for the site. Required: * `url`: absolute http(s) URL. Optional: * `render`: `"auto"` (default), `"direct"` (plain HTTP through a proxy, cheap, no JS), `"browser"` (headless browser, slower but bypasses JS-only pages), or `"stealth"` (maximum anti-bot bypass that defeats the toughest bot defenses, 40 credits). * `force_proxy`: `false` by default. Set `true` to force a higher-tier proxy (use when the site geo-blocks cheaper sources). * `auto_escalate`: `true` by default. When a non-stealth fetch is refused by the target's anti-bot, we automatically retry once in stealth mode. You are only charged the stealth rate if that retry succeeds; a request that stays blocked is free. The response sets `escalated: true` when this happened. Set `false` to keep the original tier and get the block as-is. * `timeout_seconds`: `~100` by default (accepts 5-120). Hard time budget for the whole request including any auto-escalation retry. Lower it (e.g. 20) to fail fast on hard targets; raise it for slow stealth solves. On timeout, `success` is `false` and `error_code` is `timeout`. * `block_assets`: `true` by default. Blocks images/fonts/media during rendering to save bandwidth and time. * `extract`: `true` by default. Runs readability + JSON-LD extraction so you get clean `title`, `description`, `text`, `structured`. Set to `false` to get raw HTML in `text`. * `ai_query`: natural-language extraction prompt (up to 2000 chars). Example: `"return the product name, price, and rating"`. When set, the response includes an `ai_extract` field with the AI's answer. Adds 5 premium credits. * `ai_extract_schema`: JSON Schema object constraining `ai_extract` to a typed object. Used together with `ai_query`. Max 10 KB serialized. Example: `{"type":"object","properties":{"name":{"type":"string"},"price_usd":{"type":"number"}}}`. * `include_usage`: `false` by default. When `true`, the response includes a `usage` block with credit totals and breakdown. * `include_html`: `false` by default. When `true`, the response includes a raw `html` field with the page source as received from the target (before extraction). Use this when you want both cleaned text and the raw markup in one call. No extra credits. * `return_markdown`: `false` by default. When `true`, the response includes a `markdown` field with the page converted to Markdown. Implies `include_html`. No extra credits. * `extract_rules`: `null` by default. Map of field name to CSS selector for in-response extraction. Each value can be a selector string (returns first match's text) or an object `{"selector": "...", "attr": "href", "all": true}` for attribute values or lists. Result in the `extracted` field. Implies `include_html`. Max 50 rules. No extra credits. * `wait_ms`: `null` by default. Extra wait (0-10000 ms) after load before reading the DOM. Browser render only. * `wait_for_selector`: `null` by default. CSS selector to wait for before returning. Browser render only. * `wait_for_timeout_ms`: `5000` by default (100-30000). Max ms to wait for `wait_for_selector`. * `window_width` / `window_height`: viewport dimensions for the browser. Defaults 1280x720. Browser render only. `viewport_width` / `viewport_height` are accepted as aliases. * `country`: ISO 3166-1 alpha-2 (2 chars). Routes the fetch through a residential proxy in that country. Forces a higher-tier proxy. Residential exits are available for `us`, `be`, `de`, `fr`, `it`, `sg`; other codes have no exit and fail upstream. * `device`: `"desktop"` (default) or `"mobile"`. Mobile sets a mobile User-Agent and a 390x844 viewport. Browser render only. * `block_ads`: `false` by default. When `true`, blocks known ad and tracker domains during the fetch. Browser render only. * `simulate_behavior`: `false` by default. When `true`, adds mouse-move + scroll motion before extraction to fool behavioral bot detectors (PerimeterX, DataDome, Akamai BMP). Adds ~200-600ms. Browser render only. * `session_id`: opaque sticky-session id (1-64 chars, `[A-Za-z0-9_.-]`). Repeated requests with the same id reuse the same exit IP for ~10 minutes. Use for multi-step login flows, shopping carts, or any workflow that needs IP continuity. Forces a higher-tier proxy and disables caching. Billed at the proxy tier actually used for the fetch. * `cookies`: object `{name: value}` (up to 50 entries). On this endpoint cookies are a `{name: value}` mapping only. The `[{name, value}]` list form is rejected here (it is accepted on `/v1/scrape/screenshot` and `/v1/scrape/pdf`). Cookies are injected into the browser context before navigation. * `headers`: object `{name: value}` (up to 30 entries). Extra request headers: set `User-Agent`, `Referer`, etc. `Host`, `Content-Length`, `Connection`, `Transfer-Encoding` are stripped. * `cache_ttl_seconds`: `0-86400`. Default is 900 (15 minutes). Request: ```bash curl -X POST https://scrapenest.dev/v1/scrape/url \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://example.com/article","render":"auto"}' ``` Login-flow example (sticky session + cookies). Step 1 logs in with a `session_id` you choose; step 2 reuses the same id (and therefore the same exit IP) for ~10 minutes so the target site sees a coherent session: ```bash curl -X POST https://scrapenest.dev/v1/scrape/url \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/dashboard", "session_id": "user42_login", "cookies": {"session": "abc123def456"} }' curl -X POST https://scrapenest.dev/v1/scrape/url \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/dashboard/orders", "session_id": "user42_login" }' ``` Response: ```json { "success": true, "url": "https://example.com/article", "final_url": "https://example.com/article", "status_code": 200, "method": "fetch", "proxy_used": "datacenter", "elapsed_ms": 312, "title": "Example Article", "description": "Lead summary...", "text": "Cleaned article body...", "structured": {"@type": "Article", "headline": "..."}, "cache_hit": false, "escalated": false, "credits_charged": 1, "error": null, "error_code": null } ``` IMPORTANT: this endpoint returns HTTP 200 even when the target blocks the fetch or serves no content. Always branch on the top-level `success` boolean, not the HTTP status. When `success` is `false`, the content fields are `null` and `error_code` is one of: `anti_bot_block` (anti-bot refused; already retried in stealth unless `auto_escalate:false`), `timeout` (exceeded the time budget; raise `timeout_seconds`), `not_found` (target 404/410), `empty_response` (no usable content), or `upstream_failed` (transient fetch error; retry). Blocked and empty fetches are free. `escalated: true` means we auto-upgraded to stealth to get past a block, and `credits_charged` reflects the stealth rate. Credit cost: * Direct + no proxy: 1 * Direct + premium proxy: 10 * Browser (headless) + no proxy: 5 * Browser (headless) + premium proxy: 40 * `render=stealth` (maximum anti-bot bypass): 40 * `ai_query` adds: 5 * Delivered 404/410 (page reached, not found): 1 * Undeliverable (we couldn't reach the page): 0 * Cache hit: 1 --- ### 3.3 POST /v1/search AI-friendly multi-engine web search. No fetching, just the SERP. Required: * `query`: 1-400 chars. Optional: * `num_results`: 1-50, default 10. * `language`: IETF BCP 47 tag, default `"en"` (e.g. `"en"`, `"pt-BR"`, `"zh-Hans"`). * `time_range`: `"day"`, `"week"`, `"month"`, `"year"`. * `start_date` / `end_date`: `"YYYY-MM-DD"` filters for explicit ranges. * `topic`: `"news"` only (or omit/null for general web search). The endpoint does not accept `"general"` or `"finance"`. * `depth`: `"fast"`, `"balanced"` (default), `"thorough"`, `"stealth"`. On `/v1/search` every depth queries the same engine mix; `thorough` and `stealth` allow extra time for the freshest engine data. The tier matters most on `/v1/search/deep` (see 3.4), where it governs the per-page fetch budget and anti-bot effort. * `page`: 1-based result page, default 1, max 20. `page=2` returns the next batch of results. * `engines`: array picking which engines to query. Valid values: `"google"`, `"duckduckgo"`, `"bing"`, `"bing news"`, `"duckduckgo news"`. Up to 5, must be non-empty (pass `null` or omit to use the default mix: bing, google, and duckduckgo merged; when `time_range` is set the mix drops bing, whose date filter can't be honored, and the response `note` reports the swap). Bing is the most reliable scraper-friendly engine; DuckDuckGo honors `site:` filters. The news values pull straight from the news vertical (the same lane `topic="news"` routes to). An explicit list overrides the default mix. * `country`: ISO 3166-1 alpha-2 (2 chars, e.g. `"us"`, `"de"`, `"jp"`) to bias results toward that region. * `include_domains`: array of up to 15 domains. Results are restricted to these (e.g. `["nytimes.com","bbc.co.uk"]`). The 15-entry cap is set by Google/Bing, which reject queries with more `site:` clauses. * `exclude_domains`: array of up to 15 domains to filter out. * `include_images`: `false` by default. When `true`, the response also carries an `images` array with image results. * `include_image_descriptions`: `false` by default. When `true` and `include_images` is on, populates each image's `description` field with alt-text or caption when available. * `image_sources`: array of sources to query for images. Defaults to `["bing"]` (fast, 5 credits). Add `"brave"` for a wider catalogue at +40 credits and ~8s extra latency. Example: `["bing", "brave"]`. Order doesn't matter; results are merged and deduped on the image URL. * `include_usage`: `false` by default. When `true`, the response carries a `usage` block with credit totals. * `categories`: comma-separated category filter (e.g. `"news"`). * `cache_ttl_seconds`. Request: ```bash curl -X POST https://scrapenest.dev/v1/search \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"query":"climate report 2026 ipcc","num_results":10}' ``` Response: ```json { "query": "climate report 2026 ipcc", "results": [ { "position": 1, "url": "https://...", "title": "...", "snippet": "...", "engine": "google", "engines": ["bing", "duckduckgo", "google"], "score": 1.0, "published_at": "2026-04-12T14:30:00Z", "thumbnail": null } ], "took_ms": 318, "cache_hit": false, "credits_charged": 5 } ``` Empty optional collections (`images`, `suggestions`, `answers`, `infoboxes`, `usage`, `note`) are omitted entirely. They appear only when the upstream actually returned something. With `include_images: true` the response adds an `images` array (`results` keeps the same shape as above): ```json { "query": "northern lights iceland", "results": [], "images": [ { "url": "https://.../aurora.jpg", "thumbnail": "https://.../aurora_thumb.jpg", "title": "Aurora over Reykjavik", "source_url": "https://...", "width": 1920, "height": 1080 } ], "took_ms": 318, "cache_hit": false, "credits_charged": 10 } ``` Field notes: * `engine` is the primary search engine that surfaced this URL. `engines` lists every engine that returned the same URL. When a result is confirmed by Bing + DuckDuckGo + Google, it's a stronger relevance signal than a Google-only hit. * `published_at` is parsed from the snippet when the search engine doesn't report it directly (e.g. "6 hours ago" becomes a real ISO timestamp). Useful for filtering recent news without a manual regex. Credit cost: 5 per call. 0 when no results are delivered. Cache hit: 1. --- ### 3.4 POST /v1/search/deep Search + fetch the top N pages in one round trip. Returns the same SERP plus the cleaned text of each top result. Same params as `/v1/search` (including all the filter, image, and usage flags, plus `page`), plus: * `fetch_top`: 1-10, default 5. * `render`: `"auto"`, `"direct"`, `"browser"`. * `depth`: `"fast"`, `"balanced"` (default), `"thorough"`, or `"stealth"`. On deep search `depth` governs the per-page fetch budget and anti-bot effort. `fast` (~2.5s/page) and `balanced` (~3.5s/page) fetch every page directly in parallel with no anti-bot escalation — bot-walled pages fall back to their SERP snippet. `thorough` (~6s/page) and `stealth` (~12s/page, maximum anti-bot bypass) escalate blocked pages through the real-browser bypass for higher coverage on protected sites. Request: ```bash curl -X POST https://scrapenest.dev/v1/search/deep \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"query":"how does retrieval-augmented generation work","fetch_top":3}' ``` Response (abbreviated): ```json { "query": "how does retrieval-augmented generation work", "results": [], "pages": [ {"url":"https://...","final_url":"https://...","status":200,"method":"fetch","title":"...","text":"...","structured":null,"error":null,"elapsed_ms":640} ], "took_ms": 2110, "fetched_count": 3, "cache_hit": false, "credits_charged": 20 } ``` The `results` array (omitted above for brevity) carries the same SERP shape as `/v1/search`. Each page reports `method` (`fetch`, `browser`, or `cf_bypass`), `status`, `final_url`, and `elapsed_ms`. Credit cost: `5 + 5 x pages_fetched`. Pages that fail to fetch are not counted. Cache hit: 1. --- ### 3.5 POST /v1/search/answer RAG-style: searches the web, fetches the top sources, then asks an LLM to write a cited answer. Returns the answer plus `[1]`-style citations mapping to the source URLs. Same params as `/v1/search/deep`, but `fetch_top` becomes `max_sources` (1-10, default 5). Image flags (`include_images`, `image_sources`, `include_image_descriptions`) are REJECTED here with 422. Use `/v1/search` if you need images. Request: ```bash curl -X POST https://scrapenest.dev/v1/search/answer \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"query":"is the new gpt-5 multimodal","max_sources":4}' ``` Response (abbreviated): ```json { "query": "is the new gpt-5 multimodal", "answer": "Yes. GPT-5 supports text, image, audio... [1][2]", "citations": [ {"index":1,"url":"https://openai.com/...","title":"...","snippet":"..."} ], "took_ms": 4200, "cache_hit": false, "credits_charged": 30 } ``` Credit cost: `5 + 5 x sources_fetched + 5` (answer overhead). Pages that fail to fetch are not counted. Cache hit: 1. --- ### 3.6 POST /v1/translate Translates text between languages. Required: * `text`: 1-5000 chars. * `target`: language code (`"es"`, `"ja"`, `"de"`, etc.). Accepts BCP-47 variants (`"pt-BR"`, `"zh-Hant"`, `"zh-CN"`, `"en-GB"`) and even underscore forms like `"en_US"`; they normalize through an internal alias map. Optional: * `source`: default `"auto"` (auto-detect). Same BCP-47 aliases apply. * `include_usage`: `false` by default. When `true`, response includes a `usage` block. Request: ```bash curl -X POST https://scrapenest.dev/v1/translate \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"text":"Hello world","target":"ja"}' ``` Response: ```json { "translated": "こんにちは世界", "detected_source": "en", "target": "ja", "credits_charged": 1 } ``` Fields: * `translated`: the translated text. * `detected_source`: language detected (BCP-47). * `target`: echoes the requested target. * `note`: non-null when behaviour was adjusted (source equals target, alias applied, fallback detector ran). * `credits_charged`: 1 per call, or 0 when source equals (or auto-detects to) target, since no translation is performed in that case. Credit cost: 1 per call. 0 when source equals target. --- ### 3.7 POST /v1/audio/transcript Timestamped transcript of any audio source. Supports podcast episodes (Apple Podcasts, Overcast, RSS), YouTube (captions when available, else audio transcription), X Spaces, SoundCloud, Facebook video, TikTok, and direct MP3/M4A/WAV/OGG links. Required: * `url`: source URL. Optional: * `language`: ISO 639-1 hint (`"en"`, `"es"`). * `cookies`: Netscape-format cookies.txt content as a single string (only needed for some YouTube fallbacks; use a burner Google account, never your personal one). Cookies are written to a request-scoped tempfile, used once, then deleted. Never logged or cached. * `include_usage`: `false` by default. When `true`, the response includes a `usage` block with credit totals. * `cache_ttl_seconds`: up to 30 days. Request: ```bash curl -X POST https://scrapenest.dev/v1/audio/transcript \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://podcasts.apple.com/us/podcast/.../id..."}' ``` Response (abbreviated): ```json { "url": "...", "audio_url": "https://traffic.megaphone.fm/....mp3", "canonical_url": "https://podcasts.apple.com/us/podcast/.../id...", "title": "Episode title", "channel": "Show name", "thumbnail": "https://image-resolver/.../600x600.jpg", "duration_seconds": 3618.2, "language": "en", "source": "apple_podcasts", "chunks": [ {"start": 0.0, "end": 4.2, "text": "Welcome back to the show..."} ], "text": "Welcome back to the show... [full transcript]", "took_ms": 41200, "cache_hit": false, "credits_charged": 620 } ``` `chunks` is omitted when the underlying resolver returned a transcript without segment timestamps (some caption sources, plain-text fallback). Credit cost: 20 per 2 minutes of audio (rounded up). Minimum 20 per call. YouTube anti-abuse: Shorts and certain age-gated/region-locked videos can return a `login_required` error from upstream. When that happens the error `message` instructs you to pass `cookies` (Netscape cookies.txt format) in the next request. The resolver writes them to a request-scoped tempfile, uses them once, then deletes the file before responding. Use a burner Google account, never your personal one. If the source requires downloading and processing the full audio file (long YouTube videos, full podcast episodes), the synchronous call can exceed the 100s edge timeout and fail with HTTP 524. Use the async variant below for anything you expect to take more than ~80 seconds end-to-end. --- ### 3.8 POST /v1/audio/transcript/async Identical request shape to `/v1/audio/transcript`. Returns `202` with a `run_id` immediately; poll `GET /v1/scrape/runs/{run_id}` for the completed transcript. Request: ```bash curl -X POST https://scrapenest.dev/v1/audio/transcript/async \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://www.youtube.com/watch?v=...","language":"en"}' ``` Response (202): ```json { "run_id": "f0e0c0a0-1234-5678-9abc-def012345678", "status": "queued", "status_url": "/v1/scrape/runs/f0e0c0a0-1234-5678-9abc-def012345678", "created_at": "2026-05-14T16:00:00Z", "expires_at": "2026-05-21T16:00:00Z" } ``` Then poll `/v1/scrape/runs/{run_id}` until `status="completed"`; the synchronous response body lives in `result`. Failed runs are not billed. Credit cost: same as the sync endpoint (20 per 2 minutes of audio). --- ### 3.9 POST /v1/scrape/google-maps/reviews Pull Google Maps reviews for any place. Returns up to ~10,000 unique reviews in the `reviews` array, deduplicated and always sorted newest-first. Prefer the async variant (`/v1/scrape/google-maps/reviews/async`, see 3.10) for any pull beyond ~700 reviews so the request isn't bounded by the request time limit. Required: * `place_id`: accepts two forms: - A full Google Maps URL (paste the link from the address bar; the server pulls the identifier from the URL). - A FID hex pair like `0x47e66e2964e34e2d:0x8ddca9ee380ef7e0`. This skips an internal resolution step so it's slightly faster. Optional: * `place_url`: optional canonical Maps URL. Pass when the place's FID has been rotated and the legacy short URL redirects to a generic location. * `language`: default `"en"`. Affects place name/address language only; review text is whatever the reviewer wrote. * `sort`: accepted values `"newest"` (default), `"most_relevant"`, `"highest_rating"`, `"lowest_rating"`. Reviews are returned newest-first regardless of this value. Kept for backward compatibility. * `max_reviews`: 1-10000. Default 200. Treat as a request CAP, not a promise. Returns up to ~10,000 unique reviews; the per-request ceiling lands around 10,000 unique on places with >20K total reviews. * `cache_ttl_seconds`: default 3600. * `include_usage`: `false` by default. When `true`, the response includes a `usage` block with credit totals. Auto-downgrades to 40 credits when 50 or fewer reviews are returned, so you pay only for what's available. REALISTIC DELIVERY by place size: | Place review count | Typical delivery | |--------------------|---------------------| | < 1,000 | ~100% of available | | 1,000 – 10,000 | 60–100% | | 10,000+ | ~10,000 (the per-request cap) | WHY THE PER-REQUEST CAP, and how some services claim 100K: Google itself only exposes a limited slice of any place's reviews, regardless of how many the place actually has. On places with >20K total reviews the practical ceiling lands around 10,000 unique. We cap at 10,000 because asking for more is misleading: no public scraper delivers more through any documented method. DataForSEO publishes its real per-place cap at 4,490. SerpAPI documented in their own bug tracker that pagination stops where Google's UI scroll stops. Six-figure "reviews per place" marketing numbers are aspirational ceilings, not measured deliveries against the largest places (which have hundreds of thousands of reviews). Request: ```bash curl -X POST https://scrapenest.dev/v1/scrape/google-maps/reviews \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "place_id": "0x47e66e2964e34e2d:0x8ddca9ee380ef7e0", "max_reviews": 5000 }' ``` Response (abbreviated): ```json { "place_id": "0x47e66e2964e34e2d:0x8ddca9ee380ef7e0", "name": "London Eye", "address": "Riverside Building, County Hall, London SE1 7PB, UK", "rating": 4.5, "review_count": 89234, "google_maps_url": "https://www.google.com/maps/place/...", "reviews": [ { "review_id": "ChdDSUhNMG9nS0VJQ0FnSURBNS0z...", "review_url": "https://www.google.com/maps/contrib/.../reviews?hl=en", "author_name": "Jane D.", "author_id": "1234567890", "author_profile_url": "https://www.google.com/maps/contrib/1234567890/reviews?hl=en", "author_avatar_url": "https://lh3.googleusercontent.com/...", "rating": 5, "text": "Amazing view, worth the wait.", "language": "en", "relative_time": "2 weeks ago", "posted_at": 1716480000, "posted_at_iso": "2026-05-13T00:00:00Z", "helpful_count": 4 } ], "star_histogram": [820, 540, 1480, 8930, 77464], "note": "Returned 4982 unique reviews after deduplication.", "took_ms": 197000, "cache_hit": false, "credits_charged": 10000 } ``` `star_histogram` is a 5-element list (counts in order `[1★, 2★, 3★, 4★, 5★]`) or `null` if Google didn't ship one. `photos` is omitted from each review when the reviewer didn't attach any. `owner_reply` is omitted when the business didn't respond. Credit cost: * 100 credits per 50 reviews returned, rounded up to the next 50. (Equivalently: 400 credits per 200 reviews.) Hard cap 10,000 reviews. * When 50 or fewer reviews are returned, auto-billed at the 40-credit floor. For the example above, 4982 reviews rounds up to 5000 (100 chunks of 50), billed at 100 x 100 = 10,000 credits. Failed runs are not billed. Bigger jobs can run for several minutes. Use the async variant (3.10) for any pull above ~700 reviews so the request isn't tied to a single open HTTP connection. --- ### 3.10 POST /v1/scrape/google-maps/reviews/async Identical request shape to the sync endpoint. Returns immediately with a `run_id`; poll `GET /v1/scrape/runs/{run_id}` for the result. Response (202): ```json { "run_id": "f0e0c0a0-1234-5678-9abc-def012345678", "status": "queued", "status_url": "/v1/scrape/runs/f0e0c0a0-1234-5678-9abc-def012345678", "created_at": "2026-05-13T15:00:00Z", "expires_at": "2026-05-20T15:00:00Z" } ``` Then poll: ```bash curl -X GET https://scrapenest.dev/v1/scrape/runs/f0e0c0a0-... \ -H "Authorization: Bearer sn_YOUR_API_KEY" ``` When complete, `status="completed"` and the synchronous response body is inside `result`. Credit cost is the same as the synchronous endpoint and is charged when the job completes. Failed runs are not billed. --- ### 3.11 GET /v1/scrape/runs/{run_id} Polls the status of any async run. Accepts GET or HEAD. Returns one of `queued`, `running`, `completed`, `failed`, `cancelled`. When `completed`, `result` holds the final response body. When `failed`, `error_code` and `error_message` describe the cause. Free: polling does not consume credits. Long-running scrapes (Maps reviews) populate an optional `progress` object: `{"current": int, "target": int, "detail": str}`. Use this for progress bars; null when the worker hasn't yet emitted a milestone for this run. To stop an in-flight run, hit `POST /v1/scrape/runs/{run_id}/cancel`. The background worker stops on its next checkpoint (cooperative cancellation). Any work already performed is billed; there is no refund. Polling cadence: 2-5 seconds is fine for most scrapes. For maps reviews runs targeting 5K reviews, 10-15 seconds is more polite. Runs expire after 7 days. --- ### 3.12 POST /v1/scrape/screenshot Captures a PNG or JPEG screenshot of any public URL. Returns base64-encoded image bytes inline. Auto-escalates the capture pipeline when the target site blocks the default browser engine. Required: * `url`: absolute http(s) URL. Optional: * `viewport_width`: 320-3840, default 1280. * `viewport_height`: 240-2160, default 720. * `full_page`: `false` by default. When `true`, captures the entire scrollable height instead of just the viewport. * `selector`: CSS selector (up to 300 chars). When set, captures only the bounding box of the first matching element (e.g. `"#hero"`, `".product-card:first-child"`). Overrides `full_page`. Captures the first element matching `selector`; if the selector doesn't match, the capture falls back to the full viewport; returns `422 selector_not_found` only when nothing can be rendered. * `render`: `"auto"` (default, picks the right backend), `"browser"` (forces the standard headless browser), or `"stealth"` (forces the anti-bot capture backend up front, 35 credits — use when you already know the target blocks normal browsers). * `format`: `"png"` (default) or `"jpeg"`. * `jpeg_quality`: 1-100, default 85. Ignored when format is png. * `wait_extra_ms`: 0-10000, default 0. Additional wait after page load for late-loading content (SPAs that fade in, lazy images, charts). * `cookies`: object `{name: value}` (up to 50 entries). Injected before navigation; use to screenshot authenticated/logged-in pages. The `[{name, value}]` list form is also accepted here. * `headers`: object `{name: value}` (up to 30 entries). Extra request headers (`User-Agent`, `Referer`, etc.). `Host` / `Content-Length` are ignored. * `session_id`: sticky-session id (1-64 chars, `[A-Za-z0-9_.-]`). Reuses the same exit IP for ~10 min so a screenshot matches a prior `/v1/scrape/url` session. Disables caching. * `include_usage`: `false` by default. When `true`, the response includes a `usage` block with credit totals. * `cache_ttl_seconds`: 0-86400, default 0 (off). Set non-zero to dedupe repeat captures of the same URL+viewport+format. Request: ```bash curl -X POST https://scrapenest.dev/v1/scrape/screenshot \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://news.ycombinator.com", "full_page": true, "format": "png" }' \ | jq -r .image_base64 | base64 -d > screenshot.png ``` Response: ```json { "url": "https://news.ycombinator.com", "final_url": "https://news.ycombinator.com/", "status_code": 200, "method": "browser", "proxy_used": null, "elapsed_ms": 2470, "width": 1280, "height": 1179, "bytes": 253264, "format": "png", "image_base64": "iVBORw0KGgoAAAANSUhEUgAA...", "cache_hit": false, "credits_charged": 25 } ``` The `method` field reports `"browser"` or `"stealth"` when a render succeeds, and `"unknown"` only when no engine could load the page. Credit cost: * Viewport: 15 * Full page: 25 * Selector-targeted: 15 * Stealth fallback (hostile sites, auto-escalated): 35 * Cache hit: 1 Failures and timeouts don't bill. This endpoint streams its response, so a handler error arrives as an in-body envelope `{"error": {...}, "http_status": 502}` inside an HTTP 200, not as a transport-level 502. --- ### 3.13 POST /v1/scrape/pdf Renders any URL to PDF. Returns base64-encoded bytes. Use for archiving articles, generating invoice/receipt copies, or capturing legal evidence. Required: * `url`: absolute http(s) URL. Optional: * `format`: paper size. One of `"Letter"` (default), `"Legal"`, `"Tabloid"`, `"Ledger"`, `"A0"`, `"A1"`, `"A2"`, `"A3"`, `"A4"`, `"A5"`, `"A6"`. * `landscape`: `false` by default. * `viewport_width` / `viewport_height`: render viewport in CSS pixels (320-3840 / 240-2160, defaults 1280x720). * `wait_extra_ms`: 0-10000, default 0. * `cookies`: object `{name: value}`. Use for paywalled or logged-in pages. The `[{name, value}]` list form is also accepted here. * `headers`: object `{name: value}`. * `session_id`: sticky session id (see `/v1/scrape/url`). * `include_usage`: `false` by default. * `cache_ttl_seconds`: 0-86400, default 0. Request: ```bash curl -X POST https://scrapenest.dev/v1/scrape/pdf \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://example.com/article","format":"A4"}' \ | jq -r .pdf_base64 | base64 -d > article.pdf ``` Response: ```json { "url": "https://example.com/article", "final_url": "https://example.com/article", "status_code": 200, "proxy_used": null, "elapsed_ms": 2120, "format": "A4", "landscape": false, "bytes": 20925, "pdf_base64": "JVBERi0xLjQKJdPr6eEKMSAwIG9iago...", "cache_hit": false, "credits_charged": 25 } ``` `landscape` echoes back whether the PDF was rendered in landscape orientation. Credit cost: * Per PDF: 25 * Landscape, or a large format (`Tabloid`, `Ledger`, `A0`, `A1`, `A2`): 35 * Cache hit: 1 Failures don't bill. This endpoint streams its response, so a handler error arrives as an in-body envelope `{"error": {...}, "http_status": 502}` inside an HTTP 200, not as a transport-level 502. --- ### 3.14 POST /v1/scrape/gas-prices US gas station prices for a given location, pulled from public station-finder listings. Returns the cheapest stations within range, plus state/region rollup stats when they're available. Required: * `location`: 2-200 chars. Accepts a 5-digit US ZIP (`"37122"`, `"37122-1234"`), `"City, ST"` (`"Mount Juliet, TN"`), `"City, State"` (`"Nashville, Tennessee"`), or a bare city. Ambiguous strings may resolve to the nearest match; check `location.city` / `location.state` in the response. Optional: * `grade`: fuel grade. One of `"regular"` (default), `"midgrade"`, `"premium"`, `"diesel"`, `"e85"`. * `limit`: 1-50, default 20. Hard cap on returned stations (the default listing shows ~13). * `cache_ttl_seconds`: 0-3600, default 600 (10 minutes). Capped at 1 hour because prices change slowly. * `include_usage`: `false` by default. When `true`, response includes a `usage` block. Request: ```bash curl -X POST https://scrapenest.dev/v1/scrape/gas-prices \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"location":"37122","grade":"regular","limit":10}' ``` Response: ```json { "location": { "query": "37122", "zip": "37122", "city": "Mount Juliet", "state": "TN" }, "grade": "regular", "stations": [ { "brand": "Kroger", "sub_brand": "Kroger Fuel Center", "station_id": "21847", "address": "401 S Mt Juliet Rd", "city": "Mount Juliet", "state": "TN", "price": 2.79, "currency": "USD", "unit": "USD/gallon", "reporter": "anon_18234", "updated_relative": "37 minutes ago" } ], "region_stats": { "state": "TN", "region": "Tennessee", "region_type": "state", "lowest_price": 2.69, "average_price": 2.95 }, "source_url": "https://scrapenest.dev", "fetched_at": "2026-05-19T17:30:00Z", "took_ms": 1820, "cache_hit": false, "credits_charged": 10 } ``` `region_stats.region_type` is `"state"` when the source returned a state roll-up and `"city"` when it returned city-level numbers (mainly for ZIPs that map to a city-only feed). `region_stats` is omitted when the source doesn't ship a regional summary block. Upstream caveat: some listings hide individual per-station prices (typically the cheapest few stations) behind a free account login. When the listing returned stations but some had no prices, the response sets `note` to something like `"3 stations had prices hidden."`. You'll still see those stations, just with `price: null`. Credit cost: * Per call: 10 (flat). Charged only when at least one station is returned. * Cache hit: 1 --- ### 3.15 POST /v1/scrape/batch Fetch up to 10 URLs in one call. Each URL is fetched independently and concurrently, so one URL failing never fails the rest, and you wait only as long as the slowest fetch instead of summing them. The render options apply to every URL in the batch. Required: * `urls`: array of 1-10 absolute http(s) URLs. Optional (each applies to every URL): * `render`: `"auto"` (default), `"direct"`, `"browser"`, or `"stealth"`. Same semantics as `/v1/scrape/url`. * `force_proxy`: `false` by default. Force every fetch through a higher-tier proxy. * `block_assets`: `true` by default. On browser renders, block images/fonts/media to save bandwidth. * `extract`: `true` by default. Run readability extraction so each item carries clean `title`, `description`, `text`, `structured`. * `include_html`: `false` by default. Include the raw `html` per URL alongside the cleaned text. No extra credits. * `return_markdown`: `false` by default. Add a `markdown` field per URL. Implies `include_html`. No extra credits. * `block_ads`: `false` by default. On browser renders, block known ad and tracker domains. * `country`: ISO 3166-1 alpha-2. Routes every fetch through a residential proxy in that country. Forces a higher-tier proxy. Same supported set as `/v1/scrape/url`: `us`, `be`, `de`, `fr`, `it`, `sg`. * `include_usage`: `false` by default. When `true`, the response includes a `usage` block with the batch's total credits. This endpoint does not cache: it accepts no `cache_ttl_seconds` and never returns `cache_hit`. Use `/v1/scrape/url` when you want per-URL caching. Request: ```bash curl -X POST https://scrapenest.dev/v1/scrape/batch \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": [ "https://example.com/a", "https://example.com/b", "https://example.com/missing" ], "render": "auto" }' ``` Response (abbreviated): ```json { "results": [ {"url":"https://example.com/a","final_url":"https://example.com/a","status_code":200,"method":"fetch","proxy_used":"datacenter","elapsed_ms":290,"title":"A","text":"...","ok":true,"credits_charged":1,"error":null}, {"url":"https://example.com/b","final_url":"https://example.com/b","status_code":200,"method":"browser","proxy_used":"residential","elapsed_ms":1840,"title":"B","text":"...","ok":true,"credits_charged":40,"error":null}, {"url":"https://example.com/missing","status_code":404,"method":"fetch","proxy_used":"datacenter","elapsed_ms":210,"ok":false,"credits_charged":1,"error":"http_404"} ], "requested": 3, "succeeded": 2, "credits_charged": 42 } ``` Each item carries its own `ok`, `status_code`, and `credits_charged`, so the batch's `credits_charged` is just the sum of the per-URL charges. Credit cost (per URL, then summed): * Each URL bills the tier it actually used, exactly like `/v1/scrape/url` (1 direct + no proxy, 5 browser + no proxy, 10 direct + premium proxy, 40 browser/stealth + premium proxy). * A delivered 404/410 (the page was reached, just not found) bills 1. * A URL we couldn't reach at all bills 0. * No cache hits apply (this endpoint never caches). When `include_usage` is set, `usage.breakdown` reports the batch total under a single `{"fetch": }` key rather than itemizing per URL. --- ## 4. Error codes All errors are JSON with shape: ```json {"error": {"code": "", "message": ""}} ``` Validation failures (HTTP 422 `validation_error`) include an extra structured array: ```json { "error": { "code": "validation_error", "message": "Request validation failed.", "details": [ {"loc": ["body", "max_reviews"], "type": "less_than_equal", "msg": "Input should be less than or equal to 10000"} ] } } ``` Common codes: | HTTP | Code | Meaning | What to do | |------|--------------------------|---------------------------------------------------------|---------------------------------------------| | 400 | unsafe_url | URL points at a private/internal/loopback host | Send a public http(s) URL | | 400 | redirect_to_unsafe_url | Target redirected the fetch to a private address | Use a different URL; we won't follow | | 400 | cookies_invalid | /v1/audio/transcript only: cookies field malformed | Re-export Netscape cookies.txt | | 400 | invalid_run_id | run_id is not a UUID | Use the run_id from the original 202 body | | 400 | place_id_format | place_id format is invalid | Use a Google Maps URL or `0x...:0x...` FID | | 400 | unsupported_target_language | /v1/translate: `target` is not a supported language | Use a supported language code | | 400 | translation_rejected | /v1/translate: upstream rejected the language pair | Check source/target are translatable | | 401 | unauthorized | Missing/wrong Bearer token | Check key in /dashboard/keys | | 401 | login_required | /v1/audio/transcript: source needs auth cookies | Pass `cookies` (Netscape cookies.txt) | | 402 | insufficient_credits | Out of monthly credits | Top up at /dashboard/billing | | 402 | overage_cap_reached | Hit the negative-balance floor (5,000 credits) | Top up to resume | | 402 | bandwidth_cap_exceeded | Free-tier daily bandwidth budget exhausted | Upgrade to a paid plan | | 403 | no_owner | API key has no owning account (revoked or detached) | Generate a new key | | 403 | email_not_verified | Account email not verified | Click verification link or /verify/resend | | 403 | account_suspended | Account suspended | Contact support | | 403 | audio_private | /v1/audio/transcript: audio is private or members-only | Use a public source | | 402 | team_seat_blocked | Team is over its plan's seat cap | Upgrade the plan or remove members | | 404 | not_found | Unknown route, or the requested place/resource doesn't exist | Verify path, method, and inputs | | 404 | run_not_found | run_id doesn't belong to this customer or has expired | Confirm key + check 7-day expiry | | 404 | place_id_unresolvable | Maps: couldn't resolve the shortlink/CID to a place | Pass the full Maps URL with `!1s0x...:0x...` | | 405 | method_not_allowed | Wrong HTTP method on a /v1 route | Use the documented method (check `Allow`) | | 410 | audio_not_available | /v1/audio/transcript: source is no longer available | The audio was deleted; use another source | | 422 | validation_error | Validation failed on request body | Read `details[]` and fix the field | | 422 | drm_protected | /v1/audio/transcript: source is DRM-protected | Try the same episode on another platform | | 429 | rate_limited | Hit per-minute or per-day rate limit | Back off; see `Retry-After` header | | 429 | abuse_throttled | Account throttled by abuse-detection | Wait `retry_after`; contact support if persistent | | 451 | geo_restricted | /v1/audio/transcript: source is geo-restricted | The source is blocked from our region | | 500 | server_error | Unexpected server error | Retry once with backoff; report if persistent | | 502 | upstream_failed | Target site or an upstream service errored | Retry once with backoff | | 502 | pdf_failed | Target could not be rendered to PDF | Retry once or try a different URL | | 502 | screenshot_failed | Target could not be screenshotted | Retry once or try a different URL | | 502 | no_sources | /v1/search/answer couldn't find usable sources | Rephrase the query | | 502 | parse_failed | Maps returned no parseable reviews | Retry once; if persistent, file a bug | | 502 | rpc_parse_failed | Google Maps returned a malformed response | Retry once; if persistent, file a bug | | 502 | rpc_unexpected_body | Google Maps returned an unexpected (non-JSON) response | Retry once; often a transient challenge | | 502 | llm_upstream_failed | /v1/search/answer: the answer LLM upstream errored | Retry once with backoff | | 503 | upstream_not_configured | An upstream we depend on isn't configured server-side | Contact support | | 503 | llm_not_configured | /v1/search/answer: no answer LLM is configured | Contact support | | 504 | — | Request exceeded the gateway timeout | Use the `/async` variant for long-running scrapes | Retry logic should be: on 5xx, retry with exponential backoff (1s, 3s, 9s) up to 3 times. On 429, honor `Retry-After`. On 4xx, do NOT retry; fix the request instead. --- ## 5. Rate limits per plan See `/v1/me` for the exact rate limits applied to your key. | Plan | $/mo | Credits/mo | Req/min | Req/day | |----------|------|------------|---------|---------| | Free | $0 | 1,000 | 30 | 500 | | Starter | $29 | 200,000 | 60 | 10,000 | | Growth | $99 | 1,500,000 | 120 | 50,000 | | Pro | $249 | 4,000,000 | 300 | 200,000 | | Scale | $599 | 10,000,000 | 600 | 500,000 | When you hit a rate limit you get an HTTP 429 with `Retry-After` (in seconds). Back off for the indicated duration before retrying. --- ## 6. Official SDKs and code examples Official client libraries are published for Python and Node/TypeScript. They wrap every endpoint below, handle auth and retries, and return typed results. Both default to base URL `https://scrapenest.dev`. Prefer these over hand-rolled HTTP when you can. ### Python SDK ```bash pip install scrapenest ``` ```python from scrapenest import Client client = Client("sn_YOUR_API_KEY") hits = client.search("python web scraping", num_results=5) for r in hits.results: print(r.title, r.url) page = client.scrape_url("https://example.com", render="auto", return_markdown=True) print(page.title) answer = client.search_answer("who wrote the odyssey", max_sources=3) print(answer.answer) ``` The async client exposes the same methods: ```python import asyncio from scrapenest import AsyncClient async def main(): async with AsyncClient("sn_YOUR_API_KEY") as client: result = await client.search("python web scraping") print(result.results[0].title) asyncio.run(main()) ``` Large Maps pulls run as async jobs with built-in polling: ```python run = client.maps_reviews_async("0x6b12ae665e892fdd:0x3133f8d75a1ac251", max_reviews=2000) final = client.wait_for_run(run.run_id) print(len(final.result["reviews"]), "reviews") ``` Typed errors: `RateLimitError`, `AuthenticationError`, `PaymentRequiredError`. ### Node / TypeScript SDK ```bash npm install scrapenest ``` ```typescript import { ScrapeNest } from "scrapenest"; const client = new ScrapeNest({ apiKey: "sn_YOUR_API_KEY" }); const hits = await client.search({ query: "python web scraping", num_results: 5 }); for (const r of hits.results) console.log(r.title, r.url); const page = await client.scrapeUrl({ url: "https://example.com", render: "auto" }); console.log(page.title); const run = await client.mapsReviewsAsync({ place_id: "0x6b12ae665e892fdd:0x3133f8d75a1ac251", max_reviews: 2000 }); const final = await client.waitForRun(run.run_id); console.log(final.status, final.credits_charged); ``` The package ships ESM + CJS builds and full type definitions, and exports the same typed errors (`RateLimitError`, `AuthenticationError`, `PaymentRequiredError`). Requires Node.js 18+; the Python SDK requires Python 3.9+. ### Raw HTTP: Python (httpx + asyncio) ```python import asyncio import httpx API_KEY = "sn_YOUR_API_KEY" BASE = "https://scrapenest.dev" HEADERS = {"Authorization": f"Bearer {API_KEY}"} async def fetch_reviews(place_id: str, max_reviews: int = 2000) -> dict: async with httpx.AsyncClient(base_url=BASE, headers=HEADERS, timeout=300.0) as client: r = await client.post( "/v1/scrape/google-maps/reviews", json={"place_id": place_id, "max_reviews": max_reviews}, ) r.raise_for_status() return r.json() async def fetch_reviews_async(place_id: str, max_reviews: int = 5000) -> dict: async with httpx.AsyncClient(base_url=BASE, headers=HEADERS, timeout=30.0) as client: submit = await client.post( "/v1/scrape/google-maps/reviews/async", json={"place_id": place_id, "max_reviews": max_reviews}, ) submit.raise_for_status() run_id = submit.json()["run_id"] while True: await asyncio.sleep(8) status = await client.get(f"/v1/scrape/runs/{run_id}") status.raise_for_status() body = status.json() if body["status"] in ("completed", "failed", "cancelled"): return body print(asyncio.run(fetch_reviews("0x47e66e2964e34e2d:0x8ddca9ee380ef7e0"))) ``` ### Raw HTTP: Node (fetch) ```js const API_KEY = "sn_YOUR_API_KEY"; const BASE = "https://scrapenest.dev"; async function search(query) { const r = await fetch(`${BASE}/v1/search`, { method: "POST", headers: { "Authorization": `Bearer ${API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ query, num_results: 10 }), }); if (!r.ok) throw new Error(`HTTP ${r.status}: ${await r.text()}`); return r.json(); } async function pollRun(runId, { intervalMs = 5000, maxMs = 1_800_000 } = {}) { const start = Date.now(); while (Date.now() - start < maxMs) { const r = await fetch(`${BASE}/v1/scrape/runs/${runId}`, { headers: { "Authorization": `Bearer ${API_KEY}` }, }); const body = await r.json(); if (["completed", "failed", "cancelled"].includes(body.status)) return body; await new Promise(s => setTimeout(s, intervalMs)); } throw new Error("run polling timed out"); } console.log(await search("typescript fetch retry pattern")); ``` ### Raw HTTP: curl one-liners Check your key (free): ```bash curl -s https://scrapenest.dev/v1/me \ -H "Authorization: Bearer sn_YOUR_API_KEY" | jq ``` Search (cheap): ```bash curl -s -X POST https://scrapenest.dev/v1/search \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"query":"openai gpt-5 release notes","num_results":5}' | jq ``` Kick off a big maps reviews job and poll (heavy): ```bash RUN=$(curl -s -X POST https://scrapenest.dev/v1/scrape/google-maps/reviews/async \ -H "Authorization: Bearer sn_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"place_id":"0x47e66e2964e34e2d:0x8ddca9ee380ef7e0","max_reviews":5000}' \ | jq -r .run_id) while true; do STATUS=$(curl -s https://scrapenest.dev/v1/scrape/runs/$RUN \ -H "Authorization: Bearer sn_YOUR_API_KEY") STATE=$(echo "$STATUS" | jq -r .status) [ "$STATE" = "completed" ] || [ "$STATE" = "failed" ] && { echo "$STATUS" | jq; break; } sleep 10 done ``` --- ## 7. Common workflows ### 7.1 "Reviews -> sentiment summary" for a place 1. POST `/v1/scrape/google-maps/reviews/async` with `max_reviews=5000`. 2. Poll `/v1/scrape/runs/{id}` every 10s until `status="completed"`. 3. Read `result.reviews[*].text`, batch them in chunks of 50, feed each chunk to your LLM with a prompt like "Summarize themes and rate sentiment 1-5 per review." Map results back via `review_id`. ### 7.2 "Topic research" with citations 1. POST `/v1/search/answer` with a clear question and `max_sources=6`. 2. The response gives you a written answer plus `citations[]`. 3. If the answer needs deeper backing, take `citations[*].url`, fan out to `/v1/scrape/url` (one call per URL, `extract=true`), and feed the `text` field back into your own LLM prompt. ### 7.3 "Podcast -> show notes" 1. POST `/v1/audio/transcript` with the episode URL. 2. From the response, take `chunks[]` (start/end/text) and ask your LLM: "Group these into 5-10 topic chapters, each with timestamps from the chunks." This works because chunks are already time-anchored. --- ## 8. Common mistakes * **Running large Maps scrapes on the sync endpoint.** For pulls beyond a few hundred reviews, submit via `/v1/scrape/google-maps/reviews/async` and poll `/v1/scrape/runs/{run_id}`. The sync endpoint is best for small places and quick interactive checks. Polling is free. * **Getting the FID hex for large places.** Pasting the full Google Maps URL works, but if you already have the FID hex form (`0x47e6...:0x8ddc...`) it's slightly faster since it skips an internal resolution step. The FID is in the Google Maps URL after `!1s` (e.g. `!1s0x47e6...!8m2!`). * **Assuming `max_reviews=10000` always returns 10K.** Google only makes a limited slice available, roughly 1.5K-8K reviews on most places and ~200 on small spots. No scraper can exceed what Google itself indexes. The practical ceiling lands around 10,000 unique on places with >20K total reviews. Always check `len(response.reviews)`. * **Ignoring `cache_hit: true`.** A cached response costs 1 credit and is identical to the live response. If you're comparing prices or detecting changes, set `cache_ttl_seconds: 0` to force fresh data. * **Storing API keys in client-side code.** Bearer keys give full account access. Keep them in your backend; never ship them in a browser bundle or mobile app. * **Retrying on 4xx.** 4xx errors are caused by your request, so fix it instead of retrying. Only retry 5xx and 429 (with backoff and `Retry-After`). * **Asking for `max_reviews=200` on a place that has fewer.** You'll be auto-billed at the 40-credit floor, but the wall clock still reflects however many reviews Google actually returns. Set `max_reviews` to a realistic target. * **Hammering `/v1/scrape/runs/{id}` every 100ms.** Polling is free but it counts against your rate limit. 2-10 seconds is plenty for most scrapes; 10-15s is fine for big maps runs. * **Reimplementing polling and retries by hand.** The official Python and Node SDKs (section 6) wrap auth, async job polling (`wait_for_run` / `waitForRun`), and typed errors. Reach for them before writing your own HTTP client. --- ## 9. MCP server (Model Context Protocol) ScrapeNest is also a hosted MCP server, so AI assistants (Claude, Cursor, VS Code, LM Studio, ChatGPT) can call it as native tools. * Endpoint: https://scrapenest.dev/mcp * Transport: streamable HTTP, stateless, JSON responses * Auth: `Authorization: Bearer sn_...` header, OR `?apiKey=sn_...` in the URL for clients with no header field (Claude.ai web, ChatGPT). Any key from the dashboard works; no separate MCP key. * Default tools (thin wrappers over the section-3 endpoints, same credit cost, output footer shows credits charged): web_search (/v1/search), ask_web (/v1/search/answer), deep_search (/v1/search/deep), scrape_url (/v1/scrape/url), translate_text (/v1/translate), scrapenest_account (/v1/me). * ChatGPT deep research: append `?profile=chatgpt` — exposes the `search` and `fetch` tools ChatGPT connectors require. * Claude Code one-liner: claude mcp add scrapenest --transport http --header "Authorization: Bearer $SCRAPENEST_API_KEY" https://scrapenest.dev/mcp * Tool calls are capped at ~90s; a call cancelled for exceeding the budget is not billed. For longer stealth scrapes call /v1 directly with a higher timeout_seconds. * Full setup guide: https://scrapenest.dev/docs#mcp --- ## Where to next * OpenAPI spec (machine-readable): https://scrapenest.dev/openapi.json * Interactive Swagger UI: https://scrapenest.dev/reference * Python SDK: `pip install scrapenest` (PyPI: scrapenest) * Node/TypeScript SDK: `npm install scrapenest` (npm: scrapenest) * SDK docs and usage: https://scrapenest.dev/docs * MCP server (Claude/Cursor/ChatGPT): https://scrapenest.dev/mcp — guide at https://scrapenest.dev/docs#mcp * Dashboard + key management: https://scrapenest.dev/dashboard * Status / changelog: https://scrapenest.dev/changelog * Security disclosures: https://scrapenest.dev/.well-known/security.txt * Terms of service: https://scrapenest.dev/terms * Privacy policy: https://scrapenest.dev/privacy * Acceptable use policy: https://scrapenest.dev/acceptable-use