guide

Scraping JavaScript-rendered pages with one API call

Many sites now send the browser an almost empty HTML shell and build the page with JavaScript. Fetch one of those with a plain HTTP client and you get <div id="root"></div> and a pile of script tags, not the prices or articles you came for. The usual fix is to run a headless browser yourself, which means keeping browsers alive, waiting for the right moment to read the page, and paying for machines that sit idle between jobs. This guide shows how to get the rendered page from one call to POST /v1/scrape/url, how the four render modes differ, and how to keep the cost of each page down.

The four render modes

The render field decides how a page is fetched:

ModeWhat it doesCredits
autoThe default. Picks a method per page. A page that comes back as an empty JavaScript shell gets one browser render, and a page refused by anti-bot protection gets one retry in stealth mode.The tier that delivered the page
directA plain HTTP request with no JavaScript. It never escalates, so a block comes back as it is.1
browserA headless browser renders the page before the text is extracted. Like auto, it retries once in stealth mode when anti-bot protection refuses it.5, or the stealth rate when that retry delivers the page
stealthAnti-bot evasion for pages with bot protection. Slower than the other modes.Up to 40, by the method that served it

Behind those prices are four tiers: a plain fetch is 1 credit, a browser render 5, a residential proxy without a browser 10, and a residential proxy with a browser 40. A repeat inside the cache window bills 1 credit. Every response reports the tier in method and proxy_used, and the price in credits_charged, so you can see what each page took. Set auto_escalate to false to keep the mode you picked and get a block back as it is.

Your first call

Start with render="auto" and ask for Markdown, which keeps headings, links and tables and reads well as input for a language model. In Python:

import os
from scrapenest import Client

client = Client(os.environ["SCRAPENEST_API_KEY"])

page = client.scrape_url(
    "https://example.com/products",
    render="auto",
    return_markdown=True,
)
if page.success:
    print(page.method, page.proxy_used, page.credits_charged)
    print((page.markdown or "")[:1000])
else:
    print("not delivered:", page.error_code, page.error)

In TypeScript:

import { ScrapeNest } from "scrapenest";

const client = new ScrapeNest({ apiKey: process.env.SCRAPENEST_API_KEY ?? "" });

const page = await client.scrapeUrl({
  url: "https://example.com/products",
  render: "auto",
  return_markdown: true,
});
console.log(page.success, page.method, page.credits_charged);

And with curl:

curl -X POST https://scrapenest.dev/v1/scrape/url \
  -H "Authorization: Bearer $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products", "render": "auto", "return_markdown": true}'

You need a ScrapeNest key for these: sign up, verify your email address, then mint a key on the dashboard. The free plan includes 1,000 credits a month.

Check success, not the status code

This endpoint answers 200 even when the target blocked the fetch or served nothing usable, because the call itself worked. Branch on success. When it is false, the content fields are null and error_code says why:

  • anti_bot_block: the site's bot protection refused the request, after the automatic stealth retry if escalation was on.
  • timeout: the request ran out of time. Raise timeout_seconds.
  • not_found: the target answered 404 or 410.
  • empty_response: the page had no usable content.
  • rate_limited: the target answered 429. Slow down.
  • connection_failed: the host does not resolve in DNS, or a direct fetch could not connect. In the second case, switch to auto.
  • upstream_failed: a temporary fetch error, or a bare 503 from the target. Retry.

Wait for the content you need

Some pages fill in their data a moment after the first render. Two fields help, and both apply only when a browser renders the page:

  • wait_for_selector waits until a CSS selector appears, for up to wait_for_timeout_ms (default 5,000, maximum 30,000).
  • wait_ms adds a fixed wait after the page loads, up to 10,000 milliseconds.
page = client.scrape_url(
    "https://example.com/product/123",
    render="browser",
    wait_for_selector=".price",
    extract_rules={"price": ".price", "name": "h1"},
)
print(page.extracted)

Other browser options cost nothing extra: device="mobile" for a mobile user agent and viewport, window_width and window_height for the viewport size, and block_ads for ad-heavy pages. Images, fonts and media are blocked by default to save bandwidth; set block_assets to false when a page needs them to render.

Get just the data

  • return_markdown adds a markdown field. markdown_mode="main", the default, keeps only the main content; "full" converts the whole page. No extra credits.
  • extract_rules maps names to CSS selectors and returns an extracted object, as in the example above. For a list, use {"selector": "a", "attr": "href", "all": true}. No extra credits.
  • ai_query takes a plain-language instruction such as "return the product name, price and rating" and returns ai_extract. Add ai_extract_schema to get a fixed JSON shape. It adds 5 credits when the extraction runs on a delivered page.
  • include_html returns the raw page source next to the cleaned text, at no extra cost.

Long pages can be read in windows. Set max_chars, then pass each response's next_offset (markdown_next_offset for Markdown) back as offset to read on. Or set focus to a few terms or a quoted phrase, and you get the passages that match instead of only the start of the page.

Keep cost and time under control

  • Pin static pages to direct. A blog post or a documentation page usually needs no browser, and render="direct" costs 1 credit.
  • Turn escalation off when a block is an answer. With auto_escalate=False, the mode you asked for is the mode you get.
  • Set a time budget. timeout_seconds, from 1 to 120, caps the whole request. Escalation needs at least 20 seconds left, so a tight budget also stops it.
  • Use the cache, or skip it on purpose. Successful fetches are cached for 15 minutes by default, and a cache hit bills 1 credit. For price tracking or change detection, set cache_ttl_seconds to 0.
  • Route by country only when you need to. country="jp" fetches through a residential exit in Japan, which bills at the residential tier: 10 credits, or 40 with a browser. When no exit is available for a country, the call is refused with 422 country_unavailable.

Many pages at once

POST /v1/scrape/batch takes up to 10 URLs and fetches them concurrently with shared options such as render, return_markdown and country. Each URL gets its own row with its status, text and credits_charged, and one failing URL does not fail the rest. The response reports the batch total in credits_charged.

batch = client.scrape_batch(
    ["https://example.com/a", "https://example.com/b", "https://example.com/c"],
    render="auto",
    return_markdown=True,
)
print(f"{batch.succeeded} of {batch.requested} delivered")
print(f"{batch.credits_charged} credits")

When you need pixels rather than text, screenshots start at 15 credits and PDFs at 25. AI assistants can call the same endpoint through the scrape_url tool of the MCP server.

Be considerate

A browser render costs the site you visit more than a plain request does. Fetch only the pages you need, cache what you can, and respect each site's terms and the law where you operate. The acceptable use policy sets out what the API may be used for.

1,000 free credits a month

Run this guide with free credits

Sign up, verify your email, then mint a key on the dashboard. The free plan includes 1,000 credits a month and needs no card.