Any URL

Point us at a URL. We deal with the wall.

POST /v1/scrape/url 1 – 40 credits (+5 with AI extraction) Utilities

A general-purpose fetcher with automatic escalation. We pick the cheapest method that works for each target and escalate through tougher anti-bot handling as needed, so you see one response and one tier-aware bill, not three retries.

The hard part

Generic scrapers make you pick a tier up front (datacenter or residential, render JS or not) and bill the same whether the page came back or not. Pick wrong and you're either burning credits on soft targets or staring at blocked pages on hard ones. We pick the right level per site and only charge for what actually succeeded.

What it does

`render=auto|direct|browser|stealth`: pick how much firepower you want.
`force_proxy=true` routes through residential proxies for soft-blocked targets.
Returns cleaned title/description/text and the original status_code, proxy_used, and elapsed_ms.
`include_html=true` returns the raw page source alongside the cleaned text. Same call, no extra credits.
`return_markdown=true` adds a `markdown` field, clean for LLM input. No extra credits.
`extract_rules={"price":".price","links":{"selector":"a","attr":"href","all":true}}` returns a typed `extracted` object. No extra credits.
`wait_for_selector`, `wait_ms`, `block_ads`, `device=mobile`, custom viewport: all available, all free.
`country=us` routes through an in-country residential exit (billed at the residential tier).
Pages we can't deliver (timeout, block, upstream error) don't bill. A delivered 404 'not found' bills 1 credit.

Send a call

curl -X POST https://scrapenest.dev/v1/scrape/url \
  -H "Authorization: Bearer sn_••••" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://www.zillow.com/homes/Austin-TX", "render": "auto" }'

import httpx

r = httpx.post(
    "https://scrapenest.dev/v1/scrape/url",
    headers={"Authorization": "Bearer sn_••••"},
    json={"url": "https://www.zillow.com/homes/Austin-TX", "render": "auto"},
    timeout=90,
)
page = r.json()
print(page["status_code"], page["method"], len(page["text"] or ""))

const r = await fetch("https://scrapenest.dev/v1/scrape/url", {
  method: "POST",
  headers: {
    "Authorization": "Bearer sn_••••",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://www.zillow.com/homes/Austin-TX",
    render: "auto",
  }),
});
const page = await r.json();

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
)

func main() {
    body, _ := json.Marshal(map[string]any{
        "url":    "https://www.zillow.com/homes/Austin-TX",
        "render": "auto",
    })
    req, _ := http.NewRequest("POST",
        "https://scrapenest.dev/v1/scrape/url", bytes.NewReader(body))
    req.Header.Set("Authorization", "Bearer sn_••••")
    req.Header.Set("Content-Type", "application/json")
    resp, _ := http.DefaultClient.Do(req)
    defer resp.Body.Close()
    var page struct {
        StatusCode int    `json:"status_code"`
        Method     string
        Text       string
    }
    json.NewDecoder(resp.Body).Decode(&page)
    fmt.Println(page.StatusCode, page.Method, len(page.Text))
}

Request body

{
  "url": "https://www.zillow.com/homes/Austin-TX",
  "render": "auto"
}

Response shape

{
  "url": "https://www.zillow.com/homes/Austin-TX",
  "final_url": "https://www.zillow.com/homes/Austin-TX/",
  "status_code": 200,
  "method": "browser",
  "proxy_used": "residential",
  "elapsed_ms": 3812,
  "title": "Austin TX Real Estate | Zillow",
  "description": "1,284 homes for sale in Austin, TX. See pricing and listing details.",
  "text": "1,284 homes for sale in Austin, TX …",
  "structured": null,
  "cache_hit": false,
  "credits_charged": 40,
  "error": null
}

What it costs

Tiered by what it takes to render the page. A simple HTML fetch is the floor; a full JS render with anti-bot handling is the ceiling. You only pay the tier that actually succeeded. Adding `ai_query` for AI extraction adds 5 credits.