guide
Add live web search and scraping to Claude, Cursor and ChatGPT with MCP
Language models answer from what they learned in training. Ask one about this week's release notes, a page you just linked, or the reviews of a restaurant, and it needs a tool to go and look. The Model Context Protocol (MCP) is the standard way to give an AI assistant such tools. This guide connects the hosted ScrapeNest MCP server to Claude Code, Cursor, VS Code, LM Studio and ChatGPT, so your assistant can search the web, read pages, pull Google Maps reviews and transcribe audio.
What the server gives your assistant
The server lives at https://scrapenest.dev/mcp and speaks MCP's streamable HTTP transport. Its default profile has 9 tools. Each tool calls one ScrapeNest REST endpoint and bills the same credits as that endpoint:
| Tool | What it does | Credits |
|---|---|---|
web_search | Live web search across several engines, merged into one ranked list | 5 |
ask_web | A written answer with numbered citations from freshly fetched pages | 20 + 5 per source fetched |
deep_search | Search plus the full text of the top results in one call | 5 + 5 per page fetched |
scrape_url | Read one URL as clean text or Markdown, JavaScript pages included | 1 to 40, +5 with ai_query |
translate_text | Translate text, with the source language detected | 1 |
maps_reviews | Google Maps reviews for one place, newest first | 100 per 50 reviews (40 when 50 or fewer) |
audio_transcript | Transcribe a podcast episode, YouTube video or audio file | 20 per 2 minutes |
poll_run | Collect the result of a background job | Free |
scrapenest_account | Key status, credit balance and rate limits | Free |
Before you start
Sign up, verify your email address, and mint an API key on the dashboard. Any key works for MCP; there is no separate MCP key. The free plan includes 1,000 credits a month and needs no card. Keep the key in an environment variable so it stays out of config files you might share:
export SCRAPENEST_API_KEY="sn_your_key_here"
Claude Code
Add the server with one command:
claude mcp add --transport http scrapenest https://scrapenest.dev/mcp \
--header "Authorization: Bearer $SCRAPENEST_API_KEY"
Run claude mcp list to check that it connected, then ask Claude Code about something that happened this week.
Cursor
Add the server to ~/.cursor/mcp.json to use it in every project, or to .cursor/mcp.json in one project:
{
"mcpServers": {
"scrapenest": {
"url": "https://scrapenest.dev/mcp",
"headers": { "Authorization": "Bearer sn_your_key_here" }
}
}
}
The tools then show up in Cursor's MCP settings, and its agent can call them.
VS Code
In .vscode/mcp.json, an input variable keeps the key out of the file. VS Code asks for it the first time the server starts and stores it securely:
{
"inputs": [
{ "type": "promptString", "id": "scrapenest-key", "description": "ScrapeNest API key", "password": true }
],
"servers": {
"scrapenest": {
"type": "http",
"url": "https://scrapenest.dev/mcp",
"headers": { "Authorization": "Bearer ${input:scrapenest-key}" }
}
}
}
LM Studio
Local models in LM Studio can use the same tools. Add the server to LM Studio's mcp.json:
{
"mcpServers": {
"scrapenest": {
"url": "https://scrapenest.dev/mcp",
"headers": { "Authorization": "Bearer sn_your_key_here" }
}
}
}
Local models often have short context windows. The page tools return long pages in windows, as described below, so one page does not fill the context.
ChatGPT
ChatGPT signs in with OAuth instead of a pasted key. Add https://scrapenest.dev/mcp as an MCP server in ChatGPT; when it asks you to sign in, log in to ScrapeNest and approve access. ChatGPT's menus change from time to time, so the docs keep the current steps.
Some ChatGPT features expect a server with exactly two tools, search and fetch. For those, use https://scrapenest.dev/mcp?profile=chatgpt: search returns result IDs, titles and URLs, and fetch returns the full text of one result.
Other clients
Any client that supports remote MCP servers over streamable HTTP can connect with the same URL and header. If a client takes only a URL and cannot send headers, the server also accepts the key as a query parameter: https://scrapenest.dev/mcp?apiKey=sn_your_key_here. Treat that as a last resort, because a key inside a URL ends up in config files, screenshots and logs. If a key leaks, revoke it on the dashboard and mint a new one.
Using the tools
You rarely name a tool yourself. Ask in plain language and the assistant picks one:
- "What changed in the latest Node.js release? Include links." The assistant calls
web_search, orask_webfor a written answer with numbered citations.ask_webfetches its sources live, so it takes about 10 to 30 seconds. - "Read https://example.com/pricing and list the plans." That is
scrape_url. It renders JavaScript pages when needed and can return Markdown, which keeps headings, links and tables. - "Find three recent articles on solid-state batteries and summarize them."
deep_searchsearches and returns the full text of the top results in one call. - "What do the newest Google reviews of this hotel say?"
maps_reviewstakes a place name; no ID is needed. - "Summarize this podcast episode."
audio_transcripttranscribes the audio, with timestamps on request.
Long pages come back one window at a time, so a large page cannot flood the model's context. scrape_url takes a focus argument that returns the passages matching a phrase, and an offset to read on.
maps_reviews and audio_transcript run as background jobs. Each call waits about a minute; if the job is still going, the tool returns a run_id and the assistant calls poll_run to collect the result. Polling is free. Completed results end with the credits the call charged, and scrapenest_account reports the key's remaining balance and rate limits.
Limits to know
- A tool call has a time budget of about 90 seconds, so it returns cleanly before the connection times out.
scrape_urluses up to 80 seconds of it. For slow, heavily protected pages, call the REST API with a largertimeout_seconds. maps_reviewsasks for at most 2,000 reviews per call. The REST endpoint goes up to 10,000.audio_transcriptdoes not accept cookies. Videos that need a signed-in session need the REST API'scookiesfield.- Tool calls share your account's credits and rate limits with your REST traffic.
Troubleshooting
401 unauthorizedwhen the client connects: the key is missing or wrong, or the header lacks theBearerprefix.- Tool calls fail with
email_not_verified: verify your email address, then try again. You can resend the email from the verification page. - Tool calls fail with
insufficient_credits: the balance is used up. Add credits, or wait for your plan's monthly credits. - A tool returns a
run_idinstead of a result: a background job is still running. Let the assistant callpoll_run.
The MCP server page has the tool list and client snippets in one place, and the MCP section of the docs covers setup in detail.