guide

Add live web search and scraping to Claude, Cursor and ChatGPT with MCP

Language models answer from what they learned in training. Ask one about this week's release notes, a page you just linked, or the reviews of a restaurant, and it needs a tool to go and look. The Model Context Protocol (MCP) is the standard way to give an AI assistant such tools. This guide connects the hosted ScrapeNest MCP server to Claude Code, Cursor, VS Code, LM Studio and ChatGPT, so your assistant can search the web, read pages, pull Google Maps reviews and transcribe audio.

What the server gives your assistant

The server lives at https://scrapenest.dev/mcp and speaks MCP's streamable HTTP transport. Its default profile has 9 tools. Each tool calls one ScrapeNest REST endpoint and bills the same credits as that endpoint:

ToolWhat it doesCredits
web_searchLive web search across several engines, merged into one ranked list5
ask_webA written answer with numbered citations from freshly fetched pages20 + 5 per source fetched
deep_searchSearch plus the full text of the top results in one call5 + 5 per page fetched
scrape_urlRead one URL as clean text or Markdown, JavaScript pages included1 to 40, +5 with ai_query
translate_textTranslate text, with the source language detected1
maps_reviewsGoogle Maps reviews for one place, newest first100 per 50 reviews (40 when 50 or fewer)
audio_transcriptTranscribe a podcast episode, YouTube video or audio file20 per 2 minutes
poll_runCollect the result of a background jobFree
scrapenest_accountKey status, credit balance and rate limitsFree

Before you start

Sign up, verify your email address, and mint an API key on the dashboard. Any key works for MCP; there is no separate MCP key. The free plan includes 1,000 credits a month and needs no card. Keep the key in an environment variable so it stays out of config files you might share:

export SCRAPENEST_API_KEY="sn_your_key_here"

Claude Code

Add the server with one command:

claude mcp add --transport http scrapenest https://scrapenest.dev/mcp \
  --header "Authorization: Bearer $SCRAPENEST_API_KEY"

Run claude mcp list to check that it connected, then ask Claude Code about something that happened this week.

Cursor

Add the server to ~/.cursor/mcp.json to use it in every project, or to .cursor/mcp.json in one project:

{
  "mcpServers": {
    "scrapenest": {
      "url": "https://scrapenest.dev/mcp",
      "headers": { "Authorization": "Bearer sn_your_key_here" }
    }
  }
}

The tools then show up in Cursor's MCP settings, and its agent can call them.

VS Code

In .vscode/mcp.json, an input variable keeps the key out of the file. VS Code asks for it the first time the server starts and stores it securely:

{
  "inputs": [
    { "type": "promptString", "id": "scrapenest-key", "description": "ScrapeNest API key", "password": true }
  ],
  "servers": {
    "scrapenest": {
      "type": "http",
      "url": "https://scrapenest.dev/mcp",
      "headers": { "Authorization": "Bearer ${input:scrapenest-key}" }
    }
  }
}

LM Studio

Local models in LM Studio can use the same tools. Add the server to LM Studio's mcp.json:

{
  "mcpServers": {
    "scrapenest": {
      "url": "https://scrapenest.dev/mcp",
      "headers": { "Authorization": "Bearer sn_your_key_here" }
    }
  }
}

Local models often have short context windows. The page tools return long pages in windows, as described below, so one page does not fill the context.

ChatGPT

ChatGPT signs in with OAuth instead of a pasted key. Add https://scrapenest.dev/mcp as an MCP server in ChatGPT; when it asks you to sign in, log in to ScrapeNest and approve access. ChatGPT's menus change from time to time, so the docs keep the current steps.

Some ChatGPT features expect a server with exactly two tools, search and fetch. For those, use https://scrapenest.dev/mcp?profile=chatgpt: search returns result IDs, titles and URLs, and fetch returns the full text of one result.

Other clients

Any client that supports remote MCP servers over streamable HTTP can connect with the same URL and header. If a client takes only a URL and cannot send headers, the server also accepts the key as a query parameter: https://scrapenest.dev/mcp?apiKey=sn_your_key_here. Treat that as a last resort, because a key inside a URL ends up in config files, screenshots and logs. If a key leaks, revoke it on the dashboard and mint a new one.

Using the tools

You rarely name a tool yourself. Ask in plain language and the assistant picks one:

  • "What changed in the latest Node.js release? Include links." The assistant calls web_search, or ask_web for a written answer with numbered citations. ask_web fetches its sources live, so it takes about 10 to 30 seconds.
  • "Read https://example.com/pricing and list the plans." That is scrape_url. It renders JavaScript pages when needed and can return Markdown, which keeps headings, links and tables.
  • "Find three recent articles on solid-state batteries and summarize them." deep_search searches and returns the full text of the top results in one call.
  • "What do the newest Google reviews of this hotel say?" maps_reviews takes a place name; no ID is needed.
  • "Summarize this podcast episode." audio_transcript transcribes the audio, with timestamps on request.

Long pages come back one window at a time, so a large page cannot flood the model's context. scrape_url takes a focus argument that returns the passages matching a phrase, and an offset to read on.

maps_reviews and audio_transcript run as background jobs. Each call waits about a minute; if the job is still going, the tool returns a run_id and the assistant calls poll_run to collect the result. Polling is free. Completed results end with the credits the call charged, and scrapenest_account reports the key's remaining balance and rate limits.

Limits to know

  • A tool call has a time budget of about 90 seconds, so it returns cleanly before the connection times out. scrape_url uses up to 80 seconds of it. For slow, heavily protected pages, call the REST API with a larger timeout_seconds.
  • maps_reviews asks for at most 2,000 reviews per call. The REST endpoint goes up to 10,000.
  • audio_transcript does not accept cookies. Videos that need a signed-in session need the REST API's cookies field.
  • Tool calls share your account's credits and rate limits with your REST traffic.

Troubleshooting

  • 401 unauthorized when the client connects: the key is missing or wrong, or the header lacks the Bearer prefix.
  • Tool calls fail with email_not_verified: verify your email address, then try again. You can resend the email from the verification page.
  • Tool calls fail with insufficient_credits: the balance is used up. Add credits, or wait for your plan's monthly credits.
  • A tool returns a run_id instead of a result: a background job is still running. Let the assistant call poll_run.

The MCP server page has the tool list and client snippets in one place, and the MCP section of the docs covers setup in detail.

1,000 free credits a month

Run this guide with free credits

Sign up, verify your email, then mint a key on the dashboard. The free plan includes 1,000 credits a month and needs no card.