How to Scrape Google Search Results With n8n (Legally)

Every few weeks someone asks me to scrape Google search results with n8n and shows me their first attempt. It is always the same shape: an HTTP Request node pointed at google.com/search, a regex over the HTML, and a Google Sheets node. It works for about two days.

The version that survives production looks nothing like that. It is a queue of queries, a licensed SERP API, a normalisation step, a dedupe key, a cache, and a database. I have shipped this shape inside larger crawling systems, including a distributed scraping platform running 100K+ jobs a day. Here is the whole pattern, in n8n.

Is it legal to scrape Google search results?

Scraping public search result pages is not usually a criminal matter, but pointing a bot at google.com/search breaches Google's Terms of Service and ignores its published crawl directives. Google's robots.txt file contains an explicit Disallow: /search rule for general crawlers, with narrow exceptions such as /search/about and /search/howsearchworks.

That distinction matters commercially more than legally. If you are building this for a client, a workflow that knowingly violates a platform's terms is a liability you signed for. If you are building it for yourself, it is an operational risk that surfaces at the worst possible moment, usually the day a campaign launches.

The clean answer is to buy the data from a provider whose entire business is acquiring it under its own terms and infrastructure. You get an invoice, a support contact, and a stable JSON contract. You do not get a CAPTCHA at 3am.

"If your scraping strategy depends on not being noticed, it is not a strategy. It is a countdown."

Why a licensed SERP API beats fighting anti-bot systems

A SERP API wins because the cost of maintaining an evasion stack is unbounded and the cost of an API call is a line item. Rotating residential proxies, headless Chrome fingerprint patching, CAPTCHA solving and HTML selector repair are all recurring engineering work with no ceiling.

n8n is a good host for this precisely because it does not care where the JSON came from. The HTTP Request node handles auth, pagination and batching, and everything downstream is provider-agnostic. If you are new to the tool, start with the n8n automation tutorial for beginners and come back.

Which data source should you point n8n at?

There are four realistic options, and only two of them belong in a client deliverable. The deciding factors are licensing posture, result fidelity, and who absorbs the maintenance.

One nuance people miss: Google's Custom Search JSON API is not the same corpus as a live SERP. It queries a Programmable Search Engine configuration, so a competitor-tracking or rank-tracking use case will produce numbers that do not match what a user sees. Google's own overview page documents the free daily allowance, the per-thousand paid rate above it, the daily ceiling, and the retirement date. Check that page before you build on it.

How to scrape Google search results with n8n: the architecture

Build it as a pipeline with a queue at the front and a database at the back, never as a single linear flow triggered by hand. Seven stages, each of which can fail and retry independently.

Why the queue table is non-negotiable

Without a queue table you have no idempotency. A workflow that dies on query 340 of 500 has to start over, and you pay the provider twice for the first 339. With a queue, each row carries its own state and a rerun only picks up what is genuinely pending.

This is the same discipline I apply to any batch job in a data engineering build. The unit of work is a row, not a run.

Building the workflow node by node

The whole thing is roughly nine nodes. The parsing Code node is where most of the judgement lives, because raw SERP JSON is nested, inconsistent between result types, and full of tracking parameters you do not want in your dedupe key.

// n8n Code node, run once for all items
const seen = new Set();
const out = [];

for (const item of $input.all()) {
  const query = item.json.search_parameters?.q;
  const results = item.json.organic_results || [];

  for (const r of results) {
    let host;
    let clean;
    try {
      const u = new URL(r.link);
      host = u.hostname.replace(/^www\./, "");
      clean = u.origin + u.pathname;
    } catch {
      continue;
    }

    const key = query + "|" + clean;
    if (seen.has(key)) continue;
    seen.add(key);

    out.push({
      json: {
        dedupe_key: key,
        query,
        position: r.position ?? null,
        title: r.title ?? null,
        url: clean,
        domain: host,
        snippet: r.snippet ?? null,
        fetched_at: new Date().toISOString(),
      },
    });
  }
}

return out;

Stripping the query string before hashing is the single highest-value line in that file. Providers and Google both append tracking parameters that change between runs, and if you hash the raw link every rerun looks like fresh data.

For cross-run deduplication, n8n's Remove Duplicates node has a mode that discards items it has already seen in previous executions, keyed on a field you choose. It is convenient for small workflows. For anything you plan to audit later, do the dedupe in the database with a unique constraint on dedupe_key and an upsert, so the truth lives in one place.

How do you handle rate limits and caching?

Handle rate limits with batching plus waits inside n8n, and handle cost with a cache keyed on the query. The HTTP Request node exposes batching options for items per batch and a batch interval in milliseconds, which is usually enough to stay under a provider's concurrency limit without extra nodes.

If you run this at volume, self-host. The execution counts add up fast on managed plans, and long-running loops are exactly the workload that makes the self-hosting trade-off worth doing properly.

What does it cost to run?

Cost is dominated by SERP API calls, not by n8n. The formula is queries multiplied by pages per query multiplied by refresh frequency, and every one of those three is a decision you control.

SERP vendors price in credits or searches per month, usually with tiered rates that fall as volume rises, and some charge extra for JavaScript-rendered or location-specific results. Google's Custom Search JSON API prices differently again, with a small free daily allowance and a per-thousand rate above it up to a daily ceiling. Pricing changes constantly, so read the vendor's official pricing page rather than a blog post, including this one.

The controllable half is the multiplier. Pulling three pages when page one answers the question triples your bill. Refreshing a stable prospect list daily instead of monthly multiplies it by thirty. In practice I see caching and page-depth discipline cut spend more than any vendor negotiation does.

When this workflow is the wrong tool

If you need to scrape Google search results at tens of thousands of queries an hour, with complex retry semantics and per-item observability, n8n is the wrong host. At that point you want a real job queue and workers, not a visual workflow engine holding execution state.

Below those thresholds, n8n is genuinely the fastest path from idea to running system. The same skeleton powers most of the lead generation workflows I build, and it is the backbone of the search-and-enrich layer in ProLeads. If you want it built and handed over rather than assembled from tutorials, that is what my n8n integration work and broader AI automation services cover.