How to Scrape Google Search Results | Explore SERPs the Right Way!

High Google rankings improve credibility and customer trust. This feature can significantly impact your customers' decisions. One way to ensure a higher ranking is ensuring you get positive Google Reviews and ratings.

Once in a while, you'll need to scrape Google. Although it's not legal, there are no aspects to show it is illegal. There are several ways to scrape google. Among them is the application of Google Search Application Programming Interface (API), which allows you to acquire data from its search engine. There are several other ways you can use to scrape Google. These methods may include a python application and ScrapeHero Cloud that lets you scrape Google reviews. These methods allow you to get essential information from a search engine from place IDs and Google reviews URLs.

Read through to learn the information you need to scrape Google and learn the best go-for solution on how to scrape Google Search Results.

What Are Google SERPs?

SERP is the acronym for the Search Engine Results Page. It is the page that contains all search results that associate or relate to the search query you entered on the search engine. You can also refer to it as Google Search Page or Google Page. The Search Engine Results Page is found on all search engines, including Google (the leading search engine, Yahoo, Bing, Yandex, and many more.

Understanding SERP and its works is the number one step to Google scraping. Google SERPs have made use of knowledge graphs, algorithms, and carousels. However, it has revolutionized over the years, and new features have been introduced to make it more effective.

Why Should You Scrape Google?

There are many reasons why you can scrape Google. Among these reasons are:

Integrate Customer Feedback Into Your Marketing Strategy

Digital marketing significantly relies on the online approach. One of the essential success determiners of online marketing is customer reviews and feedback. Scraping Google SERPs allows you to gather customers' information, including feedback and reviews, to know what works for them and what doesn't.

This feature lets you know your customers' critical areas of concern, improve your brand, and stand out from competitors.

Optimize Your Pay-Per-Click and Search Engine Optimization Approach

Google has a high market share and is visited more often by many users. Optimizing your keywords SEO for a higher Google ranking means many people will click on your website or article, raining your earnings. This is because Google displays ads at the start of the SERP or the end. Also, it exposes your business to many potential customers.

By scraping, you can know yourself in Google better for improved results. Building content around top Google pages makes them highly authoritative and trustworthy.

Generate Content Ideas

Scraping SERP allows you to get the most relevant content idea. Ranging from keywords to target audience. In addition, Google displays related searches, or people also ask sections to provide more content related to the search intent.

a person typing on laptop with Google search on the screen

How to Scrape Google Search Results Using Scrapy

The first step when scraping Google is to understand how it prioritizes searches. In the past, Google displayed organic search results only. But that has changed to Google focusing on answering your queries quickly and efficiently.

Follow the guide below to scrape Google results.

  1. Find your keywords. The appropriate keywords are the words customers use to search in Google. Identify the keywords related to your website or product, and don't forget to compare with top-ranking competitors.

  2. Install Scrapy in a virtual environment. We recommend using ScraperAPI, which lets you scrape Google without getting your IP banned. It integrates machine learning, third-party proxies, and statistical data to rotate the IP address for every request, ensuring that the scraper doesn't get blocked from any site.

  3. Create your project folder by entering the following snippet into your terminal.

  4. scrapy startproject google_scraper

  5. cd google_scraper

  6. scrapy genspider google api.scraperapi.com

  7. Open the Google scraper folder created and enter the "genspider" command. This function will set up a web scraper, "google."

  8. Import your dependencies into your google.py File. Add the following dependencies to the top of the file to handle JSON files and build requests.

scrapy key on the keyboard

import scrapy

from urllib.parse import urlencode

from urllib.parse import urlparse

import JSON

from datetime import datetime

API_KEY = 'YOUR_API_KEY'

  • The query will return to JSON format after sending the HTTP request. This function writes and maintains a parser which simplifies the process for you.
  1. Develop your Google search query. The standard URL structure for all Google search queries is http://www.google.com/search. To develop a Google search query, you first need to consider your data and understand your URL parameters. Below are the standard parameters for all Google search queries.

1. Construct your Google URL using the information below.

def create_google_url(query, site=''):

    google_dict = {'q': query, 'num': 100, }

    if site:

        web = urlparse(site).netloc

        google_dict['as_sitesearch'] = web

        return 'http://www.google.com/search?' + urlencode(google_dict)

    return 'http://www.google.com/search?' + urlencode(google_dict)

people in front of the monitors
  • Note that you can replace the "q" query with the search keyword.

  1. Send your search request through Scraper API's server. To do so, append your query to the proxy URL using urlencode and payload. Use the resources given below.

def get_url(url):

   payload = {'api_key': API_KEY, 'url': url, 'autoparse': 'true', 'country_code': 'us'}

   proxy_url = 'http://api.scraperapi.com/?' + urlencode(payload)

   return proxy_url

  1. Build the main spider. Scrapy allows you to create different classes (spiders) meant for different sites or pages. Use the resources given below.

class GoogleSpider(scrapy.Spider):

    name = 'google'

    allowed_domains = ['api.scraperapi.com']

    custom_settings = {'ROBOTSTXT_OBEY': False, 'LOG_LEVEL': 'INFO',

                       'CONCURRENT_REQUESTS_PER_DOMAIN': 10, 

                       'RETRY_TIMES': 5}

web tools icons on the large screen
  1. Name your spider to highlight the script you want to run. Relate the name to your search query to avoid confusion.

  2. Add "api.scraper.com" to allowed_domains to alter the IP address on retries before showing a failed message. Proceed to set constraints to avoid exceeding the limits of the free Scraper API account.

  3. Send the HTTP request which is the initial request. To do so, use the resources below.

def start_requests(self):

        queries = ['asana+reviews', 'clickup+reviews', 'best+real+estate+development', 'best+real+estate+development+for+young+professionals']

            url = create_google_url(query)

            yield scrapy.Request(get_url(url), callback=self.parse, meta={'pos': 0})

The proxy connection will send the query URL to Google search. The JSON format result will then be sent to the parse function for processing.

json with icons
  1. Enter the parse function. The scraper response to your request should be in the JSON file. Set the parameters to 'autoparse': 'true'. Use the resources given below.

def parse(self, response):

        di = json.loads(response.text)

        pos = response.meta['pos']

        dt = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

        for result in di['organic_results']:

            title = result['title']

            snippet = result['snippet']

            link = result['link']

            item = {'title': title, 'snippet': snippet, 'link': link, 'position': pos, 'date': dt}

            pos += 1

            yield item

        next_page = di['pagination']['nextPageUrl']

        if next_page:

            yield scrapy.Request(get_url(next_page), callback=self.parse, meta={'pos': pos})

different screens
  1. The Google scraper is complete. It's time to run the spider. You can change the code to add functionality. At this pouint, the google.py file should have the following display.

import scrapy

from urllib.parse import urlencode

from urllib.parse import urlparse

import json

from datetime import datetime

API_KEY = 'YOUR_API_KEY'

def get_url(url):

    payload = {'api_key': API_KEY, 'url': url, 'autoparse': 'true', 'country_code': 'us'}

    proxy_url = 'http://api.scraperapi.com/?' + urlencode(payload)

    return proxy_url

def create_google_url(query, site=''):

    google_dict = {'q': query, 'num': 100, }

    if site:

        web = urlparse(site).netloc

        google_dict['as_sitesearch'] = web

        return 'http://www.google.com/search?' + urlencode(google_dict)

    return 'http://www.google.com/search?' + urlencode(google_dict)

class GoogleSpider(scrapy.Spider):

    name = 'google'

    allowed_domains = ['api.scraperapi.com']

    custom_settings = {'ROBOTSTXT_OBEY': False, 'LOG_LEVEL': 'INFO',

                       'CONCURRENT_REQUESTS_PER_DOMAIN': 10, 

                       'RETRY_TIMES': 5}

    def start_requests(self):

        queries = ['asana+reviews', 'clickup+reviews', 'best+real+estate+development', 'best+real+estate+development+for+young+professionals']

        for query in queries:

            url = create_google_url(query)

            yield scrapy.Request(get_url(url), callback=self.parse, meta={'pos': 0})

    def parse(self, response):

        di = json.loads(response.text)

        pos = response.meta['pos']

        dt = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

        for result in di['organic_results']:

            title = result['title']

            snippet = result['snippet']

            link = result['link']

            item = {'title': title, 'snippet': snippet, 'link': link, 'position': pos, 'date': dt}

            pos += 1

            yield item

        next_page = di['pagination']['nextPageUrl']

        if next_page:

            yield scrapy.Request(get_url(next_page), callback=self.parse, meta={'pos': pos})

Scraping Google SERPs from a different state will require you to change the country_code parameter's code found in the et_url function.

python capture on the screen

How to Scrape Google Search Results Using Python

You can use Python to create a web scraper using Python. It utilizes the Requests-HTML with capabilities to give URLs feedback from a Google search.

Follow this guide to scrape Google search results using Python.

  1. Install requests_html. To do so, key in pip3 install requests_html. Ignore this step if you already have one.

  2. Open a Jupyter notebook, then upload the resources listed below.

  • import requests import urllib import pandas as pd from requests_html import HTML from requests_html import HTMLSession

  1. Send your URL to Requests-HTML. This function will display a response or an exception if not executed well. Use the resources below.

  • def get_source(url): """Return the source code for the provided URL. 

  • Args: url (string): URL of the page to scrape. 

  • Returns: response (object): HTTP response object from requests_html. """ 

  • try: session = HTMLSession() response = session.get(url) return response 

  • except requests.exceptions.RequestException as e: print(e)

  1. Run urllib.parse.quote_plus() to encode your search query. This function ensures the appending process doesn't break the URL.

  2. Run the command below to acquire a list of URLs related to the search query.

What Are the Other Methods to Scrape Google Search Results?

You can also scrape Google search results using:

  • Browser extension
  • Visual web scraper
  • Data collection service

Final Word on Google Search Scraping

Google doesn't expect you to scrape the search results. However, you can do so by employing different mechanisms. The Google search API lets you scrape Google despite its limitation and cost. You'll need to pay a certain fee per 1000 queries. The listed methods in this article are more reliable and straightforward methods you can use to scrape Google.

About Dusan Stanar

I'm the founder of VSS Monitoring. I have been both writing and working in technology in a number of roles for dozens of years and wanted to bring my experience online to make it publicly available. Visit https://www.vssmonitoring.com/about-us/ to read more about myself and the rest of the team.

Leave a Comment

PHP Code Snippets Powered By : XYZScripts.com