Table of Contents
High Google rankings improve credibility and customer trust. This feature can significantly impact your customers' decisions. One way to ensure a higher ranking is ensuring you get positive Google Reviews and ratings.
Once in a while, you'll need to scrape Google. Although it's not legal, there are no aspects to show it is illegal. There are several ways to scrape google. Among them is the application of Google Search Application Programming Interface (API), which allows you to acquire data from its search engine. There are several other ways you can use to scrape Google. These methods may include a python application and ScrapeHero Cloud that lets you scrape Google reviews. These methods allow you to get essential information from a search engine from place IDs and Google reviews URLs.
Read through to learn the information you need to scrape Google and learn the best go-for solution on how to scrape Google Search Results.
What Are Google SERPs?
SERP is the acronym for the Search Engine Results Page. It is the page that contains all search results that associate or relate to the search query you entered on the search engine. You can also refer to it as Google Search Page or Google Page. The Search Engine Results Page is found on all search engines, including Google (the leading search engine, Yahoo, Bing, Yandex, and many more.
Understanding SERP and its works is the number one step to Google scraping. Google SERPs have made use of knowledge graphs, algorithms, and carousels. However, it has revolutionized over the years, and new features have been introduced to make it more effective.
Why Should You Scrape Google?
There are many reasons why you can scrape Google. Among these reasons are:
Integrate Customer Feedback Into Your Marketing Strategy
Digital marketing significantly relies on the online approach. One of the essential success determiners of online marketing is customer reviews and feedback. Scraping Google SERPs allows you to gather customers' information, including feedback and reviews, to know what works for them and what doesn't.
This feature lets you know your customers' critical areas of concern, improve your brand, and stand out from competitors.
Optimize Your Pay-Per-Click and Search Engine Optimization Approach
Google has a high market share and is visited more often by many users. Optimizing your keywords SEO for a higher Google ranking means many people will click on your website or article, raining your earnings. This is because Google displays ads at the start of the SERP or the end. Also, it exposes your business to many potential customers.
By scraping, you can know yourself in Google better for improved results. Building content around top Google pages makes them highly authoritative and trustworthy.
Generate Content Ideas
Scraping SERP allows you to get the most relevant content idea. Ranging from keywords to target audience. In addition, Google displays related searches, or people also ask sections to provide more content related to the search intent.
How to Scrape Google Search Results Using Scrapy
The first step when scraping Google is to understand how it prioritizes searches. In the past, Google displayed organic search results only. But that has changed to Google focusing on answering your queries quickly and efficiently.
Follow the guide below to scrape Google results.
Find your keywords. The appropriate keywords are the words customers use to search in Google. Identify the keywords related to your website or product, and don't forget to compare with top-ranking competitors.
Install Scrapy in a virtual environment. We recommend using ScraperAPI, which lets you scrape Google without getting your IP banned. It integrates machine learning, third-party proxies, and statistical data to rotate the IP address for every request, ensuring that the scraper doesn't get blocked from any site.
Create your project folder by entering the following snippet into your terminal.
scrapy startproject google_scraper
cd google_scraper
scrapy genspider google api.scraperapi.com
Open the Google scraper folder created and enter the "genspider" command. This function will set up a web scraper, "google."
Import your dependencies into your google.py File. Add the following dependencies to the top of the file to handle JSON files and build requests.
import scrapy
from urllib.parse import urlencode
from urllib.parse import urlparse
import JSON
from datetime import datetime
API_KEY = 'YOUR_API_KEY'
- The query will return to JSON format after sending the HTTP request. This function writes and maintains a parser which simplifies the process for you.
- Develop your Google search query. The standard URL structure for all Google search queries is http://www.google.com/search. To develop a Google search query, you first need to consider your data and understand your URL parameters. Below are the standard parameters for all Google search queries.
- q parameter. This parameter represents the search keyword. For instance, http://www.google.com/search?q=real estate developments will display search results related to real estate information.
- hl parameter. This parameter represents the language used in your search. For example, http://www.google.com/search?q=real estate developments&hl=en.
- as_sitesearch parameter. This parameter lets you obtain search results for a website domain. For example, http://www.google.com/search?q=real estate developments&as sitesearch=amazon.com.
- num parameter. This parameter represents the number of results per search results page. For example, http://www.google.com/search?q=real estate developments&num=100 shows one hundred search results on one page. Note that the maximum number of search results per page is one hundred.
- safe parameter. This parameter displays safe search results only. An example is http://www.google.com/search?q=real estate developments&safe=active.
def create_google_url(query, site=''):
google_dict = {'q': query, 'num': 100, }
if site:
web = urlparse(site).netloc
google_dict['as_sitesearch'] = web
return 'http://www.google.com/search?' + urlencode(google_dict)
return 'http://www.google.com/search?' + urlencode(google_dict)
Note that you can replace the "q" query with the search keyword.
Send your search request through Scraper API's server. To do so, append your query to the proxy URL using urlencode and payload. Use the resources given below.
def get_url(url):
payload = {'api_key': API_KEY, 'url': url, 'autoparse': 'true', 'country_code': 'us'}
proxy_url = 'http://api.scraperapi.com/?' + urlencode(payload)
return proxy_url
Build the main spider. Scrapy allows you to create different classes (spiders) meant for different sites or pages. Use the resources given below.
class GoogleSpider(scrapy.Spider):
name = 'google'
allowed_domains = ['api.scraperapi.com']
custom_settings = {'ROBOTSTXT_OBEY': False, 'LOG_LEVEL': 'INFO',
'CONCURRENT_REQUESTS_PER_DOMAIN': 10,
'RETRY_TIMES': 5}
Name your spider to highlight the script you want to run. Relate the name to your search query to avoid confusion.
Add "api.scraper.com" to allowed_domains to alter the IP address on retries before showing a failed message. Proceed to set constraints to avoid exceeding the limits of the free Scraper API account.
Send the HTTP request which is the initial request. To do so, use the resources below.
def start_requests(self):
queries = ['asana+reviews', 'clickup+reviews', 'best+real+estate+development', 'best+real+estate+development+for+young+professionals']
url = create_google_url(query)
yield scrapy.Request(get_url(url), callback=self.parse, meta={'pos': 0})
The proxy connection will send the query URL to Google search. The JSON format result will then be sent to the parse function for processing.
Enter the parse function. The scraper response to your request should be in the JSON file. Set the parameters to 'autoparse': 'true'. Use the resources given below.
def parse(self, response):
di = json.loads(response.text)
pos = response.meta['pos']
dt = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
for result in di['organic_results']:
title = result['title']
snippet = result['snippet']
link = result['link']
item = {'title': title, 'snippet': snippet, 'link': link, 'position': pos, 'date': dt}
pos += 1
yield item
next_page = di['pagination']['nextPageUrl']
if next_page:
yield scrapy.Request(get_url(next_page), callback=self.parse, meta={'pos': pos})
The Google scraper is complete. It's time to run the spider. You can change the code to add functionality. At this pouint, the google.py file should have the following display.
import scrapy
from urllib.parse import urlencode
from urllib.parse import urlparse
import json
from datetime import datetime
API_KEY = 'YOUR_API_KEY'
def get_url(url):
payload = {'api_key': API_KEY, 'url': url, 'autoparse': 'true', 'country_code': 'us'}
proxy_url = 'http://api.scraperapi.com/?' + urlencode(payload)
return proxy_url
def create_google_url(query, site=''):
google_dict = {'q': query, 'num': 100, }
if site:
web = urlparse(site).netloc
google_dict['as_sitesearch'] = web
return 'http://www.google.com/search?' + urlencode(google_dict)
return 'http://www.google.com/search?' + urlencode(google_dict)
class GoogleSpider(scrapy.Spider):
name = 'google'
allowed_domains = ['api.scraperapi.com']
custom_settings = {'ROBOTSTXT_OBEY': False, 'LOG_LEVEL': 'INFO',
'CONCURRENT_REQUESTS_PER_DOMAIN': 10,
'RETRY_TIMES': 5}
def start_requests(self):
queries = ['asana+reviews', 'clickup+reviews', 'best+real+estate+development', 'best+real+estate+development+for+young+professionals']
for query in queries:
url = create_google_url(query)
yield scrapy.Request(get_url(url), callback=self.parse, meta={'pos': 0})
def parse(self, response):
di = json.loads(response.text)
pos = response.meta['pos']
dt = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
for result in di['organic_results']:
title = result['title']
snippet = result['snippet']
link = result['link']
item = {'title': title, 'snippet': snippet, 'link': link, 'position': pos, 'date': dt}
pos += 1
yield item
next_page = di['pagination']['nextPageUrl']
if next_page:
yield scrapy.Request(get_url(next_page), callback=self.parse, meta={'pos': pos})
Scraping Google SERPs from a different state will require you to change the country_code parameter's code found in the et_url function.
How to Scrape Google Search Results Using Python
You can use Python to create a web scraper using Python. It utilizes the Requests-HTML with capabilities to give URLs feedback from a Google search.
Follow this guide to scrape Google search results using Python.
Install requests_html. To do so, key in pip3 install requests_html. Ignore this step if you already have one.
Open a Jupyter notebook, then upload the resources listed below.
import requests import urllib import pandas as pd from requests_html import HTML from requests_html import HTMLSession
Send your URL to Requests-HTML. This function will display a response or an exception if not executed well. Use the resources below.
def get_source(url): """Return the source code for the provided URL.
Args: url (string): URL of the page to scrape.
Returns: response (object): HTTP response object from requests_html. """
try: session = HTMLSession() response = session.get(url) return response
except requests.exceptions.RequestException as e: print(e)
Run urllib.parse.quote_plus() to encode your search query. This function ensures the appending process doesn't break the URL.
Run the command below to acquire a list of URLs related to the search query.
def scrape_google(query): query = urllib.parse.quote_plus(query) response = get_source("https://www.google.co.uk/search?q=" + query) links = list(response.html.absolute_links) google_domains = ('https://www.google.', 'https://google.', 'https://webcache.googleusercontent.', 'http://webcache.googleusercontent.', 'https://policies.google.', 'https://support.google.', 'https://maps.google.') for url in links[:]: if url.startswith(google_domains): links.remove(url) return links
What Are the Other Methods to Scrape Google Search Results?
You can also scrape Google search results using:
- Browser extension
- Visual web scraper
- Data collection service
Final Word on Google Search Scraping
Google doesn't expect you to scrape the search results. However, you can do so by employing different mechanisms. The Google search API lets you scrape Google despite its limitation and cost. You'll need to pay a certain fee per 1000 queries. The listed methods in this article are more reliable and straightforward methods you can use to scrape Google.
