Scrape Google Autocomplete Suggestion with Python

Contents: intro, imports, what will be scraped, process, code, links, outro.
Intro
This blog post is a continuation of Google's web scraping series.
Here you'll see examples of how you can scrape Autocomplete Suggestion using Python. An alternative API solution will be shown.
Imports
import os, requests, json
What will be scraped
Process
All that needs to be done is to add /complete/ and client parameter (...&client=chrome...) the search URL.
This URL: https://www.google.com/search?q=minecraft is better than  # no '/complete/' word in the url
Becomes this URL: http://google.com/complete/search?client=chrome&q=minecraft is better than # added '/complete/' as well as 'client' param
Code
import requests, json

headers = {
    "User-Agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
# client param could be replaced with firefox or other browser
response = requests.get('http://google.com/complete/search?client=chrome&q=minecraft is better than', headers=headers)
for result in json.loads(response.text)[1]:
    print(result)

-----------
'''
minecraft is better than roblox
minecraft is better than fortnite
minecraft is better than terraria
minecraft is better than fortnite memes
minecraft is better than among us
minecraft is better than roblox and fortnite
minecraft is better than fortnite song
minecraft is better than fortnite reddit
'''
SerpApi is a paid API with a free trial of 5,000 searches and essentially it does the same as a code above except you can use some of the advanced autocomplete parameters such as:
  • Cursor pointer which can be used to refine completion.
  • Example with below query q: minecraft is better than and cp:8 will give completely different results as you see in the code below.
    import os 
    from serpapi import GoogleSearch
    
    params = {
      "engine": "google_autocomplete",
      "q": "minecraft",
      "api_key": os.getenv("API_KEY"), # environment variable 
    }
    
    search = GoogleSearch(params)
    results = search.get_dict()
    
    for result in results["suggestions"]:
      print(result['value'])
    
    --------------
    '''
    default (0) "cp" param
    minecraft java
    minecraft 1.17
    minecraft classic
    minecraft mods
    ....
    
    "cp" param set to 2
    minecraft classic
    minecraft 1.17
    minecraft education edition
    minecraft free
    '''
    Links
    Outro
    If you have any questions or something isn't working correctly or you want to write something else, feel free to drop a comment in the comment section or via Twitter at @serp_api.
    Yours,
    Dimitry, and the rest of SerpApi Team.

    19

    This website collects cookies to deliver better user experience

    Scrape Google Autocomplete Suggestion with Python