Get SEO Data with the Google Custom Search JSON API

Google’s Custom Search JSON API is primarily designed for creating applications that retrieve and display search results programmatically, like a search box widget that you place on your own website. However, as part of this service, Google provides a free Google Search REST API that that lets you query Google’s search engine. While constructing your queries, you can set different parameters like search country, language, exact terms, etc.

This post will describe the steps you can take to use the Google Custom Search JSON API to programmatically access search data from Google, for use in SEO analysis.

NEW! API CONNECTOR ADD-ON FOR GOOGLE SHEETS

Check out my new API Connector Add-on to import data from thousands of platforms (e.g. Shopify, Harvest, Mailchimp, ActiveCampaign, VWO, YouTube, etc.) directly into Google Sheets.

  1. Caveats
  2. Create your Custom Search Engine
  3. Modify your Search Engine to Search the Whole Web
  4. Get your Google Custom Search API Key
  5. Create a Search URL
  6. Run your Google Search API Request

CAVEATS

Before we start, please be aware of the following:

  • the results may not exactly match the results you would get by running search queries on Google directly. Google explains this here, saying that the Google Custom Search API won’t include personalized results, and may include a subset of the total data. You can compare the two sets of results to determine whether this is a big factor or not for your queries (in my own tests, it wasn’t).
  • The Custom Search JSON API provides 100 search queries per day for free. If you need more, you’ll need to sign up for a billing account. Additional requests cost $5 per 1000 queries, up to 10k queries per day.

CREATE YOUR CUSTOM SEARCH ENGINE

While logged into Google, navigate to https://cse.google.com/cse/all, and click ‘Add’.
google-search-api-img4

You’ll be prompted to configure your custom search engine. Choose a URL (any URL), and give your search engine a name. These values can be anything; for this example, I’ve entered ‘www.demo.com’ and named it ‘Demo’. Click the Create button.
google-search-api-img5

Now click Control Panel to configure your custom search engine and extend it to search the entire web.
google-search-api-img6

MODIFY YOUR SEARCH ENGINE TO SEARCH THE WHOLE WEB

You will see a long list of settings related to your new custom search engine. For our purposes you only need to pay attention to the following:

1) Scroll down the page until you see your search engine ID. Note this down, we’ll need it soon:
google-search-api-img7

2) Delete your dummy URL, and toggle on the ‘Search the entire web’ option.
google-search-api-img8

That’s all we need to do here, move on to the next step!

GET YOUR GOOGLE CUSTOM SEARCH API KEY

Accessing Google’s Custom Search JSON API requires the use of an API key. To get this key, navigate to the Custom Search JSON API page and click Get a Key.
google-search-api-img1

Choose an existing project, or create a new one, and click Next.
google-search-api-img2

Note and copy your API key. You’ll need it shortly.
google-search-api-img3

CREATE A SEARCH URL

Google’s documentation describes how to create an API URL that returns results for the search query you’re interested in.

  • API root: https://www.googleapis.com
  • Endpoint: /customsearch/v1
  • Query strings: ?key={API key}&cx={custom search engine ID}&q={search query}
  • Query strings example: ?key=AIzaSy…&016496445636816056556:qvzl4ib4xtb
    &q=google tag manager
  • You can add any of the parameters listed here to further customize your search query: https://developers.google.com/custom-search/v1/cse/list

Putting it together, we get the full API Request URL:
https://www.googleapis.com/customsearch/v1?key=AIzaSy…&016496445636816056556:qvzl4ib4xtb
&q=google tag manager

RUN YOUR GOOGLE SEARCH API REQUEST

You can now push results from this URL into a database. Google provides documentation and code samples in their developer documentation.

There are no-code options, too:
1) enter this URL into tools like Postman or Swagger Inspector to access search results. Even just entering the URL into a browser will display the API response:
google-search-api-img10

2) paste your URL into Mixed Analytics’ own API Connector, to see the results in Google Sheets.
google-search-api-img11

Whatever the method, the response data will contain a ranking of search results, as well as the URLs, titles, and text snippets that describe the results.

LIMITS

  • As noted above, the search results from the Custom Search JSON API may not exactly match the results from Google.com.
  • Google will only return 10 results at a time. To retrieve the second set of 10 results, append a ‘start’ parameter, e.g. https://www.googleapis.com/customsearch/v1?key=AIzaSy…&016496445636816056556:qvzl4ib4xtb
    &q=google tag manager&start=11. To get the third set you’d append &start=21, and so on.
  • Response data from the Google Custom Search API contains nested JSON data, so the output may require some scrubbing to get it into the desired format for analysis.

The benefit, of course, is that it’s free 😀 But if you find the usage or limits of Google’s API challenging, and don’t mind paying, there are other services that provide Google (and other) search engine data in a more convenient format, e.g. https://dataforseo.com/apis/serp-api and https://zenserp.com.

Comments:1

Leave a Reply

Your email address will not be published.