pygooglenews

If Google News had a Python library

Created by Artem from newscatcherapi.com but you do not need anything from us or from anyone else to get the software going, it just works out of the box.

My blog post about how I did it

Demo

You might also like to check our Google News API or Financial Google News API

Table of Contents

• About
• Examples of Use Cases
• Working with Google News in Production
• Motivation
• Installation
• Quickstart
• Documentation
• Advanced Query Search Examples
• About me
• Change Log

1. NewsCatcher's Google News API — all code is written for you, clean & structured JSON output. Low price. You can test it yourself with no credit card. Plus, financial version of API is also available.
2. ScrapingBee API which handles proxy rotation for you. Each function in this package has scraping_bee parameter where you paste your API key. You can also try it for free, no credit card required. See example
3. Your own proxy — already have a pool of proxies? Each function in this package has proxies parameter (python dictionary) where you just paste your own proxies.

I love working with the news data. I love it so much that I created my own company that crawls for hundreds of thousands of news articles, and allow you to search it via a news API. But this time, I want to share with the community a Python package that makes it simple to get the news data from the best search engine ever created - Google.

Most likely, you know already that Google has its own news service. It is different from the usual Google search that we use on a daily basis (sorry DuckDuckGo, maybe next time).

This package uses the RSS feed of the Google News. The top stories page, for example.

RSS is an XML page that is already well structured. I heavily rely on Feedparser package to parse the RSS feed.

Google News used to have an API but it was deprecated many years ago. (Unofficial) information about RSS syntax is decentralized over the web. There is no official documentation. So, I tried my best to collect all this informaion in one place.

$ pip install pygooglenews --upgrade

from pygooglenews import GoogleNews

gn = GoogleNews()

top = gn.top_news()

business = gn.topic_headlines('business')

headquaters = gn.geo_headlines('San Fran')

# search for the best matching articles that mention MSFT and 
# do not mention AAPL (over the past 6 month
search = gn.search('MSFT -APPL', when = '6m')

from pygooglenews import GoogleNews
# default GoogleNews instance
gn = GoogleNews(lang = 'en', country = 'US')

top = gn.top_news(proxies=None, scraping_bee = None)

business = gn.topic_headlines('BUSINESS', proxies=None, scraping_bee = None)

from pygooglenews import GoogleNews

gn = GoogleNews()
covid = gn.topic_headlines('CAAqIggKIhxDQkFTRHdvSkwyMHZNREZqY0hsNUVnSmxiaWdBUAE')

gn = GoogleNews('uk', 'UA')
kyiv = gn.geo_headlines('kyiv', proxies=None, scraping_bee = None)
# or 
kyiv = gn.geo_headlines('kiev', proxies=None, scraping_bee = None)
# or
kyiv = gn.geo_headlines('киев', proxies=None, scraping_bee = None)
# or
kyiv = gn.geo_headlines('Київ', proxies=None, scraping_bee = None)

geo['feed'].title

# 'Київ - Останні - Google Новини'

gn.search(query: str, helper = True, when = None, from_ = None, to_ = None, proxies=None, scraping_bee=None)

The official reference on what could be inserted

The biggest obstacle that you might have is to write the URL-escaping input. To ease this process, helper = True is turned on by default.

helper uses urllib.parse.quote_plus to automatically convert the input.

For example:

• 'New York metro opening' --> 'New+York+metro+opening'
• 'AAPL -MSFT' --> 'AAPL+-MSFT'
• '"Tokyo Olimpics date changes"' --> '%22Tokyo+Olimpics+date+changes%22'

You can turn it off and write your own query in case you need it by helper = False

when parameter (str) sets the time range for the published datetime. I could not find any documentation regarding this option, but here is what I deducted:

• h for hours.(For me, worked for up to 101h). when=12h will search for only the articles matching the search criteri and published for the last 12 hours
• d for days
• m for month (For me, worked for up to 48m)

I did not set any hard limit here. You may try put here anything. Probably, it will work. However, I would like to warn you that wrong inputs will not lead to an error. Instead, the when parameter will be ignored by the Google.

from_ and to_ accept the following format of date: %Y-%m-%d For example, 2020-07-01

Google's Special Query Terms Cheat Sheet

Many Google's Special Query Terms have been tested one by one. Most of the core ones have been inherited by Google News service. At first, I wanted to integrate all of those as the search() function parameters. But, I realised that it might be a bit confusing and difficult to make them all work correctly.

Instead, I decided to write some kind of a cheat sheet that should give you a decent understanding of what you could do.

• Boolean OR Search [ OR ]

from pygooglenews import GoogleNews

gn = GoogleNews()

s = gn.search('boeing OR airbus')

print(s['feed'].title)
# "boeing OR airbus" - Google News

• Exclude Query Term [-]

"The exclude (-) query term restricts results for a particular search request to documents that do not contain a particular word or phrase. To use the exclude query term, you would preface the word or phrase to be excluded from the matching documents with "-" (a minus sign)."

• Include Query Term [+]

"The include (+) query term specifies that a word or phrase must occur in all documents included in the search results. To use the include query term, you would preface the word or phrase that must be included in all search results with "+" (a plus sign).

The URL-escaped version of + (a plus sign) is %2B."

• Phrase Search

"The phrase search (") query term allows you to search for complete phrases by enclosing the phrases in quotation marks or by connecting them with hyphens.

The URL-escaped version of " (a quotation mark) is %22.

Phrase searches are particularly useful if you are searching for famous quotes or proper names."

• allintext

"The allintext: query term requires each document in the search results to contain all of the words in the search query in the body of the document. The query should be formatted as allintext: followed by the words in your search query.

If your search query includes the allintext: query term, Google will only check the body text of documents for the words in your search query, ignoring links in those documents, document titles and document URLs."

• intitle

"The intitle: query term restricts search results to documents that contain a particular word in the document title. The search query should be formatted as intitle:WORD with no space between the intitle: query term and the following word."

• allintitle

"The allintitle: query term restricts search results to documents that contain all of the query words in the document title. To use the allintitle: query term, include "allintitle:" at the start of your search query.

Note: Putting allintitle: at the beginning of a search query is equivalent to putting intitle: in front of each word in the search query."

• inurl

"The inurl: query term restricts search results to documents that contain a particular word in the document URL. The search query should be formatted as inurl:WORD with no space between the inurl: query term and the following word"

• allinurl

The allinurl: query term restricts search results to documents that contain all of the query words in the document URL. To use the allinurl: query term, include allinurl: at the start of your search query.

List of operators that do not work (for me, at least):

1. Most (probably all) of the as_* terms do not work for Google News
2. allinlinks:
3. related:

Tip. If you want to build a near real-time feed for a specific topic, use when='1h'. If Google captured fewer than 100 articles over the past hour, you should be able to retrieve all of them.

Check the Useful Links section if you want to dig into how Google Search works.

Especially, Special Query Terms section of Google XML reference.

Plus, I will provide some more examples under the Full-Text Search Examples section

Output Body

All 4 functions return the dictionary that has 2 sub-objects:

• feed - contains the information on the feed metadata
• entries - contains the parsed articles

Both are inherited from the Feedparser. The only change is that each dictionary under entries also contains sub_articles which are the similar articles found in the description. Usually, it is non-empty for top_news() and topic_headlines() feeds.

Tip To check what is the found feed's name just check the title under the feed dictionary

How to use pygooglenews with ScrapingBee

Every function has scrapingbee parameter. It accepts your ScrapingBee API key that will be used to get the response from Google's servers.

You can take a look at what exactly is happening in the source code: check for __scaping_bee_request() function under GoogleNews class

Pay attention to the concurrency of each plan at ScrapingBee

How to use example:

gn = GoogleNews()

# it's a fake API key, do not try to use it
gn.top_news(scraping_bee = 'I5SYNPRFZI41WHVQWWUT0GNXFMO104343E7CXFIISR01E2V8ETSMXMJFK1XNKM7FDEEPUPRM0FYAHFF5')

How to use pygooglenews with proxies

So, if you have your own HTTP/HTTPS proxy(s) that you want to use to make requests to Google, that's how you do it:

gn = GoogleNews()

gn.top_news(proxies = {'https':'34.91.135.38:80'})

from pygooglenews import GoogleNews

gn = GoogleNews()

s = gn.search('boeing -airbus')

print(s['feed'].title)
# "boeing -airbus" - Google News

from pygooglenews import GoogleNews

gn = GoogleNews()

s = gn.search('intitle:boeing')

print(s['feed'].title)
# "intitle:boeing" - Google News

from pygooglenews import GoogleNews

gn = GoogleNews()

s = gn.search('intitle:boeing', when = '1h')

print(s['feed'].title)
# "intitle:boeing when:1h" - Google News

from pygooglenews import GoogleNews

gn = GoogleNews()

s = gn.search('boeing OR airbus', when = '1h')

print(s['feed'].title)
# "boeing AND airbus when:1h" - Google News

Stack Overflow thread from which it all began

Google XML reference for the search query

Google News Search parameters (The Missing Manual)

Feedparser

Beutifulsoup4

My name is Artem. I ❤️ working with news data. I am a co-founder of NewsCatcherAPI - Ultra-fast API to find news articles by any topic, country, language, website, or keyword

If you are interested in hiring me, please, contact me by email - bugara.artem@gmail.com or artem@newscatcherapi.com

Follow me on 🖋 Twitter - I write about data engineering, python, entrepreneurship, and memes.

Want to read about how it all was done? Subscribe to CODARIUM

thx to Kizy

v0.1.1 -- fixed language-country issues