Disruption of equities data :: pandas_datareader dependency on Yahoo and Google Finance API

[rsvp]

Description of specific issue

We are expecting major disruption in getting data on
equities, mutual funds, and ETFs via pandas_datareader
due its dependency on API of both Yahoo and Google Finance.

• [/] Fail quote retrieval via our module lib/yi_stock.py, if no fix upstream.
• [/] Enhancement, if fecon235 takes independent action in its yi_stock module.

Observed behavior

Yahoo employee has confirmed that the free End-Of-Day data has been terminated, 2017
presumably due to acquisition by Verizon.

• Possible work-around: https://stackoverflow.com/questions/44030983/yahoo-finance-url-not-working

/r/algotrading on Google JSON termination
presumably due to cost-cutting by new product manager.
Our yi_stock module may appear to be working, but
please tail() your dataframe to verify whether quote retrieval
goes beyond 2017-09-05 [silent fail to get() current stock quotes].

Alternatives to enhance behavior

1. Switch over to Quandl, using our lib/yi_quandl.py module

   • See: https://blog.quandl.com/api-for-stock-data lists premium databases. But for free stock price data: https://www.quandl.com/product/WIKIP/WIKI/PRICES-Quandl-End-Of-Day-Stocks-Info.

2. Alpha Vantage: https://www.alphavantage.co/documentation - Requires user to get free API key

   • Python wrapper, MIT license: https://github.com/RomelTorres/alpha_vantage

3. tiingo: https://api.tiingo.com/docs/general/overview - Requires user to get free API key

4. $$$ EODhistoricaldata: https://eodhistoricaldata.com

5. Scrape Google Finance pages: https://github.com/CNuge/general_use_functions/blob/master/international_stock_scraper.py (currently also works for international stocks) -- which is not a robust solution (relative to API code) since the page layout is subject to UI changes: International market data - functionality addition request pydata/pandas-datareader#408

Why would the improvement be useful to most users?

Information from the equities markets is vital for financial economics.

Before releasing our own independent solution, we expect to
make a pull request to the pandas_datareader repository.

Please kindly propose alternative solutions
below, or at https://gitter.im/rsvp/fecon235
... and look into pandas_datareader issues

Check for revisions upstream

It is possible that a solution has been merged into pandas_datareader
and all that is necessary is an update of the package, see its CHANGELOG:
https://pandas-datareader.readthedocs.io/en/latest/whatsnew.html
and make sure the update is compatible with the latest fecon235.

Note: Anaconda distribution uses hyphen, not underscore:

$ conda update pandas-datareader

Additional helpful details for bugs

• [/] Problem started recently

• [/] Problem can be reliably reproduced

• fecon235 version: v5.17.0603

• pandas version: 0.19.2

• pandas_datareader version: 0.2.1

[rsvp]

Reports of Google limiting data length to merely one year

• Pandas data reader for google end dates are not working  pydata/pandas-datareader#395

• Unable to fetch Google finance data pydata/pandas-datareader#394

  • Also: URL to retrieve data from Google has changed
    • Unable to fetch Google finance data pydata/pandas-datareader#394 (comment)

[paintdog]

What means:

Our yi_stock module may appear to be working, but
please tail() your dataframe to verify whether quote retrieval
goes beyond 2017-09-05 [silent fail to get() current stock quotes].

Will there be no new quotes after 2017-09-05 or no data before 2017-09-05? I don't understand it.

I want to mention, that the new sources are not suitable for all users and do not cover all fields of interest :(

[rsvp]

@paintdog tail() retrieves the tail end of a dataframe.
Our yi_stock module currently has a dependency on pandas_reader
to get() equities data, but it may be re-route requests by other means
(and to other vendors besides Google Finance).

All our sources for suitable for all users in the field of financial economics.

[rsvp]

StackOverflow: DataReader google finance date not working points to change in URL at Google as cause for data disruption.

But unfortunately, for the PR to modify said URL, pydata/pandas-datareader@eac67a4 ,
the required Travis CI build has failed.

[rsvp]

Google Finance website currently displays a Yellow Warning Banner which reads as follows:

Google Finance is under renovation. As a part of this process, the Portfolios feature won't be available after mid-November 2017.

Their notice at https://support.google.com/finance indicates:

• Portfolios will be deprecated
• To download available data, follow these steps:
  • Visit that company's summary page (e.g., http://finance.google.com/finance?q=nasdaq:goog).
  • Click Historical Prices on the left side of the page.
  • Under "Export," click Download to spreadsheet.

Why? Their reasoning: "to make Google Finance more accessible and user-friendly for a wider audience." Yeah, right 👎 See also Quora: Why is the Google Finance portfolio feature disabled?

Thus any tentative fixes now for data retrieval may go to waste after renovation is completed:

• Wait to see the renovated source API for upstream repo.

Noteworthy is the secure https redirect from the "www" to "finance" subdomain for google.com.

[rsvp]

Upstream fixes not passing the tests in Travis CI build has been resolved: pydata/pandas-datareader#404

This is especially pertains to:
pydata/pandas-datareader#394
pydata/pandas-datareader#395
pydata/pandas-datareader#402

Noteworthy snippet for Google Finance URL fix:
pytest.skip("Google quote api is offline as of Oct 1, 2017")
and for Yahoo Finance, drop NaN missing values in pandas DataFrame:
return df.sort_index().dropna(how='all')