sector_etf_data

[life-elixir]

Hi @pbharrin . It is a wonderful blog and I enjoyed ingesting the fundamental data into zipline. I was wondering how I can access the file sector_etf_data.csv mentioned in risk_data_prep and also factor_loadings.h5?

[pbharrin]

I'm glad you got the fundamental data working.
Thanks for checking out my risk model. Sorry I didn't provide any documentation for that yet.
Here is how it works:
risk_data_prep.py creates the sector returns (sector_etf_data.csv) and factor loadings (factor_loadings.h5). I used this code for my thesis and I had access to the CRSP data, so I created sector_etf_data.csv from a dump CRSP data with those ~11 tickers inside. Here is the function that does it with CRSP data. It looks like I was getting started doing it with Yahoo! data as this is freely available, but I didn't finish that function. After you have the sector etf data you can generate factor_loadings.h5 with create_factor_loadings_files(). This is currently hard coded for the CRSP data. Also this function takes a long time, it took like 20hr on my laptop, for 10 years of data with the top 1000 equities. It makes a lot of calls to pipeline, when I think some refactoring of the code could allow it to do one or two big calls to pipeline.

Once you have factor_loadings.h5 you can attribute risk to each asset on each day. Similar to the Quantopian risk model. Here is a demo of how to use the risk factor loadings with Pyfolio. You can see from the code I don't do anything special, just open the DataFrame and immediately use it with Pyfolio.

I would like to provide some code snippets showing how to do this, and clean up the code a little bit so you can just change the Zipline bundle in a constant at the top.

I hope that clears things up.

[life-elixir]

Thanks @pbharrin . I will check it out and let you know. I was able to only perform with python 2.7. As you know, Zipline-live slack team developed it on 3.6 so it would be really helpful if you could tell me the changes need to be done to transfer your codes into python 3.6. Thanks again and stay safe & healthy.

[life-elixir]

Dear @pbharrin it appears that the following are not part of Quandl's SF1 fundamental table and need to be accessed from Zacks.

['cshoq', 'prccq', 'seqq', 'PS']

Will you please elaborate?

[pbharrin]

Those are actually from CRSP.

You should replace that line with a list of the fundamental values you are interested in from Quandl's SF1. For example to download data from Quandl, put the fields you care about in 'fields' in load_quandl_sf1.py, here is an example from SF1.
fields = ['netinc', 'equity', 'bvps', 'sps', 'fcfps', 'price'] # basic QV
dimensions = ['ARQ', 'ARQ', 'ARQ', 'ARQ', 'ARQ', 'ARQ']

load_quandl_sf1.py creates the fundamental data file with these fields. If you get that to work you should see a SF1.npy file in your ~/.zipline/data/ folder. If that works you can access these fields with a small fundamentals class like this (this is the example in my Fundamentals to access the above fields.

class Fundamentals(SparseDataFactor):
    outputs = ['netinc', 'equity', 'bvps', 'sps', 'fcfps', 'price']

    def __init__(self, *args, **kwargs):
        super(Fundamentals, self).__init__(*args, **kwargs)
        self.N = len(get_ticker_sid_dict_from_bundle("sep")) + 1  # max(sid)+1 get this from the bundle

        self.data_path = '/Users/peterharrington/.zipline/data/' + 'SF1.npy'

P.S. I tested the fundamental data end to end with Python 3.5.