Storage requirement estimates

[MineSafety]

I would like to download all filings as a training set for further analysis, but first need to understand the storage needs so I can plan accordingly. Does anyone know the rough storage space requirements if I wanted to download all the publicly-available filings?

Even limiting it to a few years and a few hundred companies seems like hundreds of GBs at a minimum, which is higher than I expected. Wondering if this is expected or perhaps something I am doing wrong.

[dgunning]

In your plan think about what you mean by all filings. Here is what that could mean. As you go from a to d the storage requirements drastically increase

a. Filing metadata
b. Primary filing document(s)
c. Attached documents
d. Attached Images

If you just need the filing metadata that would require about 1.5GB if you download the bulk filings.
If you need primary filing documents the that would be about 100KB * 1 million filings per year

You could sample 100 or 1000 filings and check what the download size is then scale up to 1 million

[MineSafety]

Thanks; I was hoping to have all of the above (i.e., a through d), but seems like the storage requirements are too taxing.

Sorry if this is a stupid question, but how do people usually work with all the filings / do analysis on them? Is it common to keep a local version, or only fetch them when requested? i.e., if I want to find Costco's 1998 10-K, and I have a program that only knows the metadata, it just dynamically fetches the b. primary documents, c. attached documents, and d. attached images when requested?