Option to disable automatic SEC downloads when using cloud storage

[pablograba]

Hi @dgunning (and everyone),
First of all, thank you so much for creating and maintaining edgartools! It’s become a really valuable tool for many of us working with EDGAR data.

I’ve been happily using the cloud storage feature like this: edgar.use_cloud_storage("s3://my-edgar-bucket/")

It works beautifully but I’ve noticed that when a filing isn’t already in the S3 bucket, the library automatically downloads it from the SEC EDGAR site and saves it to the bucket. This is super convenient for interactive / exploratory work!
However, in some production or restricted environments, we’d actually prefer to never make outbound requests to the SEC, even if that means getting a FileNotFoundError when a filing is missing from our bucket. The main reasons are:

• We only want to work with a pre-filtered / pre-approved subset of filings
• We want to avoid any unexpected network calls to sec.gov from our pipelines
• We want to guarantee read-only behavior from the cloud storage
• We need to test data retrieval capabilities from a specific set of filings

Would it be possible to add an option to turn off this auto-download behavior?

Thanks!

[dgunning]

Hi @pablograba,

Thank you for the kind words and for this well-articulated feature request! This is a great idea that will benefit production and enterprise users.

Current Behavior

Request filing → Check S3 → Not found → Download from SEC → Save to S3 → Return

As you noted, this is great for interactive/exploratory work but problematic for controlled production environments.

Proposed Solution

Add a fallback_to_sec parameter to use_cloud_storage():

# Current behavior (default, unchanged)
edgar.use_cloud_storage("s3://my-edgar-bucket/")
# Equivalent to:
edgar.use_cloud_storage("s3://my-edgar-bucket/", fallback_to_sec=True)

# New: Strict cloud-only mode
edgar.use_cloud_storage("s3://my-edgar-bucket/", fallback_to_sec=False)
# When filing not in bucket → raises FileNotFoundError
# No network calls to sec.gov
# Guarantees read-only behavior from cloud storage

Behavior When fallback_to_sec=False

Scenario	Result
Filing exists in S3	Returns filing
Filing missing from S3	Raises FileNotFoundError
Network calls to SEC	Never happens
Writes to S3 bucket	Never happens

This gives you exactly what you need:

• Pre-filtered/pre-approved subsets only
• No unexpected network calls
• Guaranteed read-only from cloud storage
• Predictable behavior for testing

Questions

1. Does fallback_to_sec=False clearly convey the intent, or would you prefer something like offline_mode=True or strict=True?

2. Should the error message include helpful context like:

   FileNotFoundError: Filing 0001234567-24-000001 not found in s3://my-edgar-bucket/
   (fallback_to_sec=False, no SEC download attempted)
   

3. Any other configuration options you would find useful for production pipelines?

This is a straightforward addition - I will prioritize it for an upcoming release.

Thanks again for the suggestion!

[pablograba]

Hi @dgunning,

Thank you so much for the quick response!

I like the proposed solution, adding fallback_to_sec as a parameter is clean, explicit, and keeps the default behavior unchanged, which is perfect for not breaking existing code. The table you included also makes the behavior very clear. This is exactly what I (and I’m sure many production users) need.

A couple of small thoughts on the naming (just personal preference, completely happy either way):

• While fallback_to_sec is accurate and descriptive, I find myself reading it a couple of times to fully internalize what it means.

As an alternative, something like allow_download_from_sec (or allow_sec_download) might feel a tiny bit more immediately intuitive, especially for people who are scanning the docs or code quickly:

edgar.use_cloud_storage("s3://...", allow_download_from_sec=False)

But honestly, fallback_to_sec=False is already very good and I’m 100% fine with it. It’s clear once you see the explanation.

And yes please! Including that little context in the error message would be really helpful during debugging. The suggested wording is perfect.

I can’t think of anything else right now. This flag + improved error message already covers the main production/testing needs very nicely.

Thank you again for being so open and fast to consider this!