How many requests can you place concurrently?

[baamily]

Hi!

Does anybody know how many requests I could place concurrently without throttling or having my IP blocked-- or what the rate limit for the API is?

I'm doing a project where I grabbed a list of the coordinates for the NOAA NWS historical weather stations and wanted to grab a daily forecast for those locations. Turns out there's about 85,000 U.S. weather stations.

I'm using a pyspark notebook and the asyncio library to help batch and place these requests concurrently. Below is a snippet of the configurations:

# Quick explanation: We're dividing our list into batch sizes of 500 locations = 500 async tasks.
# Semaphore is ensuring that no more than 50 requests are running at the same instant. 
# As soon as one request finishes, another starts — so you’re always keeping 50 “in flight” until the batch is done.
# When the batch finishes, we sleep for 5 seconds to apease our NOAA overlords.

MAX_CONCURRENT_REQUESTS = 50  # Setting the max requests we want to send at once (we don't want to overload the website)
BATCH_SIZE = 500              # Setting how many rows/stations to process at a time.
RETRY_LIMIT = 3               # Setting how many times we need to try again if something goes wrong.
SLEEP_TIME = 5                # Setting how many seconds we want to sleep between batches.
semaphore = asyncio.Semaphore(MAX_CONCURRENT_REQUESTS)  # Controls concurrency to avoid overloading the endpoint

[wsamoht]

From https://www.weather.gov/documentation/services-web-api:

The rate limit is not public information, but allows a generous amount for typical use. If the rate limit is execeed a request will return with an error, and may be retried after the limit clears (typically within 5 seconds). Proxies are more likely to reach the limit, whereas requests directly from clients are not likely.

Make sure to following the required "Authentication" by setting User-Agent.

[dwhitemv25]

If you need station metadata you might look into what's available from NCEI. They have their own batch data download capabilities.