mkdocs serve HTTP Error 429: Too Many Request #8213
-
|
I am trying to determine if there is an issue with a specific markdown file or a limitation with running
ProblemBuilding a Material for MkDocs locally with I have sites with far greater number of markdown files that the one which is failing, so an initial assuming was the The error refers to a specific markdown blog post. That post includes an embedded external file that has a GitHub URL. There are 5 other embedded files across other files in the website that also have GitHub URLs. Each markdown file usually contains one or more images that are served from GitHub, totalling over 100 URLs.
|
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
Hello @practicalli-johnny, There could be a couple of issues related to that:
GitHub CI executes the build only once, and the GitHub workers probably have higher rate limits for connections to GitHub itself than you local PC. Therefore, this issue won't happen in CI, because the threshold to reach the rate limit isn't reached there. Possible fixes are:
I don't think that addressing the issue in the blog plugin, which might cause multiple markdown extension executions to be worthwhile 🤔 EDIT: Reference: |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the detailed analysis, its very clear. I will avoid using embedded file snippets from the same repository as a short term solution, as this makes no significant difference to the content. Update: The timeout is happening with several embedded files in code blocks, so am avoiding this approach for now and replacing embedded file code blocks as they error. I will also review the Python markdown extensions repository and suggest additional caching, referring to this issue. |
Beta Was this translation helpful? Give feedback.
-
|
I tried to monkey patch the extendMarkdown function to control the creation of a new SnippetPreprocessor: However, this didn't work, the cache was preserved, however there were cache misses. So idk if that's a shortcoming of lru_cache when used together with a class or something else I'm missing when monkey patching it 🤷 failed attemptclass RuntimeMemory:
initial_processor = None
def wrap_extendMarkdown(func):
def wrapper(self: SnippetExtension, md: Markdown):
# Modified function from v10.14
self.md = md
md.registerExtension(self)
config = self.getConfigs()
# Preserve same instance with cache
if RuntimeMemory.initial_processor is None:
snippet = SnippetPreprocessor(config, md)
RuntimeMemory.initial_processor = snippet
else:
snippet = RuntimeMemory.initial_processor
md.preprocessors.register(snippet, "snippet", 32)
print(snippet.download.cache_info())
return wrapperSo instead I used a separate dict cache, and a wrapper around the download function to not call it when my basic cache key is hit. class RuntimeMemory:
cache = {}
def wrap_download(func):
# lru_cache needs to be added to keep the API the same
# the cache is local in the file
@functools.lru_cache
def wrapper(self, url):
if not RuntimeMemory.cache.get(url):
print("caching")
RuntimeMemory.cache[url] = func(self, url)
else:
print("reusing")
return RuntimeMemory.cache[url]
return wrapperPOC: 8213-material.zip |
Beta Was this translation helpful? Give feedback.
Hello @practicalli-johnny,
thanks for the extensive details. Looking at the traceback the issue originates inside of Snippets, so the 429 originates from a server from which you constantly download resources.
There could be a couple of issues related to that:
mkdocs serverreload it downloads the files again.Looking at the code it does cache it for the duration of the Markdown instance, and MkDocs restarts the instance for each page 🤔