我又在需求时间间隔添加了随机化请求,有这个必要吗 #748
lishujie111
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
修改 config.py
原来:CRAWLER_MAX_SLEEP_SEC = 4
CRAWLER_MIN_SLEEP_SEC = 1.2
CRAWLER_MAX_SLEEP_SEC = 4.0
def next_sleep_seconds():
import random
return random.uniform(CRAWLER_MIN_SLEEP_SEC, CRAWLER_MAX_SLEEP_SEC)
调用处
delay = config.next_sleep_seconds()
await asyncio.sleep(delay)
utils.logger.info(
f"[DouYinCrawler.get_aweme_detail] Sleeping for {delay:.2f} seconds after fetching aweme {aweme_id}"
)
return result
Beta Was this translation helpful? Give feedback.
All reactions