Failed to get crumb - too many 429 error responses

[chfiii]

Getting the above error consistently. Just started this morning - 6-May-25 EDT

To Reproduce

from yahooquery import Ticker
import UserAgent
# Generate some data
syms = 'ntc,goog,qcom,msft,gsg,meta,aapl,rtx,avgo,asml,bac,jpm,KKr,OHi,tgt,hon,nvda,ups,hack,cost,amzn,abbv,smtc'.upper().split(',')

userAgent = UserAgent().random

tics = Ticker(syms, user_agent=userAgent)

hist = tics.price

print(hist)


```**Expected behavior**
List of the price data

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 -Running on Windows 11, WSL2 - Ubuntu 22.04.5 LTS

``
Users of yfinance are seeing the same problem.  One suggestion there is to import from curl+cffi:

from curl_cffi import requests
import yfinance as yf
session = requests.Session(impersonate="chrome")
ticker = yf.Ticker('...', session=session)

Could this be done in yahooquery too?

[ericwbzhang]

confirms curl_cffi can fix this issue by passing the session variable.
@chfiii this pkg is really powerful but not well maintained. Happy to join you if you would like to resume the maintenance

[brld]

I am not as familiar with how Yahooquery handles requests, is there somewhere one could pass the session variable into the Yahooquery Ticker class?

[aggieed97]

Also seeing this same behavior. @ericwbzhang, how does this work for Yahooquery?

[bolstycjw]

from curl_cffi import Session

def init_ticker(symbols: str):
    session = Session(impersonate="chrome")
    return Ticker(symbols, session=session)

@aggieed97 you could do something similar, just initiate the session and pass it along to the Ticker constructor

[jirisarri10]

Hello,

I have been facing issues using the yahooquery library to retrieve data from Yahoo Finance. The error I keep getting is:

Failed to obtain crumb. Ability to retrieve data will be significantly limited.

After some investigation, I found that the library is not able to retrieve the "crumb" correctly for making requests to Yahoo Finance. However, I found a temporary solution:

Manually obtain the "crumb":
By sending a GET request to https://query2.finance.yahoo.com/v1/test/getcrumb, I can successfully get the "crumb" value.

Use the "crumb" with the correct URL:
After obtaining the "crumb", I can then make requests like this:

https://query1.finance.yahoo.com/v7/finance/quote?symbols=KO&crumb=<my_crumb>

This request works fine and returns the expected data.

It seems the issue lies with how the library handles "crumb" retrieval. Since this manual method works, it would be helpful if the library could implement a way to retrieve the "crumb" in a similar manner.

Is there any plan to address this in a future version of yahooquery? Or, if possible, could the documentation be updated to show how to manually retrieve the "crumb"?

Thank you for your work on the library; it’s been really useful!

Best regards,

[maread99]

(for reference, this matter was also raised in discussion #314)

Hi. For anyone just after price data I've implemented a fix in the latest version of market-prices (0.12.7) (by default market_prices uses yahooquery to get price data). Thank you to everyone who's mentioned the curl-cffi workaround!

I've had a bit of a look into the codebase here and thought it worth noting...

The crumb is currently being requested here from the address noted by @jirisarri10:

yahooquery/yahooquery/utils/__init__.py

Line 1422 in 57a73fc

response = session.get("https://query2.finance.yahoo.com/v1/test/getcrumb")

...I haven't followed it through to see whether the crumb is then being incorporated into the urls as required.

Separately, If a session isn't passed to 'yahooquery.Ticker' then it's initialised by this function...

yahooquery/yahooquery/utils/__init__.py

Lines 1345 to 1367 in 57a73fc

	def initialize_session(session=None, **kwargs):
	if session is None:
	if kwargs.get("asynchronous"):
	session = FuturesSession(max_workers=kwargs.get("max_workers", 8))
	else:
	session = requests.Session()
	if kwargs.get("proxies"):
	session.proxies = kwargs.get("proxies")
	retries = Retry(
	total=kwargs.get("retry", 5),
	backoff_factor=kwargs.get("backoff_factor", 0.3),
	status_forcelist=kwargs.get("status_forcelist", [429, 500, 502, 503, 504]),
	)
	if kwargs.get("verify") is not None:
	session.verify = kwargs.get("verify")
	session.mount(
	"https://",
	TimeoutHTTPAdapter(
	max_retries=retries, timeout=kwargs.get("timeout", DEFAULT_TIMEOUT)
	),
	)
	session.headers = random.choice(HEADERS)
	return session

i.e. this would be a place where the curl_cffi fix could be incorporated. Although, the curl_cffi.requests.Session object doesn't seem to provide the mount functionality - so the TimeoutHTTPAdapter might have to be lost or its functionality reworked somehow?

[jirisarri10]

Hi with only this yahooquery return to working. This lines must be inserted into init.py
from curl_cffi.requests import Session

def safe_get(session, url, max_retries=30, **kwargs):
for attempt in range(max_retries):
try:
return session.get(url, **kwargs)
except Exception as e:
if attempt == max_retries - 1:
raise

def initialize_session(session=None, **kwargs):
if session is None:
headers = random.choice(HEADERS)
session = Session(
impersonate="chrome110",
headers=headers,
timeout=kwargs.get("timeout", 30),
verify=kwargs.get("verify", True),
proxies=kwargs.get("proxies", None)
)
return session

def setup_session(session, url: str = None):
url = url or DEFAULT_SESSION_URL
try:
response = safe_get(session, url, allow_redirects=True)
except SSLError:
counter = 0
while counter < 5:
try:
session.headers = random.choice(HEADERS)
response = safe_get(session, url, max_retries=1, verify=False)
break
except SSLError:
counter += 1

if hasattr(response, "result"):  # por si usas FuturesSession en otro contexto
    response = response.result()

if "consent" in response.url:
    soup = BeautifulSoup(response.content, "html.parser")
    params = {}
    for param in ["csrfToken", "sessionId"]:
        try:
            params[param] = soup.find("input", attrs={"name": param})["value"]
        except Exception as exc:
            logger.critical(f'Failed to find or extract "{param}". Exception={exc}')
            return session

    response = session.post(
        "https://consent.yahoo.com/v2/collectConsent",
        data={
            "agree": ["agree", "agree"],
            "consentUUID": "default",
            "sessionId": params["sessionId"],
            "csrfToken": params["csrfToken"],
            "originalDoneUrl": url,
            "namespace": "yahoo",
        },
    )

return session

def get_crumb(session):
try:
response = safe_get(session, "https://query2.finance.yahoo.com/v1/test/getcrumb")
except (ConnectionError, RetryError, Exception):
logger.critical(CRUMB_FAILURE)
return None

if hasattr(session, "result"):  # por compatibilidad
    crumb = response.result().text
else:
    crumb = response.text

if not crumb or "<html>" in crumb:
    logger.critical(CRUMB_FAILURE)
    return None

return crumb

Resultado final:

import yahooquery as yq

Crear un objeto Ticker para AAPL

ticker = yq.Ticker('AAPL')

Obtener los datos del historial de los últimos 5 días

data = ticker.history(period="5d")

Imprimir los datos obtenidos

print(data) open high low close volume adjclose
symbol date
AAPL 2025-05-01 209.080002 214.559998 208.899994 213.320007 57365700 213.320007
2025-05-02 206.089996 206.990005 202.160004 205.350006 101010600 205.350006
2025-05-05 203.100006 204.100006 198.210007 198.889999 69018500 198.889999
2025-05-06 198.210007 200.649994 197.020004 198.509995 51216500 198.509995
2025-05-07 199.169998 199.440002 193.250000 196.250000 68536700 196.250000

Who could verify if this is correct and push it to the project to release a new version so this gem of a project doesn’t come to a halt? If you have any questions or suggestions, I’m entirely at your disposal.
Warm regards,
Javier Irisarri

[jirisarri10]

it’s a very good idea. Randomly switching the impersonate (browser agent) can help to:

Avoid blocks or detection by the server (such as Yahoo Finance).

Simulate more realistic traffic.

Bypass restrictions tied to specific user agents.

[Wermeling]

Actually, the workaround is quite simple. Use curl_cffi package and add a session.

from curl_cffi import requests
session = requests.Session(impersonate="chrome")
Ticker(tickers, asynchronous=True, progress=True, session=session)

Unfortunately, the download speed drops by 90%. I'm also curious—will the rate limit eventually be lifted, or is this workaround likely to be detected and blocked again?

[jirisarri10]

Hi! Thanks for the suggestion — it looks very helpful. Just to clarify, where exactly should I place this snippet in the context of yahooquery? I’d love to integrate it properly, but I’m not entirely sure how it fits into the current workflow. Any guidance would be much appreciated!

Best regards,
Javier

[Wermeling]

The first two lines should be on top of your codebase for clarity or anywhere nearby if you prefer. You need to install curl_cffi package using pip to use it.
The third line is my call to yahooquery, adding the session element which is the new requirement.

[james-stevens]

Actually, the workaround is quite simple. Use curl_cffi package and add a session.

from curl_cffi import requests
session = requests.Session(impersonate="chrome")
Ticker(tickers, asynchronous=True, progress=True, session=session)

I only pull about 20 Tickers every 5 mins & didn't see any performance hit for this work-around, but my service just runs in the background records the results to Prometheus, so my UI wouldn't see any hit from an extra second or two. Prometheus is currently reporting 2.1s per poll - I poll the price and major_holdersfor all tickers.

Unfortunately, the download speed drops by 90%. I'm also curious—will the rate limit eventually be lifted, or is this workaround likely to be detected and blocked again?

The history of both YahooFinance & YahooQuery is a constant arms race against Yahoo blocking or throttling access.

It's a shame Yahoo don't offer a consumer grade subscription at a reasonable price - I'd be happy to pay, but they only seem to have plans priced for Pros - tho I've not really researched it recently as YahooQuery does the job!!

[james-stevens]

Hi! Thanks for the suggestion — it looks very helpful. Just to clarify, where exactly should I place this snippet in the context of yahooquery? I’d love to integrate it properly, but I’m not entirely sure how it fits into the current workflow. Any guidance would be much appreciated!

I've run the session = requests.Session(impersonate="chrome") call every time I run the Ticker() call. I wasn't sure the difference with that & only running it once??

https://github.com/james-stevens/promyq/blob/master/python/promyq.py#L156-L157