Error: Received in valid request - Late crash inside GoBooDo.getInitialData() from GoBooDo.resethead() #70
Description
In my specific error case, the cause was something related to the class GoBooDo constructor called method resethead trying to set the future used request headers Cookie after making a request to "google."+self.domain
Frequently the response cookie does not contain the expected value with key NID, leading to a late crash inside getInitialData (where self.head it's used, but it's not defined)
As reported here, this NID is used by Google to recognize you in the next visits
However, as in the response object inside the resethead method this is not provided, it may be removed as part of a server side automatic cookie response, maybe related to the strict privacy policy
It may be skipped entirely, but if it's implemented, there might be a reason i guess...
Manually visiting Google inspecting this problem using an incognito chrome window, I found out that accepting the cookie consent popup, the cookie get populated with NID accordingly
Maybe it's time to harvest these information via an headless driver such as selenium (maybe triggering the consent for the cookie policy popup), and then keep using BeautifulSoup for the resulting response scraping if needed