It appears that Amazon has updated their bestseller page and it breaks the scraper.

[JustinGardiner]

The page https://www.amazon.com/best-sellers-books/zgbs/books/
looks to have been updated such that XPathItemLinks should be getting the div element with class 'a-column a-span12 a-text-center _p13n-zg-list-grid-desktop_style_grid-column__2hIsc' instead of getting the li element with class 'zg-item-immersion'.

There might be other changes, but I didn't see them.

[JustinGardiner]

Oh, they also made it so that not all 50 books are loaded at once. Only the first 30 are.

[davrob01]

Hey. Thanks for letting me know. I was starting to look at this last night. Yes only first 30. And the rest are retrieved as 10, then another 10 in ajax calls. Based on my investigation so far, it is going to be challenging to craft some code to replicate the POST ajax calls. They are doing a lot of things in those requests (request headers, a json body, etc).

That being said, in the short term, it will be difficult to revise the app to still get the 100 books.

However, in the short term, I may be able to revise the app so at least the top 30 books are retrieved. As a temporary fix. Then later deploy a fix to get the rest of the books, if it isnt too hard.

Thank you