amount of posts

[t-duan]

Hi Chris,
I'm not sure if I did something wrong. But the amount of scraped post seems to be very limited. For example, when I scrape images tagged with "moon" and set the parameter amp=1000, I can only get 61, although there are 34800800 totally.
Can you give me some advise to fix the limit?
Cheers,
Tinghui

[chris-greening]

Hey @t-duan! I probably should've been clearer with the documentation, unfortunately instascrape can only request and scrape the initial static HTML served back in an HTTP request.

When you make a request to Instagram at say the #moon hashtag, they serve back the first 60 or so posts to construct the initial page and then as you navigate down the page, it dynamically loads more content giving you the infinite scroll. To scrape more posts, you'll have to use something such as Selenium to render the JavaScript and collect post shortcodes and then you can instantiate Post objects with those shortcodes and the library will handle the rest from there.

Like I said, I should've been clearer with the documentation so my apologies there, now that the library is stable and I have some free time, I'll be going back and clarifying some of the vaguer aspects of the library.

I recommend you check out this blog post to see an example of a Selenium scroller that was able to gather post shortcodes similar to what I mentioned above. I've had a few people ask similar questions of extending instascrape to access dynamic content so I'll likely be doing some tutorials in the very near future covering this subject so keep an eye out for those!

Thanks so much for reaching out and let me know if you have any more questions! 😄

[t-duan]

Hi Chris,
thank you very much for your detailed answer! Now I understand the problem. I'll read your blog post about Selenium.
Best
Tinghui