JSONDecodeError within scrape() method

[santimorales0326]

Hi everyone!

Got an JSONDecodeError error after running the following code. I suspect it has to do with the source recognition as soup rather than string

from instascrape import *

'# Instantiate the scraper objects
google = Profile('https://www.instagram.com/google/')
google_post = Post('https://www.instagram.com/p/CG0UU3ylXnv/')
google_hashtag = Hashtag('https://www.instagram.com/explore/tags/google/')

'# Scrape their respective data
google.scrape()
google_post.scrape()
google_hashtag.scrape()

print(google.followers)
print(google_post['hashtags'])
print(google_hashtag.amount_of_posts)

JSONDecodeError

JSONDecodeError Traceback (most recent call last)
in
7
8 # Scrape their respective data
----> 9 google.scrape()
10 google_post.scrape()
11 google_hashtag.scrape()

~\anaconda3\lib\site-packages\instascrape\core_static_scraper.py in scrape(self, mapping, keys, exclude, headers, inplace, session, webdriver)
142 scraped_dict = self.source.to_dict()
143 else:
--> 144 return_data = self._get_json_from_source(self.source, headers=headers, session=session)
145 flat_json_dict = flatten_dict(return_data["json_dict"])
146

~\anaconda3\lib\site-packages\instascrape\core_static_scraper.py in _get_json_from_source(self, source, headers, session)
258 if initial_type:
259 soup = self.source
--> 260 json_dict_arr = json_from_soup(soup)
261 if len(json_dict_arr) == 1:
262 json_dict = json_dict_arr[0]

~\anaconda3\lib\site-packages\instascrape\scrapers\scrape_tools.py in json_from_soup(source, as_dict, flatten)
92
93 if as_dict:
---> 94 json_data = [json.loads(json_str) for json_str in json_data]
95 if flatten:
96 json_data = [flatten_dict(json_dict) for json_dict in json_data]

~\anaconda3\lib\site-packages\instascrape\scrapers\scrape_tools.py in (.0)
92
93 if as_dict:
---> 94 json_data = [json.loads(json_str) for json_str in json_data]
95 if flatten:
96 json_data = [flatten_dict(json_dict) for json_dict in json_data]

~\anaconda3\lib\json_init_.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
355 parse_int is None and parse_float is None and
356 parse_constant is None and object_pairs_hook is None and not kw):
--> 357 return _default_decoder.decode(s)
358 if cls is None:
359 cls = JSONDecoder

~\anaconda3\lib\json\decoder.py in decode(self, s, _w)
335
336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()
339 if end != len(s):

~\anaconda3\lib\json\decoder.py in raw_decode(self, s, idx)
351 """
352 try:
--> 353 obj, end = self.scan_once(s, idx)
354 except StopIteration as err:
355 raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

To Reproduce
Steps to reproduce the behavior:

1. Run the example provided here https://github.com/chris-greening/instascrape
2. See error described above

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Windows 10

[esvban]

i tried going through this with the debugger.

I don't know the original intent of this code, or how it worked before https://github.com/chris-greening/instascrape/blob/master/instascrape/core/json_algos.py#L75

but basically the json strings returned by this filter are not all pure json data, but also contain javascript code. Checking for any script tag with 'config' in it is not the correct way to find json script tags, at least anymore. I tried lookin at the source html and couldn't figure out which json data is actually needed. The media html is using a bunch of async loading of json data embedded in javascript code, as well, that will be difficult to parse correctly, probably designed to make scraping the website very difficult without using the Facebook developer API. I couldn't even figure out how to manually download a single video from a reel page (i.e. using chrome developer tools / inspector) either. I suppose long gone are the days of just searching for a ".mp4" link in the source and downloading it, with all the obfuscation and anti-scraping / anti-downloading going on...

There hasnt been any commit on this package in over a year, probably have to fix it our selves / fork it if the developers are no longer active.

edit: after a big more digging around, finally found the mp4 sources for reels in the network profiler with some url editing... The download speed is super slow, and the audio / video are split into separate files, but this is good enough for me, at least for now.

[santimorales0326]

Thank you for your quick response. Do you know how efficient scrapping with Facebook dev API is? I'd appreciate any resource that you can provide.

…

On Thu, Jul 28, 2022 at 9:43 AM Stephen Hopkins ***@***.***> wrote: i tried going through this with the debugger. I don't know the original intent of this code, or how it worked before https://github.com/chris-greening/instascrape/blob/master/instascrape/core/json_algos.py#L75 but basically the json strings returned by this filter are not all pure json data, but also contain javascript code. Checking for any script tag with 'config' in it is not the correct way to find json script tags, at least anymore. I tried lookin at the source html and couldn't figure out which json data is actually needed. The media html is using a bunch of async loading of json data embedded in javascript code, as well, that will be difficult to parse correctly, probably designed to make scraping the website very difficult without using the Facebook developer API. There hasnt been any commit on this package in over a year, probably have to fix it our selves / fork it if the developers are no longer active. — Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2HQJCULX3YIJRJDWOYH4UTVWI2ZPANCNFSM54VZVNIQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[esvban]

I don't know too much about the API, except it is only usable by registered and most likely required to be approved, facebook integrated applications.

[ivangoranov]

Hi there, has anybody solved this yet?

[furmanlukasz]

Hi, I'm hiting the same error. anybody have a clue how to solve it?

[nsde]

This project isn't maintained anymore ._.