Update scraper to work with latest xbox.com update

[jasonclemons]

With the latest xbox.com update, the scraper no longer returns the proper information. Rewrite it so that it can scrape the right information.

[bartekcc]

Just ignore /profile.json?gamertag= and try to get games.json. It should contains all information u need.

I have random errors when trying to get games.json, but usually it works well.
profile.json always returns error.

[djekl]

The latest updates to xbox.com mean that the profile pages REQUIRE javascript. Without using a JS Scraper solution (PhantomJS/CasperJS) you can't get the pages. This is because cURL and file_get_contents() can't mock javascript.

[jasonclemons]

@djekl is correct. I am looking into other libraries that can accomplish what needs to be done. With the 3.0 rewrite I have been slowly working on, it will allow the use of other libraries more easily by making it more Composer-friendly.

[rubenyc]

Is there any update guys?

[reidmain]

It doesn't look like all the profile information is on the redesigned pages. Things like the user's online status and what game they are playing are absent.

[jasonclemons]

I'm currently exploring alternate methods of obtaining this info. The rewrite may take some time.

[djekl]

@reidmain Please remember that this is just a side project for @jasonclemons and I contribute when I can. The rewrite to get around the new issue of pages requiring javascript could take some time

[reidmain]

I totally understand @djekl. I was actually scraping the Xbox.com website just like you guys for a side project I was working on. I was about a week away from launching it when the redesigned happened and so the project has basically been scuttled until I find a workaround. That search lead me to this project.

If you guys need any help investigating alternate data sources just ask. So far the new Xbox.com website seems to be void of the information I need and sniffing traffic from the Xbox SmartGlass app has also proved fruitless.

[djekl]

@reidmain What information are you looking for? Scraping xbox.com will only ever get you the information they want you to have. I have a closed source site that uses the smartglass api (https://xboxapi.com). Its just going through an upgrade today, so may not be 100% stable.

As for this project, I think that the only real option moving forward would be to implement PhantomJS or CasperJS to scrape xbox.com. The only problem with these scripts is that they can be rather slow to run.

[reidmain]

@djekl I was scraping the user's gamerpic, online status and the name of the game they were playing before. Even if you weren't friends with that person their online status used to be displayed if they didn't have the privacy settings turned on. I have been able to make the "profile" request to the SmartGlass API but it seems incredibly stale.

For example while I was testing the SmartGlass API I had the Twitch app running on my Xbox One. I then logged into my dummy account on xbox.com and saw that my main account was using the Twitch app. However according to the SmartGlass app I had "been last seen on Xbox.com 30 minutes ago" because at some point AFTER I started the Twitch app I had visited xbox.com and it was only remember the last thing I did.

[djekl]

@reidmain If you have access to the smartglass code, then its the presence your after. That shit is so real time its scary, you can also see if they have anything snapped etc.

[reidmain]

@djekl just to be sure we're talking bout the same thing I am looking at https://uds-part.xboxlive.com/profile.svc/profile?sectionFlags=97&gamertag=Blah and using the PresenceInfo element.