Scrape from Pokemon Zone

[greg19]

Problems with limitlesstcg

There has been #749 which arisen from inconsistency in promo cards rarity on limitlesstcg. Also identifying reprints is currently rather fragile, and might break with new sets.

Change card database structure

I also argued during the migration from card_id to internal_id that we shouldn't duplicate cards in the cards.json array when there appear in multiple expansion. Meaning:

// currently
[
  {
    "expansion": "A2b",
    "card_id": "A2b-1",
    "internal_id": 327744,
    // all other properties
  },
  {
    "expansion": "A4b",
    "card_id": "A4b-6",
    "internal_id": 327744,
    // every single property repeated again
  },
  // other cards
]

// should be
[
  {
    "expansions": [
      { "expansion": "A2b", "number": 1 },
      { "expansion": "A4b", "number": 6 },
    ],
    "internal_id": 65600,
    // all other properties
  },
  // other cards
]

Then in another file we can for each expansion store the list of cards (internal ids) that appear in this expansion.

Advantages of Pokemon Zone

https://www.pokemon-zone.com/sets/b1/
https://www.pokemon-zone.com/cards/a4b/1/bulbasaur/

First of all, they actually scrape the game data, so their data should be the highest quality and we can more or less trust it that there won't be any inconsistencies caused by them (like the inconsistency of promo cards rarity on limitlesstcg).

Promo rarities

When on the topic of promo rarities, they actually also assign them (to all promo cards, including P-A). This suggests that they actually have it somewhere in the game code. But the game interface puts them into a separate "promo" rarity (and you can't find them when filtering for non-promo rarities). Next thing that the game is so messy about...

One card in multiple expansions

They actually have a separate table for this, so it will be much more robust.

Evolution line

They include what pokemon the current one can evolve into, allowing us to implement #656.

[marcelpanse]

Yeah seems good. Limitless has some other annoying problems, like not implementing the shiny rarities at all. If they release new sets pretty fast then that would be great.

[greg19]

Note: They had B1a at the day of the release, before limitless.

[afu-dev]

I'm already scrapping Pokémon Zone for my personal use and I'm using the data mined Google Drive for the assets.

Using Pokémon Zone is a must to associate correctly the internal PTCGP id (example: PK_10_013070_00 for the first Bulbasaur in B1a) with the collection number of the Pokémon.

I don't want to share my scripts (it has my personal info from PZone in it), but I think I can dump everything (assets + cards, sets, expansions json) in a GitHub repo. I can add a packages.json and some exports to allow this project to just require the repo to use the data?

I would prefer to setup everything to only scrap the game itself, but I haven't dig deep enough to fetch the required data 😅

Ref: TCGPocket Data Mined Google Drive link

[greg19]

I'm already scrapping Pokémon Zone for my personal use and I'm using the data mined Google Drive for the assets.

Using Pokémon Zone is a must to associate correctly the internal PTCGP id (example: PK_10_013070_00 for the first Bulbasaur in B1a) with the collection number of the Pokémon.

I don't want to share my scripts (it has my personal info from PZone in it), but I think I can dump everything (assets + cards, sets, expansions json) in a GitHub repo. I can add a packages.json and some exports to allow this project to just require the repo to use the data?

I assume you are calling their backend? The /api/game/game-data/ looks promising and can be called without logging in. Looks to me like the mapping you are talking about is under expansionCollectionNumbers[i].collectionNumbers. This endpoint doesn't provide info about the cards (like hp, attacks, etc.) and I didn't see any request for it when viewing individual cards. Do you maybe know if they have such endpoint or they are serving a prerendered page?

I would prefer to setup everything to only scrap the game itself, but I haven't dig deep enough to fetch the required data 😅

I'd assume it would be against the terms of service of the game. When dealing with other third party services they probably can't do anything when we scrape them other than send an angry email and change their data representation. But for example Nintendo (IDK if The Pokemon Company also) is known from suing everyone over everything. Probably not that big of a concern (our site is rather small at the end of the day), but still.

[afu-dev]

The card info is at /api/game/card-data/, it should have everything you want. Indeed it has no auth on it, but on /api/players/mine/ it does, that's why my setup is using my credentials.

This way I can sync my collection on Pokemon-Zone, export to CSV with my scripts and import the CSV into this (better 😬) tracker.

PZ is for sure datamining the game, and even the Nintendo API to fetch user collection informations (completly against the ToS too). But because the data is owned by The Pokemon Company, scraping it from PZ or the game itself is the same problem. At least, when scrapping the game, we are only against ToS from Dena/Pkmn Company and not against PZ haha.

Limitless hasn't updated yet. For the next expansion, I'll have an external repo ready with the data from the game (assets + cards info).

[greg19]

The card info is at /api/game/card-data/, it should have everything you want.

Cool, then it will be so much easier to scrape than XPath'ing the HTML from limitless.

But because the data is owned by The Pokemon Company.

Hm okay. Though, I agree that using Dena API would be nicer because we don't have one more failure point in form of PZ.

[marcelpanse]

Getting it directly from the original source would be preferred. How do we get the data?

• the card data updates separately from the APK, so no use trying to decompile the APK file.
• when the app updates, listen for the requests using a debugger and figure out the API. It could be this requires you to be logged in with a Nintendo account
• figure out where the data is stored on the phone after downloading and see if we can use that. That would be ideal.

[greg19]

figure out where the data is stored on the phone after downloading and see if we can use that. That would be ideal.

GPT says this: https://chatgpt.com/share/69430efb-af1c-8012-b744-eb4e9f3b5cbb

I checked and the app is (obviously) not debuggable, so this would require root/emulator. I don't currently have any rooted device on hand. I also earlier tried to set up an emulator with TCP Pocket, but it doesn't run for me for some reason.

[greg19]

The card info is at /api/game/card-data/, it should have everything you want.

Cool, then it will be so much easier to scrape than XPath'ing the HTML from limitless.

Pokemon Zone uses Cloudflare bot protection that effectively blocks these endpoints outside of a real browser 😕

[afu-dev]

Yes PZ is completely protected. I tried to fetch the data from the game without any success for now.

[SantiagoRR2004]

Hello, I also maintain a repository that also scrapes limitlesstcg. I wanted to ask some questions on how you are scrapping and obtaining some data. Do I create a new issue or can I ask in this one? All of my questions are related to scraping.

[marcelpanse]

The code is open source, you can just check the files ;-)

[afu-dev]

Hello, I also maintain a repository that also scrapes limitlesstcg. I wanted to ask some questions on how you are scrapping and obtaining some data. Do I create a new issue or can I ask in this one? All of my questions are related to scraping.

See https://github.com/marcelpanse/tcg-pocket-collection-tracker/blob/main/scripts/scraper.ts

[SantiagoRR2004]

I was mostly wondering about the translations. Do you rely contributors or are you scraping them?

I would also like to comment that you can see the shiny rarity in limitlesstcg. In theory, you could cross-reference using the Advanced Search, with something like this. I haven't implemented it yet, but it seems possible.