Add Supplements spider + integrated crawling/postprocessing #5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduce ScpSupplement class for item representation - Implement ScpSupplementSpider to crawl supplement pages - Update makefile to include supplement in data targets
- Implement run_postproc_supplement to process SCP supplement data - Create necessary directories and handle data extraction - Store processed supplements in JSON format for further use
- Added instructions for crawling pages tagged as 'supplement' - Updated content structure to include multiple content types - Clarified post-processing details for supplements
- Updated the LinkExtractor in ScpTaleSpider to deny links matching specific patterns, improving the relevance of parsed tales.
- for reference
tedivm
requested changes
Dec 15, 2025
Member
tedivm
left a comment
tedivm
left a comment
There was a problem hiding this comment.
Overall this looks really good, but there are some changes to be made to the github workflows.
- This change prevents automatic pushing to the SCP API during CI - it should run on pull requests to allow for manual review first - remove clone step as it's unnecessary - resolves scp-data#5
- this fixes unintend removal of Linkextractor Rule
Contributor
Author
|
I adjusted the workflow file, but keep in mind I really added it by accident. So if the edits are not working out, I would prefer removing the yml file - I really do not know much about github actions :) Also For testing purpose I forked the api repo, too. See: Result Data: Note: I actually remove a linkrule for "TALES", so in the testrun "Tales" is empty. I reverted it with the last commit. I will start another workflow run to make sure this works out. |
- Add checks for empty responses and missing 'body' in JSON - Log errors for various failure scenarios to improve debugging - Ensure robust parsing of history HTML to prevent crashes
- Allows manual triggering of the workflow - Improves flexibility for testing and updates
- Handle empty history cases by returning an empty list - Support both dict and list formats for history input - Safely parse date strings with error handling - Sort revisions by date, ensuring robustness against missing values
- Use `get` method to safely access history in hubs and items - Prevent potential KeyError by ensuring history key is present
heroheman
added a commit
to heroheman/scp_crawler
that referenced
this pull request
Dec 17, 2025
- This change prevents automatic pushing to the SCP API during CI - it should run on pull requests to allow for manual review first - remove clone step as it's unnecessary - resolves scp-data#5
Contributor
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds first-class support for “supplement” pages from the SCP Wiki and integrates them into the existing crawl + post-processing workflow.
scp_supplementto crawl pages tagged supplement and export them toscp_supplement.json.run_postproc_supplementto generate processed outputs under supplement:content_supplement.json(full content + history/source/images)index.json(metadata + content_file)parent_scp(best-effort extracted from the link) and parent_tale (best-effort extracted from *- patterns).scp_crawlandscp_postprocess(and still available via dedicatedsupplement_*targets).How to test
make scp(includes supplements crawl + postprocess)Or individually:
make supplement_crawlmake supplement_postprocessscrapy crawl scp_supplement -o data/scp_supplement.jsonThis PR should fix #2