Open
Conversation
kp-automata
reviewed
Aug 23, 2025
Collaborator
kp-automata
left a comment
kp-automata
left a comment
There was a problem hiding this comment.
Left some feedback! Very cool stuff. Let me know when it's addressed in this PR
| filepath = os.path.join(destination, f"image_{idx}.jpg") | ||
| with open(filepath, "wb") as f: | ||
| f.write(r.content) | ||
| print(f"Downloaded: {filepath}") |
Collaborator
There was a problem hiding this comment.
May you change all the print statement to logging?
| new_height = driver.execute_script("return document.body.scrollHeight") | ||
| if new_height == last_height: | ||
|
|
||
| def human_pause(a=1.2, b=2.4): |
| def batch_data_downloader_selenium(url=None, max_pages=9): | ||
| """Downloads images from a Flickr album using Selenium.""" | ||
| url = url or "https://www.flickr.com/photos/esa_events/albums/72157716491073681/" | ||
| destination = "../data/labeled/no" |
Collaborator
There was a problem hiding this comment.
Let's save it somewhere else. Let's say ../data/flickr_captures
Ideally use the path utils utility function to resolve the data directory
| """ | ||
| # TODO: Hardcoded url for now, if needed expose this for customization | ||
| url = "https://www.flickr.com/photos/esa_events/albums/72157716491073681/" | ||
| def batch_data_downloader_selenium(url=None, max_pages=9): |
Collaborator
There was a problem hiding this comment.
Let's make it a class because it has nested functions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Downloads all visible photos from a Flickr album using Selenium, handling infinite scroll and pagination. It collects only real photo URLs (live.staticflickr.com), de-duplicates with a seen set, throttles with human-like pauses, and writes images to ../data/labeled/no. Returns the total number of files saved.