Feat: Pre-check for existing images using in_img_dir column (Closes #34) #43

EmersonFras · 2025-11-24T17:25:41Z

Summary

Implements a feature to check for and skip downloading images that already exist in the target image directory (img_dir), addressing Issue #34.

Changes Implemented

A new method, check_exisiting_images, utilizes sum-buddy.gather_file_paths to collect file names from the existing img_dir.
A boolean column, in_img_dir, is added to the source_df to track which images are already present.
The download_images function is now passed a filtered dataframe containing only images that need to be downloaded (in_img_dir == False).
Added an early exit condition: if the filtered dataframe is empty (all images exist), the process exits with the message: '{img_dir}' already contains all images. Exited without executing.
Updated the output message regarding file counts to: There are {checksum_df.shape[0]} files in {img_dir}. Based on {csv_path}, there should be {expected_num_imgs} images.
Updated the README's CLI Examples section regarding new outputs and cases where relevant to these changes

Closes #34

…list

…directory cases, update test_download_images for new logic flow

Copilot

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

src/cautiousrobot/__main__.py

tests/test_existing_images.py

tests/test_download_images.py

src/cautiousrobot/__main__.py

src/cautiousrobot/utils.py

README.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…mageomics/cautious-robot into feature/issue-34/check-existing-images

tests/test_existing_images.py

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

tests/test_download_images.py

tests/test_existing_images.py

src/cautiousrobot/utils.py

src/cautiousrobot/__main__.py

src/cautiousrobot/utils.py

egrace479

I pushed one update to the description (beginning of the README) to reflect the updated functionality. A few other suggestions for clarity are noted below.

A bigger issue I noticed is that the check_existing_images function will have a conflict if the --starting-idx parameter is passed and I don't believe that's addressed.

Suggested method: run the existing image check for all filenames starting at a passed starting index and then have the download start from only the images that aren't already in the output folder following the starting index. This should maintain the current, expected behavior.

README.md

src/cautiousrobot/utils.py

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

src/cautiousrobot/utils.py

tests/test_existing_images.py

egrace479

A couple more items. I am also concerned that process_checksums needs an edit here.

tests/test_existing_images.py

src/cautiousrobot/utils.py

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

src/cautiousrobot/utils.py

egrace479

One last thing I noticed while running through the README examples. I made the suggestion on the line I could and just pointed to the line above. It's the new in_img_dir column that now shows up in the missing CSV.

README.md

egrace479

🚀

EmersonFras added 5 commits October 30, 2025 12:18

Add check_existing_images() to compare existing image files with CSV …

d35e2b2

…list

Integrate existing image check into main download flow

32ae64d

Only print pre-download directory status if missing images

489a896

Add tests for check_existing_images() including partial and complete …

aa434ef

…directory cases, update test_download_images for new logic flow

Update CLI Examples in README for new check existing image examples

37f4099

egrace479 requested a review from Copilot November 24, 2025 17:35

Copilot started reviewing on behalf of egrace479 November 24, 2025 17:35 View session

Copilot finished reviewing on behalf of egrace479 November 24, 2025 17:41

Copilot AI reviewed Nov 24, 2025

View reviewed changes

EmersonFras and others added 10 commits November 25, 2025 10:06

Fix CLI Examples wrong output dir

bf39663

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Remove validate_output_directory call

f73f16e

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Combine imports in main

a0f839c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Complete docstring for check_existing_images

1997b6d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Remove useless test assert

2cc1c16

Remove unused variable

6ffa15a

Use .code over str() for consistency

90653cf

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Use fullpath when checking existing files

17f49a2

Merge branch 'feature/issue-34/check-existing-images' of github.com:I…

c9a8d39

…mageomics/cautious-robot into feature/issue-34/check-existing-images

Handle directory existing but empty case

6fdcc47

egrace479 reviewed Nov 25, 2025

View reviewed changes

tests/test_existing_images.py Show resolved Hide resolved

egrace479 requested a review from Copilot November 25, 2025 19:20

Copilot started reviewing on behalf of egrace479 November 25, 2025 19:20 View session

Copilot finished reviewing on behalf of egrace479 November 25, 2025 19:24

Copilot AI reviewed Nov 25, 2025

View reviewed changes

Update description to match enhanced functionality

8d19c70

egrace479 requested changes Nov 26, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

src/cautiousrobot/utils.py Outdated Show resolved Hide resolved

EmersonFras and others added 4 commits December 1, 2025 11:38

Use bash section in MD for proper rendering

3810adb

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Move comments in README code section to new lines

fa8e01b

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Move comments to new lines

1c46e94

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Renamed 'subfolders_col' -> 'subfolders' for consistency

2001973

EmersonFras added 7 commits December 1, 2025 11:44

Spacing after imports in main

ae5c211

Use os.path.normpath to normalize pathing for comparison

29dffc4

Use copy of source_df to avoid future side-effects

a2ad790

Update docstring for test_directory_does_note_exist

641e4f4

Implement check_existing_images with starting_idx

0612a26

Add subfolders handling check for existing images

78fbfde

Remove test_main_directory_exists

212011f

egrace479 reviewed Dec 2, 2025

View reviewed changes

src/cautiousrobot/utils.py Outdated Show resolved Hide resolved

egrace479 reviewed Dec 2, 2025

View reviewed changes

tests/test_existing_images.py Outdated Show resolved Hide resolved

EmersonFras added 2 commits December 3, 2025 14:20

Rename missing_df to filtered_df

ced40e9

Pass actual DF in tests

ce1aab2

egrace479 reviewed Dec 3, 2025

View reviewed changes

tests/test_existing_images.py Outdated Show resolved Hide resolved

src/cautiousrobot/utils.py Outdated Show resolved Hide resolved

src/cautiousrobot/utils.py Outdated Show resolved Hide resolved

EmersonFras and others added 4 commits December 4, 2025 12:42

Update docstring wording in tests/test_existing_images.py

b8e6da0

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Update docstring wording in src/cautiousrobot/utils.py

923e243

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Return filtered_df when directory does not exist

385b12f

Update print message in check_existing_images

1092d74

egrace479 mentioned this pull request Dec 4, 2025

Update number of expected images validation check #45

Closed

Update test assert messages

b9fcf09

egrace479 reviewed Dec 8, 2025

View reviewed changes

src/cautiousrobot/utils.py Outdated Show resolved Hide resolved

Update phrasing for how many desired images downloaded

b62a09f

egrace479 requested changes Dec 8, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

Update README for extra column

ab3f398

egrace479 approved these changes Dec 8, 2025

View reviewed changes

EmersonFras merged commit fe031cb into main Dec 8, 2025
7 checks passed

egrace479 deleted the feature/issue-34/check-existing-images branch December 8, 2025 21:11

egrace479 mentioned this pull request Dec 11, 2025

Update AGENTS.md to match change in existing image handling #46

Closed

This was referenced Dec 19, 2025

feat: Remove starting_idx parameter (closes #45) #47

Merged

feat: Update AGENTS.md to match new feature for existing image handling (closes #46) #48

Merged

Feat: Pre-check for existing images using in_img_dir column (Closes #34) #43

Feat: Pre-check for existing images using in_img_dir column (Closes #34) #43

Uh oh!

Conversation

EmersonFras commented Nov 24, 2025

Summary

Changes Implemented

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

egrace479 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

egrace479 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

egrace479 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

egrace479 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

egrace479 left a comment •

edited

Loading