Add workflow to create and upload package to track downloads #1031

amontanez24 · 2025-09-03T20:12:02Z

resolves #1030

This PR adds a script and github workflow that:

On release, creates a python package called rdt-download-tracker
Matches the version of this package to the version of the rdt release
uploads the package to S3
Updates the index.html page in the S3 bucket to have the new files (this is required for pip to be able to install it)

codecov · 2025-09-03T20:12:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (dbfd782) to head (9f9fb70).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main     #1031   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           19        19           
  Lines         2642      2642           
=========================================
  Hits          2642      2642

Flag	Coverage Δ
integration	`83.49% <ø> (ø)`
unit	`100.00% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

lint Beginning to add code to update index Adding workflow and updating script to update index.html Commenting out branch for now and renaming workflow Adding id-token Only adding new files to index Adding dryrun and ability to create index file for first time Fixing format of index.html adding new line using actual s3 link

amontanez24 · 2025-09-04T23:17:35Z

.github/workflows/publish_download_tracker.yml

+        # with:
+        #   ref: 'stable'


I will uncomment this before merging because we want it to install the rdt release version

amontanez24 · 2025-09-04T23:17:50Z

.github/workflows/publish_download_tracker.yml

+on:
+  release:
+    types: [published]
+  pull_request:


I will remove this before merging

gsheni · 2025-09-05T13:10:20Z

scripts/create_download_tracker.py

+    text_list = [current_text]
+    for file in files:
+        download_link = f'https://{BUCKET}.s3.us-east-1.amazonaws.com/{S3_PACKAGE_PATH}{file}'
+        new_link = f"<a href='{download_link}'>{file}</a>"


Do you need to include the sha256 hash?

Each file URL SHOULD include a hash in the form of a URL fragment with the following syntax: #=, where is the lowercase name of the hash function (such as sha256) and is the hex encoded digest.

https://packaging.python.org/en/latest/specifications/simple-repository-api/#project-detail

gsheni · 2025-09-05T13:13:51Z

scripts/create_download_tracker.py

+            Key=index_file_path,
+            Body=new_index,
+            ContentType='text/html',
+            CacheControl='no-cache',


Can we specify the ChecksumAlgorithm to be SHA256? It defaults to CRC64NVME

does the index file need a checksum? Wouldn't it just be needed for the actual wheels and tar.gz files?

Oh right, yah just specify the Checksum for wheel/tar.gz files

ok, good catch. I'll add that

amontanez24 · 2025-09-09T15:49:37Z

@gsheni @rwedge I added a refactoring change to pull the publishing workflows into workflows that have to always be called with inputs. The release workflow will just call them with the desired inputs. Let me know what you think

gsheni · 2025-09-09T20:03:27Z

.github/workflows/publish.yml

@@ -0,0 +1,87 @@
+name: Publish


Suggested change

name: Publish

name: Publish RDT

gsheni · 2025-09-09T20:06:37Z

.github/workflows/publish.yml

+        type: boolean
+        default: false
+jobs:
+  release:


Suggested change

release:

publish-rdt::

gsheni · 2025-09-09T20:10:40Z

.github/workflows/release.yml

+      candidate: false
+      test_pypi: false
+  release-download-tracker:
+    uses: ./.github/workflows/publish_download_tracker.yml


What happens if the download tracker version already exists on PyPI? Will version of download tracker match RDT's version?

The script will use the version of RDT that it installs. If it already exists, it will be overridden. That shouldn't really happen but if it does I think it's ok

Gotcha, as long as the hash of the files doesn't change we should be good.

If the hash does change, will it work if I remove the old link in the index and replace it with a link with the new hash?

It should. pip does verify the wheel/tar.gz file hash matches. That is, the hash given by index.html should match hash generated by the user's local machine (pip downloads the file and hashes it itself). If the mismatch occurs, pip will throw an error.
In addition,pip on user's machine will cache files. So if they have downloaded a wheel file before, the hash changes, then pip will complain. I believe the user would have to run pip cache purge in that case.

So is it better to do that, or just prevent it from uploading a package that is already there?

We should prevent it from uploading if the package is there (similar to public PyPI where you cannot re-upload an already published version).

gsheni

Looks good. Just 1 suggestion.

gsheni · 2025-09-15T14:19:55Z

scripts/create_download_tracker.py

+            if file_name not in links:
+                filepath = os.path.join('dist', file_name)
+                file_hash = _get_file_hash(filepath)
+                s3_client.upload_file(filepath, BUCKET, dest)


Suggested change

s3_client.upload_file(filepath, BUCKET, dest)

s3_client.upload_file(filepath, BUCKET, dest, ExtraArgs={'ChecksumAlgorithm': 'SHA256'})

rwedge · 2025-09-15T14:51:15Z

scripts/rdt-download-tracker/README.md

@@ -0,0 +1 @@
+# TODO: fill this in


Should we create a follow up issue to document this?

amontanez24 · 2025-09-23T16:30:29Z

Closing this PR because we will no longer be adding this package

amontanez24 force-pushed the issue-1030-tracking-package branch from cbbd203 to 3b2cfa3 Compare September 4, 2025 19:22

amontanez24 added 2 commits September 4, 2025 17:54

Setting content type

2d58947

Lint

9b16b0b

amontanez24 commented Sep 4, 2025

View reviewed changes

amontanez24 marked this pull request as ready for review September 4, 2025 23:23

amontanez24 requested a review from a team as a code owner September 4, 2025 23:23

amontanez24 requested review from gsheni and removed request for a team September 4, 2025 23:23

gsheni reviewed Sep 5, 2025

View reviewed changes

Adding hash to file links

20f10e0

amontanez24 requested a review from rwedge September 8, 2025 20:11

amontanez24 added 3 commits September 8, 2025 15:14

Adding filepath

6b5e331

Fixing file loading

9f94f75

refactoring workflows

976e3ea

gsheni reviewed Sep 9, 2025

View reviewed changes

amontanez24 added 2 commits September 11, 2025 12:43

PR comments

903ce7c

Not uploading files if the package is already in the index

ac9a717

amontanez24 requested a review from gsheni September 11, 2025 19:55

gsheni approved these changes Sep 15, 2025

View reviewed changes

rwedge approved these changes Sep 15, 2025

View reviewed changes

amontanez24 added 2 commits September 15, 2025 15:21

Adding checksum to aws package uploads

ac415a2

adding issue to todo

9f9fb70

amontanez24 closed this Sep 23, 2025

	s3_client.upload_file(filepath, BUCKET, dest)
	s3_client.upload_file(filepath, BUCKET, dest, ExtraArgs={'ChecksumAlgorithm': 'SHA256'})

Add workflow to create and upload package to track downloads #1031

Add workflow to create and upload package to track downloads #1031

Uh oh!

Conversation

amontanez24 commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gsheni Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amontanez24 commented Sep 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gsheni Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gsheni left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amontanez24 commented Sep 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

amontanez24 commented Sep 3, 2025 •

edited

Loading

codecov bot commented Sep 3, 2025 •

edited

Loading

gsheni Sep 5, 2025 •

edited

Loading

gsheni Sep 10, 2025 •

edited

Loading