Conversation
added 15 commits
July 28, 2025 15:56
…to edawson/scdl-schema
Signed-off-by: Eric T. Dawson <edawson@nvidia.com>
…to edawson/scdl-schema
Signed-off-by: Eric T. Dawson <edawson@nvidia.com>
…ives separately from core implementation testing Signed-off-by: Eric T. Dawson <edawson@nvidia.com>
Signed-off-by: Eric T. Dawson <edawson@nvidia.com>
edawson
requested review from
broland-hat,
cspades,
dorotat-nv,
jomitchellnv,
jstjohn,
jwilber,
polinabinder1,
pstjohn,
skothenhill-nv,
trvachov,
tshimko-nv and
yzhang123
as code owners
added 2 commits
August 14, 2025 15:10
Signed-off-by: Eric T. Dawson <edawson@nvidia.com>
Signed-off-by: Eric T. Dawson <edawson@nvidia.com>
Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Collaborator
|
/ok to test a720862 |
Collaborator
|
/ok to test 9719cee |
Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
…mo-framework into edawson/scdl-schema
Collaborator
|
/ok to test 6698414 |
Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Collaborator
|
/ok to test 33d6fa8 |
jwilber
reviewed
Aug 19, 2025
Signed-off-by: polinabinder1 <pbinder@nvidia.com>
Collaborator
|
/ok to test 74af02e |
Collaborator
|
/ok to test e4acd0f |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This MR implements a strict schema-defined header for SCDL archives. This header stores metadata about the archive and its composite arrays, including a version, the array lengths and data types, and information about the RowFeatureIndexes. This adds the features necessary to fix #999 as well as implement simple bit-packing of the rowptr, colptr, and data arrays. It also should make SCDL more secure, enable strict compatibility checking, and open the door to more performance improvements.
Note: I am still wiring up the header to the archive. I will make a note here when the MR is ready.
Type of changes
CI Pipeline Configuration
Configure CI behavior by applying the relevant labels:
Note
By default, the notebooks validation tests are skipped unless explicitly enabled.
Authorizing CI Runs
We use copy-pr-bot to manage authorization of CI
runs on NVIDIA's compute resources.
automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
/ok to testcomment on the pull request to trigger CI. This will need to be done for each new commit.Usage
This change is opaque to the user - the headers are not human-readable on disk. For a full description of the format and how to interact with it, see the
schemadirectory in SCDL's source directory.Pre-submit Checklist