deep-registry-archive will grab the latest LIDVIDs only, not the LIDVIDs associated with the bundle VID

[tbarnes4]

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

Two, I believe related, bugs:

(1) When I run pds-deep-registry-archive on a bundle that has a collection with which the inventory file contains multiple VIDs of a single LID, only the most recent LIDVID product files are included in the manifest files. This is a problem for archiving as all the products listed should be included. This is a greater problem for us in that the first version of many of our collections contain multiple versions of a product due to migration. I will note that this issue did not exist in April 2024 when I ran the utility on urn:nasa:pds:nh_mvic::1.0.

(2) When I run pds-deep-registry-archive on bundle VID that is not the most recent, the utility correctly grabs the bundle VID requested bundle products, but incorrectly grabs the latest collection VID instead of the collection VID associated with this older bundle VID as specified in the bundle product.

Note that I discovered these issues when I saw that the produced manifest files record count did match the number of files I expected to be included.

🕵️ Expected behavior

For (1) and (2), I would expect the utility grab all collection VIDs from the bundle product and all these collection's child products from the correct collection VID's collection product inventory file. I believe that this is the case when running pds-deep-archive (which cannot be run on *.lblx, see #157 ), and was the case with a prior version of pds-deep-registry-archive.

📜 To Reproduce

For (1) run:
pds-deep-registry-archive -s PDS_SBN urn:nasa:pds:nh_mvic::1.0

Sort and compare these results with those posted in the previously produced manifests (see nh_mvic_v1.0_20240415* files, noting that several versioned files are now missing).

For (2) run:
pds-deep-registry-archive -s PDS_SBN urn:nasa:pds:nh_documents::2.0

Note that the collection urn:nasa:pds:nh_documents:mission::2.0 child products (seen with pds4-nh_documents:mission-v2.0/" are picked up even though this bundle should be picking up urn:nasa:pds:nh_documents:mission::1.0 files only.

🖥 Environment Info

• Version of this software: pds-deep-registry-archive 1.3.0
• Operating System: macOS Monterey Version 12.7.6

📚 Version of Software Used

pds-deep-registry-archive 1.3.0

🩺 Test Data / Additional context

No response

🦄 Related requirements

🦄 #xyz

Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

No response

🎉 Integration & Test

No response

[jordanpadams]

@tbarnes4

(1) When I run pds-deep-registry-archive on a bundle that has a collection with which the inventory file contains multiple VIDs of a single LID, only the most recent LIDVID product files are included in the manifest files. This is a problem for archiving as all the products listed should be included. This is a greater problem for us in that the first version of many of our collections contain multiple versions of a product due to migration. I will note that this issue did not exist in April 2024 when I ran the utility on urn:nasa:pds:nh_mvic::1.0.

As long as you run deep-registry archive on older versions of the bundle, this should not be a problem. The deliveries to NSSDCA are for new versions of data. The older versions would have already been delivered.

(2) Note that the collection urn:nasa:pds:nh_documents:mission::2.0 child products (seen with pds4-nh_documents:mission-v2.0/" are picked up even though this bundle should be picking up urn:nasa:pds:nh_documents:mission::1.0 files only

Hmmm. when I look at the bundle label it says this:

<Bundle_Member_Entry>
  <lidvid_reference>urn:nasa:pds:nh_documents:mission::2.0</lidvid_reference>
  <member_status>Primary</member_status>
  <reference_type>bundle_has_document_collection</reference_type>
</Bundle_Member_Entry>

which would indicate it should be picking up the products associated with version 2.0 of that collection. are you saying that is not happening?

[tbarnes4]

As long as you run deep-registry archive on older versions of the bundle, this should not be a problem.

For (1), the problem is there are multiple LIDVIDs for a single LID within a collection, and the tool is only grabbing the latest version, not all of them. There are no older bundle versions, so it is the current version that is having the problem. Example (1) shows this. But if I were run this on an older bundle like mentioned in (2), it grabs the correct collection LIDs, but then grabs the most recent LIDVIDs of the collection and child products, instead of the first version. See at the bottom for the example in (2).

But again, this was not a problem in the prior version of the software.

The deliveries to NSSDCA are for new versions of data. The older versions would have already been delivered.

True, but if that was fully the case, why does the tool add old collections already delivered?

Hmmm. when I look at the bundle label it says this:

<Bundle_Member_Entry>
  <lidvid_reference>urn:nasa:pds:nh_documents:mission::2.0</lidvid_reference>
  <member_status>Primary</member_status>
  <reference_type>bundle_has_document_collection</reference_type>
</Bundle_Member_Entry>

which would indicate it should be picking up the products associated with version 2.0 of that collection. are you saying that is not happening?

You are correct. I apologize. The offending collection was ralph not mission. In the produced manifest files ralph::1.0 was not included but ralph:2.0 was included. ralph::2.0 was delivered in nh_documents::3.0 not 2.0.

[jordanpadams]

Per discussions today, for (1) above, support for a collection referencing multiple versions of a product is not supported by the API, and therefore will not be supported by deep archive for right now.

For this ticket, we will document that this is not possible using this utility.

For (2), we will need to investigate further.