[User Interface: Infrastructure]: Improving the sitemaps

[gtsueng]

Issue Name

Improving the sitemaps

Issue Description

Currently, google search is indexing what we include in the sitemaps. All other records not included in the sitemaps are crawled by google, but google decides which record to index and which record not to.

As of June 2025, our sitemaps only include records from NIAID-funded repositories

To improve the number of records indexed, we should also include records from all IID repositories in the sitemaps. Note that this can be a separate sitemap file.

Additionally, we may want to consider including records in our sitemap that meet some sort of minimum metadata completeness threshold. To ensure that records indexed in google are of high quality.

To do:

• Generate list of top 5 records by metadata compatibility score per repository and note the record url and required ratio score
• For OMICS-DI (which aggregates from multiple sources and has large variability), do the top 5 records by metadata compatibility score per sdPublisher
• Evaluate the appearance of the records based on the card view and the page view
• Identify a stringent threshold for which the records generally look nice
• Develop a heuristic based on the thresholds that will ensure our sitemaps only include high quality and relevant records
• Develop a helper or util that will apply the heuristic and generate the sitemap file

Issue Discussion

No response

Related WBS task

https://github.com/NIAID-Data-Ecosystem/nde-roadmap/issues/33

For internal use only. Assignee, please select the status of this issue

• Not yet started
• In progress
• Blocked
• Will not address

Status Description

No response

UI change status check list

• This ui infrastructure change has yet to be discussed between NIAID, Scripps, Leidos
• This ui infrastructure change has been discussed/reported between NIAID, Scripps, Leidos
• This ui infrastructure change has been mocked up
• This ui infrastructure change has been implemented on Dev
• This ui infrastructure change has been approved for Staging
• This ui infrastructure change has been implemented on Staging
• This ui infrastructure change has been approved for Production
• This ui infrastructure change has been implemented on Production

[gtsueng]

Thanks to @candicecz the top records by metadata score per repo are available here: https://docs.google.com/spreadsheets/d/1WiXPfpBUP_qLUyDuTqGbu09GRcqt_uTzuKKD128sEP4/edit?pli=1&gid=719458099#gid=719458099

I've done a first pass on them. @andrewsu @newgene, I think the follow heuristics for extending the sitemaps beyond the IID repositories would be stringent enough to ensure that the records indexed by google appear to be high quality IID records:

1. All NIAID-funded Repositories (current sitemap)
2. All IID Repositories (not in 1) (to be added)
3. All records not in 1 or 2 that meet the following criteria: (any opinions on this last one?)
   a. Has metadata completeness score of >=17
   b. Has ratio of required score of >=0.75
   c. Has a value for infectiousAgent

[gtsueng]

For now, we will proceed with the above heuristics which are expected to nearly double the number of records that we include on our sitemaps and monitor the results of doing so.

Lowering the threshold in 3a to >=15 will potentially expand the number of records in the sitemap between 4-5x