Skip to content

Conversation

@luddaniel
Copy link
Contributor

@luddaniel luddaniel commented May 28, 2025

Hi everyone :)
Our team identified a issue with api/admin/makeDataCount/:persistentId/updateCitationsForDataset?persistentId=$DOI that could end up in infinite loop if Datacite result is paginated.

While using Datacite API pagination by page number + using next link : it will keep sending the same content page after page 10 and next link will exists forever.
Moreover, using this method is limited to a retrieve of 10,000 records (After page 10, all pages display page content of page n°10)

Ex:
First page : https://api.datacite.org/events?doi=10.12763/SMDGR1&source=crossref&page[size]=1000

"meta": {
    "total": 10276,
    "total-pages": 11,

"links": {
    "self": "https://api.datacite.org/events?doi=10.12763/SMDGR1&source=crossref&page[size]=1000",
    "next": "https://api.datacite.org/events?doi=10.12763%2FSMDGR1&page%5Bnumber%5D=2&page%5Bsize%5D=1000"
  }

Page number 11 : https://api.datacite.org/events?doi=10.12763%2FSMDGR1&page%5Bnumber%5D=11&page%5Bsize%5D=1000
Still displays a next link :

"links": {
    "self": "https://api.datacite.org/events?doi=10.12763%2FSMDGR1&page%5Bnumber%5D=11&page%5Bsize%5D=1000",
    "next": "https://api.datacite.org/events?doi=10.12763%2FSMDGR1&page%5Bnumber%5D=11&page%5Bsize%5D=1000"
  }

This code is now using Datacite API pagination by Cursor that will remove next link at last page, fixing the infinite loop issue while increasing the search capability to its maximum.

@pdurbin pdurbin moved this to Ready for Triage in IQSS Dataverse Project May 28, 2025
@jeromeroucou jeromeroucou moved this to 🚧 Dev by Recherche Data Gouv in Recherche Data Gouv Jun 2, 2025
@ofahimIQSS ofahimIQSS added the Size: 3 A percentage of a sprint. 2.1 hours. label Jun 3, 2025
@ofahimIQSS ofahimIQSS moved this from Ready for Triage to Ready for Review ⏩ in IQSS Dataverse Project Jun 3, 2025
@cmbz cmbz added FY25 Sprint 24 FY25 Sprint 24 (2025-05-21 - 2025-06-04) FY25 Sprint 25 FY25 Sprint 25 (2025-06-04 - 2025-06-18) labels Jun 4, 2025
@stevenwinship stevenwinship self-assigned this Jun 16, 2025
@stevenwinship stevenwinship moved this from Ready for Review ⏩ to In Review 🔎 in IQSS Dataverse Project Jun 16, 2025
@github-project-automation github-project-automation bot moved this from In Review 🔎 to Ready for QA ⏩ in IQSS Dataverse Project Jun 16, 2025
@stevenwinship stevenwinship removed their assignment Jun 16, 2025
@cmbz cmbz added the FY25 Sprint 26 FY25 Sprint 26 (2025-06-18 - 2025-07-02) label Jun 19, 2025
@cmbz cmbz added the FY26 Sprint 1 FY26 Sprint 1 (2025-07-02 - 2025-07-16) label Jul 2, 2025
@DS-INRAE DS-INRAE moved this from 🚧 Dev by Recherche Data Gouv to 🙏 Wanted for next version in Recherche Data Gouv Jul 16, 2025
@pdurbin pdurbin added this to the 6.8 milestone Jul 16, 2025
@cmbz cmbz added the FY26 Sprint 2 FY26 Sprint 2 (2025-07-16 - 2025-07-30) label Jul 17, 2025
@ofahimIQSS ofahimIQSS self-assigned this Jul 22, 2025
@ofahimIQSS ofahimIQSS moved this from Ready for QA ⏩ to QA ✅ in IQSS Dataverse Project Jul 22, 2025
@ofahimIQSS
Copy link
Contributor

looks good - merging

@ofahimIQSS ofahimIQSS merged commit c856b07 into IQSS:develop Jul 23, 2025
17 checks passed
@github-project-automation github-project-automation bot moved this from QA ✅ to Merged 🚀 in IQSS Dataverse Project Jul 23, 2025
@github-project-automation github-project-automation bot moved this from 🙏 Wanted for next version to Done in Recherche Data Gouv Jul 23, 2025
@ofahimIQSS ofahimIQSS removed their assignment Jul 23, 2025
@DS-INRAE DS-INRAE moved this from Done to 🚀 Done (Recherche Data Gouv) in Recherche Data Gouv Jul 24, 2025
@scolapasta scolapasta moved this from Merged 🚀 to Done 🧹 in IQSS Dataverse Project Jul 24, 2025
@cmbz cmbz added the FY26 Sprint 4 FY26 Sprint 4 (2025-08-13 - 2025-08-27) label Aug 16, 2025
luddaniel added a commit to Recherche-Data-Gouv/dataverse that referenced this pull request Dec 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

FY25 Sprint 24 FY25 Sprint 24 (2025-05-21 - 2025-06-04) FY25 Sprint 25 FY25 Sprint 25 (2025-06-04 - 2025-06-18) FY25 Sprint 26 FY25 Sprint 26 (2025-06-18 - 2025-07-02) FY26 Sprint 1 FY26 Sprint 1 (2025-07-02 - 2025-07-16) FY26 Sprint 2 FY26 Sprint 2 (2025-07-16 - 2025-07-30) FY26 Sprint 4 FY26 Sprint 4 (2025-08-13 - 2025-08-27) Size: 3 A percentage of a sprint. 2.1 hours.

Projects

Status: Done 🧹
Status: 🚀 Done (Recherche Data Gouv)

Development

Successfully merging this pull request may close these issues.

5 participants