Integrating PubMed Central As a DataSource to Improve Quantifying Creative Commons

[Goziee-git]

Problem

Improving Quantifying Creative commons with Medical and life Science records from PubMed.

Description

This Issue aims to integrate PubMed as a new data source for improving Quantifying Creative Commons project. PubMed provides access to biomedical literature with Creative Commons licensing information, contributing valuable insights into open access scientific publishing trends.

Additional context

API Documentation

• NCBI E-utilities Overview - Complete guide to Entrez Programming Utilities
• ESearch Documentation - Search parameter reference and examples
• EFetch Documentation - Data retrieval specifications and XML schemas
• PubMed Search Field Descriptions - Complete field tag reference
• E-utilities Usage Guidelines - Rate limits, best practices, and compliance requirements

API Limitations and Constraints

Rate Limiting

• Maximum Rate: 3 requests per second (0.34-second intervals)
• Enforcement: Implemented via time.sleep(0.34) between requests
• Retry Strategy: 3 retries with exponential backoff for failed requests

Data Retrieval Limits

• Per Request: Maximum 9,999 records per esearch call
• Batch Size: 200 papers per efetch request for optimal performance
• Total Limit: Configurable via --fetch-limit parameter (default: 5,000)

PubMed Data Source Information

• PubMed Homepage - Main search interface and database statistics
• PubMed Data Distribution - Bulk data access and licensing information
• MEDLINE/PubMed Data Element Descriptions - Complete metadata field specifications
• PubMed Central (PMC) - Full-text article repository with CC license information
• NIH Public Access Policy - Mandate requiring open access for NIH-funded research

Implementation

• I would be interested in implementing this feature.

[TimidRobot]

Before a pull request (PR) is submitted, please document the fetch strategy:

• Which API(s)?
• Which fields?
• Anticipated number of queries and/or expected duration?