Skip to content

Commit 765df8d

Browse files
committed
Add v5 datasheets and semantic evaluations to documentation
- Add new 'Latest: v5 Datasheets and Semantic Evaluations' section - Include links to all 8 v5 HTML files (4 datasheets + 4 evaluations) - Document rubric20-semantic evaluation framework and results - Show evaluation scores: VOICE (96.4%), AI-READI (94.0%), CM4AI (91.7%), CHORUS (84.5%) - Add comprehensive About section explaining v5 generation and evaluation process This enables GitHub Pages deployment of v5 HTML files.
1 parent 4fe8f30 commit 765df8d

File tree

1 file changed

+63
-0
lines changed

1 file changed

+63
-0
lines changed

docs/d4d_examples.md

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,30 @@ These datasheets were automatically synthesized using Claude Sonnet 4.5 with **d
7070
- [Download YAML](yaml_output/concatenated/claudecode/VOICE_d4d.yaml)
7171
- [Download Metadata](yaml_output/concatenated/claudecode/VOICE_d4d_metadata.yaml)
7272

73+
## Latest: v5 Datasheets and Semantic Evaluations (December 2025)
74+
75+
**NEW**: These are the latest datasheets (v5) with comprehensive semantic evaluation results.
76+
77+
### AI-READI v5
78+
- [D4D Datasheet (Human-Readable HTML)](html_output/D4D_-_AI-READI_v5_human_readable.html) - Comprehensive project metadata
79+
- [Semantic Evaluation Report](html_output/D4D_-_AI-READI_v5_evaluation.html) - Rubric20 assessment (79/84, 94.0%)
80+
81+
### CHORUS v5
82+
- [D4D Datasheet (Human-Readable HTML)](html_output/D4D_-_CHORUS_v5_human_readable.html) - Comprehensive project metadata
83+
- [Semantic Evaluation Report](html_output/D4D_-_CHORUS_v5_evaluation.html) - Rubric20 assessment (71/84, 84.5%)
84+
85+
### CM4AI v5
86+
- [D4D Datasheet (Human-Readable HTML)](html_output/D4D_-_CM4AI_v5_human_readable.html) - Comprehensive project metadata
87+
- [Semantic Evaluation Report](html_output/D4D_-_CM4AI_v5_evaluation.html) - Rubric20 assessment (77/84, 91.7%)
88+
89+
### VOICE v5
90+
- [D4D Datasheet (Human-Readable HTML)](html_output/D4D_-_VOICE_v5_human_readable.html) - Comprehensive project metadata
91+
- [Semantic Evaluation Report](html_output/D4D_-_VOICE_v5_evaluation.html) - Rubric20 assessment (81/84, 96.4%)
92+
93+
**About v5 Datasheets**: Generated using Claude Sonnet 4.5 with deterministic settings (temperature=0.0, model: claude-sonnet-4-5-20250929) on December 20, 2025. Each datasheet has been evaluated using the Rubric20-Semantic framework, which assesses 20 questions across 4 categories (Structural Completeness, Metadata Quality, Technical Documentation, FAIRness & Accessibility) with semantic validation of correctness and consistency.
94+
95+
**Average Score**: 77.0/84 (91.7%) across all 4 projects.
96+
7397
## Individual Dataset Datasheets
7498

7599
These datasheets were created from specific dataset metadata sources:
@@ -134,6 +158,45 @@ See [DETERMINISM.md](https://github.com/bridge2ai/data-sheets-schema/blob/main/D
134158
### Individual Dataset Datasheets
135159
The **Individual Dataset Datasheets** provide detailed metadata for specific datasets from each project's primary data repository (FAIRHub, Dataverse, PhysioNet). These focus on individual dataset instances rather than project-level metadata.
136160

161+
### v5 Datasheets and Semantic Evaluations
162+
The **v5 Datasheets** represent the latest generation (December 2025) of comprehensive project metadata created using Claude Sonnet 4.5 with fully deterministic settings:
163+
164+
**Generation Process:**
165+
1. Multiple project-related documents concatenated in reproducible order
166+
2. AI-powered extraction and synthesis using Claude Sonnet 4.5
167+
3. Temperature=0.0 for deterministic output
168+
4. Pinned model version (claude-sonnet-4-5-20250929) for reproducibility
169+
5. Validation against the LinkML D4D schema
170+
171+
**Semantic Evaluation Framework:**
172+
Each v5 datasheet has been evaluated using the **Rubric20-Semantic** framework, which provides:
173+
174+
- **20 Questions** across 4 categories:
175+
1. Structural Completeness (max 24 points) - Schema field population and required elements
176+
2. Metadata Quality (max 22 points) - Accuracy, specificity, and completeness of information
177+
3. Technical Documentation (max 25 points) - Collection, preprocessing, and reproducibility details
178+
4. FAIRness & Accessibility (max 13 points) - Findability, Accessibility, Interoperability, Reusability
179+
180+
- **Semantic Analysis:**
181+
- Correctness validation (DOI formats, grant IDs, RRID identifiers)
182+
- Consistency checking (cross-field validation, logical coherence)
183+
- Content accuracy assessment
184+
- Evidence-based scoring with detailed justifications
185+
186+
**Evaluation Results:**
187+
- **VOICE**: 81/84 points (96.4%) - Highest scoring project
188+
- **AI-READI**: 79/84 points (94.0%)
189+
- **CM4AI**: 77/84 points (91.7%)
190+
- **CHORUS**: 71/84 points (84.5%)
191+
- **Average**: 77.0/84 points (91.7%)
192+
193+
Each evaluation report provides:
194+
- Category-level score breakdown
195+
- Question-by-question assessment with evidence
196+
- Semantic analysis findings (correctness and consistency checks)
197+
- Strengths, weaknesses, and recommendations
198+
- Detailed justifications for all scoring decisions
199+
137200
## Schema Information
138201

139202
All datasheets conform to the [Datasheets for Datasets](https://arxiv.org/abs/1803.09010) framework by Gebru et al., implemented using the [Bridge2AI LinkML schema](https://github.com/bridge2ai/data-sheets-schema).

0 commit comments

Comments
 (0)