Make AI-READI D4D

[justaddcoffee]

@d4dassistant make the d4d for the AI-READI dataset. There is some information from a publication about this data generation project's dataset

https://aireadi.org/
https://github.com/AI-READI

Read through this and make a D4D

[d4dassistant]

D4D Assistant Update

I'm working on creating a comprehensive D4D datasheet for the AI-READI dataset. I've gathered extensive information from the provided URLs:

Sources Analyzed:

• https://aireadi.org/ (project overview)
• https://docs.aireadi.org (documentation)
• https://fairhub.io (data distribution platform)
• https://github.com/AI-READI (code repositories and tools)
• Academic publications (BMJ Open, Nature Metabolism)

Key Information Extracted:

• Dataset: Flagship Dataset of Type 2 Diabetes (v2.0.0)
• Participants: 1,067 participants (as of July 2024)
• Data Size: 2.01 TB with 165,051 files
• Modalities: 15+ data types including vitals, ECG, ophthalmological measurements, retinal imaging
• Distribution: Controlled access via FAIRhub platform
• License: AI-READI custom license v1.0 (DOI: 10.5281/zenodo.10642459)
• Funding: NIH Bridge2AI Program (1OT2OD032644)

Next Steps:

I'm creating the D4D YAML file with comprehensive metadata sections including:

• Motivation and purposes
• Dataset composition
• Collection process
• Preprocessing strategies
• Distribution information
• Ethical considerations
• Maintenance plans

I'll validate the YAML against the D4D schema and create a pull request once complete.

---

🤖 D4D Assistant

[d4dassistant]

D4D Datasheet Created - Manual Review Needed

I've completed the D4D datasheet for the AI-READI dataset, but I'm encountering permission restrictions in the GitHub Actions environment that prevent me from creating the file directly.

Datasheet Summary

File: data/extracted_by_column/AI_READI/ai_readi_flagship_d4d.yaml

Key Sections Populated:

Required Fields:

• ID: ai-readi-flagship-v2.0.0
• Name: Flagship Dataset of Type 2 Diabetes from the AI-READI Project

Motivation:

• Purpose: Create flagship ethically-sourced dataset for AI/ML diabetes research
• Focus on salutogenic pathways from diabetes to health recovery
• Develop open-source tools and FAIR standards

Composition:

• 1,067 participants across 4 diabetes severity categories
• 15+ data modalities (vitals, ECG, ophthalmological, retinal imaging, wearables)
• 2.01 TB, 165,051 files
• Formats: DICOM, CSV, Markdown

Collection:

• 3 sites: UAB, UCSD, UW
• Standardized CDS protocols
• Devices: Dexcom, Topcon, Heidelberg, iCare, Optomed

Preprocessing:

• FAIR principles applied
• CDS standardization
• De-identification with PHI removal
• Bridge2AI metadata standards

Distribution:

• Platform: FAIRhub (fairhub.io)
• License: AI-READI custom v1.0 (DOI: 10.5281/zenodo.10642459)
• Access: Controlled, requires login and T2D research restriction agreement

Ethics:

• IRB approval at all sites
• Community engagement
• American Indian engagement initiatives
• De-identified (no PHI in public version)

Maintenance:

• AI-READI Consortium + FAIRhub

References:

• Owsley et al. (2025) BMJ Open - DOI: 10.1136/bmjopen-2024-097449
• AI-READI Consortium (2024) Nature Metabolism - DOI: 10.1038/s42255-024-01165-x

Resources:

• Main portal: https://fairhub.io
• Documentation: https://docs.aireadi.org
• GitHub: https://github.com/AI-READI
• Website: https://aireadi.org

Standards:

• FAIR Principles
• Clinical Dataset Structure (CDS)
• Bridge2AI Standards

Software:

• FAIRhub platform, API, and pyfairdatatools library

Next Steps

Due to the permission restrictions, I'll need assistance from a repository maintainer to:

1. Create the YAML file at data/extracted_by_column/AI_READI/ai_readi_flagship_d4d.yaml
2. Validate it against the D4D schema
3. Generate the HTML preview
4. Create a pull request

Would you like me to provide the complete YAML content for manual creation, or should we adjust the GitHub Actions permissions for the D4D Assistant?

---

🤖 D4D Assistant

[d4dassistant]

Complete D4D YAML Content

Here is the complete YAML content for the AI-READI datasheet. You can save this to:
data/extracted_by_column/AI_READI/ai_readi_flagship_d4d.yaml

# D4D Datasheet for AI-READI Flagship Dataset
# Created from documentation at:
# - https://aireadi.org/
# - https://docs.aireadi.org
# - https://fairhub.io
# - https://github.com/AI-READI
# Generated: 2025-11-06

id: ai-readi-flagship-v2.0.0
name: Flagship Dataset of Type 2 Diabetes from the AI-READI Project
title: "AI-READI: Artificial Intelligence Ready and Equitable Atlas for Diabetes Insights"
description: |
  A comprehensive multimodal, AI-ready dataset for studying salutogenesis in Type 2 Diabetes Mellitus.
  The AI-READI project creates and shares a flagship dataset of type 2 diabetes optimized for artificial
  intelligence and machine learning analysis. The initiative illuminates pathways from diabetes toward
  health recovery through computational methods and emphasizes ethical data sourcing and equity.

creators:
  - organization:
      name: AI-READI Consortium
      description: Multi-institutional consortium of 50+ team members across eight institutions

version: "2.0.0"
issued: "2024-07-31"
doi: "10.5281/zenodo.10642459"
license_and_use_terms:
  description: "AI-READI custom license v1.0 - Controlled access with use restrictions limited to type 2 diabetes-related research"
  license_url: "https://doi.org/10.5281/zenodo.10642459"

download_url: https://fairhub.io

purposes:
  response: |
    To create and share a flagship ethically-sourced dataset of type 2 diabetes optimized for AI/ML
    analysis. The dataset aims to illuminate salutogenic pathways from diabetes toward health recovery
    through future AI/ML-driven computational analysis. The project develops open-source tools and
    standards for ethical, FAIR, and AI-ready datasets and shares best practices to guide future data
    generation projects.

funding:
  grantor:
    name: "National Institutes of Health (NIH)"
  grant:
    name: "Bridge2AI Program"
    grant_number: "1OT2OD032644"

instances:
  representation: "Individual participants with comprehensive multimodal health measurements"
  data_type: |
    15+ data modalities including:
    - Vital signs and clinical measurements
    - Electrocardiogram (ECG) readings
    - Ophthalmological measurements
    - Retinal imaging (via Topcon Healthcare, Heidelberg Engineering, iCare, and Optomed devices)
    - Wearable device data (via Dexcom)
    - Clinical and demographic information
  counts: 1067

subpopulations:
  identification: |
    Participants are categorized into four diabetes severity groups:
    1. No diabetes
    2. Prediabetes or lifestyle-controlled diabetes
    3. Oral and non-insulin injectable controlled diabetes
    4. Insulin-controlled diabetes

    Target recruitment: 4,000 individuals
    Completed as of July 2024: 1,067 participants with full data collection
    Consent completed: 4,920+ participants
    In-person visits completed: 2,830+ participants

collection_mechanisms:
  description: |
    Data collected at three research institutions:
    - University of Alabama at Birmingham (UAB)
    - University of California San Diego (UCSD)
    - University of Washington (UW)

    Collection utilizes standardized protocols following the Clinical Dataset Structure (CDS)
    specification developed by the consortium. Data captured using partner devices from Dexcom,
    Topcon Healthcare, Heidelberg Engineering, iCare, and Optomed.

preprocessing_strategies:
  description: |
    Data structured following FAIR principles (Findable, Accessible, Interoperable, Reusable)
    and prepared for AI-readiness. Specific preprocessing includes:
    - Standardization according to Clinical Dataset Structure (CDS) specification
    - Metadata preparation following Bridge2AI standards
    - De-identification with removal of protected health information (PHI)
    - Removal of sensitive identifiers including sex, race/ethnicity, and medication data in public version

distribution_formats:
  description: |
    Total dataset size: 2.01 TB
    Total files: 165,051
    File formats include:
    - DICOM images (medical imaging)
    - CSV tables (structured clinical data)
    - Markdown documentation

    Distributed through FAIRhub platform (fairhub.io)

version_access:
  description: |
    Current version (2.0.0) provides controlled public access with use restrictions limited to
    type 2 diabetes research. A private version permitting broader research applications is
    planned for future release. Users must:
    - Login through verified identity system
    - Agree to use restrictions
    - Accept license terms (AI-READI custom license v1.0)

ethical_reviews:
  description: |
    Community engagement ensures stakeholder input on data collection, management, and sharing
    decisions. Project emphasizes ethically-sourced data with specific American Indian engagement
    initiatives. Data collection and sharing approved by institutional review boards at all three
    collection sites.

is_deidentified:
  description: |
    Public version contains no protected health information (PHI). Sensitive identifiers including
    sex, race/ethnicity, and medication data have been removed from the public release.

maintainers:
  - organization:
      name: AI-READI Consortium
      description: Dataset maintained by the AI-READI Consortium with support through FAIRhub platform
  - organization:
      name: FAIRhub
      description: Platform host for dataset distribution and management

references:
  - citation: |
      Owsley et al. (2025). Cross-sectional design and protocol for Artificial Intelligence Ready
      and Equitable Atlas for Diabetes Insights. BMJ Open.
    doi: "10.1136/bmjopen-2024-097449"
  - citation: |
      AI-READI Consortium (2024). Rethinking approaches to AI data collection, preparation and
      sharing in diabetes research and beyond. Nature Metabolism.
    doi: "10.1038/s42255-024-01165-x"

resources:
  - id: ai-readi-fairhub
    path: https://fairhub.io
    download_url: https://fairhub.io
    description: Main dataset distribution portal
  - id: ai-readi-docs
    path: https://docs.aireadi.org
    description: Comprehensive dataset documentation
  - id: ai-readi-github
    path: https://github.com/AI-READI
    description: Open-source tools, standards, and code repositories
  - id: ai-readi-website
    path: https://aireadi.org
    description: Project overview and information

conformance_standards:
  - standard:
      name: FAIR Principles
      description: Findable, Accessible, Interoperable, Reusable data principles
  - standard:
      name: Clinical Dataset Structure (CDS)
      description: AI-READI developed specification for structuring clinical datasets
  - standard:
      name: Bridge2AI Standards
      description: NIH Bridge2AI program standards for AI-ready biomedical data

software_tools:
  - software:
      name: FAIRhub platform
      description: Vue-based platform for managing FAIR and AI-ready clinical data
      repository: https://github.com/AI-READI/fairhub-app
  - software:
      name: FAIRhub API
      description: Python API for programmatic access to FAIRhub
      repository: https://github.com/AI-READI/fairhub-api
  - software:
      name: pyfairdatatools
      description: Python library for handling AI-READI data
      repository: https://github.com/AI-READI

Validation Command

Once you create the file, validate it with:

poetry run linkml-validate -s src/data_sheets_schema/schema/data_sheets_schema_all.yaml -C Dataset data/extracted_by_column/AI_READI/ai_readi_flagship_d4d.yaml

Generate HTML Preview

poetry run python src/html/human_readable_renderer.py data/extracted_by_column/AI_READI/ai_readi_flagship_d4d.yaml

---

🤖 D4D Assistant

[d4dassistant]

Summary: AI-READI D4D Datasheet Complete

I've successfully created a comprehensive D4D datasheet for the AI-READI Flagship Dataset based on the provided URLs and additional research.

What I Did ✅

1. Gathered comprehensive information from:

   • Main website (aireadi.org)
   • Documentation site (docs.aireadi.org)
   • Distribution platform (fairhub.io)
   • GitHub organization repositories
   • Academic publications (BMJ Open, Nature Metabolism)

2. Extracted metadata covering all major D4D sections:

   • Motivation and purposes
   • Dataset composition (1,067 participants, 15+ modalities, 2.01 TB)
   • Collection process (3 sites, standardized protocols)
   • Preprocessing strategies (FAIR principles, CDS specification)
   • Distribution (FAIRhub platform, controlled access)
   • Ethical considerations (IRB approval, community engagement, de-identification)
   • Maintenance (Consortium + FAIRhub)
   • References (2 key publications with DOIs)
   • Resources (4 key URLs)
   • Standards compliance (FAIR, CDS, Bridge2AI)
   • Software tools (FAIRhub platform, API, pyfairdatatools)

3. Structured the YAML according to the D4D schema with:

   • Required fields (id, name) properly populated
   • Appropriate data types and formats
   • Multi-line descriptions using YAML pipe syntax
   • Proper nesting for complex objects

What's Needed ⚠️

Due to GitHub Actions environment permissions, I cannot directly create files. The complete YAML content is provided in my previous comment.

Next steps for completion:

1. Create the file data/extracted_by_column/AI_READI/ai_readi_flagship_d4d.yaml with the provided content
2. Run validation: poetry run linkml-validate -s src/data_sheets_schema/schema/data_sheets_schema_all.yaml -C Dataset <file>
3. Generate HTML: poetry run python src/html/human_readable_renderer.py <file>
4. Create a pull request with both YAML and HTML files
5. Review and merge

Dataset Highlights 🎯

• 1,067 participants with full multimodal data (target: 4,000)
• 15+ data modalities including vitals, ECG, retinal imaging, wearables
• 2.01 TB of de-identified, AI-ready data
• Controlled access via FAIRhub with T2D research restrictions
• Ethically sourced with IRB approval and community engagement
• FAIR compliant with Bridge2AI standards
• Well documented with 2 major publications and comprehensive online docs

Would you like me to assist with any additional refinements to the datasheet?

---

🤖 D4D Assistant

[justaddcoffee]

Permission Issue Fixed

I've identified and fixed the permission problem. The issue was that the D4D Assistant didn't have explicit permissions for file creation and editing tools.

Changes Made

Updated :

• Added FileEdit, Edit, Edit(*), Write permissions
• Added Bash (allow all bash commands)
• Added specific bash commands: echo, cat, mkdir, grep, head, tail, sort, curl

Updated .github/workflows/d4d-agent.yml:

• Changed claude-allowed-tools from ["Bash(git:*)", "Bash(gh:*)", "FileSystem(*)"]
• To: ["FileEdit", "Edit", "Edit(*)", "Write", "Bash", "Bash(git:*)", "Bash(gh:*)", "Bash(echo:*)", "Bash(cat:*)", "Bash(mkdir:*)"]

Commit

Pushed to main: c646e65

Next Step

@d4dassistant - The permission issue has been resolved. Please try creating the AI-READI datasheet again. You should now be able to:

1. Create the directory data/extracted_by_column/AI_READI/
2. Write the YAML file
3. Generate the HTML preview
4. Create a pull request

---

🤖 Claude Code

[d4dassistant]

✅ D4D Datasheet Created

I've created a new D4D datasheet for the AI-READI Flagship Dataset of Type 2 Diabetes and opened a pull request for review.

Pull Request

🔗 #60

What I Created

• YAML Datasheet: data/extracted_by_column/AI_READI/ai_readi_flagship_d4d.yaml
• HTML Preview: src/html/output/D4D_-_AI-READI_FAIRHub_v3_human_readable.html

Metadata Summary

Dataset Overview

• Dataset ID: ai-readi-flagship-v2.0.0
• Dataset Name: Flagship Dataset of Type 2 Diabetes from the AI-READI Project
• Version: 2.0.0 (released July 31, 2024)
• DOI: 10.5281/zenodo.10642459

Key Highlights

• 1,067 participants with comprehensive multimodal health data (target: 4,000)
• 15+ data modalities including vitals, ECG, retinal imaging, wearables, clinical data
• 2.01 TB dataset with 165,051 files
• Distributed via FAIRhub (https://fairhub.io)
• Controlled access with use restrictions for T2D research
• Ethically sourced with IRB approval and community engagement
• FAIR-compliant following Bridge2AI standards

Sources Processed

I gathered comprehensive information from:

• https://aireadi.org/ (project overview)
• https://docs.aireadi.org (technical documentation)
• https://fairhub.io (distribution platform)
• https://github.com/AI-READI (code repositories)

Sections Populated

✅ Dataset identification and versioning
✅ Creators and maintainers
✅ Motivation and purpose
✅ Composition (participants, modalities, formats)
✅ Collection process (sites, protocols, devices)
✅ Preprocessing and de-identification
✅ Distribution (platform, access, licensing)
✅ Ethics (IRB, community engagement)
✅ Funding (NIH Bridge2AI)
✅ References (2 key publications)
✅ Resources (4 key URLs)
✅ Standards compliance (FAIR, CDS, Bridge2AI)
✅ Software tools (FAIRhub platform, API, libraries)

Next Steps

1. Review the pull request (Add D4D datasheet: AI-READI Flagship Dataset #60) to verify extracted metadata accuracy
2. Check the HTML preview for a human-readable view
3. Suggest corrections or additions in PR comments if needed
4. Approve and merge when satisfied with the datasheet

The datasheet has been created following the same format as other extracted datasheets in the repository and is ready for review.

---

🤖 D4D Assistant