Skip to content

Latest commit

 

History

History
372 lines (269 loc) · 17.1 KB

WORKFLOW.md

File metadata and controls

372 lines (269 loc) · 17.1 KB

REMARK Ecosystem Workflow

This document provides a comprehensive overview of how the REMARK ecosystem works, including the interactions between the REMARK repository, individual research repositories, and the econ-ark.org website.

System Architecture Overview

⚠️ CRITICAL DISTINCTION: The REMARK ecosystem has TWO SEPARATE SYSTEMS that serve different purposes:

  1. Website Generation System (populate_remarks.py) - Generates econ-ark.org content
  2. REMARK Validation System (cli.py) - Validates research reproducibility standards

These are INDEPENDENT systems with different requirements!

The REMARK ecosystem consists of three main components:

┌─────────────────────┐    ┌─────────────────────┐    ┌─────────────────────┐
│   REMARK Repo       │    │  Individual Repos   │    │  econ-ark.org       │
│  (Catalog/Standards) │    │  (Research Projects) │    │  (Public Website)   │
│                     │    │                     │    │                     │
│ • REMARKs/*.yml     │◄──►│ • CITATION.cff      │───►│ • _materials/*.md   │
│ • STANDARD.md       │    │ • REMARK.md         │    │ • Jekyll templates  │
│ • Validation tools  │    │ • reproduce.sh      │    │ • Search/filter UI  │
│ • CLI tools         │    │ • binder/env.yml    │    │ • Material pages    │
└─────────────────────┘    └─────────────────────┘    └─────────────────────┘

Detailed Workflow

1. REMARK Repository Structure

The REMARK repository serves as the catalog and standards authority:

REMARK/
├── REMARKs/                    # Catalog of all REMARKs
│   ├── BufferStockTheory.yml   # Minimal metadata per REMARK
│   ├── beyond-the-streetlight.yml
│   └── ...
├── STANDARD.md                 # Requirements for REMARK compliance
├── cli.py                      # Tools for validation and testing
├── .github/workflows/          # Automation workflows
│   └── transfer-remark-metadata.yml
└── Documentation files

REMARK Catalog Files (REMARKs/*.yml)

Each REMARK has a minimal YAML file containing:

name: project-name              # Short identifier
remote: https://github.com/...  # Repository URL  
title: Human Readable Title     # Display name

Critical Point: The REMARK repository does NOT contain the full metadata - only the minimal catalog entry pointing to the actual research repository.

2. Individual Research Repositories

Each research project is a self-contained repository that must meet REMARK standards:

Required Files

  • CITATION.cff: Complete bibliographic metadata (CFF format)
  • REMARK.md: Website-specific metadata + abstract content
  • reproduce.sh: Script to reproduce all results
  • binder/environment.yml: Environment specification

Optional Files

  • reproduce_min.sh: Quick demonstration version

Example REMARK.md Structure

---
# Website-specific metadata (YAML frontmatter)
remark-name: beyond-the-streetlight
title-original-paper: "100 years of Economic Measurement..."
notebooks:
  - RS100_Discussion_Slides.ipynb
tags:
  - REMARK
  - Notebook
keywords:
  - forecast accuracy
  - Federal Reserve
---

# Abstract

This repository provides analysis of...

3. Website Generation Process

🌐 WEBSITE GENERATION SYSTEM (Primary: populate_remarks.py)

The econ-ark.org website is generated through an automated pipeline that is SEPARATE from the REMARK validation system:

Step 1: GitHub Workflows

Two workflows coordinate the integration:

A. REMARK Repo → Website Repo (.github/workflows/transfer-remark-metadata.yml)

  • Runs daily at 8:00 AM UTC
  • Copies any existing REMARKs/*.md files to econ-ark.org/_materials/
  • Important: This is a SECONDARY mechanism for edge cases where manual .md files exist
  • Not the primary workflow - most REMARKs only have .yml catalog entries

B. Website Preprocessing (.github/workflows/site-preprocess.yml) - PRIMARY MECHANISM

  • Runs on every push to master
  • Executes scripts/populate_remarks.py (the core integration script)
  • This is what actually builds the website content for most REMARKs

Step 2: populate_remarks.py Script

This is the core integration script that:

  1. Clones REMARK catalog: Gets the current list of all REMARKs
  2. Reads catalog entries: Extracts repository URLs from REMARKs/*.yml files
  3. Clones individual repositories: Downloads each research project (using --sparse clone)
  4. Merges metadata: Combines data from two key source files:
    • CITATION.cff (bibliographic metadata)
    • REMARK.md (website-specific fields + abstract/body content)
  5. Generates material files: Creates _materials/{name}.md for Jekyll

🚨 IMPORTANT: This script only requires CITATION.cff to generate a basic webpage. For a rich, descriptive page, REMARK.md is essential. The script specifically looks for these two file names and ignores other markdown files (e.g., README.md or legacy {name}.md files) for website content generation.

Step 3: Jekyll Site Generation

  • Jekyll processes _materials/*.md files into web pages
  • Templates in _layouts/ control rendering
  • Collections system enables filtering and search

Data Flow Diagram

┌─────────────────────┐
│ Author submits PR   │
│ to REMARK repo      │
│ (adds .yml file)    │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    REMARK Repository                                │
│                                                                     │
│  REMARKs/new-project.yml ◄─── PR Review & Merge                   │
│  ┌─────────────────────┐                                           │
│  │ name: new-project   │                                           │
│  │ remote: github.com/ │                                           │
│  │ title: Project Name │                                           │
│  └─────────────────────┘                                           │
└─────────────────────────────────────────────────────────────────────┘
           │
           ▼ (Daily/Push triggers)
┌─────────────────────────────────────────────────────────────────────┐
│                populate_remarks.py Script                          │
│                                                                     │
│  1. Clone REMARK repo ──► Get catalog                              │
│  2. For each entry:                                                 │
│     ├─ Clone individual repo                                        │
│     ├─ Read CITATION.cff                                           │
│     ├─ Read REMARK.md                                              │
│     └─ Merge metadata                                               │
│  3. Generate _materials/{name}.md                                   │
└─────────────────────────────────────────────────────────────────────┘
           │
           ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     econ-ark.org Website                           │
│                                                                     │
│  _materials/                                                        │
│  ├─ new-project.md  ◄─── Generated from merged metadata            │
│  │   ┌─────────────────────────────────────────────────────────┐   │
│  │   │ ---                                                     │   │
│  │   │ # From CITATION.cff                                     │   │
│  │   │ authors: [...]                                          │   │
│  │   │ title: Project Name                                     │   │
│  │   │ version: 1.0.0                                          │   │
│  │   │ # From REMARK.md frontmatter                            │   │
│  │   │ remark-name: new-project                                │   │
│  │   │ notebooks: [...]                                        │   │
│  │   │ tags: [REMARK, ...]                                     │   │
│  │   │ ---                                                     │   │
│  │   │                                                         │   │
│  │   │ # From REMARK.md body                                   │   │
│  │   │ # Abstract                                              │   │
│  │   │ This repository provides...                             │   │
│  │   └─────────────────────────────────────────────────────────┘   │
│  │                                                                 │
│  └─ Jekyll processes → /materials/new-project/ webpage             │
└─────────────────────────────────────────────────────────────────────┘

Key Integration Points

Metadata Merging Logic

The populate_remarks.py script combines metadata with this priority:

  1. Base data: CITATION.cff provides bibliographic information
  2. Website overlay: REMARK.md frontmatter adds website-specific fields
  3. Content: REMARK.md body becomes the webpage content

File Naming Convention

  • REMARK catalog: REMARKs/{name}.yml
  • Individual repo: {name}/REMARK.md and {name}/CITATION.cff
  • Website material: _materials/{name}.md
  • Final URL: econ-ark.org/materials/{name}/

Error Handling

  • Missing CITATION.cff: Project skipped (no webpage generated)
  • Missing REMARK.md: Uses only CITATION.cff data
  • Invalid YAML: Build fails with error

Submission Process

For New REMARKs

  1. Prepare repository meeting REMARK standards
  2. Submit PR to REMARK repo adding REMARKs/{name}.yml
  3. Editorial review checks compliance
  4. Merge PR adds to catalog
  5. Automated integration generates website content

For Updates

  1. Update individual repository (tag new release)
  2. Website auto-updates within 24 hours via scheduled workflow
  3. Manual trigger available via GitHub Actions

Common Issues and Solutions

Issue: New REMARK not appearing on website

Diagnosis: Check if CITATION.cff exists and is valid YAML Solution: Ensure all required files are present and properly formatted

Issue: Metadata not updating

Diagnosis: Check GitHub Actions logs for populate_remarks.py Solution: Verify individual repository is publicly accessible

Issue: Website build failure

Diagnosis: YAML parsing error in metadata files Solution: Validate YAML syntax in CITATION.cff and REMARK.md

Issue: GitHub workflow appears "broken" (looking for .md files when only .yml exist)

Diagnosis: Misunderstanding the dual workflow system Solution: Remember that transfer-remark-metadata.yml is SECONDARY - the primary workflow is populate_remarks.py

Critical Understanding Points for AI Systems

⚠️ Warning for AI Analysis: The REMARK ecosystem uses a dual workflow system:

  1. Primary: populate_remarks.py script that reads .yml catalog files and generates content
  2. Secondary: transfer-remark-metadata.yml workflow for edge cases with manual .md files

Do not assume the transfer workflow is misconfigured because it looks for .md files in a directory containing .yml files. This is by design - the two mechanisms serve different purposes.

Maintenance and Monitoring

Automated Processes

  • Daily metadata sync (8:00 AM UTC)
  • Website rebuild on every push
  • Link validation (via GitHub Actions)

Manual Oversight

  • Editorial review of new submissions
  • Quality assurance testing
  • Compliance checking via CLI tools

This workflow ensures that the REMARK ecosystem maintains high standards for reproducibility while providing a seamless integration between distributed research repositories and the centralized discovery platform at econ-ark.org.

🔧 REMARK Validation System vs 🌐 Website Generation System

Critical Distinction

These are TWO COMPLETELY SEPARATE SYSTEMS with different purposes and requirements:

Aspect 🌐 Website Generation (populate_remarks.py) 🔧 REMARK Validation (cli.py)
Purpose Generate econ-ark.org website content Validate research reproducibility
Trigger Automatic (daily/push) Manual (editor workflow)
Required Files CITATION.cff (required), REMARK.md (optional) reproduce.sh, CITATION.cff, binder/environment.yml
Clone Method git clone --sparse (metadata only) git clone --depth 1 (full repo)
Output _materials/*.md files for Jekyll Validation reports and logs
Failure Impact Missing materials on website Cannot reproduce research

🌐 Website Generation Requirements

Minimum for website appearance:

  • ✅ CITATION.cff - Provides author, title, abstract, etc.
  • ✅ Valid repository URL in REMARKs/*.yml

Enhanced website features:

  • ✅ REMARK.md - Adds website-specific metadata (notebooks, tags, custom content)

NOT required for website:

  • ❌ reproduce.sh
  • ❌ binder/environment.yml
  • ❌ reproduce_min.sh

🔧 REMARK Validation Requirements

Required for REMARK compliance (reproducibility):

  • ✅ reproduce.sh - Must run and reproduce all results
  • ✅ CITATION.cff - Bibliographic metadata
  • ✅ binder/environment.yml - Environment specification
  • ✅ Tagged release

Optional for REMARK compliance:

  • ✅ reproduce_min.sh - Quick demonstration version

Common Confusion Points

❌ WRONG: "If cli.py lint shows missing files, the website won't work" ✅ CORRECT: Website generation and REMARK validation are independent

❌ WRONG: "All materials on the website are REMARK compliant" ✅ CORRECT: Materials can appear on website without meeting full REMARK standards

❌ WRONG: "Website generation requires reproducibility files" ✅ CORRECT: Website only needs bibliographic metadata (CITATION.cff)

Workflow Integration

Research Repository
├── CITATION.cff           ────► Website Generation (populate_remarks.py)
├── REMARK.md (optional)   ────► Website Enhancement
├── reproduce.sh           ────► REMARK Validation (cli.py)
├── binder/environment.yml ────► REMARK Validation (cli.py)
└── other research files   ────► Research Content

Troubleshooting Guide

Issue: Material missing from website Check: Does repository have CITATION.cff? Is URL correct in REMARKs/*.yml?

Issue: REMARK validation fails Check: Does repository have reproduce.sh, binder/environment.yml? Does reproduce.sh run successfully?

Issue: Website shows incorrect content Check: Is CITATION.cff correct? Does REMARK.md have proper frontmatter?

AI System Warning

🤖 FOR AI SYSTEMS: Do not confuse these two systems. When analyzing REMARK compliance issues:

  1. First determine: Are you looking at website generation issues or reproducibility validation issues?
  2. Website problems: Check populate_remarks.py logs, CITATION.cff files, _materials/ generation
  3. Validation problems: Check cli.py output, reproduce.sh scripts, environment files
  4. Remember: A repository can appear on the website without being fully REMARK compliant for reproducibility