Support parallel processing of multiple consent groups with shared schema

[amc-corey-cox]

Problem Statement

Currently, running the pipeline for multiple consent groups within the same study requires fully separate runs with different DM_SCHEMA_NAME values (e.g., StudyA_c1, StudyA_c2). This duplicates work:

• Data preparation runs multiple times on the same raw files
• Schema generation produces identical schemas for each consent group
• No coordination between parallel runs

Desired Behavior

A single pipeline command that:

1. Prepares data once - Raw dbGaP files are cleaned/standardized once per study
2. Generates schema once - A single schema is created from the prepared data
3. Validates and maps per consent group in parallel - These steps run concurrently for all consent groups, sharing the prepared data and schema

Example Usage

make pipeline DM_SCHEMA_NAME=StudyA DM_CONSENT_GROUPS="c1 c2 c3"

This would:

1. Run prepare-input once → output/StudyA/prepared/
2. Run schema-create once → output/StudyA/StudyA.yaml
3. Run validation + mapping in parallel for c1, c2, c3 → output/StudyA/c1/, output/StudyA/c2/, output/StudyA/c3/

Questions to Resolve

• Where does consent group information come from? (directory structure in raw data, explicit list, config file?)
• How do we handle partial failures? (one consent group fails, others succeed)

Acceptance Criteria

• Single command processes all consent groups for a study
• Data preparation and schema generation run only once
• Validation and mapping run in parallel across consent groups
• Output structure keeps consent groups separate
• Clear documentation on multi-consent workflow