Epic: Push System Core Framework

[deepakduggirala]

Implement the base infrastructure for push support including abstractions, metadata models, and Celery task plumbing.

Tasks:

• push-core-define-strategy-pattern

  • Implement PushStrategy abstract base class
  • Define push() interface

• push-core-implement-hpfs-strategy

  • Implement HPFSPush subclass
  • Support atomic copy or rsync to target HPFS directory

• push-core-integrate-push-celery

  • Add push_dataset() Celery task
  • Wire push stage into workflow pipeline

• push-core-idempotency-handling

  • Flag file (HPFS), object existence check (cloud), checksum/sync-level (Globus)

• push-core-database-schema-extension

  • Add push_job and push_workflow Prisma models
  • Migrate DB and update ORM access

Implement robust error handling and retry support.

• push-error-logging

  • Centralized error capture and logging

• push-task-failure-marking

  • Update push job record with failure status and cause

• push-operator-retry-ui

  • Admin UI to re-run failed pushes from dashboard