Multimodal Spatiotemporal Forecasting of Deepfake Propagation

This repository provides an implementation for multimodal spatiotemporal forecasting of deepfake content propagation.
The model integrates:

• Temporal Graph Transformer to encode early propagation sequences
• FiLM-based perceptual conditioning to incorporate image metadata
• Cross-modal contrastive alignment (InfoNCE) between image and graph embeddings
• Temporal prediction via Contrastive Predictive Coding (CPC)
• Multi-task regression predicting:
  • early-window Share Count
  • propagation Wiener Index

The codebase enables reproducible training and evaluation of multimodal forecasting models for understanding and predicting how deepfake content spreads over time.

---

Multimodal-Spatiotemporal-Forecasting-of-Deepfake-Propagation/
│
├── data/
│ ├── train.csv / val.csv / test.csv
│ ├── repost_info.csv
│ └── npy_vectors/ # pre-extracted 128-d image features
│
├── dataset/
│ ├── build_graphs.py
│ ├── add_features.py
│ ├── deepfake_meta.py
│ ├── dfd_dataset.py
│ └── collate.py
│
├── models/
│ ├── film.py # FiLM perceptual conditioning
│ ├── graph_encoder.py # GraphSAGE encoder
│ ├── transformer.py # Temporal Graph Transformer
│ ├── projection.py # Projection heads + InfoNCE + CPC
│ ├── multitask.py # Multi-task regression heads
│ └── pt_dgcl.py # Full multimodal spatiotemporal model
│
├── train.py
├── eval.py
└── requirements.txt

---

🚀 Training

To train the multimodal spatiotemporal forecasting model:

python train.py

Training will:

Construct day-by-day propagation graph sequences

Combine image features, metadata, and propagation structure

Optimize supervised + self-supervised objectives

---

🧪 Evaluation

Evaluate the trained model on the test split:

python eval.py

This computes:

MAE / RMSE for Share Count prediction

MAE / RMSE for Wiener Index prediction

nDCG@10 to assess ranking quality of early-window forecasts

All metrics are printed directly in the console.