The issue regarding the experimental effect

[GSW2221]

Dear author, hello. I conducted local experiments on the FactorizePhys you proposed on my computer, but the experimental results have never matched those in your paper. I have no idea where the problem lies in my experiment. The following are my computer equipment and yaml configuration files.
My experimental result:
FFT MAE (FFT Label): 5.146815413135593 +/- 1.9381495224469212
FFT RMSE (FFT Label): 15.751784013476142 +/- 3.951787453784477
FFT MAPE (FFT Label): 9.809061510340282 +/- 3.808499814347441
FFT MACC (FFT Label): 0.5950231712559986 +/- 0.025860228403292003
FFT Pearson (FFT Label): 0.8065964056138266 +/- 0.07829344693185468
FFT SNR (FFT Label): -0.030858069351716838 +/- 0.4764039084402862 (dB)

yaml file:
It is worth noting that I modified the CHUNK_LENGTH to 161 as described in your paper and conducted experiments on the rPPG_Toolbox. The experimental results have been listed above.
I saw that the CHUNK_LENGTH in the file you uploaded is 160. I also conducted an experiment based on 160, but the experimental result still has a significant difference from the result in the paper.

BASE: ['']
TOOLBOX_MODE: "train_and_test" # "train_and_test" or "only_train" or "only_test"
TRAIN:
BATCH_SIZE: 4
EPOCHS: 10
LR: 1e-3
MODEL_FILE_NAME: UBFC-rPPG_FactorizePhys_FSAM_Res
PLOT_LOSSES_AND_LR: True
DATA:
FS: 30
DATASET: UBFC-rPPG
DO_PREPROCESS: True # if first time, should be true
DATA_FORMAT: NCDHW
DATA_PATH: "D:/CUDA/DATA/UBFCrPPG" # Raw dataset path, need to be updated
CACHED_PATH: "D:/CUDA/pre" # Processed dataset save path, need to be updated
EXP_DATA_NAME: ""
BEGIN: 0.0
END: 1.0
PREPROCESS:
DATA_TYPE: ['Raw']
DATA_AUG: ['None'] # 'None' or 'Motion' is supported, used if the data path points to an augmented dataset or requires augmentation
LABEL_TYPE: Standardized
DO_CHUNK: True
CHUNK_LENGTH: 161
CROP_FACE:
DO_CROP_FACE: True
BACKEND: 'Y5F' # HC for Haar Cascade, RF for RetinaFace, Y5F for YOLO5Face
USE_LARGE_FACE_BOX: True
LARGE_BOX_COEF: 1.5
DETECTION:
DO_DYNAMIC_DETECTION: True
DYNAMIC_DETECTION_FREQUENCY : 30
USE_MEDIAN_FACE_BOX: False # This should be used ONLY if dynamic detection is used
RESIZE:
H: 72
W: 72
VALID:
DATA:
FS: 30
DATASET: UBFC-rPPG
DO_PREPROCESS: True # if first time, should be true
DATA_FORMAT: NCDHW
DATA_PATH: "D:/CUDA/DATA/UBFCrPPG" # Raw dataset path, need to be updated
CACHED_PATH: "D:/CUDA/pre" # Processed dataset save path, need to be updated
EXP_DATA_NAME: ""
BEGIN: 0.8
END: 1.0
PREPROCESS:
DATA_TYPE: ['Raw']
DATA_AUG: ['None'] # 'None' or 'Motion' is supported, used if the data path points to an augmented dataset or requires augmentation
LABEL_TYPE: Standardized
DO_CHUNK: True
CHUNK_LENGTH: 161
CROP_FACE:
DO_CROP_FACE: True
BACKEND: 'Y5F' # HC for Haar Cascade, RF for RetinaFace, Y5F for YOLO5Face
USE_LARGE_FACE_BOX: True
LARGE_BOX_COEF: 1.5
DETECTION:
DO_DYNAMIC_DETECTION: True
DYNAMIC_DETECTION_FREQUENCY : 30
USE_MEDIAN_FACE_BOX: False # This should be used ONLY if dynamic detection is used
RESIZE:
H: 72
W: 72
TEST:
METRICS: ['MAE', 'RMSE', 'MAPE', 'MACC', 'Pearson', 'SNR', 'BA']
USE_LAST_EPOCH: True # to use provided validation dataset to find the best epoch, should be false
DATA:
FS: 30
DATASET: PURE
DO_PREPROCESS: True # if first time, should be true
DATA_FORMAT: NCDHW
DATA_PATH: "D:/CUDA/DATA/PURE" # Raw dataset path, need to be updated
CACHED_PATH: "D:/CUDA/pre" # Processed dataset save path, need to be updated
EXP_DATA_NAME: ""
BEGIN: 0.0
END: 1.0
PREPROCESS:
DATA_TYPE: ['Raw']
LABEL_TYPE: Standardized
DO_CHUNK: True
CHUNK_LENGTH: 161
CROP_FACE:
DO_CROP_FACE: True
BACKEND: 'Y5F' # HC for Haar Cascade, RF for RetinaFace, Y5F for YOLO5Face
USE_LARGE_FACE_BOX: True
LARGE_BOX_COEF: 1.5
DETECTION:
DO_DYNAMIC_DETECTION: True
DYNAMIC_DETECTION_FREQUENCY : 30
USE_MEDIAN_FACE_BOX: False # This should be used ONLY if dynamic detection is used
RESIZE:
H: 72
W: 72
DEVICE: cuda:0
NUM_OF_GPU_TRAIN: 1
LOG:
PATH: runs/exp
MODEL:
DROP_RATE: 0.1
NAME: FactorizePhys
FactorizePhys:
CHANNELS: 3
FRAME_NUM: 160
MD_FSAM: True
MD_TYPE: "NMF"
MD_TRANSFORM: "T_KAB" #Default, if not specified
MD_R: 1
MD_S: 1
MD_STEPS: 4
MD_RESIDUAL: True
MD_INFERENCE: True
INFERENCE:
BATCH_SIZE: 4
EVALUATION_METHOD: "FFT" # "FFT" or "peak detection"
EVALUATION_WINDOW:
USE_SMALLER_WINDOW: False # Change this if you'd like an evaluation window smaller than the test video length
WINDOW_SIZE: 30 # In seconds
MODEL_PATH: ""

Equipment:
CPU: AMD Ryzen9 7900X
GPU: NVIDIA 3080 Ti
RAM: 64GB
pytorch 1.10.1
numpy 1.23.5
efficientnet-pytorch 0.7.1 pypi_0 pypi
torchaudio 0.10.1 py38_cu113 pytorch
torchinfo 1.8.0 pypi_0 pypi
torchstat 0.0.7 pypi_0 pypi
torchsummary 1.5.1 pypi_0 pypi
torchvision 0.11.2 py38_cu113 pytorch

This problem has bothered me for a long time and I am very much looking forward to getting your help. If there is any information that needs to be supplemented, I can provide it for you.

Looking forward to your reply.

[GSW2221]

The yaml file I am currently using is the one named [ubffc -rPPG_PURE_FactorizePhys_FSAM_Res.yaml] that you uploaded on GitHub. I have only modified the location where the data is stored.

[jnj256]

Hi @GSW2221 ,
Thank you for reaching out for the issue that your are facing.

I would first like to verify the data preparation step. Were both UBFC-rPPG and PURE datasets pre-processed using the YAML file for FactorizePhys, or were they prepared/ pre-processed using the YAML file of some other model?

[GSW2221]

This time, I completely used the project code you published on github for my experiment. During the data preprocessing stage, I directly called [configs-train_configs-UBFC-rPPG_PURE_FactorizePhys_FSAM_Res.yaml] and modified the storage locations of the files [DATA_PATH and CACHED_PATH]. All other parameters remained unchanged.

Hi @GSW2221 , Thank you for reaching out for the issue that your are facing.

I would first like to verify the data preparation step. Were both UBFC-rPPG and PURE datasets pre-processed using the YAML file for FactorizePhys, or were they prepared/ pre-processed using the YAML file of some other model?

[jnj256]

@GSW2221 : Thank you for clarifying this.
Other thing we can try before debugging deeper is to match the Python envionment. I notice that your PyTorch version (1.10.1) is bit older, and than the one used for FactorizePhys (2.0.0). If possible, please try to setup a new Python environment - as mentioned on the ReadMe of this Repo - also sharing it here:
STEP 1: bash setup.sh
STEP 2: conda activate fsam
STEP 3: pip install -r requirements.txt

Let's see if this helps, otherwise we will debug thoroughly.

[GSW2221]

@GSW2221 : Thank you for clarifying this. Other thing we can try before debugging deeper is to match the Python envionment. I notice that your PyTorch version (1.10.1) is bit older, and than the one used for FactorizePhys (2.0.0). If possible, please try to setup a new Python environment - as mentioned on the ReadMe of this Repo - also sharing it here: STEP 1: bash setup.sh STEP 2: conda activate fsam STEP 3: pip install -r requirements.txt

Let's see if this helps, otherwise we will debug thoroughly.

Thank you very much for your explanation. I will reconfigure the environment and conduct the experiment again next.