Reproduce the results - Repeat results

[zwl8979]

I encountered some problems when reproducing the experiment according to the paper:
1. For different --lof_k values, the same experimental data was obtained;
2. When verifying that the noise level is 0.1 (--noise 0.1, No overlap, using soft weights, no noise enhancement), the same experimental results were obtained after three trainings. What is the reason? When using the same noise discriminator for detection, do I need to delete the index of the randomly sampled abnormal samples?
The following is my main experimental configuration：
--gpu
0
--seed
0
--results_path
result
--log_project
MVTecAD-wideresnet50-noiseo0.1-different-lof
--log_group
No-overlap-SW-NA-noise0.1-lof-6
--save_segmentation_images
--sampler_name
approx_greedy_coreset
--sampling_ratio
0.1
--faiss_on_gpu
--faiss_num_workers
8
--weight_method
lof
--threshold
0.15
--lof_k
6
--dataset
mvtec
--data_path
./MVTecAD
--subdatasets
bottle
cable
capsule
carpet
grid
hazelnut
leather
metal_nut
pill
screw
tile
toothbrush
transistor
wood
zipper
--batch_size
16
--resize
256
--imagesize
224
--noise
0.1
--fold
0
The screenshot shows the results of three identical experimental settings：

@jam-cc @swordlidev

[jam-cc]

Your situation is quite unusual, and we haven't encountered it before.

1. Slightly adjusting --lof_k may have a small impact on performance, but the results should not be exactly the same.
2. Changing the noise ratio should definitely affect the performance metrics. The index of randomly sampled abnormal samples does not need to be deleted.

In summary, there may be an issue with how your parameters are being passed.

[zwl8979]

Your situation is quite unusual, and we haven't encountered it before.

1. Slightly adjusting --lof_k may have a small impact on performance, but the results should not be exactly the same.
2. Changing the noise ratio should definitely affect the performance metrics. The index of randomly sampled abnormal samples does not need to be deleted.

In summary, there may be an issue with how your parameters are being passed.

Thanks to the author for his positive response. I have two more questions to ask:

1. When changing the value of --lof k, I understand that it is not necessary to change the index of sampling abnormal samples to verify the results. Then for the same --lof k value (for example, --lof 6), do we need different abnormal sample indexes when taking the average of three sets of experiments?
2. For the same discriminator (for example, lof, no overlap, --ratio 0.1, using soft weights, not using noise enhancement), when running three times to take the average, do we need to sample different samples each time to calculate the average of the experimental results?
3. During the calculation process, the seed of the random number is fixed, so the abnormal samples are the same. Shouldn’t the calculation results under the unified settings be the same?@jam-cc