Seg Eval

[xing0047]

Hi @helblazer811 Thanks for your wonderful work!

I am trying to reproduce the results in this paper. I came across error when running the provided script: https://github.com/helblazer811/ConceptAttention/blob/master/experiments/imagenet_segmentation/run_cross_self_ablation.sh as mentioned in Issue #12.

I use the dataloader from ImageNet-S and use your pipeline implementation instead, the reproduced results from my side is as follows.

ImageNet-S	acc	iou
reproduced	0.62	0.45
report	0.83	0.71

May I ask how the evaluation setup may be corrected to reproduce the paper results, thanks!

[helblazer811]

Hey. I'm glad you are interested in our work.

Could you be more specific about the error you ran into when running the script?

Also it is hard for me to diagnose the discrepancy in the evaluation if I don't know exactly the evaluation set up you used. Are you able to reproduce the other baseline evaluation results with your set up?

Small differences in how these metrics are computed can cause significant differences. For my evaluation I followed the set up from this repository by copying their data loading/evaluation pipeline as to reproduce their results https://github.com/hila-chefer/Transformer-Explainability.

Cheers.

[xing0047]

Hi @helblazer811

Thanks for your kind reply! The reported error is as follows,

Exception has occurred: ValueError
not enough values to unpack (expected 3, got 2)
  File "ConceptAttention/concept_attention/binary_segmentation_baselines/raw_output_space.py", line 191, in segment_individual_image
    pred, _, concept_heatmaps = self.generator.model(
  File "ConceptAttention/concept_attention/segmentation.py", line 49, in __call__
    coefficients, reconstructed_image = self.segment_individual_image(
  File "ConceptAttention/experiments/imagenet_segmentation/run_experiment.py", line 143, in <module>
    mask, coefficients, reconstructed_image = segmentation_model(
ValueError: not enough values to unpack (expected 3, got 2)

when I further investigate this error, it appears that the type of segmentation_model here is ModifiedFluxDiT, whose forward returns two values (img, combined_concept_attention_dict), while this line requires 3 (mask, coefficients, reconstructed_image).

For the alternative evaluation, I use the pipeline implementation you provided. The code snippet related to ConceptAttention is as follows,

pipeline_output = pipeline.encode_image(
    image=img,
    concepts=[simplified_name] + args.background_concepts,
    prompt=f"a {simplified_name}",
    width=1024,
    height=1024,
    return_pil_heatmaps=False,
    num_samples=args.num_samples,
    num_steps=args.num_steps,
    noise_timestep=args.noise_timestep,
    softmax=True
)
mask = pipeline_output.concept_heatmaps[0]

Hope this clarifies my question.