Reproduction of results not possible, due to errors in code.

[anshul013]

Respected authors,

Firstly, I would like to congratulate you on the publication of your paper named "From Similarity to Superiority: Channel Clustering
for Time Series Forecasting". I have been conducting some research in the field of Time Series Forecasting and I really liked your paper, so, I chose it as a baseline for my research.

But when I went to reproduce the results, I encountered 3 issues which were:

1. In the exp_ccm.py file under the similarity_loss_batch() method, you are multiplying the probability membership and similarity matrix but the similarity matrix is a tensor which results in an error.
2. The similarity and the membership matrix were not the same datatype.
3. In layers.py file, line 287 under the forward() method I received this error:
   prob = torch.mm(self.l2norm(x_emb), self.l2norm(cluster_emb).t()).reshape(bs, n_vars, self.n_cluster)
   RuntimeError: t() expects a tensor with <= 2 dimensions, but self is 3D

I fixed these errors and the code started working for me but the results were not even close to the reported results from the paper. Here are the results after running the code for ETTh1 for TSMixer model with the CCM module.
96 | 0.4652 | 0.4645
192 | 0.4162 | 0.4258
336 | 0.4530 | 0.4556
720 | 0.4981 | 0.5079

I would like to request you to please update your code to your final run which you utilised for the results that are reported in the research paper. I would also like to request you to please undertake the updation as soon as possible, as your implementation is quite integral for the research that I am carrying out. Thank you for your time and looking forward to hearing from you.

[CSyyn]

I had the same problem. Could you please update the final code?

[Stark897]

I had the same problem. Could you please update the final code?

[137315198]

I had the same problem. Could you please update the final code?

[RakibulHaqueSajal]

https://github.com/RakibulHaqueSajal/CCM_Module.git

• I have tried to fix the issues. I was able to replicate the results for ETTm1 dataset.(Not the exact numbers but close enough)

[137315198]

https://github.com/RakibulHaqueSajal/CCM_Module.git

• I have tried to fix the issues. I was able to replicate the results for ETTm1 dataset.(Not the exact numbers but close enough)

Thank you so much！

[JamesHInGit]

https://github.com/RakibulHaqueSajal/CCM_Module.git

• I have tried to fix the issues. I was able to replicate the results for ETTm1 dataset.(Not the exact numbers but close enough)

Thank you so much！

[pallav-vunet]

https://github.com/RakibulHaqueSajal/CCM_Module.git

• I have tried to fix the issues. I was able to replicate the results for ETTm1 dataset.(Not the exact numbers but close enough)

Why don't you raise a PR?

[yi-zhi111]

https://github.com/RakibulHaqueSajal/CCM_Module.git

• I have tried to fix the issues. I was able to replicate the results for ETTm1 dataset.(Not the exact numbers but close enough)

Thank you very much!

[anjeta]

https://github.com/RakibulHaqueSajal/CCM_Module.git

* I have tried to fix the issues. I was able to replicate the results for the ETTm1 dataset.(Not the exact numbers but close enough)

Hey,

I think the fixed implementation still does not calculate the channel similarities correctly. In fact, the get_similarity_matrix_update function in exp_ccm.py now computes the point-by-point differences along the temporal dimension first. However, I think this should only be done when calculating similarity for univariate time series. On the other hand, when having multivariate time series, I believe the similarities should be obtained just by calculating Euclidean distances between channel sequences (without the temporal shifting). After updating the functions get_similarity_matrix_update and similarity_loss_batch, I was able to get results that are closer to the ones reported in the paper for the ETTm1 dataset and the DLinear model:
96 | Test MSE: 0.299 Test MAE: 0.343
192 | Test MSE: 0.336 Test MAE: 0.367
336 | Test MSE: 0.370 Test MAE: 0.384
720 | Test MSE: 0.423 Test MAE: 0.417

def get_similarity_matrix_update(self, batch_data, sigma=5.0):
        x = batch_data.transpose(1, 2)  
        cdist = torch.cdist(x, x, p=2.0)
        
        # Similarity matrix per input example
        sim_matrix = torch.exp(-cdist / (2 * sigma ** 2))

        return sim_matrix.to(self.device)

def similarity_loss_batch(self, prob, simMatrix):
        def concrete_bern(prob, temp = 0.07):
            random_noise = torch.empty_like(prob).uniform_(1e-10, 1 - 1e-10).to(self.device)
            random_noise = torch.log(random_noise) - torch.log(1.0 - random_noise)
            prob = torch.log(prob + 1e-10) - torch.log(1.0 - prob + 1e-10)
            prob_bern = ((prob + random_noise) / temp).sigmoid()
            return prob_bern
        membership = concrete_bern(prob)  #[n_channels, n_clusters]
        
        S = simMatrix.mean(axis=0)
        M = membership.to(device=S.device, dtype=S.dtype)
        
        temp_1 = torch.mm(M.t(), S) 
        SAS = torch.mm(temp_1, M)
        _SS = 1 - torch.mm(M, M.t())
        loss = - torch.trace(SAS) + torch.trace(torch.mm(_SS, S)) + M.shape[0]
        
        ent_loss = (-prob * torch.log(prob + 1e-15)).sum(dim=-1).mean()
        return loss + ent_loss

Another thing that confuses me is whether 1 in _SS = 1 - torch.mm(M, M.t()) should be replaced with an identity matrix, and why is there the + M.shape[0] term in the loss?

Please correct me if I'm seeing something wrong here!