Visualising the last layer feature map with tSNE

[akashtadwai]

Hi,
I am using a SimpleMLP model with num_classes=15, hidden_size=100 with a GEM plugin. I wanted to visualize the last feature map before the classifier head with tSNE so that I could visually see the 2D projection of what my model is learning (similar to what is done in this) in subsequent tasks. For that, I should have access to the test_stream labels and the model after the test_stream. Can anyone help me with this? My basic code for the model is uploaded here.

[vlomonaco]

Hi @akashtadwai! Adding a metric to visualize feature maps is something we definitely would like to integrate in the near future. I suggest to go through the From Zero to Hero tutorial to see how you can add custom metrics.

As for your request, you can access the dataset of each experience at any time with test_stream[idx_exp].dataset and you can extract targets iterating trough it with a data loader or by accessing the targets field: test_stream[idx_exp].dataset.targets directly.

The final model at the end of the training can be recovered with model = cl_strategy.model.

Let me know if you need further assistance! :)

[EdenBelouadah]

@akashtadwai, I just wanted to remind you to return to the training mode in case you will continue training your model after extracting the scores (I think it's even done automatically in the BaseStrategy's training loop). If you don't need it, remove it.

Yes, SimpleMLP and strategy.model are the same. It's just that strategy.model points to the trained model. However the instantiation of SimpleMLP is randomly initialized.

@AndreaCossu, please correct me if I'm wrong.

[vlomonaco]

@akashtadwai let us know if we can mark your question as "answered", thanks! :)

[akashtadwai]

@vlomonaco Yes Thanks! I have done it using hooks!

[akashtadwai]

test_stream[idx_exp].dataset

@vlomonaco @AndreaCossu Can someone clarify what exactly the train_stream[exp_idx].dataset[idx] returns.
I was doing stg like this,

for idx in range(len(train_stream)):
    print(type(train_stream[idx].dataset[0]))
    break

I was getting at tuple with 3 attributes when I ran locally but when I saw the original code in avalanche_dataset.py It says that it consists of a tuple with 4 attributes.
Can you exactly specify what are the attributes in the tuple?
I guess the attributes I got are (features[idx],label,task_label). Is this correct??? and what does *element[2:] mean in the code avalanche_dataset.py.

[vlomonaco]

Hi @akashtadwai! As you correctly derived train_stream[exp_idx].dataset[idx] returns an AvalancheDataset. an AvalancheDataset returns a tuple that may contain a variable number of elements based on the specific dataset/task. For classification (data, label, task_label) is often enough. However, for detection or segmentation you often need more data: (data, label, bounding_box, task_label) for example. So in general the AvalancheDataset returns a tuple of this form (data, label, ... , task_label).

[AndreaCossu]

You are correct @EdenBelouadah, if you have direct access to model you can use that instance. Otherwise, if you have access to strategy you can access the model by strategy.model. This is the same.