How to get vectors after running with test data?

[sudarshan85]

Hi,

I'm doing text classification using HuggingFace. I would like access to the [CLS] vectors of the test data after I finish my testing. Currently, these are my methods for testing:

  @torch.no_grad()
  def test_epoch_end(self, outputs):
    loss = torch.stack(list(zip(*outputs))[0])    
    logits = torch.cat(list(zip(*outputs))[1])    
    preds = logits.argmax(axis=1).cpu()
    labels = torch.stack(list(zip(*outputs))[2]).view(logits.shape[0]).to(torch.int).cpu()   
    cls_vectors = torch.stack(list(zip(*outputs))[3]).view(logits.shape[0], -1).cpu()
    self.log('test_loss', loss)
    self.log('accuracy', accuracy_score(labels, preds))
    self.log('precision', precision_score(labels, preds))
    self.log('recall', recall_score(labels, preds))
    self.log('f1', f1_score(labels, preds))
    self.log('cls', cls_vectors)

  @torch.no_grad()
  def test_step(self, batch, batch_idx):
    outputs = self(**batch, output_hidden_states=True)    
    labels = batch['labels']
    loss = outputs[0]
    logits = outputs[1]    
    cls_vectors = outputs[2][-1][:,0,:]
    return loss, logits, labels, cls_vectors

and run the model like this result = test_trainer.test(clf_model, dataloaders=test_dl)

However, only 1 float value of the CLS vector seems to get logged: result[0]['cls'] returns -0.008937936276197433

How do get access to the all the CLS vectors (essentially a tensor of shape (test_data_size, 768)?

Thanks.

[SkafteNicki]

Hi @sudarshan85,
Have you tried returning the cls_vectors from the test_epoch_end method.
Please note that if you are using self.log that it only accepts scalar input and that is probably the reason why you are only seeing one float value being logged.