Metrics for BNN

[alexandruvesa]

Hello,
Can you help me please with some suggestions about metrics?
I would like for example to obtain a confusion matrix . more exactly i would like to measure the performance of my model.
Thanks a lot,
Alex

[giuliogamba]

Hi,
I am not sure about what's your question.
Anyway, you can easily obtain a confusion matrix by classifying with your model on your target testset and compare against the label.

[alexandruvesa]

Thanks for the answer @giuliogamba . Sorry for my unclear question. In this code. For example in cifar10.py , what variables should I use for the confusion matrix?
I know, for example i can import confusion matrix from the scikit learn , but I need two parameters there.
Thanks a lot,
Alex

[nickfraser]

Do you want to compute the confusion matrix from the output of the hardware, or from the output of the training software?

While both of the these should achieve the same exact same result, they require different development effort.

[alexandruvesa]

Hello. From the training software.

[nickfraser]

OK, I'll provide a worked example in the coming weeks. Unfortunately, I have some other work I need to finish before I can work on this.

In the meantime, if you want to do this yourself - please send us a PR! I'll give you a few pointers:

1. You need to make a new theano function which returns the 'test_output' result (rather than classification error) - see these lines of code 1, 2
2. Then pass the test set through this function, as shown here.
3. After this, you should have a tensor with the shape (num_images, num_classes). You can take the max value for each image to know the predicted class, you can then compare this with the maximum value for each image of the label
4. Putting this together, you should be able to generate your confusion matrix.

You may also find the Theano and Lasagne documentation to be very useful.