Question regarding accuracy problem.

[ubuntulim]

Hello,
Your project is very interesting! I looked at your project and I have a question I would like to ask you.

I trained CNV-W1A1 model with binary in Theano, then finnthesized it to get '.bin' parameters.
Then I generated bitstreams (.bit, .hwh, .tcl files) using make-hw.sh file
and then using make-sw.sh file I made '.so' file.
With the files that generate I just tested 10,000 test images on my PYNQ-Z2 board.
When I tested the 10,000 test images on Theano the error rate was 20.9, meaning the accuracy was about 79.1%.
When I tested the same test images on the PYNQ-Z2 board I got a 74.47% accuracy rate.
To my understanding the network topology of Theano and PYNQ-Z2 should be the same, then why is this problem of different accuracy occurring?
I read both your FINN papers and I couldn't find any information regarding this problem.
It would be a lot of help, if you could explain this problem for me.
I hope you are doing well during this COVID-19 crisis.
Thank you.

Theano results.

PYNQ-Z2 results

[chaeeon-lim]

Actually, this issue is written by me and @ubuntulim together because we are studying it together in our school.

I'll write details about our test as much as I can. If there is any information that I obscured, please let me know to explain again.

Firstly, we tested Theano with 'cifar10-1w-1a.npz' parameter which is uploaded in this repository (I didn't train a new one, just check the result). The test is done in docker.
Test codes on Theano are this. We just modified 'quantiezed_net.py' by giving boolean variable named in 'validation_mode'

Then we ran 'cifar10-gen-weights-W1A1.py' to make parameters for our PYNQ-Z2 board
After 'pip install BNN-PYNQ' on our board, we moved the parameter folder which is 'cnvW1A1' to python package folder in the board (/usr/local/lib/python3.6/dist-packages/bnn/params/cifar10/).
Also, we uploaded cifar10 test images which are consisted of 10,000 images on the board.
So, we tested the 10,000 images while we did not change anything but 'params'.

The above codes are our testing codes on the board. We did not interfere with this system much. I barely understand what makes this accuracy drop between Theano and the board?

[nickfraser]

There should not be an accuracy drop between Theano and the board. Can you explain in more detail what you have done so far, and what modifications you have made to the code (if any) for both hardware and software.

For example:

• are you training the network yourself? Or using the pretrained weights from the repo?
• do you get the same accuracy with the '.bin' that come with the distribution? Or only the ones you generate yourself?
• what accuracy do you get with the '.bin' files that come with the distribution?