Training or Learning curve?

[ynandwan]

Hi
Is it possible for you to share the learning curves: plot of train/val accuracy or val/train loss vs epoch or train time?
I ran run_sr10to40.sh script and it has already run for 47 of 200 epochs (total 50 hrs of running on K40) but the loss hasn't reduced at all. Accuracy is at 0.5 since the beginning. Unless some inflection point comes, I don't see it converging.
One difference that I noticed is that the paper mentions using layer norm lstm whereas you have used normal lstm. Not sure how much impact it has on the performance.

Thanks
Yatin

[ryanzhangfan]

Sry, since I have graduated from my college so I can't access to the server to get the detailed log. But I found another log file which shows that our model starts to converge since epoch 23 (train on sr10to40, and i killed it when it appeared to converge). I had also noticed that we did not use the layer norm lstm but the experiment results showed that this did not influence the convergence. We had trained our model for multiple times.

So I recommend that you can first try to train the model on sr3to10 to validate the correctness. It will take quiet less time than sr10to40. You can first generate 300k instance offline by using the data_maker scripts and than train the with this data file by specifying "--train-file" flag.

[lee-man]

@ryanzhangfan
Hi Fan,
I am trying to run your code, starting from sr3to10 problems.
Could you offer any ideas about the meaning of 'val_v10_vpb12000_b148.pkl'? In particular, what 'b148' refers to? I guess it's related to the problem size but I am not quite sure.

Thanks!

Min

[ryanzhangfan]

@lee-man This file is generated offline and custom named by myself. v denotes the number of variable of each problem. vpb denotes "variables per batch", it restricts the max node number in one batch. b denotes the batch size. For more data generation details, please refer to link

[lee-man]

Got it. May I know the number of instances of SR10 and SR40 you used for validation? Thanks!

[ryanzhangfan]

@lee-man I can't remember clearly. It might be 5000 or 10000 instances for validation

[x-lad]

@ynandwan have you been able to overcome this issue?