What are the training parameters for pre-trained networks?

[xphoniex]

I've been training for cubism style with 500 images (I know I should have around ~3k but there's not enough sample) with nz=300 ndf=20 ngf=100 (if I make them higher D or G would fixate on 0) and here are my results after 750, 1500 and 2250 iterations respectively:

Any suggestions for parameter tuning?

[robbiebarrat]

first off - try horizontally mirroring the images to get more data.

your network is spitting out the same images over and over (or variants of them) - which usually means that the generator has outsmarted the discriminator. try either reducing the number of filters in the generator, or increasing the ones in the discriminator.

[robbiebarrat]

or - use a pre-trained network (like the landscapes one) - and train that on cubism for a few epochs. you will definitely get more stable results that way.

[xphoniex]

I mirrored the images like you said, then trained the network using ndf=50 ngf=125:

Somewhere past the 750th iteration one of the networks reached 0 and the other 27 so I stopped the training. Is this because my data sample is still low? Or did it get solved in 750 iteration - is that ever supposed to happen?

I tried again using ndf=45 ngf=120:

your network is spitting out the same images over and over (or variants of them) - which usually means that the generator has outsmarted the discriminator.

at this point, if I modify ndf or ngf training is stuck after second iteration:

ndf : 75
ngf : 140

Epoch: [1][       0 /       15]	 Time: 11.247  DataTime: 0.001    Err_G: 1.9977  Err_D: 2.3039	
Epoch: [1][       1 /       15]	 Time: 0.934  DataTime: 0.005    Err_G: 0.0000  Err_D: 17.6696	
Epoch: [1][       2 /       15]	 Time: 1.540  DataTime: 0.001    Err_G: 3.0010  Err_D: 0.4860	
Epoch: [1][       3 /       15]	 Time: 1.538  DataTime: 0.001    Err_G: 0.0545  Err_D: 5.4351	
Epoch: [1][       4 /       15]	 Time: 1.537  DataTime: 0.001    Err_G: 11.4275  Err_D: 2.3259	
Epoch: [1][       5 /       15]	 Time: 1.542  DataTime: 0.001    Err_G: 0.0000  Err_D: 14.4293	
Epoch: [1][       6 /       15]	 Time: 1.551  DataTime: 0.001    Err_G: 11.4585  Err_D: 1.7864	
Epoch: [1][       7 /       15]	 Time: 1.541  DataTime: 0.001    Err_G: 0.0009  Err_D: 9.7629	
Epoch: [1][       8 /       15]	 Time: 1.540  DataTime: 0.001    Err_G: 19.6811  Err_D: 1.5082	
Epoch: [1][       9 /       15]	 Time: 2.630  DataTime: 0.001    Err_G: 0.0101  Err_D: 6.8360	
Epoch: [1][      10 /       15]	 Time: 0.919  DataTime: 0.001    Err_G: 20.2226  Err_D: 2.1656	
Epoch: [1][      11 /       15]	 Time: 1.541  DataTime: 0.001    Err_G: 0.0001  Err_D: 12.2572	
Epoch: [1][      12 /       15]	 Time: 1.549  DataTime: 0.001    Err_G: 20.4725  Err_D: 1.3243	
Epoch: [1][      13 /       15]	 Time: 1.549  DataTime: 0.001    Err_G: 0.0028  Err_D: 9.1036	
Epoch: [1][      14 /       15]	 Time: 1.543  DataTime: 0.001    Err_G: 25.9297  Err_D: 2.1749	
Epoch: [1][      15 /       15]	 Time: 1.538  DataTime: 0.001    Err_G: 5.9669  Err_D: 0.7090	
End of epoch 1 / 2500 	 Time Taken: 34.242	
Epoch: [2][       0 /       15]	 Time: 1.537  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.7518	
Epoch: [2][       1 /       15]	 Time: 1.476  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.7053	
Epoch: [2][       2 /       15]	 Time: 1.477  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6438	
Epoch: [2][       3 /       15]	 Time: 1.474  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6322	
Epoch: [2][       4 /       15]	 Time: 1.475  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6385	
Epoch: [2][       5 /       15]	 Time: 1.477  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6317	
Epoch: [2][       6 /       15]	 Time: 1.479  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6500	
Epoch: [2][       7 /       15]	 Time: 1.472  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6529	
Epoch: [2][       8 /       15]	 Time: 1.473  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6390	
Epoch: [2][       9 /       15]	 Time: 2.478  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6334	
Epoch: [2][      10 /       15]	 Time: 0.920  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6319	
Epoch: [2][      11 /       15]	 Time: 1.477  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6328	
Epoch: [2][      12 /       15]	 Time: 1.473  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6319	
Epoch: [2][      13 /       15]	 Time: 1.479  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6314	
Epoch: [2][      14 /       15]	 Time: 1.477  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6318	
Epoch: [2][      15 /       15]	 Time: 1.484  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.6339	

ndf : 55
ngf : 120

Epoch: [1][       0 /       15]	 Time: 9.819  DataTime: 0.001    Err_G: 0.6422  Err_D: 2.2376	
Epoch: [1][       1 /       15]	 Time: 0.598  DataTime: 0.001    Err_G: 0.0480  Err_D: 4.9154	
Epoch: [1][       2 /       15]	 Time: 1.001  DataTime: 0.001    Err_G: 3.0255  Err_D: 2.1711	
Epoch: [1][       3 /       15]	 Time: 0.979  DataTime: 0.001    Err_G: 0.0001  Err_D: 11.6199	
Epoch: [1][       4 /       15]	 Time: 0.986  DataTime: 0.001    Err_G: 1.9036  Err_D: 1.5011	
Epoch: [1][       5 /       15]	 Time: 0.970  DataTime: 0.001    Err_G: 0.0116  Err_D: 6.4144	
Epoch: [1][       6 /       15]	 Time: 0.978  DataTime: 0.001    Err_G: 8.6712  Err_D: 1.4926	
Epoch: [1][       7 /       15]	 Time: 0.976  DataTime: 0.001    Err_G: 0.0000  Err_D: 12.0402	
Epoch: [1][       8 /       15]	 Time: 0.983  DataTime: 0.001    Err_G: 4.9386  Err_D: 1.1580	
Epoch: [1][       9 /       15]	 Time: 1.763  DataTime: 0.001    Err_G: 0.0001  Err_D: 11.5025	
Epoch: [1][      10 /       15]	 Time: 0.583  DataTime: 0.001    Err_G: 9.6556  Err_D: 1.3580	
Epoch: [1][      11 /       15]	 Time: 0.976  DataTime: 0.001    Err_G: 0.0019  Err_D: 7.9405	
Epoch: [1][      12 /       15]	 Time: 0.978  DataTime: 0.001    Err_G: 14.5594  Err_D: 1.2251	
Epoch: [1][      13 /       15]	 Time: 0.983  DataTime: 0.001    Err_G: 0.0127  Err_D: 7.0154	
Epoch: [1][      14 /       15]	 Time: 0.986  DataTime: 0.001    Err_G: 15.0553  Err_D: 0.7528	
Epoch: [1][      15 /       15]	 Time: 0.980  DataTime: 0.001    Err_G: 0.0270  Err_D: 6.3222	
End of epoch 1 / 2500 	 Time Taken: 24.540	
Epoch: [2][       0 /       15]	 Time: 0.978  DataTime: 0.001    Err_G: 17.3309  Err_D: 1.5832	
Epoch: [2][       1 /       15]	 Time: 0.979  DataTime: 0.001    Err_G: 0.0668  Err_D: 4.5823	
Epoch: [2][       2 /       15]	 Time: 0.977  DataTime: 0.001    Err_G: 16.7114  Err_D: 1.4077	
Epoch: [2][       3 /       15]	 Time: 0.992  DataTime: 0.001    Err_G: 0.0003  Err_D: 9.9892	
Epoch: [2][       4 /       15]	 Time: 0.979  DataTime: 0.001    Err_G: 18.4478  Err_D: 1.3203	
Epoch: [2][       5 /       15]	 Time: 0.980  DataTime: 0.001    Err_G: 0.3714  Err_D: 2.5584	
Epoch: [2][       6 /       15]	 Time: 0.997  DataTime: 0.001    Err_G: 8.9893  Err_D: 1.0589	
Epoch: [2][       7 /       15]	 Time: 0.987  DataTime: 0.001    Err_G: 0.0000  Err_D: 27.0925	
Epoch: [2][       8 /       15]	 Time: 0.978  DataTime: 0.001    Err_G: 0.0000  Err_D: 26.1768	
Epoch: [2][       9 /       15]	 Time: 1.763  DataTime: 0.001    Err_G: 0.0004  Err_D: 9.1433	
Epoch: [2][      10 /       15]	 Time: 0.577  DataTime: 0.001    Err_G: 27.6180  Err_D: 0.5351	
Epoch: [2][      11 /       15]	 Time: 0.990  DataTime: 0.001    Err_G: 27.6310  Err_D: 0.7237	
Epoch: [2][      12 /       15]	 Time: 0.982  DataTime: 0.001    Err_G: 27.6310  Err_D: 0.3762	
Epoch: [2][      13 /       15]	 Time: 0.979  DataTime: 0.001    Err_G: 27.6310  Err_D: 0.0788	
Epoch: [2][      14 /       15]	 Time: 0.978  DataTime: 0.001    Err_G: 27.6310  Err_D: 0.1221	
Epoch: [2][      15 /       15]	 Time: 0.980  DataTime: 0.001    Err_G: 27.6310  Err_D: 0.0090