a graph that shows the training loss

[sperzieb00n]

i know it makes a csv file, but still it would be nice if the webui could show a loss graph that gets updated every second or so

[ghost]

I think graphical updates can cause significant performance loss, so it's kinda better not. You can't really tell if the loss is going up or down on big datasets initially anyway, so you don't need to see it often, so might as well use convenient https://chart-studio.plotly.com/create/#/

Also from personal experience... Sometimes loss going up is not reeeeeally a bad thing, and you can continue training like that, but that's rare of course.

[5KilosOfCheese]

Loss going up at times is not an big issue, as long as the long term trend goes downwards. You don't see this when taking 10-100 samples. You need 1000s of points to see the trend for real. The slope is always really shallow all things considered.
However a good way to see this is to look at the highest numbers you see in the loss, and as long as you see them going down somewhat then the training is going towards the right direction.

But the training tends to always go towards lower noise, whether that low noise actually represents the correct results is totally another thing. From the AI's perspective it is going towards the right direction, it is up to you as a human to confirm whether it is.
But if you see that you results go wonky at certain point, you can back up to that point and restart there with different settings - if you don't want to restart completely.

[sperzieb00n]

ah, thanks for the info