Learn Rate, Loss, TensorBoard Discussion

[JustAnOkapi]

• Constant lr_scheduler is slightly better
• Using TensorBoard

Using constant give slightly lower loss and is easier to resume off. Same conclution as this reddit post.

You can enable tensorboard pretty easily with two blocks:

!pip3 uninstall tensorboard # these three fix ValueError: Duplicate plugins for name projector
!pip3 uninstall tensorflow
!pip3 install --ignore-installed tf-nightly # theres probably a correct version to not have to do nightly

# !kill 10459
%reload_ext tensorboard
%tensorboard --logdir /content/models/$Session_Name/logs/

import IPython
display(IPython.display.HTML('''
<button id='open_tb'>Open TensorBoard in a new tab.</button> 
<button id='hide_tb'>Hide TensorBoard from colab.</button> 
<script>document.querySelector('#open_tb').onclick = () => { window.open(document.querySelector('iframe').src, "__blank") }
        document.querySelector('#hide_tb').onclick = () => { document.querySelector('iframe').style.display = "none" }</script>'''))

This will stay running in the background so you can start training.

I manually reorganize the log files into this structure:

(run) (unet or text) (steps)

[JustAnOkapi]

• Loss Graphs

Text encoder:

Unet:

[JustAnOkapi]

• Why is loss all over the place, and not going down?
• How come the losses are almost identical when resuming?
  Looks like the seed locked from cross-model testing. After changing the loss no longer overlaps:
  

[TheLastBen]

in the case of dreambooth, a constant learning rate for a small dataset is not ideal

[JustAnOkapi]

seems that the constant option, doesnt even work on unet, it still runs linears anyways. weird

I cant get small datasets to work, it goes straight from not looking like the face, to overfitting.

[JustAnOkapi]

why is loss so all over the place anyways, even the avg doesnt go down

[TheLastBen]

the loss is displayed for each instance image, not the avg

[JustAnOkapi]

yea but smoothed it should be the avg around each image? and that should get better?

[JustAnOkapi]

lora training has the classic loss curve