Tensorboard, prediction and more

[Samael-1976]

Preface 1: I'm using Windows, so I wouldn't know how to troubleshoot issues with Linux or other systems.
Preface 2: The code was created for personal use, without any claims.
Preface 3: It was helped by my friend (Franci) and I used CHATGPT a lot.
Preface 4: I do my best for the calculation of convergence, Over Training, and Over Fitting, but I couldn't improve it with my limited knowled
Preface 5: The code is completely open source, so use it and modify it as you wish. If you make significant improvements, please share them as a response to this message to help everyone.

What the code does:

• The code start to calculate everything starting from second epoch (or after the warmup steps)
• The code creates a graph and provides a terminal response with the following values while the training is in progress (or finished) - The data and graph update automatically every 10 minutes:

Total Epochs Value from JSON: 300
Warmup Steps Value from JSON: 500
Repetitions Value from JSON: 1.0
Batch Size Value from JSON: 1
Results:

Best step: (99612.0, 0.10412197560071945)
Worst step: (8377.0, 0.15289561450481415)
Best epoch: 83 (Step: 98936.0) - Mean Loss: 0.12
Worst epoch: 78 (Step: 92976.0) - Mean Loss: 0.14
Convergence Epoch: 972 (Step: 1158624.0)
Overtraining Epoch: 1000 (Step: 1192000.0)
Overfitting Epoch: 974 (Step: 1161008.0)

How to make it work:

1. Save the PreCog-EN.py inside the OneTrainer path C:\OneTrainer\Workspace (my OneTrainer is installed on C:, but your path may differ).
2. In command prompt go inside your C:\OneTrainer\venv\Lib\site-packages, type: pip install matplotlib
3. Start OneTrainer and begin the training.
4. Once started, use the command line to navigate to this path C:\OneTrainer\Workspace\tensorboard\
5. You should find folders named similar to these: 2024-05-22_09-04-48
6. Copy the complete path somewhere. Example: C:\OneTrainer\Workspace\tensorboard\2024-05-22_09-04-48
7. Navigate to the venv\script\ folder (example: C:\OneTrainer\venv\Scripts)
8. In the window, where the path is shown, delete the path, type CMD, and the terminal window will open.
9. In the terminal window, type: tensorboard --logdir C:\OneTrainer\Workspace\tensorboard\2024-05-22_09-04-48 --host=127.0.0.1
   Of course, you need to replace 2024-05-22_09-04-48 with the name of your folder. A window will open, leave it open and return to OneTrainer.
10. Optional) In the bottom right, click on the Tensorboard button, from which you can retrieve the exact path of the CSV file with the SMOOTH LOSS data in case you need to change it within the code.
11. (If you have executed step 9, minimize the window) Open Explorer, and go to the folder where you saved the PreCog-EN.py file (which should be C:\OneTrainer\Workspace)
12. At the top in the path bar, type CMD and press enter.
13. In the command prompt, type the name of the PreCog-EN.py and press enter.
14. Now it will ask you to enter the number of images in your DATASET, do it and press enter.
15. A window will open again where you will need to select the JSON file related to TRAINING (it will ask you on the prompt). Usually, you can find the file here: C:\OneTrainer\training_presets\name_of_your_preset.json
16. Another window will open where you need to select the JSON file related to the CONCEPT DATASET (it will ask you on the prompt). Usually, you can find the file here: I:\OneTrainer\training_concepts\name_of_your_concept_datset.json
17. Now it will ask which smoothing method to use to calculate the best convergence, over training and over fitting. You can choose between:

• Savitzky-Golay
• Moving Average
• Gaussian Kernel
• Exponential Moving Average
• Loess
• Butterworth Filter

Done.

If it works, a window with a graph similar to this one will open:

and a sort of this command prompt:
!

To Download PreCog-EN: https://silverider76.iliadboxos.it:27440/share/rC8xVK_qv6BbRjH3/PreCog-EN.py

@Nerogar if you are interested to integrate this code in OneTrainer, you are welcome. I'm glad to help your fantastic job ;)

[Samael-1976]

I managed to improve the predictive code slightly.
Now in the initial phase it also asks which smoothing method to use to calculate the best convergence, over training and over fitting.

Choosing between:

• Savitzky-Golay
• Moving Average
• Gaussian Kernel

The "Moving Average" choice is the one that was the default in the previous code.

The other two choices, however, seem more "true" to me as a prediction.

I'll leave you the screenshots

I've updated the link for the download the file, also in the first message:
https://silverider76.iliadboxos.it:27440/share/rC8xVK_qv6BbRjH3/PreCog-EN.py

[Samael-1976]

Fixed on data and graph.
https://silverider76.iliadboxos.it:27440/share/rC8xVK_qv6BbRjH3/PreCog-EN.py

[Samael-1976]

Corretced two bugs about :

• animation loop can cause issues
• Initialize Tkinter properly and ensure the mainloop runs without interruptions.

https://silverider76.iliadboxos.it:27440/share/rC8xVK_qv6BbRjH3/PreCog-EN.py

[Samael-1976]

Updated, now the graph is dynamic.
When you move the mouse over the graph it show you the Step, Epoch and loss.

[Samael-1976]

Updated, now the prevision methods are 6:

Savitzky-Golay
Moving Average
Gaussian Kernel
Exponential Moving Average
Loess
Butterworth Filter

Exponential Moving Average -> expressly designed for Stable Diffusion XL training

[Samael-1976]

Updated again:
all predictive systems have been improved, they are now all dedicated for the Stable Diffusion XL models

[djp3k05]

Hello,
I tried to test your tool, but at step 12, when opening the PreCog-EN.py, the new windows close without any message. So your tool it is closing before the 21 line, where it should have asked me about "Enter the number of images in the Dataset:". I will try to debug it.

[Samael-1976]

Hi! it's very strange, because I'm still using it and also a my friend without any problem, let me know :)

Probably it's a problem with Matplotlib

[djp3k05]

indeed... the first issue is that i'm missing the mentioned lib:
import matplotlib.pyplot as plt ModuleNotFoundError: No module named 'matplotlib'

[Samael-1976]

very easy to resolve with:
with cmd go inside your C:\OneTrainer\venv\Lib\site-packages
type: pip install matplotlib

[djp3k05]

success, it is working. the file could be run as "python -i Path_to_file
PreCog-EN.py" (interactive mode. this way will shown any errors, like I had).
thank you! i will return with feedback.

[Samael-1976]

Thanx to you!!!

[MegaCocos]

Hello,
I tried to use your tool, but after step 9 no window opens and when I run PreCog-EN.py it gives errors after choosing the method. I added the screenshots. At the same time in the OneTrainer if I press TensorBoard button, it opens normally. I added screenshots too. Can you help? What it could be? Windows 10, RTX 4080 Super