Improve mlflow detection#1455

Open

AlyaGomaa wants to merge 461 commits intodevelopfrom

improve-mlflow-detection

This pull request is big! We're only showing the most recent 250 commits

Commits on May 5, 2025

Fix plot

eldraco
authored and
AlyaGomaa
committed
Plot testing performance from a log

eldraco
authored and
AlyaGomaa
committed
Fix the plot

eldraco
authored and
AlyaGomaa
committed
Fix the plots

eldraco
authored and
AlyaGomaa
committed
Fix plot

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Change plot names

eldraco
authored and
AlyaGomaa
committed
Rename file

eldraco
authored and
AlyaGomaa
committed
Recover good flowmldetection deleted by mistake

eldraco
authored and
AlyaGomaa
committed
Fix plot test

eldraco
authored and
AlyaGomaa
committed
Add testing code to evaluate performance. It is optional with a varible

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Fix train plot

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Add performance metrics to the training evaluation

eldraco
authored and
AlyaGomaa
committed
Fix experiment names

eldraco
authored and
AlyaGomaa
committed
Merge remote-tracking branch 'origin/improve-mlflow-detection' into improve-mlflow-detection
AlyaGomaa
committed
test_profiler: update unit tests
AlyaGomaa
committed
Fix that the training and testing logs files were appened instead of rewritten
eldraco
committed
Fix an issue of storing the new log files
eldraco
committed

Commits on May 10, 2025

enable/ disable training and testing.log with a param in the config file
AlyaGomaa
committed
dont create an empty logfile when create_performance_metrics_log_files is set to false
AlyaGomaa
committed
when enabled, create testing.log or training.log in the current output dir
AlyaGomaa
committed
Add an enum called labels with either Benign or Malicious so the labels are unified.
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
state_handler: split get_final_state_from_flags() into smaller functions
AlyaGomaa
committed
state_handler: refactor get_final_state_from_flags()
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
Re add function that alya added

eldraco
authored and
AlyaGomaa
committed
delete sys

eldraco
authored and
AlyaGomaa
committed
Delete file that was deleted from develop

eldraco
authored and
AlyaGomaa
committed
Flowmldetection. Fix missing db reference

eldraco
authored and
AlyaGomaa
committed
Fix the training of flows with ML in new version

eldraco
authored and
AlyaGomaa
committed
Fix the profiler handler for cases of nan in state

eldraco
authored and
AlyaGomaa
committed
slips.yaml. Update to have correct labels. By default test. Defaul training lbel is benign

eldraco
authored and
AlyaGomaa
committed
First ipython to tst ML flow related models

eldraco
authored and
AlyaGomaa
committed
flowml. If the dataset has one flow and that is deleted, then return empty fast.

eldraco
authored and
AlyaGomaa
committed
flowml. If the datasert is empty. Return none

eldraco
authored and
AlyaGomaa
committed
profile_handler. Small bug in how we handled the profiles, we were using 'in' instead of == for established. Some not established MAY not have been correctly captured

eldraco
authored and
AlyaGomaa
committed
First new version of the model and scaler. Not good yet, but working.

eldraco
authored and
AlyaGomaa
committed
model and scaler with 1 malicious and 1 benign

eldraco
authored and
AlyaGomaa
committed
cleaner jupyter

eldraco
authored and
AlyaGomaa
committed
New models after 3rd ttrain

eldraco
authored and
AlyaGomaa
committed
Models after 4th train

eldraco
authored and
AlyaGomaa
committed
Models of ml flow with the first good performance in small tests

eldraco
authored and
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
state_handler: split get_final_state_from_flags() into smaller functions
AlyaGomaa
committed
state_handler: refactor get_final_state_from_flags()
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
state_handler: split get_final_state_from_flags() into smaller functions
AlyaGomaa
committed
state_handler: refactor get_final_state_from_flags()
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
state_handler: split get_final_state_from_flags() into smaller functions
AlyaGomaa
committed
state_handler: refactor get_final_state_from_flags()
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
state_handler: split get_final_state_from_flags() into smaller functions
AlyaGomaa
committed
state_handler: refactor get_final_state_from_flags()
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Ignore UID column

eldraco
authored and
AlyaGomaa
committed
Re add function that alya added

eldraco
authored and
AlyaGomaa
committed
Delete file that was deleted from develop

eldraco
authored and
AlyaGomaa
committed
Flowmldetection. Fix missing db reference

eldraco
authored and
AlyaGomaa
committed
Fix the training of flows with ML in new version

eldraco
authored and
AlyaGomaa
committed
flowml. If the dataset has one flow and that is deleted, then return empty fast.

eldraco
authored and
AlyaGomaa
committed
flowml. If the datasert is empty. Return none

eldraco
authored and
AlyaGomaa
committed
First new version of the model and scaler. Not good yet, but working.

eldraco
authored and
AlyaGomaa
committed
model and scaler with 1 malicious and 1 benign

eldraco
authored and
AlyaGomaa
committed
cleaner jupyter

eldraco
authored and
AlyaGomaa
committed
New models after 3rd ttrain

eldraco
authored and
AlyaGomaa
committed
Models after 4th train

eldraco
authored and
AlyaGomaa
committed
Models of ml flow with the first good performance in small tests

eldraco
authored and
AlyaGomaa
committed
Add plot for flowml train scores

eldraco
authored and
AlyaGomaa
committed
Add a log file to store the training data output

eldraco
authored and
AlyaGomaa
committed
Store data in the log file of training

eldraco
authored and
AlyaGomaa
committed
better comments

eldraco
authored and
AlyaGomaa
committed
Fix issue not dropping detailed labels

eldraco
authored and
AlyaGomaa
committed
Fix issue that not all labels sere given to the partial fit

eldraco
authored and
AlyaGomaa
committed
count partial labels in this epoch

eldraco
authored and
AlyaGomaa
committed
Dont print training in screen

eldraco
authored and
AlyaGomaa
committed
Add function to write to train log

eldraco
authored and
AlyaGomaa
committed
Fix label in dummy flow

eldraco
authored and
AlyaGomaa
committed
Fix dummy flow

eldraco
authored and
AlyaGomaa
committed
Rename variable

eldraco
authored and
AlyaGomaa
committed
Fix dummy flow label

eldraco
authored and
AlyaGomaa
committed
Pass values to train function

eldraco
authored and
AlyaGomaa
committed
import os

eldraco
authored and
AlyaGomaa
committed
Get issue of total flows zero

eldraco
authored and
AlyaGomaa
committed
Add comments

eldraco
authored and
AlyaGomaa
committed
Rename var name to be more clear

eldraco
authored and
AlyaGomaa
committed
Rename var name

eldraco
authored and
AlyaGomaa
committed
Fix processeed flows being zero

eldraco
authored and
AlyaGomaa
committed
Delete old comments

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Fix plot

eldraco
authored and
AlyaGomaa
committed
Fix plot

eldraco
authored and
AlyaGomaa
committed
Fix plot

eldraco
authored and
AlyaGomaa
committed
Plot testing performance from a log

eldraco
authored and
AlyaGomaa
committed
Fix the plot

eldraco
authored and
AlyaGomaa
committed
Fix the plots

eldraco
authored and
AlyaGomaa
committed
Fix plot

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Change plot names

eldraco
authored and
AlyaGomaa
committed
Rename file

eldraco
authored and
AlyaGomaa
committed
Recover good flowmldetection deleted by mistake

eldraco
authored and
AlyaGomaa
committed
Fix plot test

eldraco
authored and
AlyaGomaa
committed
Add testing code to evaluate performance. It is optional with a varible

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Fix train plot

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Add performance metrics to the training evaluation

eldraco
authored and
AlyaGomaa
committed
Fix experiment names

eldraco
authored and
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
state_handler: split get_final_state_from_flags() into smaller functions
AlyaGomaa
committed
state_handler: refactor get_final_state_from_flags()
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
Re add function that alya added

eldraco
authored and
AlyaGomaa
committed
delete sys

eldraco
authored and
AlyaGomaa
committed
Delete file that was deleted from develop

eldraco
authored and
AlyaGomaa
committed
Flowmldetection. Fix missing db reference

eldraco
authored and
AlyaGomaa
committed
Fix the training of flows with ML in new version

eldraco
authored and
AlyaGomaa
committed
Fix the profiler handler for cases of nan in state

eldraco
authored and
AlyaGomaa
committed
flowml. If the dataset has one flow and that is deleted, then return empty fast.

eldraco
authored and
AlyaGomaa
committed
flowml. If the datasert is empty. Return none

eldraco
authored and
AlyaGomaa
committed
First new version of the model and scaler. Not good yet, but working.

eldraco
authored and
AlyaGomaa
committed
model and scaler with 1 malicious and 1 benign

eldraco
authored and
AlyaGomaa
committed
cleaner jupyter

eldraco
authored and
AlyaGomaa
committed
New models after 3rd ttrain

eldraco
authored and
AlyaGomaa
committed
Models after 4th train

eldraco
authored and
AlyaGomaa
committed
Models of ml flow with the first good performance in small tests

eldraco
authored and
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
state_handler: split get_final_state_from_flags() into smaller functions
AlyaGomaa
committed
state_handler: refactor get_final_state_from_flags()
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
state_handler: split get_final_state_from_flags() into smaller functions
AlyaGomaa
committed
state_handler: refactor get_final_state_from_flags()
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Add a function to convert the state again

eldraco
authored and
AlyaGomaa
committed
delete get_final_state_from_flags() from flowmldetection, profiler, and the db
AlyaGomaa
committed
mlflow. Ignore UID column

eldraco
authored and
AlyaGomaa
committed
Re add function that alya added

eldraco
authored and
AlyaGomaa
committed
Delete file that was deleted from develop

eldraco
authored and
AlyaGomaa
committed
Flowmldetection. Fix missing db reference

eldraco
authored and
AlyaGomaa
committed
Fix the training of flows with ML in new version

eldraco
authored and
AlyaGomaa
committed
flowml. If the dataset has one flow and that is deleted, then return empty fast.

eldraco
authored and
AlyaGomaa
committed
flowml. If the datasert is empty. Return none

eldraco
authored and
AlyaGomaa
committed
First new version of the model and scaler. Not good yet, but working.

eldraco
authored and
AlyaGomaa
committed
model and scaler with 1 malicious and 1 benign

eldraco
authored and
AlyaGomaa
committed
cleaner jupyter

eldraco
authored and
AlyaGomaa
committed
New models after 3rd ttrain

eldraco
authored and
AlyaGomaa
committed
Models after 4th train

eldraco
authored and
AlyaGomaa
committed
Models of ml flow with the first good performance in small tests

eldraco
authored and
AlyaGomaa
committed
Add plot for flowml train scores

eldraco
authored and
AlyaGomaa
committed
Add a log file to store the training data output

eldraco
authored and
AlyaGomaa
committed
Store data in the log file of training

eldraco
authored and
AlyaGomaa
committed
better comments

eldraco
authored and
AlyaGomaa
committed
Fix issue not dropping detailed labels

eldraco
authored and
AlyaGomaa
committed
Fix issue that not all labels sere given to the partial fit

eldraco
authored and
AlyaGomaa
committed
count partial labels in this epoch

eldraco
authored and
AlyaGomaa
committed
Dont print training in screen

eldraco
authored and
AlyaGomaa
committed
Add function to write to train log

eldraco
authored and
AlyaGomaa
committed
Fix label in dummy flow

eldraco
authored and
AlyaGomaa
committed
Fix dummy flow

eldraco
authored and
AlyaGomaa
committed
Rename variable

eldraco
authored and
AlyaGomaa
committed
Fix dummy flow label

eldraco
authored and
AlyaGomaa
committed
Pass values to train function

eldraco
authored and
AlyaGomaa
committed
import os

eldraco
authored and
AlyaGomaa
committed
Delete old comments

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Fix plot

eldraco
authored and
AlyaGomaa
committed
Fix plot

eldraco
authored and
AlyaGomaa
committed
Fix plot

eldraco
authored and
AlyaGomaa
committed
Plot testing performance from a log

eldraco
authored and
AlyaGomaa
committed
Fix the plot

eldraco
authored and
AlyaGomaa
committed
Fix the plots

eldraco
authored and
AlyaGomaa
committed
Fix plot

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Change plot names

eldraco
authored and
AlyaGomaa
committed
Rename file

eldraco
authored and
AlyaGomaa
committed
Recover good flowmldetection deleted by mistake

eldraco
authored and
AlyaGomaa
committed
Fix plot test

eldraco
authored and
AlyaGomaa
committed
Add testing code to evaluate performance. It is optional with a varible

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Fix train plot

eldraco
authored and
AlyaGomaa
committed
Fix plots

eldraco
authored and
AlyaGomaa
committed
Add performance metrics to the training evaluation

eldraco
authored and
AlyaGomaa
committed
Fix experiment names

eldraco
authored and
AlyaGomaa
committed
test_profiler: update unit tests
AlyaGomaa
committed
Fix that the training and testing logs files were appened instead of rewritten

eldraco
authored and
AlyaGomaa
committed
Fix an issue of storing the new log files

eldraco
authored and
AlyaGomaa
committed
enable/ disable training and testing.log with a param in the config file
AlyaGomaa
committed
dont create an empty logfile when create_performance_metrics_log_files is set to false
AlyaGomaa
committed
when enabled, create testing.log or training.log in the current output dir
AlyaGomaa
committed
Add an enum called labels with either Benign or Malicious so the labels are unified.
AlyaGomaa
committed
update branch with the latest develop
AlyaGomaa
committed

Commits on May 12, 2025

set the config label as the GT label if not founf in the given file
AlyaGomaa
committed

Commits on May 20, 2025

Commits on May 29, 2025

Fix plot testing bug with zeros
eldraco
committed

Commits on Jul 11, 2025

Improve the testing of performance
eldraco
committed

Commits on Oct 3, 2025

Commits on Nov 5, 2025

New model.bin model
eldraco
committed