Base strategies (LwF, Native) do not advance to the next experience after evaluating on the first experience

[JothamWong]

I am using the standard template from Avalanche's tutorial page "Putting it all together" found here https://avalanche.continualai.org/from-zero-to-hero-tutorial/6.-putting-all-together
I am looping through the scenario's train stream which has 8 experiences
The issue arises when I use one of the base strategies included in the Avalanche framework. The strategies that I have tried to use are LwF and Native.
What happens is that the strategy will train and evaluate fine, it will print the results to the standard output, and then just pause there. I have included a print statement after the strategy.eval line but it never prints.
I have tried to reproduce this issue with a strategy that I have implemented myself and it works fine, after the eval results are printed, it moves onto the next experience.
I'm hoping someone could help me find out the cause of this issue.

[Mattdl]

Could you provide the main script (with the main loop over experiences etc.) so we can have a look? Thanks!

[JothamWong]

from avalanche.evaluation.metrics import ExperienceForgetting, accuracy_metrics, loss_metrics, timing_metrics, \
    cpu_usage_metrics, StreamConfusionMatrix, disk_usage_metrics
from avalanche.logging import InteractiveLogger, TextLogger, TensorboardLogger
from avalanche.training.plugins import EvaluationPlugin

import utils


def parse_arguments():
    argparser = argparse.ArgumentParser("Run continual learning experiments")
    argparser.add_argument("--model", type=str)
    argparser.add_argument("--scenario", choices=["NI", "NIC", "NC"])
    argparser.add_argument("--dataset", choices=["MNIST", "CIFAR10", "Core50"])
    argparser.add_argument("--task_list", nargs="+")
    argparser.add_argument("--nb_tasks", type=int)
    argparser.add_argument("--optimizer", type=str, default="SGD")
    argparser.add_argument("--lr", type=float, default=0.001)
    argparser.add_argument("--momentum", type=float, default=0.9)
    argparser.add_argument("--criterion", type=str, default="CrossEntropyLoss")
    argparser.add_argument("--strategy", type=str, choices=["Naive", "DAT", "LwF"])
    argparser.add_argument("--epochs", type=int, default=10)
    argparser.add_argument("--batch_size", type=int, default=16)
    argparser.add_argument("--file_name", type=str, default="log.txt")
    argparser.add_argument("--n_classes", type=int)
    return argparser.parse_args()


print("Parsing arguments...")
args = parse_arguments()
print("Arguments parsed.")
scenario = utils.get_scenario(args)
model = utils.get_model(scenario, args)

tb_logger = TensorboardLogger()
filename = args.file_name
text_logger = TextLogger(open(filename, "a"))
interactive_logger = InteractiveLogger()

eval_plugin = EvaluationPlugin(
    accuracy_metrics(minibatch=True, epoch=True, epoch_running=True, experience=True, stream=True),
    loss_metrics(minibatch=True, epoch=True, epoch_running=True, experience=True, stream=True),
    timing_metrics(minibatch=True, epoch=True, epoch_running=True, experience=True, stream=True),
    cpu_usage_metrics(minibatch=True, epoch=True, epoch_running=True, experience=True, stream=True),
    ExperienceForgetting(),
    StreamConfusionMatrix(num_classes=args.n_classes, save_image=True),
    disk_usage_metrics(minibatch=True, epoch=True, experience=True, stream=True),
    loggers=[interactive_logger, text_logger, tb_logger]
)

cl_strategy = utils.get_strategy(model, eval_plugin, args)

print("Starting experiment...")
print(len(scenario.train_stream))

for experience in scenario.train_stream:
    print("Start of experience ", experience.current_experience)
    print("Current classes: ", experience.classes_in_this_experience)

    cl_strategy.train(experience)
    print("Training complete.")

    print("Computing accuracy on the test set")
    cl_strategy.eval(scenario.test_stream)
    print("End of experience ", experience.current_experience)

Hi here is the main script. When I use my own custom strategy, it works fine, and the next experience is reached.
When I use any of the base strategies, the results of the test stream is printed to the Interactive Logger and the next print line statement is never executed.

[vlomonaco]

Hi @psychicmario, it seems the problem is related to the StreamConfusionMatrix metric, can you try re-run your script commenting out that metric? On my Linux server with a fresh env it doesn't stall anymore.

[AndreaCossu]

Hi @psychicmario ! We are working on solving the issue, it should not take long. You can track the advancements here #513

[AndreaCossu]

#518 should have solved this. Please, @psychicmario try again and see if it works now. Check out the class documentation of ConfusionMatrix and StreamConfusionMatrix for more details.

[vlomonaco]

Marking this as solved due to inactivity, @psychicmario let us know in case you want to re-open it.

[JothamWong]

Hey, sorry for the late reply. I just tested my script with the ConfusionMatrix metric and I was able to get it training without a delay. Thanks @AndreaCossu, @vlomonaco for solving the problem :)