AGEM hyperparameter tuning

[en-tropyc]

Hi all,

I'm trying to tune AGEM's hyperparameters with the code below.

model = LeNet()
optimizer = SGD(model.parameters(), lr=0.001, momentum=0.9)
criterion = CrossEntropyLoss()

cl_strategy = AGEM(
    model, optimizer, criterion, 
    train_mb_size=50, eval_mb_size=50, 
    train_epochs=3,
    evaluator=eval_plugin,
    # ewc_lambda=0.01,
    patterns_per_exp=10,
    sample_size=500,
    device=device
)

patterns_per_experience = [10, 30]
sample_size = [250, 500]

for ppe in patterns_per_experience:
    for ss in sample_size:
        # Class-IL scenario creation
        scenario = PermutedMNIST2(
            n_experiences=10,
            seed=0)

        train_stream = scenario.train_stream
        test_stream = scenario.test_stream

        # TRAINING LOOP
        print('Starting experiment...')
        results = []
        for experience in scenario.train_stream:
            print("Start of experience: ", experience.current_experience)
            print("Current Classes: ", experience.classes_in_this_experience)
            
            res = cl_strategy.train(experience)
            print('Training completed')

            print('Computing accuracy on test set')
            results.append(cl_strategy.eval(scenario.test_stream[0:experience.current_experience+1]))

In my output logs, I noticed the following pattern:
For first set of hyperparams (ppe = 10, ss = 250):

-- Starting eval on experience 0 (Task 0) from test stream --
> Eval on experience 0 (Task 0) from test stream ended.
	Loss_Exp/eval_phase/test_stream/Task000/Exp000 = 7.9767
	Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0000
-- Starting eval on experience 1 (Task 0) from test stream --
> Eval on experience 1 (Task 0) from test stream ended.
	Loss_Exp/eval_phase/test_stream/Task000/Exp001 = 0.2386
	Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.9268
-- >> End of eval phase << --
	Loss_Stream/eval_phase/test_stream = 4.1076
	Top1_Acc_Stream/eval_phase/test_stream = 0.4634 

For last set of hyperparams (ppe = 30, ss = 500):

-- >> Start of eval phase << --
-- Starting eval on experience 0 (Task 0) from test stream --
> Eval on experience 0 (Task 0) from test stream ended.
	Loss_Exp/eval_phase/test_stream/Task000/Exp000 = 0.6890
	Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.7993
-- Starting eval on experience 1 (Task 0) from test stream --
> Eval on experience 1 (Task 0) from test stream ended.
	Loss_Exp/eval_phase/test_stream/Task000/Exp001 = 0.1504
	Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.9560
-- >> End of eval phase << --
	Loss_Stream/eval_phase/test_stream = 0.4197
	Top1_Acc_Stream/eval_phase/test_stream = 0.8777

However, if I train only using last set of hyperparams (ppe = 30, ss = 500):

-- >> Start of eval phase << --
-- Starting eval on experience 0 (Task 0) from test stream --
> Eval on experience 0 (Task 0) from test stream ended.
	Loss_Exp/eval_phase/test_stream/Task000/Exp000 = 8.0198
	Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0000
-- Starting eval on experience 1 (Task 0) from test stream --
> Eval on experience 1 (Task 0) from test stream ended.
	Loss_Exp/eval_phase/test_stream/Task000/Exp001 = 0.2302
	Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.9292
-- >> End of eval phase << --
	Loss_Stream/eval_phase/test_stream = 4.1250
	Top1_Acc_Stream/eval_phase/test_stream = 0.4646

So there is clearly a bug in the code I've written. I can't quite put a finger on it, but perhaps I need to re-initialize the model before each iteration of tuning? I'm also confused about the 0.000 eval accuracy since the lower bound for completely random guesses should be ~10%, yet later on in the tuning process the eval accuracy for the same experience improves to ~80%. Any ideas for how I should go about hyperparameter tuning correctly?

[vlomonaco]

@AndreaCossu maybe you had a bit more experience with tuning AGEM and you can give @christam96 a quick feedback?
I don't know if this is related to Avalanche though. But maybe should be discussed in https://github.com/ContinualAI/reproducible-continual-learning if we discover a bug in AGEM.

[AndreaCossu]

Hi @christam96 ! The AGEM is currently being fixed (there is an open PR about it) because it does not use multi-head implementation. With single-head, catastrophic forgetting can completely destroy the previous performance, making it lower than the random one for certain configurations. The original AGEM implementation does not work well in class-incremental scenarios.

[en-tropyc]

Ah, thank you for the response. I'm seeing similar behavior (0.000 eval accuracy for all previously seen tasks) using EWC in class-IL as well. I know EWC does not work well in this scenario, but could a multi-head approach prevent catastrophic forgetting from completely destroying previous performance in this scenario too?

[AndreaCossu]

Yes, EWC in class-incremental may produce zero accuracies, that behavior is possible. Using multi-head usually helps. You can use the avalanche multi-headed model (you have to use task labels in your CL benchmark).

[en-tropyc]

Will try, thanks for the help :)