Pretrain epoch loss strange behaviour

[gustavengstrom]

Plotting epoch_loss from log.jsonl from pretraining which yields strange results. Not sure what is going wrong?!

Pretraining is on a Swedish language dataset consisting of approx. 7.6 miljon sentences and 129 million words...

Losses plotted from the log.jsonl file:

Which is generated using the following script:

import plotly.express as px
import pandas as pd
import json

def plot_losses(title, loss):
    df = pd.DataFrame([[i, loss] for i, loss in enumerate(loss)], columns=["epoch", "loss"])
    fig = px.line(df, x="epoch", y="loss", hover_name="epoch", line_shape="spline", render_mode="svg", title=title)
    fig.show()
    fig.write_image("fig1.png")
    
def get_losses(path):
    records = [json.loads(line) for line in open(path)]
    losses = [rec["epoch_loss"] for rec in records]
    return losses

loss = get_losses("log.jsonl")

plot_losses("Losses", loss)

The config pretraining section is taken directly from the pretraining section in the docs:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null
raw_text = "./corpus/riksdagen_unique_sents.jsonl"

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "sv"
pipeline = ["tok2vec","textcat"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.textcat]
factory = "textcat"
threshold = 0.5

[components.textcat.model]
@architectures = "spacy.TextCatEnsemble.v2"
nO = null

[components.textcat.model.linear_model]
@architectures = "spacy.TextCatBOW.v2"
exclusive_classes = true
ngram_size = 1
no_output_layer = false
nO = null

[components.textcat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 256
depth = 8
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.pretrain]
@readers = "spacy.JsonlCorpus.v1"
path = ${paths.raw_text}
min_length = 5
max_length = 500
limit = 0

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
cats_score = 1.0
cats_score_desc = null
cats_micro_p = null
cats_micro_r = null
cats_micro_f = null
cats_macro_p = null
cats_macro_r = null
cats_macro_f = null
cats_macro_auc = null
cats_f_per_type = null
cats_macro_auc_per_type = null

[pretraining]
max_epochs = 1000
dropout = 0.2
n_save_every = 5
component = "textcat"
layer = "tok2vec"
corpus = "corpora.pretrain"

[pretraining.batcher]
@batchers = "spacy.batch_by_words.v1"
size = 3000
discard_oversize = false
tolerance = 0.2
get_length = null

[pretraining.objective]
@architectures = "spacy.PretrainCharacters.v1"
maxout_pieces = 3
hidden_size = 300
n_characters = 4

[pretraining.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = true
eps = 0.00000001
learn_rate = 0.001

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[polm]

Thanks for the report, that does look weird.

I don't see anything wrong with your graph code, but could you upload the raw data for us to look at?

[polm]

The log file in the zip cuts off in the middle of a line in epoch 25...?

I found the issue though, I thought 80k was a lot of epochs. You're treating every line in the file as an epoch, but that's not the case. If you look at the actual epoch value you'll see it only goes up occasionally. The loss is accumulated through each epoch and reset at the start of the next epoch. So there's nothing weird about this at all.

[gustavengstrom]

Ah... I suspected this was the case since one can see how the peaks start to level off but I could not find any documentation with regards to how the losses are calculated. Thanks anhow!

[gustavengstrom]

Hi again, I amended the plot code which now produces a much smoother curve. I am however still a bit puzzled about the behaviour around epoch 57. Is this something you have seen before?!

import plotly.express as px
import pandas as pd
import json

def plot_losses(title, losses):
    df = pd.DataFrame(losses, columns=["epoch", "loss"])
    dfg= df.groupby('epoch').tail(1).reset_index()
    fig = px.line(dfg, x="epoch", y="loss", hover_name="epoch", line_shape="spline", render_mode="svg", title=title)
    fig.show()
    fig.write_image("fig1.png")
    
def get_losses(path):
    records = [json.loads(line) for line in open(path)]
    losses = [(rec["epoch"], rec["epoch_loss"]) for rec in records]
    return losses

loss = get_losses("/home/gustav/dev/projects/academic/climate-politics/riksdagen_project/pretraining/log.jsonl")
plot_losses("Losses", loss)

log.jsonl 2.zip

[polm]

Your graph is wrong again. Because of the way you're accumulating the loss you're only getting one point (the last one) for each epoch. Because your last epoch is shorter for some reason the cumulative epoch loss is much lower.

[gustavengstrom]

Ok. Thanks! The last epoch is clearly shorter since I was generating the plots while the training process was ongoing. My misstake...