why value in confusion matrix can decrease when evaluating (no normalization is applied)

[tabVersion]

I use multiple files as testsets, and I print confusion matrix when one file is completed. However, I find some value can decrease compared to the previous iteration. I checked the source code and I could not find any op can do so, which means the behavior is unexpected.

Here is the code:

test_acc = metrics.Accuracy()
confmat = metrics.ConfusionMatrix(num_classes=2)

def validate(model, dataloader, acc_rec, confusion_matrix):
    model.eval()
    output_total, label_total = None, None
    for rec, label in dataloader:
        outputs = model(rec.float())
        acc_rec(outputs, label)
        confusion_matrix(outputs, label)

for idx in range(0):
    with open(f'data/part-{idx}', 'r') as f:
        lines = [line.strip() for line in f]
    test_dataloader = DataLoader(RecordDataset(lines), batch_size=512, shuffle=True)
    validate(model, test_dataloader, test_acc, confmat)
    print(f"[test] epoch: {epoch}: test acc: {test_acc.compute()}, confusion: {confmat.compute()}")

And here are part of the output:

...
[test] epoch: 0: test acc: 0.7447031736373901, confusion: tensor([[77757.,  2197.],
        [25300.,  2452.]])
[test] epoch: 0: test acc: 0.7578601837158203, confusion: tensor([[66211.,  1665.],
        [20107.,  1932.]])
[test] epoch: 0: test acc: 0.7820931077003479, confusion: tensor([[74930.,  1711.],
        [19733.,  2035.]])
[test] epoch: 0: test acc: 0.826224684715271, confusion: tensor([[191914.,  10425.],
        [ 30831.,   4240.]])
[test] epoch: 0: test acc: 0.8618631958961487, confusion: tensor([[149753.,   8602.],  <-- drop from 191914 to 149753
        [ 15771.,   2315.]])
...