Replicate v2.3 textcat in 3.3 config

[python3Berg]

Hello,
Summer's the time for some long overdue maintenance, but I am having a puzzling issue with the most basic of my pipelines. Using spacy 2.3, I can train a classifier to about a 93% score over the course of 40-50 epochs. It quickly reaches about 70% about 2-3 and then slow evolution. In spacy 3.3, I cannot achieve even 50% regardless of the epochs, architecture or learning rate. My data has about 150 labels trained across about 8000, often large documents. It is normalized to provide a set of training and testing split at the classification level and then randomized. Lots of stuff below, but looking for ideas here. Eager to take advantage of new 3.3 capabilities but must first reestablish baseline. Has anyone experienced same?

Spacy 2.3 pipeline code is almost a cut and paste from old docs...

def main(model='devModel',  n_iter=100, n_texts=2500,version='2.3',clean=0):
    output_dir = os.path.join('/home/training',version,'models','docClassifier',model)
    model = output_dir
    try:
        nlp = spacy.load(output_dir)  # load existing spaCy model
        print("Loaded model '%s'" % model)
    except:
        nlp = spacy.blank('en')  # create blank Language class
        print("Created blank 'en' model")

    if 'textcat' in nlp.pipe_names and clean:
        nlp.remove_pipe('textcat')

    addLabels = False
    if 'textcat' not in nlp.pipe_names:
        addLabels=True
        textcat = nlp.create_pipe('textcat')
        nlp.add_pipe(textcat, last=True)
    else:
        textcat = nlp.get_pipe('textcat')

    limit=0
    split=.80
    docSorter = defaultdict(list)
    train_data=[]
    traincats=[]
    useCats=[]
    
    rc1 = cur.execute("select classification from document_class_map where sub_document_type = 1")
    catData = cur.fetchall()
    rc = cur.execute("""select classification, text,document_id,sub_document_id 
    from sub_document 
    where classification is not null and classification != '' and classification != 'Not Classified' and reviewed=1""")
    data1 = cur.fetchall()

    for row in catData:
        cat = row[0]
        useCats.append(cat)
    dcount=0
    for row in data1:
        txt = row[1]
        txt = txt.strip()
        if len(txt) > 300000:
            txt= txt[0:300000]
        docSorter[row[0]].append([txt,row[0]])
        dcount+=1
        # train_data.append([txt,row[0]])
        
    train_data=[]
    test_data=[]

    for k in sorted(docSorter):
        v = docSorter[k]
        traincats.append(k)
        random.shuffle(v)
        dsplit = max(2,int(len(v) * split))
        trd = v[:dsplit]
        train_data.extend(trd)
        tsd = v[dsplit:]
        test_data.extend(tsd)
        print(k,"training: ",len(trd),'testing: ',len(tsd))

    useCats=set(traincats)
    random.shuffle(train_data)
    train_texts, labels = zip(*train_data)
    train_cats=[]
    for lab in labels:
        cd = {}                           ##set 0 or 1 values for all labels
        for cat in useCats:
            cd[cat] = bool(cat==lab)
        train_cats.append(cd)

    random.shuffle(test_data)
    dev_texts, labels = zip(*test_data)
    dev_cats=[]
    for lab in labels:
        cd = {}
        for cat in useCats:
            cd[cat] = bool(cat==lab)
        dev_cats.append(cd)

    if addLabels:
        for lab in useCats:
            textcat.add_label(lab)

    print("Using {} examples ({} training, {} evaluation)"
          .format(n_texts, len(train_texts), len(dev_texts)))
    train_data = list(zip(train_texts,
                          [{'cats': cats} for cats in train_cats]))

    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'textcat']
    with nlp.disable_pipes(*other_pipes):  # only train textcat
        optimizer = nlp.begin_training()
        print("Training the model...")
        print('{:^5}\t{:^5}\t{:^5}\t{:^5}'.format('LOSS', 'P', 'R', 'F'))
        for i in range(n_iter):
            sTime = time.ctime()
            print("Iteration",i,sTime)
            losses = {}
            batches = minibatch(train_data, size=compounding(4., 20., 1.002))
            for batch in batches:
                texts, annotations = zip(*batch)
                nlp.update(texts, annotations, sgd=optimizer, drop=0.2,
                           losses=losses)
            with textcat.model.use_params(optimizer.averages):
                eTime=time.ctime()
                scores = evaluate(nlp.tokenizer, textcat, dev_texts, dev_cats)
                print("Interation: {} Loss: {} P: {} R: {} F:{} Start: {} End: {}".format(i, losses['textcat'], scores['textcat_p'], scores['textcat_r'], scores['textcat_f'], sTime, eTime))
            if model is not None:
                output_dir = Path(model)
                if not output_dir.exists():
                    output_dir.mkdir()
                nlp.to_disk(output_dir)
                print("Saved model # TODO: o", output_dir)

---

I am using a very similar normalization to generate the spacy binary files required by 3.3...

def main(version='3.3'):
    dataPath= os.path.join('C:/home/training/',version,'docClassifier')

    trainPath= os.path.join(dataPath,'train.spacy')
    testPath= os.path.join(dataPath,'test.spacy')
    labelPath= os.path.join(dataPath,'textcat.json')
    print("Training sub classifier...loading data")
    docSorter = defaultdict(list)

    cur = connections[0].cursor()
    rc = cur.execute("""select distinct classification from document_class_map where sub_document_type=1""")
    traincats = [r[0] for r in cur.fetchall()]
    cur.close()
    with open(labelPath, 'w') as f:
        json.dump(traincats,f)

    for idx,con in enumerate(connections):
        cur = con.cursor()
        rc = cur.execute("""select classification, text,document_id,sub_document_id 
        from sub_document 
        where classification is not null and classification != '' and classification != 'Not Classified' and reviewed=1""")
        
        data1 = cur.fetchall()
        for row in data1:
            txt = row[1]
            txt = txt.strip()
            docSorter[row[0]].append(txt)
        cur.close()
        
    nlp = spacy.blank("en")
    dbTrain = DocBin()
    dbTest = DocBin()
    preTrain=DocBin()
    
    print("{} documents found".format(len(data1)))
    totTrain=0
    totTest=0
    for k in sorted(docSorter):
        trainCount=0
        testCount=0
        v = docSorter[k]
        random.shuffle(v)
        if k in dumb:
            v = v[:200]
        for vidx,txt in enumerate(v):
            doc = nlp(txt)  ##have also tried nlp.make_doc(txt)
            doc.cats[k]=1.0
            # cd = {}  ##have also tried setting all values since it was unclear if all architectures could use single.
            # for cat in traincats:
            #     cd[cat] = 1.0 if k == cat else 0.0
            # doc.cats = cd
            if vidx % 5 == 0 and vidx != 0:
                dbTest.add(doc)
                testCount+=1
            else:
                dbTrain.add(doc)
                trainCount+=1
        totTrain += trainCount
        totTest += testCount
        print("Document: {} Training Count: {} Testing Count: {}".format(k,trainCount,testCount))
        
    dbTrain.to_disk(trainPath)
    dbTest.to_disk(testPath)

    print("Document: TOTAL Training Count: {} Testing Count: {}".format(totTrain,totTest))

---

I've tried the fast/simple and the slow/accurate configs generated from docs. I settled on the textcat config in project/templates/textcat_demo. Using the ensemble model with embeddings seems closest to 2.3 defaults.

Working config...

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
seed = 0
gpu_allocator = null

[nlp]
lang = "en"
pipeline = ["textcat"]
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 10000
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.textcat]
factory = "textcat"
threshold = 0.5

[components.textcat.model]
@architectures = "spacy.TextCatEnsemble.v2"
nO = null

[components.textcat.model.linear_model]
@architectures = "spacy.TextCatBOW.v1"
exclusive_classes = true
ngram_size = 1
no_output_layer = false
nO = null

[components.textcat.model.tok2vec]
@architectures = "spacy.Tok2Vec.v2"

[components.textcat.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v1"
width = 64
rows = [2000,2000,1000,1000,1000,1000]
attrs = ["ORTH","LOWER","PREFIX","SUFFIX","SHAPE","ID"]
include_static_vectors = false

[components.textcat.model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 64
window_size = 1
maxout_pieces = 3
depth = 2

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
gold_preproc = false
max_length = 0
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
gold_preproc = false
max_length = 0
limit = 0
augmenter = null

[training]
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.2
accumulate_gradient = 1
patience = 2000
max_epochs = 0
max_steps = 100000
eval_frequency = 100
frozen_components = []
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_sequence.v1"
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 4
stop = 30
compound = 1.05
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.002

[training.score_weights]
cats_score_desc = null
cats_micro_p = null
cats_micro_r = null
cats_micro_f = null
cats_macro_p = null
cats_macro_r = null
cats_macro_f = null
cats_macro_auc = null
cats_f_per_type = null
cats_macro_auc_per_type = null
cats_score = 1.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.components.textcat]

[initialize.components.textcat.labels]
@readers = "spacy.read_labels.v1"
path = "./docClassifier/textcat.json"


[initialize.tokenizer]

[svlandeg]

My data has about 150 labels trained across about 8000, often large documents

And just to double check, the labels are exclusive, right? One doc has exactly one label in your gold standard?

[python3Berg]

I was not aware that textcat in v2 ran multilabel by default...particularly when the "What's new" for 3.0 specified multilabel as a new trainable pipeline. To test impact, I updated config to use textcat_multilabel pipeline. After 1 epoch, it is now testing better than after 20 with exclusive categories. This is particularly confusing since every tuple has only one classification = 1.0, with the rest 0. My documents can be quite similar...is it possible that the multilabel pipeline is more tolerant of this similarity? Is it possible that exclusive category would eventualy move past the 5-10 epochs of hovering about 45% and move on the better accurancy? Very interesting and very confusing...

[svlandeg]

I know it's confusing - it was a bit of a necessary breaking change we had to make between v2 and v3. We do have a note on this API change in the "What's new in 3.0"

The textcat pipeline component is now only applicable for classification of mutually exclusives classes - i.e. one predicted class per input sentence or document. To perform multi-label classification, use the new textcat_multilabel component instead.

and also on the textcat API page: https://spacy.io/api/textcategorizer

But anyway, while the behaviour does sound weird I'm happy to hear that at least it seems like you're getting better performance now. Can you now get to the same F-scores when running training/testing on the same datasets for v2 and v3?

It's difficult to answer your follow-up questions without having gone through the datasets/experiments myself in detail, but I guess it could be an interesting error analysis on your end to try and find those cases that the textcat configuration consistently gets wrong (as opposed to multilabel_textcat) - maybe that gives a hint as to what could be going on. For instance, if you have two very similar docs but one labeled A and another B, then yes the multilabel one could be more robust to training on that.

[python3Berg]

Using textcat_multilabel finishes at score around 59%...better, but obliviously nowhere close to v2.3 93% results...

[python3Berg]

I think I have an answer here and perhaps need some help to replicate my evaluate function vs your. It appears that the scores presented during training are an arithmetic mean of the scores of each category. My 2.3 scores used a weighted average. Looking at the spacy evaluate results (could I display that result every batch instead of current summary?), the documents where my sample size is large all score 1.0, where those that are smaller score None or 0. Not surprising....Is there a way to create a custom evaluate in the config? Thanks

[adrianeboyd]

As of spacy v3.2 all components support registered scorers. It does look like the textcat API docs got missed in the update (I will get that updated), but you can see the registered scorer in the config:

[components.textcat]
factory = "textcat"
scorer = {"@scorers":"spacy.textcat_scorer.v1"}
threshold = 0.5

You can replace this with any scorer.