[ehealth] add sequence classification example

Author: LemonNoel

examples/biomedical/cblue/sequence_classification/README.md

@@ -0,0 +1,63 @@
+# 使用医疗领域预训练模型Fine-tune完成中文医疗文本分类任务
+
+近年来，预训练语言模型（Pre-trained Language Model，PLM）逐渐成为自然语言处理（Natural Language Processing，NLP）的主流方法。这类模型可以利用大规模的未标注语料进行训练，得到的模型在下游NLP任务上效果明显提升，在通用领域和特定领域均有广泛应用。在医疗领域，早期的做法是在预先训练好的通用语言模型上进行Fine-tune。后来的研究发现直接使用医疗相关语料学习到的预训练语言模型在医疗文本任务上的效果更好，采用的模型结构也从早期的BERT演变为更新的RoBERTa、ALBERT和ELECTRA。
+
+本示例展示了中文医疗预训练模型eHealth（[Building Chinese Biomedical Language Models via Multi-Level Text Discrimination](https://arxiv.org/abs/2110.07244)）如何Fine-tune完成中文医疗文本分类任务。
+
+## 模型介绍
+
+本项目针对中文医疗文本分类任务，开源了中文医疗预训练模型eHealth（简写`chinese-ehealth`）。eHealth（[Building Chinese Biomedical Language Models via Multi-Level Text Discrimination](https://arxiv.org/abs/2110.07244)）使用了医患对话、科普文章、病历档案、临床病理学教材等脱敏中文语料进行预训练，通过预训练任务设计来学习词级别和句级别的文本信息。该模型的整体结构与ELECTRA相似，包括生成器和判别器两部分。 而Fine-tune过程只用到了判别器模块，由12层Transformer网络组成。
+
+## 快速开始
+
+### 代码结构说明
+
+以下是本项目主要代码结构及说明：
+
+```text
+sequence_classification/
+├── README.md # 使用说明
+└── train.py # 训练评估脚本
+```
+
+### 模型训练
+
+我们以中文医疗文本数据集CBLUE中的文本分类数据集为示例数据集，包括：
+
+* CHIP-CDN：给定病历档案，预测其中包含的规范化诊断实体。本项目使用了检索后重新构建的二分类数据集，给定病历档案和规范化诊断实体，预测前者是否包含后者（简写`CHIP-CDN-2C`）。
+* CHIP-CTC：给定医疗文本描述，按照中国临床筛选标准进行分类。
+* CHIP-STS：给定两个涉及5种不同疾病的句子，预测二者语义是否相似。
+* KUAKE-QIC：给定医疗问句，对患者咨询目的进行分类。
+* KUAKE-QTR：给定医疗问句和文章标题，预测二者内容是否一致。
+* KUAKE-QQR：给定两个医疗问句，预测二者描述内容是否一致。
+
+可以运行下边的命令，在训练集上进行训练，并在开发集上进行验证。
+```shell
+$ unset CUDA_VISIBLE_DEVICES
+$ python -m paddle.distributed.launch --gpus "0" train.py --dataset CHIP-CDN-2C --batch_size 256 --max_seq_length 32 --weight_decay 0.01 --warmup_proportion 0.1
+```
+
+可支持配置的参数：
+
+* `save_dir`：可选，保存训练模型的目录；默认保存在当前目录checkpoints文件夹下。
+* `dataset`：可选，CHIP-CDN-2C CHIP-CTC CHIP-STS KUAKE-QIC KUAKE-QTR KUAKE-QQR，默认为KUAKE-QIC数据集。
+* `max_seq_length`：可选，ELECTRA模型使用的最大序列长度，最大不能超过512, 若出现显存不足，请适当调低这一参数；默认为128。
+* `batch_size`：可选，批处理大小，请结合显存情况进行调整，若出现显存不足，请适当调低这一参数；默认为32。
+* `learning_rate`：可选，Fine-tune的最大学习率；默认为6e-5。
+* `weight_decay`：可选，控制正则项力度的参数，用于防止过拟合，默认为0.01。
+* `epochs`: 训练轮次，默认为3。
+* `valid_steps`: evaluate的间隔steps数，默认100。
+* `save_steps`: 保存checkpoints的间隔steps数，默认100。
+* `logging_steps`: 日志打印的间隔steps数，默认10。
+* `warmup_proption`：可选，学习率warmup策略的比例，如果0.1，则学习率会在前10%训练step的过程中从0慢慢增长到learning_rate, 而后再缓慢衰减，默认为0.1。
+* `init_from_ckpt`：可选，模型参数路径，热启动模型训练；默认为None。
+* `seed`：可选，随机种子，默认为1000.
+* `device`: 选用什么设备进行训练，可选cpu或gpu。如使用gpu训练则参数gpus指定GPU卡号。
+* `use_amp`: 是否使用混合精度训练，默认为False。
+* `use_ema`: 是否使用Exponential Moving Average预测，默认为False。
+
+### 依赖安装
+
+```shell
+pip install xlrd==1.2.0
+```

examples/biomedical/cblue/sequence_classification/train.py

@@ -0,0 +1,270 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from functools import partial
+import argparse
+import os
+import random
+import time
+import distutils.util
+
+import numpy as np
+import paddle
+import paddle.nn.functional as F
+from paddle.metric import Accuracy
+import paddlenlp as ppnlp
+from paddlenlp.data import Stack, Tuple, Pad
+from paddlenlp.datasets import load_dataset
+from paddlenlp.transformers import LinearDecayWithWarmup
+from paddlenlp.metrics import MultiLabelsMetric
+from paddlenlp.ops.optimizer import ExponentialMovingAverage
+
+from utils import convert_example
+
+METRIC_CLASSES = {
+    'KUAKE-QIC': Accuracy,
+    'KUAKE-QQR': Accuracy,
+    'KUAKE-QTR': Accuracy,
+    'CHIP-CTC': partial(
+        MultiLabelsMetric, name='macro'),
+    'CHIP-STS': partial(
+        MultiLabelsMetric, name='macro'),
+    'CHIP-CDN-2C': partial(
+        MultiLabelsMetric, name='micro')
+}
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument('--dataset', choices=['KUAKE-QIC', 'KUAKE-QQR', 'KUAKE-QTR', 'CHIP-STS', 'CHIP-CTC', 'CHIP-CDN-2C'],
+                                 default='KUAKE-QIC', type=str, help='Dataset for token classfication tasks.')
+parser.add_argument('--seed', default=1000, type=int, help='Random seed for initialization.')
+parser.add_argument('--device', choices=['cpu', 'gpu', 'xpu', 'npu'], default='gpu', help='Select which device to train model, default to gpu.')
+parser.add_argument('--epochs', default=3, type=int, help='Total number of training epochs to perform.')
+parser.add_argument('--batch_size', default=32, type=int, help='Batch size per GPU/CPU for training.')
+parser.add_argument('--learning_rate', default=6e-5, type=float, help='Learning rate for fine-tuning sequence classification task.')
+parser.add_argument('--weight_decay', default=0.01, type=float, help="Weight decay if we apply some.")
+parser.add_argument('--warmup_proportion', default=0.1, type=float, help='Linear warmup proportion over the training process.')
+parser.add_argument('--max_seq_length', default=128, type=int, help='The maximum total input sequence length after tokenization.')
+parser.add_argument('--init_from_ckpt', default=None, type=str, help='The path of checkpoint to be loaded.')
+parser.add_argument('--logging_steps', default=10, type=int, help='The interval steps to logging.')
+parser.add_argument('--save_dir', default='./checkpoint', type=str, help='The output directory where the model checkpoints will be written.')
+parser.add_argument('--save_steps', default=100, type=int, help='The interval steps to save checkppoints.')
+parser.add_argument('--valid_steps', default=100, type=int, help='The interval steps to evaluate model performance.')
+parser.add_argument('--use_ema', default=False, type=bool, help='Use exponential moving average for evaluation.')
+parser.add_argument('--use_amp', default=False, type=distutils.util.strtobool, help='Enable mixed precision training.')
+parser.add_argument('--scale_loss', default=128, type=float, help='The value of scale_loss for fp16.')
+
+args = parser.parse_args()
+# yapf: enable
+
+
+def set_seed(seed):
+    """set random seed"""
+    random.seed(seed)
+    np.random.seed(seed)
+    paddle.seed(seed)
+
+
+@paddle.no_grad()
+def evaluate(model, criterion, metric, data_loader):
+    """
+    Given a dataset, it evals model and compute the metric.
+
+    Args:
+        model(obj:`paddle.nn.Layer`): A model to classify texts.
+        dataloader(obj:`paddle.io.DataLoader`): The dataset loader which generates batches.
+        criterion(obj:`paddle.nn.Layer`): It can compute the loss.
+        metric(obj:`paddle.metric.Metric`): The evaluation metric.
+    """
+    model.eval()
+    metric.reset()
+    losses = []
+    for batch in data_loader:
+        input_ids, token_type_ids, position_ids, labels = batch
+        logits = model(input_ids, token_type_ids, position_ids)
+        loss = criterion(logits, labels)
+        losses.append(loss.numpy())
+        correct = metric.compute(logits, labels)
+        metric.update(correct)
+    if isinstance(metric, Accuracy):
+        metric_name = 'accuracy'
+        result = metric.accumulate()
+    else:
+        metric_name = metric._name + ' f1'
+        _, _, result = metric.accumulate(metric._name)
+    print('eval loss: %.5f, %s: %.5f' % (np.mean(losses), metric_name, result))
+    model.train()
+    metric.reset()
+
+
+def create_dataloader(dataset,
+                      mode='train',
+                      batch_size=1,
+                      batchify_fn=None,
+                      trans_fn=None):
+    if trans_fn:
+        dataset = dataset.map(trans_fn)
+
+    shuffle = True if mode == 'train' else False
+    if mode == 'train':
+        batch_sampler = paddle.io.DistributedBatchSampler(
+            dataset, batch_size=batch_size, shuffle=shuffle)
+    else:
+        batch_sampler = paddle.io.BatchSampler(
+            dataset, batch_size=batch_size, shuffle=shuffle)
+
+    return paddle.io.DataLoader(
+        dataset=dataset,
+        batch_sampler=batch_sampler,
+        collate_fn=batchify_fn,
+        return_list=True)
+
+
+def do_train():
+    paddle.set_device(args.device)
+    rank = paddle.distributed.get_rank()
+    if paddle.distributed.get_world_size() > 1:
+        paddle.distributed.init_parallel_env()
+
+    set_seed(args.seed)
+
+    train_ds, dev_ds, test_ds = load_dataset(
+        'cblue', args.dataset, splits=['train', 'dev', 'test'])
+
+    model = ppnlp.transformers.ElectraForSequenceClassification.from_pretrained(
+        'chinese-ehealth', num_classes=len(train_ds.label_list))
+    tokenizer = ppnlp.transformers.ElectraTokenizer.from_pretrained(
+        'chinese-ehealth')
+
+    trans_func = partial(
+        convert_example,
+        tokenizer=tokenizer,
+        max_seq_length=args.max_seq_length)
+    batchify_fn = lambda samples, fn=Tuple(
+        Pad(axis=0, pad_val=tokenizer.pad_token_id),  # input
+        Pad(axis=0, pad_val=tokenizer.pad_token_type_id),  # segment
+        Pad(axis=0, pad_val=args.max_seq_length - 1),  # position
+        Stack(dtype='int64')): [data for data in fn(samples)]
+    train_data_loader = create_dataloader(
+        train_ds,
+        mode='train',
+        batch_size=args.batch_size,
+        batchify_fn=batchify_fn,
+        trans_fn=trans_func)
+    dev_data_loader = create_dataloader(
+        dev_ds,
+        mode='dev',
+        batch_size=args.batch_size,
+        batchify_fn=batchify_fn,
+        trans_fn=trans_func)
+
+    if args.init_from_ckpt and os.path.isfile(args.init_from_ckpt):
+        state_dict = paddle.load(args.init_from_ckpt)
+        model.set_dict(state_dict)
+    model = paddle.DataParallel(model)
+
+    num_training_steps = len(train_data_loader) * args.epochs
+
+    lr_scheduler = LinearDecayWithWarmup(args.learning_rate, num_training_steps,
+                                         args.warmup_proportion)
+
+    # Generate parameter names needed to perform weight decay.
+    # All bias and LayerNorm parameters are excluded.
+    decay_params = [
+        p.name for n, p in model.named_parameters()
+        if not any(nd in n for nd in ['bias', 'norm'])
+    ]
+
+    optimizer = paddle.optimizer.AdamW(
+        learning_rate=lr_scheduler,
+        parameters=model.parameters(),
+        weight_decay=args.weight_decay,
+        apply_decay_param_fun=lambda x: x in decay_params)
+
+    criterion = paddle.nn.loss.CrossEntropyLoss()
+    if METRIC_CLASSES[args.dataset] is Accuracy:
+        metric = METRIC_CLASSES[args.dataset]()
+        metric_name = 'accuracy'
+    else:
+        metric = METRIC_CLASSES[args.dataset](
+            num_labels=len(train_ds.label_list))
+        metric_name = metric._name + ' f1'
+    if args.use_amp:
+        scaler = paddle.amp.GradScaler(init_loss_scaling=args.scale_loss)
+    if args.use_ema and rank == 0:
+        ema = ExponentialMovingAverage(model)
+        ema.register()
+    global_step = 0
+    tic_train = time.time()
+    total_train_time = 0
+    for epoch in range(1, args.epochs + 1):
+        for step, batch in enumerate(train_data_loader, start=1):
+            input_ids, token_type_ids, position_ids, labels = batch
+            with paddle.amp.auto_cast(
+                    args.use_amp,
+                    custom_white_list=['layer_norm', 'softmax', 'gelu'], ):
+                logits = model(input_ids, token_type_ids, position_ids)
+                loss = criterion(logits, labels)
+            probs = F.softmax(logits, axis=1)
+            correct = metric.compute(probs, labels)
+            metric.update(correct)
+
+            if isinstance(metric, Accuracy):
+                result = metric.accumulate()
+            else:
+                _, _, result = metric.accumulate(metric._name)
+
+            if args.use_amp:
+                scaler.scale(loss).backward()
+                scaler.minimize(optimizer, loss)
+            else:
+                loss.backward()
+                optimizer.step()
+            lr_scheduler.step()
+            if args.use_ema and rank == 0:
+                ema.update()
+            optimizer.clear_grad()
+
+            global_step += 1
+            if global_step % args.logging_steps == 0 and rank == 0:
+                time_diff = time.time() - tic_train
+                total_train_time += time_diff
+                print(
+                    'global step %d, epoch: %d, batch: %d, loss: %.5f, %s: %.5f, speed: %.2f step/s'
+                    % (global_step, epoch, step, loss, metric_name, result,
+                       args.logging_steps / time_diff))
+                tic_train = time.time()
+
+            if global_step % args.valid_steps == 0 and rank == 0:
+                if args.use_ema:
+                    ema.apply_shadow()
+                    evaluate(model, criterion, metric, dev_data_loader)
+                    ema.restore()
+                else:
+                    evaluate(model, criterion, metric, dev_data_loader)
+                tic_train = time.time()
+
+            if global_step % args.save_steps == 0 and rank == 0:
+                save_dir = os.path.join(args.save_dir, 'model_%d' % global_step)
+                if not os.path.exists(save_dir):
+                    os.makedirs(save_dir)
+                model._layers.save_pretrained(save_dir)
+                tokenizer.save_pretrained(save_dir)
+                tic_train = time.time()
+
+    print('Speed: %.2f steps/s' % (global_step / total_train_time))
+
+
+if __name__ == "__main__":
+    do_train()