Fine-tuning with DocLayNet

[yflam1]

This may not be a proper request, but can you try running the following script

import os
import deepdoctection as dd

doclaynet = dd.get_dataset("doclaynet")

doclaynet.dataflow.categories.set_cat_to_sub_cat({
dd.LayoutType.caption: dd.DatasetType.publaynet,
dd.LayoutType.footnote: dd.DatasetType.publaynet,
dd.LayoutType.formula: dd.DatasetType.publaynet,
dd.LayoutType.list: dd.DatasetType.publaynet,
dd.LayoutType.page_footer: dd.DatasetType.publaynet,
dd.LayoutType.page_header: dd.DatasetType.publaynet,
dd.LayoutType.figure: dd.DatasetType.publaynet,
dd.LayoutType.section_header: dd.DatasetType.publaynet,
dd.LayoutType.table: dd.DatasetType.publaynet,
dd.LayoutType.text: dd.DatasetType.publaynet,
dd.LayoutType.title: dd.DatasetType.publaynet})

doclaynet.dataflow.categories._categories_update = [dd.LayoutType.text,
                                                    dd.LayoutType.title,
                                                    dd.LayoutType.list,
                                                    dd.LayoutType.table,
                                                    dd.LayoutType.figure]

config_yaml_path = dd.ModelCatalog.get_full_path_configs("dd/d2/layout/CASCADE_RCNN_R_50_FPN_GN.yaml")
weights_path = dd.ModelDownloadManager.maybe_download_weights_and_configs("layout/d2_model_0829999_layout_inf_only.pt")

coco_metric = dd.get_metric("coco")

config_overwrite=["SOLVER.MAX_ITER=100000",
                  "TEST.EVAL_PERIOD=20000",
                  "SOLVER.CHECKPOINT_PERIOD=20000",
                  "MODEL.BACKBONE.FREEZE_AT=0",
                  "SOLVER.BASE_LR=1e-3"]

build_train_config = ["max_datapoints=86000"]

dd.train_d2_faster_rcnn(
    path_config_yaml=config_yaml_path,
    dataset_train=doclaynet,
    path_weights=weights_path,
    config_overwrite=config_overwrite,
    log_dir="train_log",
    build_train_config=build_train_config,
    dataset_val=doclaynet,
    build_val_config=None,
    metric_name=None,
    metric=coco_metric,
    pipeline_component_name="ImageLayoutService")

and show me what should be the console output? It is quite frustrating that I keep getting CUDA out of memory error even though my script is almost identical to your tutorial.

[JaMe76]

Let me check. Will get back to you..

[JaMe76]

Finally, got the issue after reporting the same CUDA OOM on my machine.

Problem is, that Detectron2 training config changed over time. The default train batch size is 16 which is way to large and needs to be reduced when training on one GPU.

Reduce SOLVER.IMS_PER_BATCH e.g.

 config_overwrite = ["SOLVER.MAX_ITER=100000",
                        "TEST.EVAL_PERIOD=20000",
                        "SOLVER.CHECKPOINT_PERIOD=20000",
                        "MODEL.BACKBONE.FREEZE_AT=0",
                        "SOLVER.BASE_LR=1e-3",
                        "SOLVER.IMS_PER_BATCH=2"]

Thank you for pointing out. Will update the tutorial accordingly.

After adding this, the training script runs with PyTorch version 1.9.0+cu111 on my machine.

[yflam1]

THANKS!!!!!!!!! You truly saved me!

[himasai9711]

i want to know how you are finetuning cause when i tried training it struck at iteration 0