very slow data_time compared to released logs #1924
Closed
TongkunGuan
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Screenshot of the official training log （DBNet18_TotalText）：
This is my log:
2023/06/07 16:45:03 - mmengine - INFO -
System environment:
sys.platform: linux
Python: 3.8.16 (default, Mar 2 2023, 03:21:46) [GCC 11.2.0]
CUDA available: True
numpy_random_seed: 1528941036
GPU 0,1,2,3: NVIDIA GeForce RTX 3090
CUDA_HOME: /usr/local/cuda-11.1/bin
NVCC: Not Available
GCC: gcc (Ubuntu 10.3.0-1ubuntu1~20.04) 10.3.0
PyTorch: 1.10.2
PyTorch compiling details: PyTorch built with:
GCC 7.3
C++ Version: 201402
Intel(R) oneAPI Math Kernel Library Version 2023.1-Product Build 20230303 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 11.3
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
CuDNN 8.2
Magma 2.5.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.11.3
OpenCV: 4.7.0
MMEngine: 0.7.3
Runtime environment:
cudnn_benchmark: False
mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
dist_cfg: {'backend': 'nccl'}
seed: None
Distributed launcher: pytorch
Distributed training: True
GPU number: 4
2023/06/07 16:45:03 - mmengine - INFO - Config:
model = dict(
type='DBNet',
backbone=dict(
type='mmdet.ResNet',
depth=18,
num_stages=4,
out_indices=(0, 1, 2, 3),
frozen_stages=-1,
norm_cfg=dict(type='BN', requires_grad=True),
init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet18'),
norm_eval=False,
style='caffe'),
neck=dict(
type='FPNC', in_channels=[64, 128, 256, 512], lateral_channels=256),
det_head=dict(
type='DBHead',
in_channels=256,
module_loss=dict(type='DBModuleLoss'),
postprocessor=dict(type='DBPostprocessor', text_repr_type='quad')),
data_preprocessor=dict(
type='TextDetDataPreprocessor',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
bgr_to_rgb=True,
pad_size_divisor=32))
train_pipeline = [
dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
dict(
type='LoadOCRAnnotations',
with_polygon=True,
with_bbox=True,
with_label=True),
dict(type='FixInvalidPolygon', min_poly_points=4),
dict(
type='TorchVisionWrapper',
op='ColorJitter',
brightness=0.12549019607843137,
saturation=0.5),
dict(
type='ImgAugWrapper',
args=[['Fliplr', 0.5], {
'cls': 'Affine',
'rotate': [-10, 10]
}, ['Resize', [0.5, 3.0]]]),
dict(type='RandomCrop', min_side_ratio=0.1),
dict(type='Resize', scale=(640, 640), keep_ratio=True),
dict(type='Pad', size=(640, 640)),
dict(
type='PackTextDetInputs',
meta_keys=('img_path', 'ori_shape', 'img_shape'))
]
test_pipeline = [
dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
dict(type='Resize', scale=(1333, 736), keep_ratio=True),
dict(
type='LoadOCRAnnotations',
with_polygon=True,
with_bbox=True,
with_label=True),
dict(type='FixInvalidPolygon', min_poly_points=4),
dict(
type='PackTextDetInputs',
meta_keys=('img_path', 'ori_shape', 'img_shape', 'scale_factor'))
]
totaltext_textdet_data_root = 'data/totaltext'
totaltext_textdet_train = dict(
type='OCRDataset',
data_root='data/totaltext',
ann_file='textdet_train.json',
filter_cfg=dict(filter_empty_gt=True, min_size=32),
pipeline=[
dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
dict(
type='LoadOCRAnnotations',
with_polygon=True,
with_bbox=True,
with_label=True),
dict(type='FixInvalidPolygon', min_poly_points=4),
dict(
type='TorchVisionWrapper',
op='ColorJitter',
brightness=0.12549019607843137,
saturation=0.5),
dict(
type='ImgAugWrapper',
args=[['Fliplr', 0.5], {
'cls': 'Affine',
'rotate': [-10, 10]
}, ['Resize', [0.5, 3.0]]]),
dict(type='RandomCrop', min_side_ratio=0.1),
dict(type='Resize', scale=(640, 640), keep_ratio=True),
dict(type='Pad', size=(640, 640)),
dict(
type='PackTextDetInputs',
meta_keys=('img_path', 'ori_shape', 'img_shape'))
])
totaltext_textdet_test = dict(
type='OCRDataset',
data_root='data/totaltext',
ann_file='textdet_test.json',
test_mode=True,
pipeline=[
dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
dict(type='Resize', scale=(1333, 736), keep_ratio=True),
dict(
type='LoadOCRAnnotations',
with_polygon=True,
with_bbox=True,
with_label=True),
dict(type='FixInvalidPolygon', min_poly_points=4),
dict(
type='PackTextDetInputs',
meta_keys=('img_path', 'ori_shape', 'img_shape', 'scale_factor'))
])
default_scope = 'mmocr'
env_cfg = dict(
cudnn_benchmark=False,
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
dist_cfg=dict(backend='nccl'))
randomness = dict(seed=None)
default_hooks = dict(
timer=dict(type='IterTimerHook'),
logger=dict(type='LoggerHook', interval=5),
param_scheduler=dict(type='ParamSchedulerHook'),
checkpoint=dict(type='CheckpointHook', interval=20),
sampler_seed=dict(type='DistSamplerSeedHook'),
sync_buffer=dict(type='SyncBuffersHook'),
visualization=dict(
type='VisualizationHook',
interval=1,
enable=False,
show=False,
draw_gt=False,
draw_pred=False))
log_level = 'INFO'
log_processor = dict(type='LogProcessor', window_size=10, by_epoch=True)
load_from = None
resume = False
val_evaluator = dict(type='HmeanIOUMetric')
test_evaluator = dict(type='HmeanIOUMetric')
vis_backends = [dict(type='LocalVisBackend')]
visualizer = dict(
type='TextDetLocalVisualizer',
name='visualizer',
vis_backends=[dict(type='LocalVisBackend')])
opencv_num_threads = 0
mp_start_method = 'fork'
optim_wrapper = dict(
type='OptimWrapper',
optimizer=dict(type='SGD', lr=0.007, momentum=0.9, weight_decay=0.0001))
train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=1200, val_interval=20)
val_cfg = dict(type='ValLoop')
test_cfg = dict(type='TestLoop')
param_scheduler = [dict(type='PolyLR', power=0.9, eta_min=1e-07, end=1200)]
train_dataloader = dict(
batch_size=16,
num_workers=16,
pin_memory=True,
persistent_workers=True,
sampler=dict(type='DefaultSampler', shuffle=True),
dataset=dict(
type='OCRDataset',
data_root='data/totaltext',
ann_file='textdet_train.json',
filter_cfg=dict(filter_empty_gt=True, min_size=32),
pipeline=[
dict(
type='LoadImageFromFile',
color_type='color_ignore_orientation'),
dict(
type='LoadOCRAnnotations',
with_polygon=True,
with_bbox=True,
with_label=True),
dict(type='FixInvalidPolygon', min_poly_points=4),
dict(
type='TorchVisionWrapper',
op='ColorJitter',
brightness=0.12549019607843137,
saturation=0.5),
dict(
type='ImgAugWrapper',
args=[['Fliplr', 0.5], {
'cls': 'Affine',
'rotate': [-10, 10]
}, ['Resize', [0.5, 3.0]]]),
dict(type='RandomCrop', min_side_ratio=0.1),
dict(type='Resize', scale=(640, 640), keep_ratio=True),
dict(type='Pad', size=(640, 640)),
dict(
type='PackTextDetInputs',
meta_keys=('img_path', 'ori_shape', 'img_shape'))
]))
val_dataloader = dict(
batch_size=1,
num_workers=1,
pin_memory=True,
persistent_workers=True,
sampler=dict(type='DefaultSampler', shuffle=False),
dataset=dict(
type='OCRDataset',
data_root='data/totaltext',
ann_file='textdet_test.json',
test_mode=True,
pipeline=[
dict(
type='LoadImageFromFile',
color_type='color_ignore_orientation'),
dict(type='Resize', scale=(1333, 736), keep_ratio=True),
dict(
type='LoadOCRAnnotations',
with_polygon=True,
with_bbox=True,
with_label=True),
dict(type='FixInvalidPolygon', min_poly_points=4),
dict(
type='PackTextDetInputs',
meta_keys=('img_path', 'ori_shape', 'img_shape',
'scale_factor'))
]))
test_dataloader = dict(
batch_size=1,
num_workers=1,
pin_memory=True,
persistent_workers=True,
sampler=dict(type='DefaultSampler', shuffle=False),
dataset=dict(
type='OCRDataset',
data_root='data/totaltext',
ann_file='textdet_test.json',
test_mode=True,
pipeline=[
dict(
type='LoadImageFromFile',
color_type='color_ignore_orientation'),
dict(type='Resize', scale=(1333, 736), keep_ratio=True),
dict(
type='LoadOCRAnnotations',
with_polygon=True,
with_bbox=True,
with_label=True),
dict(type='FixInvalidPolygon', min_poly_points=4),
dict(
type='PackTextDetInputs',
meta_keys=('img_path', 'ori_shape', 'img_shape',
'scale_factor'))
]))
auto_scale_lr = dict(base_batch_size=16)
launcher = 'pytorch'
work_dir = './work_dirs/dbnet_resnet18_fpnc_1200e_totaltext'
2023/06/07 16:45:05 - mmengine - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH ) RuntimeInfoHook
(BELOW_NORMAL) LoggerHook
before_train:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(VERY_LOW ) CheckpointHook
before_train_epoch:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(NORMAL ) DistSamplerSeedHook
before_train_iter:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
after_train_iter:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook
after_train_epoch:
(NORMAL ) IterTimerHook
(NORMAL ) SyncBuffersHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook
before_val_epoch:
(NORMAL ) IterTimerHook
(NORMAL ) SyncBuffersHook
before_val_iter:
(NORMAL ) IterTimerHook
after_val_iter:
(NORMAL ) IterTimerHook
(NORMAL ) VisualizationHook
(BELOW_NORMAL) LoggerHook
after_val_epoch:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook
after_train:
(VERY_LOW ) CheckpointHook
before_test_epoch:
(NORMAL ) IterTimerHook
before_test_iter:
(NORMAL ) IterTimerHook
after_test_iter:
(NORMAL ) IterTimerHook
(NORMAL ) VisualizationHook
(BELOW_NORMAL) LoggerHook
after_test_epoch:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
after_run:
(BELOW_NORMAL) LoggerHook
2023/06/07 16:45:06 - mmengine - INFO - load model from: torchvision://resnet18
2023/06/07 16:45:06 - mmengine - INFO - Loads checkpoint by torchvision backend from path: torchvision://resnet18
2023/06/07 16:45:06 - mmengine - WARNING - The model and loaded state dict do not match exactly
unexpected key in source state_dict: fc.weight, fc.bias
Name of parameter - Initialization information
backbone.conv1.weight - torch.Size([64, 3, 7, 7]):
PretrainedInit: load from torchvision://resnet18
backbone.bn1.weight - torch.Size([64]):
PretrainedInit: load from torchvision://resnet18
backbone.bn1.bias - torch.Size([64]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.0.conv1.weight - torch.Size([64, 64, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.0.bn1.weight - torch.Size([64]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.0.bn1.bias - torch.Size([64]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.0.conv2.weight - torch.Size([64, 64, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.0.bn2.weight - torch.Size([64]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.0.bn2.bias - torch.Size([64]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.1.conv1.weight - torch.Size([64, 64, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.1.bn1.weight - torch.Size([64]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.1.bn1.bias - torch.Size([64]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.1.conv2.weight - torch.Size([64, 64, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.1.bn2.weight - torch.Size([64]):
PretrainedInit: load from torchvision://resnet18
backbone.layer1.1.bn2.bias - torch.Size([64]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.0.conv1.weight - torch.Size([128, 64, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.0.bn1.weight - torch.Size([128]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.0.bn1.bias - torch.Size([128]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.0.conv2.weight - torch.Size([128, 128, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.0.bn2.weight - torch.Size([128]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.0.bn2.bias - torch.Size([128]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.0.downsample.0.weight - torch.Size([128, 64, 1, 1]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.0.downsample.1.weight - torch.Size([128]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.0.downsample.1.bias - torch.Size([128]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.1.conv1.weight - torch.Size([128, 128, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.1.bn1.weight - torch.Size([128]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.1.bn1.bias - torch.Size([128]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.1.conv2.weight - torch.Size([128, 128, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.1.bn2.weight - torch.Size([128]):
PretrainedInit: load from torchvision://resnet18
backbone.layer2.1.bn2.bias - torch.Size([128]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.0.conv1.weight - torch.Size([256, 128, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.0.bn1.weight - torch.Size([256]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.0.bn1.bias - torch.Size([256]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.0.conv2.weight - torch.Size([256, 256, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.0.bn2.weight - torch.Size([256]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.0.bn2.bias - torch.Size([256]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.0.downsample.0.weight - torch.Size([256, 128, 1, 1]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.0.downsample.1.weight - torch.Size([256]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.0.downsample.1.bias - torch.Size([256]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.1.conv1.weight - torch.Size([256, 256, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.1.bn1.weight - torch.Size([256]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.1.bn1.bias - torch.Size([256]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.1.conv2.weight - torch.Size([256, 256, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.1.bn2.weight - torch.Size([256]):
PretrainedInit: load from torchvision://resnet18
backbone.layer3.1.bn2.bias - torch.Size([256]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.0.conv1.weight - torch.Size([512, 256, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.0.bn1.weight - torch.Size([512]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.0.bn1.bias - torch.Size([512]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.0.conv2.weight - torch.Size([512, 512, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.0.bn2.weight - torch.Size([512]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.0.bn2.bias - torch.Size([512]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.0.downsample.0.weight - torch.Size([512, 256, 1, 1]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.0.downsample.1.weight - torch.Size([512]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.0.downsample.1.bias - torch.Size([512]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.1.conv1.weight - torch.Size([512, 512, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.1.bn1.weight - torch.Size([512]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.1.bn1.bias - torch.Size([512]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.1.conv2.weight - torch.Size([512, 512, 3, 3]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.1.bn2.weight - torch.Size([512]):
PretrainedInit: load from torchvision://resnet18
backbone.layer4.1.bn2.bias - torch.Size([512]):
PretrainedInit: load from torchvision://resnet18
neck.lateral_convs.0.conv.weight - torch.Size([256, 64, 1, 1]):
Initialized by user-defined
init_weightsin ConvModuleneck.lateral_convs.1.conv.weight - torch.Size([256, 128, 1, 1]):
Initialized by user-defined
init_weightsin ConvModuleneck.lateral_convs.2.conv.weight - torch.Size([256, 256, 1, 1]):
Initialized by user-defined
init_weightsin ConvModuleneck.lateral_convs.3.conv.weight - torch.Size([256, 512, 1, 1]):
Initialized by user-defined
init_weightsin ConvModuleneck.smooth_convs.0.conv.weight - torch.Size([64, 256, 3, 3]):
Initialized by user-defined
init_weightsin ConvModuleneck.smooth_convs.1.conv.weight - torch.Size([64, 256, 3, 3]):
Initialized by user-defined
init_weightsin ConvModuleneck.smooth_convs.2.conv.weight - torch.Size([64, 256, 3, 3]):
Initialized by user-defined
init_weightsin ConvModuleneck.smooth_convs.3.conv.weight - torch.Size([64, 256, 3, 3]):
Initialized by user-defined
init_weightsin ConvModuledet_head.binarize.0.weight - torch.Size([64, 256, 3, 3]):
The value is the same before and after calling
init_weightsof DBNetdet_head.binarize.1.weight - torch.Size([64]):
The value is the same before and after calling
init_weightsof DBNetdet_head.binarize.1.bias - torch.Size([64]):
The value is the same before and after calling
init_weightsof DBNetdet_head.binarize.3.weight - torch.Size([64, 64, 2, 2]):
The value is the same before and after calling
init_weightsof DBNetdet_head.binarize.3.bias - torch.Size([64]):
The value is the same before and after calling
init_weightsof DBNetdet_head.binarize.4.weight - torch.Size([64]):
The value is the same before and after calling
init_weightsof DBNetdet_head.binarize.4.bias - torch.Size([64]):
The value is the same before and after calling
init_weightsof DBNetdet_head.binarize.6.weight - torch.Size([64, 1, 2, 2]):
The value is the same before and after calling
init_weightsof DBNetdet_head.binarize.6.bias - torch.Size([1]):
The value is the same before and after calling
init_weightsof DBNetdet_head.threshold.0.weight - torch.Size([64, 256, 3, 3]):
The value is the same before and after calling
init_weightsof DBNetdet_head.threshold.1.weight - torch.Size([64]):
The value is the same before and after calling
init_weightsof DBNetdet_head.threshold.1.bias - torch.Size([64]):
The value is the same before and after calling
init_weightsof DBNetdet_head.threshold.3.weight - torch.Size([64, 64, 2, 2]):
The value is the same before and after calling
init_weightsof DBNetdet_head.threshold.3.bias - torch.Size([64]):
The value is the same before and after calling
init_weightsof DBNetdet_head.threshold.4.weight - torch.Size([64]):
The value is the same before and after calling
init_weightsof DBNetdet_head.threshold.4.bias - torch.Size([64]):
The value is the same before and after calling
init_weightsof DBNetdet_head.threshold.6.weight - torch.Size([64, 1, 2, 2]):
The value is the same before and after calling
init_weightsof DBNetdet_head.threshold.6.bias - torch.Size([1]):
The value is the same before and after calling
init_weightsof DBNet2023/06/07 16:45:06 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io
2023/06/07 16:45:06 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
2023/06/07 16:45:06 - mmengine - INFO - Checkpoints will be saved to /home/guantongkun/Projects/STD/mmocr/work_dirs/dbnet_resnet18_fpnc_1200e_totaltext.
2023/06/07 16:47:08 - mmengine - INFO - Epoch(train) [1][ 5/20] lr: 7.0000e-03 eta: 6 days, 18:43:28 time: 24.4138 data_time: 12.6575 memory: 6773 loss: 15.3831 loss_prob: 12.2361 loss_thr: 2.2011 loss_db: 0.9459
2023/06/07 16:47:14 - mmengine - INFO - Epoch(train) [1][10/20] lr: 7.0000e-03 eta: 3 days, 13:07:39 time: 12.7745 data_time: 6.3293 memory: 6773 loss: 10.8112 loss_prob: 8.1802 loss_thr: 1.6932 loss_db: 0.9379
2023/06/07 16:47:19 - mmengine - INFO - Epoch(train) [1][15/20] lr: 7.0000e-03 eta: 2 days, 11:11:58 time: 1.1214 data_time: 0.0011 memory: 6773 loss: 5.7262 loss_prob: 3.5901 loss_thr: 1.1740 loss_db: 0.9620
2023/06/07 16:47:25 - mmengine - INFO - Exp name: dbnet_resnet18_fpnc_1200e_totaltext_20230607_164457
2023/06/07 16:47:25 - mmengine - INFO - Epoch(train) [1][20/20] lr: 7.0000e-03 eta: 1 day, 22:10:27 time: 1.0894 data_time: 0.0012 memory: 6773 loss: 5.1352 loss_prob: 2.9833 loss_thr: 1.1549 loss_db: 0.9971
2023/06/07 16:49:04 - mmengine - INFO - Epoch(train) [2][ 5/20] lr: 6.9947e-03 eta: 2 days, 15:22:40 time: 10.4632 data_time: 8.4200 memory: 6773 loss: 5.0528 loss_prob: 2.9071 loss_thr: 1.1457 loss_db: 1.0000
2023/06/07 16:49:09 - mmengine - INFO - Epoch(train) [2][10/20] lr: 6.9947e-03 eta: 2 days, 6:02:17 time: 10.4838 data_time: 8.4199 memory: 6773 loss: 5.0326 loss_prob: 2.8929 loss_thr: 1.1397 loss_db: 1.0000
2023/06/07 16:49:15 - mmengine - INFO - Epoch(train) [2][15/20] lr: 6.9947e-03 eta: 1 day, 23:21:27 time: 1.1075 data_time: 0.0015 memory: 6773 loss: 5.0037 loss_prob: 2.8684 loss_thr: 1.1353 loss_db: 1.0000
2023/06/07 16:49:20 - mmengine - INFO - Exp name: dbnet_resnet18_fpnc_1200e_totaltext_20230607_164457
2023/06/07 16:49:20 - mmengine - INFO - Epoch(train) [2][20/20] lr: 6.9947e-03 eta: 1 day, 18:17:36 time: 1.0708 data_time: 0.0018 memory: 6773 loss: 4.9785 loss_prob: 2.8439 loss_thr: 1.1347 loss_db: 1.0000
2023/06/07 16:51:03 - mmengine - INFO - Epoch(train) [3][ 5/20] lr: 6.9895e-03 eta: 2 days, 4:49:18 time: 10.8226 data_time: 8.2966 memory: 6773 loss: 4.9583 loss_prob: 2.8286 loss_thr: 1.1297 loss_db: 1.0000
2023/06/07 16:51:09 - mmengine - INFO - Epoch(train) [3][10/20] lr: 6.9895e-03 eta: 2 days, 0:16:51 time: 10.8679 data_time: 8.2962 memory: 6773 loss: 4.9430 loss_prob: 2.8215 loss_thr: 1.1216 loss_db: 0.9999
2023/06/07 16:51:15 - mmengine - INFO - Epoch(train) [3][15/20] lr: 6.9895e-03 eta: 1 day, 20:34:28 time: 1.1370 data_time: 0.0010 memory: 6773 loss: 4.9207 loss_prob: 2.8178 loss_thr: 1.1056 loss_db: 0.9973
2023/06/07 16:51:20 - mmengine - INFO - Exp name: dbnet_resnet18_fpnc_1200e_totaltext_20230607_164457
2023/06/07 16:51:20 - mmengine - INFO - Epoch(train) [3][20/20] lr: 6.9895e-03 eta: 1 day, 17:25:11 time: 1.0849 data_time: 0.0009 memory: 6773 loss: 4.8702 loss_prob: 2.8197 loss_thr: 1.1268 loss_db: 0.9237
2023/06/07 16:52:58 - mmengine - INFO - Epoch(train) [4][ 5/20] lr: 6.9842e-03 eta: 2 days, 0:15:56 time: 10.3282 data_time: 5.7911 memory: 6773 loss: 4.8344 loss_prob: 2.8201 loss_thr: 1.1547 loss_db: 0.8596
2023/06/07 16:53:03 - mmengine - INFO - Epoch(train) [4][10/20] lr: 6.9842e-03 eta: 1 day, 21:19:33 time: 10.3602 data_time: 5.7912 memory: 6773 loss: 4.7782 loss_prob: 2.8201 loss_thr: 1.1319 loss_db: 0.8262
2023/06/07 16:53:09 - mmengine - INFO - Epoch(train) [4][15/20] lr: 6.9842e-03 eta: 1 day, 18:48:50 time: 1.1299 data_time: 0.0013 memory: 6773 loss: 4.6503 loss_prob: 2.8263 loss_thr: 1.1211 loss_db: 0.7029
2023/06/07 16:53:14 - mmengine - INFO - Exp name: dbnet_resnet18_fpnc_1200e_totaltext_20230607_164457
2023/06/07 16:53:14 - mmengine - INFO - Epoch(train) [4][20/20] lr: 6.9842e-03 eta: 1 day, 16:33:49 time: 1.1076 data_time: 0.0016 memory: 6773 loss: 4.5212 loss_prob: 2.8308 loss_thr: 1.0923 loss_db: 0.5981
2023/06/07 16:54:48 - mmengine - INFO - Epoch(train) [5][ 5/20] lr: 6.9790e-03 eta: 1 day, 21:28:58 time: 9.8801 data_time: 7.8095 memory: 6773 loss: 4.4301 loss_prob: 2.8336 loss_thr: 1.0420 loss_db: 0.5545
2023/06/07 16:54:54 - mmengine - INFO - Epoch(train) [5][10/20] lr: 6.9790e-03 eta: 1 day, 19:22:58 time: 9.9481 data_time: 7.8095 memory: 6773 loss: 4.3626 loss_prob: 2.8351 loss_thr: 1.0086 loss_db: 0.5188
2023/06/07 16:55:00 - mmengine - INFO - Epoch(train) [5][15/20] lr: 6.9790e-03 eta: 1 day, 17:29:20 time: 1.1599 data_time: 0.0017 memory: 6773 loss: 4.3495 loss_prob: 2.8334 loss_thr: 0.9989 loss_db: 0.5172
2023/06/07 16:55:05 - mmengine - INFO - Exp name: dbnet_resnet18_fpnc_1200e_totaltext_20230607_164457
2023/06/07 16:55:05 - mmengine - INFO - Epoch(train) [5][20/20] lr: 6.9790e-03 eta: 1 day, 15:44:32 time: 1.0755 data_time: 0.0012 memory: 6773 loss: 4.3041 loss_prob: 2.8167 loss_thr: 0.9812 loss_db: 0.5062
main issues:
Although num_works=16, only a few CPU cores are occupied ---> slow data_time? --->where is the problem？
Beta Was this translation helpful? Give feedback.
All reactions