The bbox_loss was high when training smoke on waymo.

[JiSuanJiDaWang]

Hi, I found that the model seems not to converge when I trained smoke on waymo dataset. The value of loss_bbox maintain at around 3-4 and the loss_cls is around 1.5. Do you have any idea why this happen?

Here is my config file.

dataset_type = 'WaymoDataset'
data_root = '/home/sysadmin/temp/waymo1.4/'
class_names = ['Car', 'Pedestrian', 'Cyclist']
input_modality = dict(use_lidar=False, use_camera=True)
file_client_args = dict(backend='disk')
train_pipeline = [
    dict(type='LoadImageFromFileMono3D'),
    dict(
        type='LoadAnnotations3D',
        with_bbox=True,
        with_label=True,
        with_attr_label=False,
        with_bbox_3d=True,
        with_label_3d=True,
        with_bbox_depth=True),
    dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5),
    dict(type='RandomShiftScale', shift_scale=(0.2, 0.4), aug_prob=0.3),
    dict(type='AffineResize', img_scale=(1248, 832), down_ratio=4),
    dict(
        type='Pack3DDetInputs',
        keys=[
            'img', 'gt_bboxes', 'gt_bboxes_labels', 'gt_bboxes_3d',
            'gt_labels_3d', 'centers_2d', 'depths'
        ])
]
test_pipeline = [
    dict(type='LoadImageFromFileMono3D'),
    dict(type='AffineResize', img_scale=(1248, 832), down_ratio=4),
    dict(type='Pack3DDetInputs', keys=['img'])
]
eval_pipeline = [
    dict(type='LoadImageFromFileMono3D'),
    dict(type='AffineResize', img_scale=(1248, 832), down_ratio=4),
    dict(type='Pack3DDetInputs', keys=['img'])
]
metainfo = dict(CLASSES=['Car', 'Pedestrian', 'Cyclist'])
train_dataloader = dict(
    batch_size=4,
    num_workers=1,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=True),
    dataset=dict(
        type='WaymoDataset',
        data_root='/home/sysadmin/temp/waymo1.4/kitti2/kitti_format/',
        ann_file='waymo_infos_train.pkl',
        data_prefix=dict(
            pts='training/velodyne',
            CAM_FRONT='training/image_0',
            CAM_FRONT_RIGHT='training/image_1',
            CAM_FRONT_LEFT='training/image_2',
            CAM_SIDE_RIGHT='training/image_3',
            CAM_SIDE_LEFT='training/image_4'),
        pipeline=[
            dict(type='LoadImageFromFileMono3D'),
            dict(
                type='LoadAnnotations3D',
                with_bbox=True,
                with_label=True,
                with_attr_label=False,
                with_bbox_3d=True,
                with_label_3d=True,
                with_bbox_depth=True),
            dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5),
            dict(
                type='RandomShiftScale', shift_scale=(0.2, 0.4), aug_prob=0.3),
            dict(type='AffineResize', img_scale=(1248, 832), down_ratio=4),
            dict(
                type='Pack3DDetInputs',
                keys=[
                    'img', 'gt_bboxes', 'gt_bboxes_labels', 'gt_bboxes_3d',
                    'gt_labels_3d', 'centers_2d', 'depths'
                ])
        ],
        modality=dict(use_lidar=False, use_camera=True),
        test_mode=False,
        metainfo=dict(CLASSES=['Car', 'Pedestrian', 'Cyclist']),
        box_type_3d='Camera',
        load_type='fov_image_based',
        load_interval=5))
val_dataloader = dict(
    batch_size=1,
    num_workers=1,
    persistent_workers=True,
    drop_last=False,
    sampler=dict(type='DefaultSampler', shuffle=False),
    dataset=dict(
        type='WaymoDataset',
        data_root='/home/sysadmin/temp/waymo1.4/kitti2/kitti_format/',
        data_prefix=dict(
            pts='training/velodyne',
            CAM_FRONT='training/image_0',
            CAM_FRONT_RIGHT='training/image_1',
            CAM_FRONT_LEFT='training/image_2',
            CAM_SIDE_RIGHT='training/image_3',
            CAM_SIDE_LEFT='training/image_4'),
        ann_file='waymo_infos_val.pkl',
        pipeline=[
            dict(type='LoadImageFromFileMono3D'),
            dict(type='AffineResize', img_scale=(1248, 832), down_ratio=4),
            dict(type='Pack3DDetInputs', keys=['img'])
        ],
        modality=dict(use_lidar=False, use_camera=True),
        test_mode=True,
        metainfo=dict(CLASSES=['Car', 'Pedestrian', 'Cyclist']),
        box_type_3d='Camera',
        load_type='fov_image_based'))
test_dataloader = dict(
    batch_size=1,
    num_workers=1,
    persistent_workers=True,
    drop_last=False,
    sampler=dict(type='DefaultSampler', shuffle=False),
    dataset=dict(
        type='WaymoDataset',
        data_root='/home/sysadmin/temp/waymo1.4/kitti2/kitti_format/',
        data_prefix=dict(
            pts='training/velodyne',
            CAM_FRONT='training/image_0',
            CAM_FRONT_RIGHT='training/image_1',
            CAM_FRONT_LEFT='training/image_2',
            CAM_SIDE_RIGHT='training/image_3',
            CAM_SIDE_LEFT='training/image_4'),
        ann_file='waymo_infos_val.pkl',
        pipeline=[
            dict(type='LoadImageFromFileMono3D'),
            dict(type='AffineResize', img_scale=(1248, 832), down_ratio=4),
            dict(type='Pack3DDetInputs', keys=['img'])
        ],
        modality=dict(use_lidar=False, use_camera=True),
        test_mode=True,
        metainfo=dict(CLASSES=['Car', 'Pedestrian', 'Cyclist']),
        box_type_3d='Camera',
        load_type='fov_image_based'))
val_evaluator = dict(
    type='WaymoMetric',
    ann_file=
    '/home/sysadmin/temp/waymo1.4/kitti2/kitti_format/waymo_infos_val.pkl',
    waymo_bin_file='/home/sysadmin/temp/waymo1.4/waymo_format/cam_gt.bin',
    data_root='/home/sysadmin/temp/waymo1.4/waymo_format',
    metric='mAP',
    load_type='fov_image_based')
test_evaluator = dict(
    type='WaymoMetric',
    ann_file=
    '/home/sysadmin/temp/waymo1.4/kitti2/kitti_format/waymo_infos_val.pkl',
    waymo_bin_file='/home/sysadmin/temp/waymo1.4/waymo_format/cam_gt.bin',
    data_root='/home/sysadmin/temp/waymo1.4/waymo_format',
    metric='mAP',
    load_type='fov_image_based')
vis_backends = [dict(type='LocalVisBackend')]
visualizer = dict(
    type='Det3DLocalVisualizer',
    vis_backends=[dict(type='LocalVisBackend')],
    name='visualizer')
model = dict(
    type='SMOKEMono3D',
    data_preprocessor=dict(
        type='Det3DDataPreprocessor',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        bgr_to_rgb=True,
        pad_size_divisor=32),
    backbone=dict(
        type='DLANet',
        depth=34,
        in_channels=3,
        norm_cfg=dict(type='GN', num_groups=32),
        init_cfg=dict(
            type='Pretrained',
            checkpoint='http://dl.yf.io/dla/models/imagenet/dla34-ba72cf86.pth'
        )),
    neck=dict(
        type='DLANeck',
        in_channels=[16, 32, 64, 128, 256, 512],
        start_level=2,
        end_level=5,
        norm_cfg=dict(type='GN', num_groups=32)),
    bbox_head=dict(
        type='SMOKEMono3DHead',
        num_classes=3,
        in_channels=64,
        dim_channel=[3, 4, 5],
        ori_channel=[6, 7],
        stacked_convs=0,
        feat_channels=64,
        use_direction_classifier=False,
        diff_rad_by_sin=False,
        pred_attrs=False,
        pred_velo=False,
        dir_offset=0,
        strides=None,
        group_reg_dims=(8, ),
        cls_branch=(256, ),
        reg_branch=((256, ), ),
        num_attrs=0,
        bbox_code_size=7,
        dir_branch=(),
        attr_branch=(),
        bbox_coder=dict(
            type='SMOKECoder',
            base_depth=(41.01, 18.44),
            base_dims=((4.73, 1.77, 2.08), (0.91, 1.74, 0.84), (1.81, 1.77,
                                                                0.84)),
            code_size=7),
        loss_cls=dict(type='mmdet.GaussianFocalLoss', loss_weight=1.0),
        loss_bbox=dict(
            type='mmdet.L1Loss',
            reduction='sum',
            loss_weight=0.0033333333333333335),
        loss_dir=dict(
            type='mmdet.CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
        loss_attr=None,
        conv_bias=True,
        dcn_on_last_conv=False),
    train_cfg=None,
    test_cfg=dict(topK=100, local_maximum_kernel=3, max_per_img=100))
lr = 0.001
optim_wrapper = dict(
    type='OptimWrapper',
    optimizer=dict(type='SGD', lr=0.002, weight_decay=0.01),
    clip_grad=dict(max_norm=35, norm_type=2),
    paramwise_cfg=dict(bias_lr_mult=2.0, bias_decay_mult=0.0))
train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=24, val_interval=24)
val_cfg = dict(type='ValLoop')
test_cfg = dict(type='TestLoop')
param_scheduler = [
    dict(
        type='LinearLR',
        start_factor=0.3333333333333333,
        by_epoch=False,
        begin=0,
        end=500),
    dict(
        type='MultiStepLR',
        begin=0,
        end=24,
        by_epoch=True,
        milestones=[16, 22],
        gamma=0.1)
]
auto_scale_lr = dict(enable=False, base_batch_size=3)
default_scope = 'mmdet3d'
default_hooks = dict(
    timer=dict(type='IterTimerHook'),
    logger=dict(type='LoggerHook', interval=50),
    param_scheduler=dict(type='ParamSchedulerHook'),
    checkpoint=dict(type='CheckpointHook', interval=-1),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    visualization=dict(type='Det3DVisualizationHook'))
env_cfg = dict(
    cudnn_benchmark=False,
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
    dist_cfg=dict(backend='nccl'))
log_processor = dict(type='LogProcessor', window_size=50, by_epoch=True)
log_level = 'INFO'
load_from = None
resume = False
total_epochs = 24

[ZCMax]

I recommend you to visualize the actual training data on waymo dataset to check whether the annotations are right. Besides, some hyper-parameters should be adjusted on waymo dataset such as base_depth and base_dims.

[JiSuanJiDaWang]

Hi,
I did modify the base_depth and the base_dims by waymo dataset. By the way, how can I visualize the actual training data? by using the tools/misc/browse_dataset.py?
Best wishes.

[FlyingAnt2018]

I recommend you to visualize the actual training data on waymo dataset to check whether the annotations are right. Besides, some hyper-parameters should be adjusted on waymo dataset such as base_depth and base_dims.

Hi, how to design waymo dataset traing and test resolution? default img0-2 is (1280, 1920) and img3-4 is (886, 1920)

[FlyingAnt2018]

Hi, I did modify the base_depth and the base_dims by waymo dataset. By the way, how can I visualize the actual training data? by using the tools/misc/browse_dataset.py? Best wishes.

Have you solved this problem? I found this is caused by data preperation pipeline "Resize" which only resize image and 2d bbox, and 3d bboxes are not resized, and i have visualize the data augment result, as follow.