Question about "bash tests/minimal_test.sh" #15
Description
"TypeError: 'NoneType' object is not subscriptable" when I run this instruction.
Here is the detail:
| Wrote preprocessed oracle data to DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//
| Wrote preprocessed embedding data to DATA/wiki25/embeddings/RoBERTa-large-top24
[Training:]
[Configuration file:]
configs/wiki25.sh
| distributed init (rank 0): tcp://localhost:16604
| distributed init (rank 5): tcp://localhost:16604
| initialized host localhost as rank 5
| distributed init (rank 6): tcp://localhost:16604
| initialized host localhost as rank 6
| distributed init (rank 3): tcp://localhost:16604
| initialized host localhost as rank 3
| distributed init (rank 4): tcp://localhost:16604
| initialized host localhost as rank 4
| distributed init (rank 1): tcp://localhost:16604
| initialized host localhost as rank 1
| distributed init (rank 2): tcp://localhost:16604
| initialized host localhost as rank 2
| distributed init (rank 7): tcp://localhost:16604
| initialized host localhost as rank 7
| initialized host localhost as rank 0
Namespace(activation_dropout=0.0, activation_fn='relu', adam_betas='(0.9,0.98)', adam_eps=1e-08, adaptive_input=False, adaptive_softmax_cutoff=None, adaptive_softmax_dropout=0, append_eos_to_target=0, apply_tgt_actnode_masks=0, apply_tgt_input_src=0, apply_tgt_src_align=1, apply_tgt_vocab_masks=1, arch='transformer_tgt_pointer_graphmp', attention_dropout=0.0, bert_backprop=False, best_checkpoint_metric='loss', bpe=None, bucket_cap_mb=25, clip_norm=0.0, collate_tgt_states=1, cpu=False, criterion='label_smoothed_cross_entropy_pointer', curriculum=0, data='DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//', dataset_impl=None, ddp_backend='c10d', decoder_attention_heads=4, decoder_embed_dim=256, decoder_embed_path=None, decoder_ffn_embed_dim=512, decoder_input_dim=256, decoder_layers=6, decoder_learned_pos=False, decoder_normalize_before=False, decoder_output_dim=256, device_id=0, disable_validation=False, distributed_backend='nccl', distributed_init_method='tcp://localhost:16604', distributed_no_spawn=False, distributed_port=-1, distributed_rank=0, distributed_world_size=8, dropout=0.3, emb_dir='DATA/wiki25/embeddings/RoBERTa-large-top24', encode_state_machine=None, encoder_attention_heads=4, encoder_embed_dim=256, encoder_embed_path=None, encoder_ffn_embed_dim=512, encoder_layers=6, encoder_learned_pos=False, encoder_normalize_before=False, find_unused_parameters=False, fix_batches_to_gpus=False, fp16=False, fp16_init_scale=128, fp16_scale_tolerance=0.0, fp16_scale_window=None, keep_interval_updates=-1, keep_last_epochs=6, label_smoothing=0.01, lazy_load=False, left_pad_source='True', left_pad_target='False', log_format='json', log_interval=1000, loss_coef=1.0, lr=[0.0005], lr_scheduler='inverse_sqrt', max_epoch=10, max_sentences=None, max_sentences_valid=None, max_source_positions=1024, max_target_positions=1024, max_tokens=3584, max_tokens_valid=3584, max_update=0, maximize_best_checkpoint_metric=False, memory_efficient_fp16=False, min_loss_scale=0.0001, min_lr=1e-09, no_bert_precompute=False, no_epoch_checkpoints=False, no_last_checkpoints=False, no_progress_bar=False, no_save=False, no_save_optimizer_state=False, no_token_positional_embeddings=False, num_workers=1, optimizer='adam', optimizer_overrides='{}', pointer_dist_decoder_selfattn_avg=0, pointer_dist_decoder_selfattn_heads=1, pointer_dist_decoder_selfattn_infer=5, pointer_dist_decoder_selfattn_layers=[5], pretrained_embed_dim=1024, raw_text=False, required_batch_size_multiple=8, reset_dataloader=False, reset_lr_scheduler=False, reset_meters=False, reset_optimizer=False, restore_file='checkpoint_last.pt', save_dir='DATA/wiki25/models/exp_cofill_o8.3_act-states_RoBERTa-large-top24/_act-pos-grh_vmask1_shiftpos1_ptr-lay6-h1_grh-lay123-h2-allprev_1in1out_cam-layall-h2-abuf/ep10-seed42', save_interval=1, save_interval_updates=0, seed=42, sentence_avg=False, share_all_embeddings=False, share_decoder_input_output_embed=0, shift_pointer_value=1, skip_invalid_size_inputs_valid_test=False, source_lang=None, target_lang=None, task='amr_action_pointer_graphmp', tbmf_wrapper=False, tensorboard_logdir='DATA/wiki25/models/exp_cofill_o8.3_act-states_RoBERTa-large-top24/_act-pos-grh_vmask1_shiftpos1_ptr-lay6-h1_grh-lay123-h2-allprev_1in1out_cam-layall-h2-abuf/ep10-seed42', tgt_factored_emb_out=0, tgt_graph_heads=2, tgt_graph_layers=[0, 1, 2], tgt_graph_mask='allprev_1in1out', tgt_input_src_backprop=1, tgt_input_src_combine='add', tgt_input_src_emb='top', tgt_src_align_focus=['p0c1n0', 'p0c0n*'], tgt_src_align_heads=2, tgt_src_align_layers=[0, 1, 2, 3, 4, 5], threshold_loss_scale=None, tokenizer=None, train_subset='train', update_freq=[1], upsample_primary=1, use_bmuf=False, user_dir='../fairseq_ext', valid_subset='valid', validate_interval=1, warmup_init_lr=1e-07, warmup_updates=4000, weight_decay=0.0)
| [en] dictionary: 248 types
| [actions_nopos] dictionary: 128 types
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.en
| loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/valid.en-actions.en.bert
| loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/valid.en-actions.en.wordpieces
| loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/valid.en-actions.en.wp2w
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.nopos_in
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.nopos_out
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.pos
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.vocab_masks
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.src_cursors
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actnode_masks
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_masks
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_1stnode_masks
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_cur_node_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_cur_1stnode_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_pre_node_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_directions
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_allpre_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_allpre_pre_node_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_allpre_directions
| model transformer_tgt_pointer_graphmp, criterion LabelSmoothedCrossEntropyPointerCriterion
| num. model params: 8298496 (num. trained: 8298496)
| training on 8 GPUs
| max tokens per GPU = 3584 and max sentences per GPU = None
| no existing checkpoint found DATA/wiki25/models/exp_cofill_o8.3_act-states_RoBERTa-large-top24/_act-pos-grh_vmask1_shiftpos1_ptr-lay6-h1_grh-lay123-h2-allprev_1in1out_cam-layall-h2-abuf/ep10-seed42/checkpoint_last.pt
| loading train data for epoch 0
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.en
| loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/train.en-actions.en.bert
| loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/train.en-actions.en.wordpieces
| loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/train.en-actions.en.wp2w
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.nopos_in
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.nopos_out
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.pos
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.vocab_masks
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.src_cursors
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actnode_masks
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_masks
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_1stnode_masks
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_cur_node_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_cur_1stnode_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_pre_node_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_directions
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_allpre_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_allpre_pre_node_indexes
| loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_allpre_directions
/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
Traceback (most recent call last):
File "fairseq_ext/train.py", line 341, in
cli_main()
File "fairseq_ext/train.py", line 333, in cli_main
nprocs=args.distributed_world_size,
File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
while not spawn_context.join():
File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:
-- Process 5 terminated with the following error:
Traceback (most recent call last):
File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/data4/yhchen/transition-amr-parser/fairseq_ext/train.py", line 297, in distributed_main
main(args, init_distributed=True) # 分布式训练
File "/data4/yhchen/transition-amr-parser/fairseq_ext/train.py", line 103, in main
train(args, trainer, task, epoch_itr)
File "/data4/yhchen/transition-amr-parser/fairseq_ext/train.py", line 149, in train
log_output = trainer.train_step(samples)
File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/fairseq/trainer.py", line 264, in train_step
ignore_grad
File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/fairseq_ext/tasks/amr_action_pointer_graphmp.py", line 462, in train_step
loss, sample_size, logging_output = criterion(model, sample)
File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/fairseq_ext/criterions/label_smoothed_cross_entropy_pointer.py", line 104, in forward
net_output = model(**sample['net_input'])
TypeError: 'NoneType' object is not subscriptable