How can I get the data?

[YeJinPaark]

First, I downloaded 'Transformer-en' and renamed it like './model/syngec/english_transformer_baseline.pt'.
Then, I downloaded the preprocessed data.

And I run the code './pipeline_gopar.sh'.
But the error is:

Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../src/src_gopar/parse.py", line 17, in
input_sentences = load(sys.argv[1])
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../src/src_gopar/parse.py", line 9, in load
with open(filename, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/wi_locness_train/tgt.txt'
Loading resources...
Processing parallel files...
Traceback (most recent call last):
File "/opt/conda/bin/errant_parallel", line 8, in
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/errant/commands/parallel_to_m2.py", line 16, in main
in_files = [stack.enter_context(open(i)) for i in [args.orig]+args.cor]
File "/opt/conda/lib/python3.10/site-packages/errant/commands/parallel_to_m2.py", line 16, in
in_files = [stack.enter_context(open(i)) for i in [args.orig]+args.cor]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/wi_locness_train/tgt.txt'
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../src/src_gopar/convert_gec_data_to_parsing_data_english.py", line 153, in
with open(conll_file, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/wi_locness_train/tgt.txt.conll_predict'
/opt/conda/lib/python3.10/site-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use-env is set by default in torchrun.
If your script expects --local-rank argument to be set, please
change it to read from os.environ['LOCAL_RANK'] instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions

warnings.warn(
WARNING:torch.distributed.run:

---

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 30676) of binary: /opt/conda/bin/python
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launch.py", line 196, in
main()
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launch.py", line 192, in main
launch(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launch.py", line 177, in launch
run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

supar.cmds.biaffine_dep FAILED

Failures:
[1]:
time : 2023-08-24_08:10:36
host : 309e7fc0781e
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 30677)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2023-08-24_08:10:36
host : 309e7fc0781e
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 30678)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2023-08-24_08:10:36
host : 309e7fc0781e
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 30679)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
time : 2023-08-24_08:10:36
host : 309e7fc0781e
rank : 4 (local_rank: 4)
exitcode : 1 (pid: 30680)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
time : 2023-08-24_08:10:36
host : 309e7fc0781e
rank : 5 (local_rank: 5)
exitcode : 1 (pid: 30681)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
time : 2023-08-24_08:10:36
host : 309e7fc0781e
rank : 6 (local_rank: 6)
exitcode : 1 (pid: 30682)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
time : 2023-08-24_08:10:36
host : 309e7fc0781e
rank : 7 (local_rank: 7)
exitcode : 1 (pid: 30683)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Root Cause (first observed failure):
[0]:
time : 2023-08-24_08:10:36
host : 309e7fc0781e
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 30676)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

nohup: appending output to 'nohup.out'
nohup: appending output to 'nohup.out'
nohup: appending output to 'nohup.out'
nohup: appending output to 'nohup.out'
nohup: appending output to 'nohup.out'
nohup: appending output to 'nohup.out'

How can I fix it? plz help me...

[HillZhang1999]

I notice that the file path in your error message seems strange, such as ``/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../src/src_gopar/parse.py''.
Please try to enter the corresponding directory and then re-run the bash file.

[YeJinPaark]

Ok, I'll try soon

Then, I wonder that the how can I get the data like:

FileNotFoundError: [Errno 2] No such file or directory: '../../data/wi_locness_train/tgt.txt'

[HillZhang1999]

You should download the preprocessed data, unzip them, and put them into https://github.com/HillZhang1999/SynGEC/tree/main/data

[YeJinPaark]

Is that preprocessed data same the link of data:
https://drive.google.com/file/d/1dIDfYhELrh3BEKgGpsPYAy5ehcobmMov/view

So I downloaded the data and unzip ./data/
but I got the error like

Apply BPE...
./preprocess_syngec_transformer.sh: line 22: ../../data/clang8_train/src.txt: No such file or directory
./preprocess_syngec_transformer.sh: line 23: ../../data/clang8_train/tgt.txt: No such file or directory
./preprocess_syngec_transformer.sh: line 24: ../../data/bea19_dev/src.txt: No such file or directory
./preprocess_syngec_transformer.sh: line 25: ../../data/bea19_dev/tgt.txt: No such file or directory
Align subwords and words...
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/subword_align.py", line 41, in
with open(file_word, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/clang8_train/src.txt'
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/subword_align.py", line 41, in
with open(file_word, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt'
cp: cannot stat '../../data/clang8_train/src.txt': No such file or directory
cp: cannot stat '../../data/clang8_train/src.txt.bpe': No such file or directory
cp: cannot stat '../../data/clang8_train/tgt.txt': No such file or directory
cp: cannot stat '../../data/clang8_train/tgt.txt.bpe': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.bpe': No such file or directory
cp: cannot stat '../../data/bea19_dev/tgt.txt': No such file or directory
cp: cannot stat '../../data/bea19_dev/tgt.txt.bpe': No such file or directory
cp: cannot stat '../../data/clang8_train/src.txt.swm': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.swm': No such file or directory
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/clang8_train/src.txt.swm'
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/clang8_train/src.txt.swm'
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt.swm'
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt.swm'
cp: cannot stat '../../data/clang8_train/src.txt.conll_predict_gopar_np': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.conll_predict_gopar_np': No such file or directory
Calculate dependency distance...
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/calculate_dependency_distance.py", line 126, in
with open(conll_file, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/clang8_train/src.txt.conll_predict_gopar'
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/calculate_dependency_distance.py", line 126, in
with open(conll_file, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt.conll_predict_gopar'
cp: cannot stat '../../data/clang8_train/src.txt.conll_predict_gopar_np.dpd': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.conll_predict_gopar_np.dpd': No such file or directory
cp: cannot stat '../../data/clang8_train/src.txt.conll_predict_gopar_np.probs': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.conll_predict_gopar_np.probs': No such file or directory
Preprocess...
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
usage: preprocess.py [-h] [--no-progress-bar]
[--log-interval LOG_INTERVAL]
[--log-format LOG_FORMAT]
[--tensorboard-logdir TENSORBOARD_LOGDIR]
[--seed SEED] [--cpu] [--tpu] [--bf16]
[--memory-efficient-bf16] [--fp16]
[--memory-efficient-fp16]
[--fp16-no-flatten-grads]
[--fp16-init-scale FP16_INIT_SCALE]
[--fp16-scale-window FP16_SCALE_WINDOW]
[--fp16-scale-tolerance FP16_SCALE_TOLERANCE]
[--min-loss-scale MIN_LOSS_SCALE]
[--threshold-loss-scale THRESHOLD_LOSS_SCALE]
[--user-dir USER_DIR]
[--empty-cache-freq EMPTY_CACHE_FREQ]
[--all-gather-list-size ALL_GATHER_LIST_SIZE]
[--model-parallel-size MODEL_PARALLEL_SIZE]
[--checkpoint-suffix CHECKPOINT_SUFFIX]
[--checkpoint-shard-count CHECKPOINT_SHARD_COUNT]
[--quantization-config-path QUANTIZATION_CONFIG_PATH]
[--profile]
[--criterion {label_smoothed_cross_entropy,composite_loss,ctc,masked_lm,legacy_masked_lm_loss,wav2vec,sentence_prediction,label_smoothed_cross_entropy_with_alignment,cross_entropy,nat_loss,adaptive_loss,sentence_ranking,vocab_parallel_cross_entropy}]
[--tokenizer {space,moses,nltk}]
[--bpe {characters,bert,subword_nmt,fastbpe,byte_bpe,bytes,sentencepiece,gpt2,hf_byte_bpe}]
[--optimizer {adafactor,nag,adam,adagrad,adamax,lamb,sgd,adadelta}]
[--lr-scheduler {cosine,triangular,inverse_sqrt,tri_stage,fixed,reduce_lr_on_plateau,polynomial_decay}]
[--scoring {wer,sacrebleu,bleu,chrf}]
[--task TASK] [-s SRC] [-t TARGET]
[--source-lang-with-nt SRC] [--trainpref FP]
[--validpref FP] [--testpref FP]
[--align-suffix FP] [--conll-suffix FP [FP ...]]
[--dpd-suffix FP [FP ...]]
[--probs-suffix FP [FP ...]] [--swm-suffix FP]
[--destdir DIR] [--thresholdtgt N]
[--thresholdsrc N] [--tgtdict FP] [--srcdict FP]
[--labeldict FP [FP ...]] [--nwordstgt N]
[--nwordssrc N] [--alignfile ALIGN]
[--dataset-impl FORMAT] [--joined-dictionary]
[--only-source] [--padding-factor N]
[--workers N]
preprocess.py: error: argument --user-dir: invalid Optional value: '../../src/src_syngec/syngec_model'
Finished!
Apply BPE...
./preprocess_syngec_transformer.sh: line 112: ../../data/error_coded_train/src.txt: No such file or directory
./preprocess_syngec_transformer.sh: line 113: ../../data/error_coded_train/tgt.txt: No such file or directory
./preprocess_syngec_transformer.sh: line 114: ../../data/bea19_dev/src.txt: No such file or directory
./preprocess_syngec_transformer.sh: line 115: ../../data/bea19_dev/tgt.txt: No such file or directory
Align subwords and words...
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/subword_align.py", line 41, in
with open(file_word, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/error_coded_train/src.txt'
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/subword_align.py", line 41, in
with open(file_word, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt'
cp: cannot stat '../../data/error_coded_train/src.txt': No such file or directory
cp: cannot stat '../../data/error_coded_train/src.txt.bpe': No such file or directory
cp: cannot stat '../../data/error_coded_train/tgt.txt': No such file or directory
cp: cannot stat '../../data/error_coded_train/tgt.txt.bpe': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.bpe': No such file or directory
cp: cannot stat '../../data/bea19_dev/tgt.txt': No such file or directory
cp: cannot stat '../../data/bea19_dev/tgt.txt.bpe': No such file or directory
cp: cannot stat '../../data/error_coded_train/src.txt.swm': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.swm': No such file or directory
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/error_coded_train/src.txt.swm'
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/error_coded_train/src.txt.swm'
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt.swm'
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt.swm'
cp: cannot stat '../../data/error_coded_train/src.txt.conll_predict_gopar_np': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.conll_predict_gopar_np': No such file or directory
Calculate dependency distance...
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/calculate_dependency_distance.py", line 126, in
with open(conll_file, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/error_coded_train/src.txt.conll_predict_gopar'
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/calculate_dependency_distance.py", line 126, in
with open(conll_file, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt.conll_predict_gopar'
cp: cannot stat '../../data/error_coded_train/src.txt.conll_predict_gopar_np.dpd': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.conll_predict_gopar_np.dpd': No such file or directory
cp: cannot stat '../../data/error_coded_train/src.txt.conll_predict_gopar_np.probs': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.conll_predict_gopar_np.probs': No such file or directory
Preprocess...
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
usage: preprocess.py [-h] [--no-progress-bar]
[--log-interval LOG_INTERVAL]
[--log-format LOG_FORMAT]
[--tensorboard-logdir TENSORBOARD_LOGDIR]
[--seed SEED] [--cpu] [--tpu] [--bf16]
[--memory-efficient-bf16] [--fp16]
[--memory-efficient-fp16]
[--fp16-no-flatten-grads]
[--fp16-init-scale FP16_INIT_SCALE]
[--fp16-scale-window FP16_SCALE_WINDOW]
[--fp16-scale-tolerance FP16_SCALE_TOLERANCE]
[--min-loss-scale MIN_LOSS_SCALE]
[--threshold-loss-scale THRESHOLD_LOSS_SCALE]
[--user-dir USER_DIR]
[--empty-cache-freq EMPTY_CACHE_FREQ]
[--all-gather-list-size ALL_GATHER_LIST_SIZE]
[--model-parallel-size MODEL_PARALLEL_SIZE]
[--checkpoint-suffix CHECKPOINT_SUFFIX]
[--checkpoint-shard-count CHECKPOINT_SHARD_COUNT]
[--quantization-config-path QUANTIZATION_CONFIG_PATH]
[--profile]
[--criterion {label_smoothed_cross_entropy,composite_loss,ctc,masked_lm,legacy_masked_lm_loss,wav2vec,sentence_prediction,label_smoothed_cross_entropy_with_alignment,cross_entropy,nat_loss,adaptive_loss,sentence_ranking,vocab_parallel_cross_entropy}]
[--tokenizer {space,moses,nltk}]
[--bpe {characters,bert,subword_nmt,fastbpe,byte_bpe,bytes,sentencepiece,gpt2,hf_byte_bpe}]
[--optimizer {adafactor,nag,adam,adagrad,adamax,lamb,sgd,adadelta}]
[--lr-scheduler {cosine,triangular,inverse_sqrt,tri_stage,fixed,reduce_lr_on_plateau,polynomial_decay}]
[--scoring {wer,sacrebleu,bleu,chrf}]
[--task TASK] [-s SRC] [-t TARGET]
[--source-lang-with-nt SRC] [--trainpref FP]
[--validpref FP] [--testpref FP]
[--align-suffix FP] [--conll-suffix FP [FP ...]]
[--dpd-suffix FP [FP ...]]
[--probs-suffix FP [FP ...]] [--swm-suffix FP]
[--destdir DIR] [--thresholdtgt N]
[--thresholdsrc N] [--tgtdict FP] [--srcdict FP]
[--labeldict FP [FP ...]] [--nwordstgt N]
[--nwordssrc N] [--alignfile ALIGN]
[--dataset-impl FORMAT] [--joined-dictionary]
[--only-source] [--padding-factor N]
[--workers N]
preprocess.py: error: argument --user-dir: invalid Optional value: '../../src/src_syngec/syngec_model'
Finished!
Apply BPE...
./preprocess_syngec_transformer.sh: line 202: ../../data/wi_locness_train/src.txt: No such file or directory
./preprocess_syngec_transformer.sh: line 203: ../../data/wi_locness_train/tgt.txt: No such file or directory
./preprocess_syngec_transformer.sh: line 204: ../../data/bea19_dev/src.txt: No such file or directory
./preprocess_syngec_transformer.sh: line 205: ../../data/bea19_dev/tgt.txt: No such file or directory
Align subwords and words...
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/subword_align.py", line 41, in
with open(file_word, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/wi_locness_train/src.txt'
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/subword_align.py", line 41, in
with open(file_word, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt'
cp: cannot stat '../../data/wi_locness_train/src.txt': No such file or directory
cp: cannot stat '../../data/wi_locness_train/src.txt.bpe': No such file or directory
cp: cannot stat '../../data/wi_locness_train/tgt.txt': No such file or directory
cp: cannot stat '../../data/wi_locness_train/tgt.txt.bpe': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.bpe': No such file or directory
cp: cannot stat '../../data/bea19_dev/tgt.txt': No such file or directory
cp: cannot stat '../../data/bea19_dev/tgt.txt.bpe': No such file or directory
cp: cannot stat '../../data/wi_locness_train/src.txt.swm': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.swm': No such file or directory
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/wi_locness_train/src.txt.swm'
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/wi_locness_train/src.txt.swm'
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt.swm'
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt.swm'
cp: cannot stat '../../data/wi_locness_train/src.txt.conll_predict_gopar_np': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.conll_predict_gopar_np': No such file or directory
Calculate dependency distance...
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/calculate_dependency_distance.py", line 126, in
with open(conll_file, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/wi_locness_train/src.txt.conll_predict_gopar'
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/calculate_dependency_distance.py", line 126, in
with open(conll_file, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_dev/src.txt.conll_predict_gopar'
cp: cannot stat '../../data/wi_locness_train/src.txt.conll_predict_gopar_np.dpd': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.conll_predict_gopar_np.dpd': No such file or directory
cp: cannot stat '../../data/wi_locness_train/src.txt.conll_predict_gopar_np.probs': No such file or directory
cp: cannot stat '../../data/bea19_dev/src.txt.conll_predict_gopar_np.probs': No such file or directory
Preprocess...
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
usage: preprocess.py [-h] [--no-progress-bar]
[--log-interval LOG_INTERVAL]
[--log-format LOG_FORMAT]
[--tensorboard-logdir TENSORBOARD_LOGDIR]
[--seed SEED] [--cpu] [--tpu] [--bf16]
[--memory-efficient-bf16] [--fp16]
[--memory-efficient-fp16]
[--fp16-no-flatten-grads]
[--fp16-init-scale FP16_INIT_SCALE]
[--fp16-scale-window FP16_SCALE_WINDOW]
[--fp16-scale-tolerance FP16_SCALE_TOLERANCE]
[--min-loss-scale MIN_LOSS_SCALE]
[--threshold-loss-scale THRESHOLD_LOSS_SCALE]
[--user-dir USER_DIR]
[--empty-cache-freq EMPTY_CACHE_FREQ]
[--all-gather-list-size ALL_GATHER_LIST_SIZE]
[--model-parallel-size MODEL_PARALLEL_SIZE]
[--checkpoint-suffix CHECKPOINT_SUFFIX]
[--checkpoint-shard-count CHECKPOINT_SHARD_COUNT]
[--quantization-config-path QUANTIZATION_CONFIG_PATH]
[--profile]
[--criterion {label_smoothed_cross_entropy,composite_loss,ctc,masked_lm,legacy_masked_lm_loss,wav2vec,sentence_prediction,label_smoothed_cross_entropy_with_alignment,cross_entropy,nat_loss,adaptive_loss,sentence_ranking,vocab_parallel_cross_entropy}]
[--tokenizer {space,moses,nltk}]
[--bpe {characters,bert,subword_nmt,fastbpe,byte_bpe,bytes,sentencepiece,gpt2,hf_byte_bpe}]
[--optimizer {adafactor,nag,adam,adagrad,adamax,lamb,sgd,adadelta}]
[--lr-scheduler {cosine,triangular,inverse_sqrt,tri_stage,fixed,reduce_lr_on_plateau,polynomial_decay}]
[--scoring {wer,sacrebleu,bleu,chrf}]
[--task TASK] [-s SRC] [-t TARGET]
[--source-lang-with-nt SRC] [--trainpref FP]
[--validpref FP] [--testpref FP]
[--align-suffix FP] [--conll-suffix FP [FP ...]]
[--dpd-suffix FP [FP ...]]
[--probs-suffix FP [FP ...]] [--swm-suffix FP]
[--destdir DIR] [--thresholdtgt N]
[--thresholdsrc N] [--tgtdict FP] [--srcdict FP]
[--labeldict FP [FP ...]] [--nwordstgt N]
[--nwordssrc N] [--alignfile ALIGN]
[--dataset-impl FORMAT] [--joined-dictionary]
[--only-source] [--padding-factor N]
[--workers N]
preprocess.py: error: argument --user-dir: invalid Optional value: '../../src/src_syngec/syngec_model'
Finished!
Apply BPE...
./preprocess_syngec_transformer.sh: line 290: ../../data/conll14_test/src.txt: No such file or directory
Align subwords and words...
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/subword_align.py", line 41, in
with open(file_word, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/conll14_test/src.txt'
cp: cannot stat '../../data/conll14_test/src.txt': No such file or directory
cp: cannot stat '../../data/conll14_test/src.txt.bpe': No such file or directory
cp: cannot stat '../../data/conll14_test/src.txt.swm': No such file or directory
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/conll14_test/src.txt.swm'
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/conll14_test/src.txt.swm'
cp: cannot stat '../../data/conll14_test/src.txt.conll_predict_gopar_np': No such file or directory
Calculate dependency distance...
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/calculate_dependency_distance.py", line 126, in
with open(conll_file, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/conll14_test/src.txt.conll_predict_gopar'
cp: cannot stat '../../data/conll14_test/src.txt.conll_predict_gopar_np.dpd': No such file or directory
cp: cannot stat '../../data/conll14_test/src.txt.conll_predict_gopar_np.probs': No such file or directory
Preprocess...
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
usage: preprocess.py [-h] [--no-progress-bar]
[--log-interval LOG_INTERVAL]
[--log-format LOG_FORMAT]
[--tensorboard-logdir TENSORBOARD_LOGDIR]
[--seed SEED] [--cpu] [--tpu] [--bf16]
[--memory-efficient-bf16] [--fp16]
[--memory-efficient-fp16]
[--fp16-no-flatten-grads]
[--fp16-init-scale FP16_INIT_SCALE]
[--fp16-scale-window FP16_SCALE_WINDOW]
[--fp16-scale-tolerance FP16_SCALE_TOLERANCE]
[--min-loss-scale MIN_LOSS_SCALE]
[--threshold-loss-scale THRESHOLD_LOSS_SCALE]
[--user-dir USER_DIR]
[--empty-cache-freq EMPTY_CACHE_FREQ]
[--all-gather-list-size ALL_GATHER_LIST_SIZE]
[--model-parallel-size MODEL_PARALLEL_SIZE]
[--checkpoint-suffix CHECKPOINT_SUFFIX]
[--checkpoint-shard-count CHECKPOINT_SHARD_COUNT]
[--quantization-config-path QUANTIZATION_CONFIG_PATH]
[--profile]
[--criterion {label_smoothed_cross_entropy,composite_loss,ctc,masked_lm,legacy_masked_lm_loss,wav2vec,sentence_prediction,label_smoothed_cross_entropy_with_alignment,cross_entropy,nat_loss,adaptive_loss,sentence_ranking,vocab_parallel_cross_entropy}]
[--tokenizer {space,moses,nltk}]
[--bpe {characters,bert,subword_nmt,fastbpe,byte_bpe,bytes,sentencepiece,gpt2,hf_byte_bpe}]
[--optimizer {adafactor,nag,adam,adagrad,adamax,lamb,sgd,adadelta}]
[--lr-scheduler {cosine,triangular,inverse_sqrt,tri_stage,fixed,reduce_lr_on_plateau,polynomial_decay}]
[--scoring {wer,sacrebleu,bleu,chrf}]
[--task TASK] [-s SRC] [-t TARGET]
[--source-lang-with-nt SRC] [--trainpref FP]
[--validpref FP] [--testpref FP]
[--align-suffix FP] [--conll-suffix FP [FP ...]]
[--dpd-suffix FP [FP ...]]
[--probs-suffix FP [FP ...]] [--swm-suffix FP]
[--destdir DIR] [--thresholdtgt N]
[--thresholdsrc N] [--tgtdict FP] [--srcdict FP]
[--labeldict FP [FP ...]] [--nwordstgt N]
[--nwordssrc N] [--alignfile ALIGN]
[--dataset-impl FORMAT] [--joined-dictionary]
[--only-source] [--padding-factor N]
[--workers N]
preprocess.py: error: argument --user-dir: invalid Optional value: '../../src/src_syngec/syngec_model'
Finished!
Apply BPE...
./preprocess_syngec_transformer.sh: line 358: ../../data/bea19_test/src.txt: No such file or directory
Align subwords and words...
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/subword_align.py", line 41, in
with open(file_word, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_test/src.txt'
cp: cannot stat '../../data/bea19_test/src.txt': No such file or directory
cp: cannot stat '../../data/bea19_test/src.txt.bpe': No such file or directory
cp: cannot stat '../../data/bea19_test/src.txt.swm': No such file or directory
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_test/src.txt.swm'
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/syntax_information_reprocess.py", line 37, in
swm_list = [[int(i) for i in line.rstrip("\n").split()] for line in open(swm_file, "r").readlines()]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_test/src.txt.swm'
cp: cannot stat '../../data/bea19_test/src.txt.conll_predict_gopar_np': No such file or directory
Calculate dependency distance...
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../utils/calculate_dependency_distance.py", line 126, in
with open(conll_file, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/bea19_test/src.txt.conll_predict_gopar'
cp: cannot stat '../../data/bea19_test/src.txt.conll_predict_gopar_np.dpd': No such file or directory
cp: cannot stat '../../data/bea19_test/src.txt.conll_predict_gopar_np.probs': No such file or directory
Preprocess...
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
usage: preprocess.py [-h] [--no-progress-bar]
[--log-interval LOG_INTERVAL]
[--log-format LOG_FORMAT]
[--tensorboard-logdir TENSORBOARD_LOGDIR]
[--seed SEED] [--cpu] [--tpu] [--bf16]
[--memory-efficient-bf16] [--fp16]
[--memory-efficient-fp16]
[--fp16-no-flatten-grads]
[--fp16-init-scale FP16_INIT_SCALE]
[--fp16-scale-window FP16_SCALE_WINDOW]
[--fp16-scale-tolerance FP16_SCALE_TOLERANCE]
[--min-loss-scale MIN_LOSS_SCALE]
[--threshold-loss-scale THRESHOLD_LOSS_SCALE]
[--user-dir USER_DIR]
[--empty-cache-freq EMPTY_CACHE_FREQ]
[--all-gather-list-size ALL_GATHER_LIST_SIZE]
[--model-parallel-size MODEL_PARALLEL_SIZE]
[--checkpoint-suffix CHECKPOINT_SUFFIX]
[--checkpoint-shard-count CHECKPOINT_SHARD_COUNT]
[--quantization-config-path QUANTIZATION_CONFIG_PATH]
[--profile]
[--criterion {label_smoothed_cross_entropy,composite_loss,ctc,masked_lm,legacy_masked_lm_loss,wav2vec,sentence_prediction,label_smoothed_cross_entropy_with_alignment,cross_entropy,nat_loss,adaptive_loss,sentence_ranking,vocab_parallel_cross_entropy}]
[--tokenizer {space,moses,nltk}]
[--bpe {characters,bert,subword_nmt,fastbpe,byte_bpe,bytes,sentencepiece,gpt2,hf_byte_bpe}]
[--optimizer {adafactor,nag,adam,adagrad,adamax,lamb,sgd,adadelta}]
[--lr-scheduler {cosine,triangular,inverse_sqrt,tri_stage,fixed,reduce_lr_on_plateau,polynomial_decay}]
[--scoring {wer,sacrebleu,bleu,chrf}]
[--task TASK] [-s SRC] [-t TARGET]
[--source-lang-with-nt SRC] [--trainpref FP]
[--validpref FP] [--testpref FP]
[--align-suffix FP] [--conll-suffix FP [FP ...]]
[--dpd-suffix FP [FP ...]]
[--probs-suffix FP [FP ...]] [--swm-suffix FP]
[--destdir DIR] [--thresholdtgt N]
[--thresholdsrc N] [--tgtdict FP] [--srcdict FP]
[--labeldict FP [FP ...]] [--nwordstgt N]
[--nwordssrc N] [--alignfile ALIGN]
[--dataset-impl FORMAT] [--joined-dictionary]
[--only-source] [--padding-factor N]
[--workers N]
preprocess.py: error: argument --user-dir: invalid Optional value: '../../src/src_syngec/syngec_model'
Finished!

[HillZhang1999]

please enter the directory of this bash file, then run cat ../../data/clang8_train/src.txt, check whether there is actually a file. If not, please check the way you unzip the data.

[YeJinPaark]

First I use the unzip like "tar -zxvf syngec_preprocess.tar.gz"

and then the log is
preprocess/
preprocess/chinese_hsk+lang8_with_syntax_transformer/
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/dict.label.txt
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/dict.src.txt
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/dict.tgt.txt
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/preprocess.log
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/train.conll.src-tgt.src.bin
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/train.conll.src-tgt.src.idx
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/train.dpd.src-tgt.src.bin
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/train.dpd.src-tgt.src.idx
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/train.probs.src-tgt.src.bin
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/train.probs.src-tgt.src.idx
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/train.src-tgt.src.bin
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/train.src-tgt.src.idx
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/train.src-tgt.tgt.bin
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/train.src-tgt.tgt.idx
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/valid.conll.src-tgt.src.bin
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/valid.conll.src-tgt.src.idx
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/valid.dpd.src-tgt.src.bin
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/valid.dpd.src-tgt.src.idx
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/valid.probs.src-tgt.src.bin
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/valid.probs.src-tgt.src.idx
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/valid.src-tgt.src.bin
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/valid.src-tgt.src.idx
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/valid.src-tgt.tgt.bin
preprocess/chinese_hsk+lang8_with_syntax_transformer/bin/valid.src-tgt.tgt.idx
preprocess/chinese_mucgec_with_syntax_transformer/
preprocess/chinese_mucgec_with_syntax_transformer/bin/
preprocess/chinese_mucgec_with_syntax_transformer/bin/dict.label.txt
preprocess/chinese_mucgec_with_syntax_transformer/bin/dict.src.txt
preprocess/chinese_mucgec_with_syntax_transformer/bin/preprocess.log
preprocess/chinese_mucgec_with_syntax_transformer/bin/test.conll.src-tgt.src.bin
preprocess/chinese_mucgec_with_syntax_transformer/bin/test.conll.src-tgt.src.idx
preprocess/chinese_mucgec_with_syntax_transformer/bin/test.dpd.src-tgt.src.bin
preprocess/chinese_mucgec_with_syntax_transformer/bin/test.dpd.src-tgt.src.idx
preprocess/chinese_mucgec_with_syntax_transformer/bin/test.probs.src-tgt.src.bin
preprocess/chinese_mucgec_with_syntax_transformer/bin/test.probs.src-tgt.src.idx
preprocess/chinese_mucgec_with_syntax_transformer/bin/test.src-tgt.src.bin
preprocess/chinese_mucgec_with_syntax_transformer/bin/test.src-tgt.src.idx
preprocess/english_bea19_with_syntax_bart/
preprocess/english_bea19_with_syntax_bart/bin/
preprocess/english_bea19_with_syntax_bart/bin/dict.label.txt
preprocess/english_bea19_with_syntax_bart/bin/dict.src.txt
preprocess/english_bea19_with_syntax_bart/bin/preprocess.log
preprocess/english_bea19_with_syntax_bart/bin/test.conll.src-tgt.src.bin
preprocess/english_bea19_with_syntax_bart/bin/test.conll.src-tgt.src.idx
preprocess/english_bea19_with_syntax_bart/bin/test.dpd.src-tgt.src.bin
preprocess/english_bea19_with_syntax_bart/bin/test.dpd.src-tgt.src.idx
preprocess/english_bea19_with_syntax_bart/bin/test.probs.src-tgt.src.bin
preprocess/english_bea19_with_syntax_bart/bin/test.probs.src-tgt.src.idx
preprocess/english_bea19_with_syntax_bart/bin/test.src-tgt.src.bin
preprocess/english_bea19_with_syntax_bart/bin/test.src-tgt.src.idx
preprocess/english_bea19_with_syntax_transformer/
preprocess/english_bea19_with_syntax_transformer/dict.label.txt
preprocess/english_bea19_with_syntax_transformer/dict.src.txt
preprocess/english_bea19_with_syntax_transformer/preprocess.log
preprocess/english_bea19_with_syntax_transformer/test.conll.src-tgt.src.bin
preprocess/english_bea19_with_syntax_transformer/test.conll.src-tgt.src.idx
preprocess/english_bea19_with_syntax_transformer/test.dpd.src-tgt.src.bin
preprocess/english_bea19_with_syntax_transformer/test.dpd.src-tgt.src.idx
preprocess/english_bea19_with_syntax_transformer/test.probs.src-tgt.src.bin
preprocess/english_bea19_with_syntax_transformer/test.probs.src-tgt.src.idx
preprocess/english_bea19_with_syntax_transformer/test.src-tgt.src.bin
preprocess/english_bea19_with_syntax_transformer/test.src-tgt.src.idx
preprocess/english_clang8_with_syntax_bart/
preprocess/english_clang8_with_syntax_bart/bin/
preprocess/english_clang8_with_syntax_bart/bin/dict.label.txt
preprocess/english_clang8_with_syntax_bart/bin/dict.src.txt
preprocess/english_clang8_with_syntax_bart/bin/dict.tgt.txt
preprocess/english_clang8_with_syntax_bart/bin/preprocess.log
preprocess/english_clang8_with_syntax_bart/bin/train.conll.src-tgt.src.bin
preprocess/english_clang8_with_syntax_bart/bin/train.conll.src-tgt.src.idx
preprocess/english_clang8_with_syntax_bart/bin/train.dpd.src-tgt.src.bin
preprocess/english_clang8_with_syntax_bart/bin/train.dpd.src-tgt.src.idx
preprocess/english_clang8_with_syntax_bart/bin/train.probs.src-tgt.src.bin
preprocess/english_clang8_with_syntax_bart/bin/train.probs.src-tgt.src.idx
preprocess/english_clang8_with_syntax_bart/bin/train.src-tgt.src.bin
preprocess/english_clang8_with_syntax_bart/bin/train.src-tgt.src.idx
preprocess/english_clang8_with_syntax_bart/bin/train.src-tgt.tgt.bin
preprocess/english_clang8_with_syntax_bart/bin/train.src-tgt.tgt.idx
preprocess/english_clang8_with_syntax_bart/bin/valid.conll.src-tgt.src.bin
preprocess/english_clang8_with_syntax_bart/bin/valid.conll.src-tgt.src.idx
preprocess/english_clang8_with_syntax_bart/bin/valid.dpd.src-tgt.src.bin
preprocess/english_clang8_with_syntax_bart/bin/valid.dpd.src-tgt.src.idx
preprocess/english_clang8_with_syntax_bart/bin/valid.probs.src-tgt.src.bin
preprocess/english_clang8_with_syntax_bart/bin/valid.probs.src-tgt.src.idx
preprocess/english_clang8_with_syntax_bart/bin/valid.src-tgt.src.bin
preprocess/english_clang8_with_syntax_bart/bin/valid.src-tgt.src.idx
preprocess/english_clang8_with_syntax_bart/bin/valid.src-tgt.tgt.bin
preprocess/english_clang8_with_syntax_bart/bin/valid.src-tgt.tgt.idx
preprocess/english_clang8_with_syntax_transformer/
preprocess/english_clang8_with_syntax_transformer/bin/
preprocess/english_clang8_with_syntax_transformer/bin/dict.label.txt
preprocess/english_clang8_with_syntax_transformer/bin/dict.src.txt
preprocess/english_clang8_with_syntax_transformer/bin/dict.tgt.txt
preprocess/english_clang8_with_syntax_transformer/bin/preprocess.log
preprocess/english_clang8_with_syntax_transformer/bin/train.conll.src-tgt.src.bin
preprocess/english_clang8_with_syntax_transformer/bin/train.conll.src-tgt.src.idx
preprocess/english_clang8_with_syntax_transformer/bin/train.dpd.src-tgt.src.bin
preprocess/english_clang8_with_syntax_transformer/bin/train.dpd.src-tgt.src.idx
preprocess/english_clang8_with_syntax_transformer/bin/train.probs.src-tgt.src.bin
preprocess/english_clang8_with_syntax_transformer/bin/train.probs.src-tgt.src.idx
preprocess/english_clang8_with_syntax_transformer/bin/train.src-tgt.src.bin
preprocess/english_clang8_with_syntax_transformer/bin/train.src-tgt.src.idx
preprocess/english_clang8_with_syntax_transformer/bin/train.src-tgt.tgt.bin
preprocess/english_clang8_with_syntax_transformer/bin/train.src-tgt.tgt.idx
preprocess/english_clang8_with_syntax_transformer/bin/valid.conll.src-tgt.src.bin
preprocess/english_clang8_with_syntax_transformer/bin/valid.conll.src-tgt.src.idx
preprocess/english_clang8_with_syntax_transformer/bin/valid.dpd.src-tgt.src.bin
preprocess/english_clang8_with_syntax_transformer/bin/valid.dpd.src-tgt.src.idx
preprocess/english_clang8_with_syntax_transformer/bin/valid.probs.src-tgt.src.bin
preprocess/english_clang8_with_syntax_transformer/bin/valid.probs.src-tgt.src.idx
preprocess/english_clang8_with_syntax_transformer/bin/valid.src-tgt.src.bin
preprocess/english_clang8_with_syntax_transformer/bin/valid.src-tgt.src.idx
preprocess/english_clang8_with_syntax_transformer/bin/valid.src-tgt.tgt.bin
preprocess/english_clang8_with_syntax_transformer/bin/valid.src-tgt.tgt.idx
preprocess/english_conll14_with_syntax_bart/
preprocess/english_conll14_with_syntax_bart/bin/
preprocess/english_conll14_with_syntax_bart/bin/dict.label.txt
preprocess/english_conll14_with_syntax_bart/bin/dict.src.txt
preprocess/english_conll14_with_syntax_bart/bin/preprocess.log
preprocess/english_conll14_with_syntax_bart/bin/test.conll.src-tgt.src.bin
preprocess/english_conll14_with_syntax_bart/bin/test.conll.src-tgt.src.idx
preprocess/english_conll14_with_syntax_bart/bin/test.dpd.src-tgt.src.bin
preprocess/english_conll14_with_syntax_bart/bin/test.dpd.src-tgt.src.idx
preprocess/english_conll14_with_syntax_bart/bin/test.probs.src-tgt.src.bin
preprocess/english_conll14_with_syntax_bart/bin/test.probs.src-tgt.src.idx
preprocess/english_conll14_with_syntax_bart/bin/test.src-tgt.src.bin
preprocess/english_conll14_with_syntax_bart/bin/test.src-tgt.src.idx
preprocess/english_conll14_with_syntax_transformer/
preprocess/english_conll14_with_syntax_transformer/dict.label.txt
preprocess/english_conll14_with_syntax_transformer/dict.src.txt
preprocess/english_conll14_with_syntax_transformer/preprocess.log
preprocess/english_conll14_with_syntax_transformer/test.conll.src-tgt.src.bin
preprocess/english_conll14_with_syntax_transformer/test.conll.src-tgt.src.idx
preprocess/english_conll14_with_syntax_transformer/test.dpd.src-tgt.src.bin
preprocess/english_conll14_with_syntax_transformer/test.dpd.src-tgt.src.idx
preprocess/english_conll14_with_syntax_transformer/test.probs.src-tgt.src.bin
preprocess/english_conll14_with_syntax_transformer/test.probs.src-tgt.src.idx
preprocess/english_conll14_with_syntax_transformer/test.src-tgt.src.bin
preprocess/english_conll14_with_syntax_transformer/test.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_bart/
preprocess/english_error_coded_with_syntax_bart/bin/
preprocess/english_error_coded_with_syntax_bart/bin/dict.label.txt
preprocess/english_error_coded_with_syntax_bart/bin/dict.src.txt
preprocess/english_error_coded_with_syntax_bart/bin/dict.tgt.txt
preprocess/english_error_coded_with_syntax_bart/bin/preprocess.log
preprocess/english_error_coded_with_syntax_bart/bin/train.conll.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_bart/bin/train.conll.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_bart/bin/train.dpd.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_bart/bin/train.dpd.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_bart/bin/train.probs.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_bart/bin/train.probs.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_bart/bin/train.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_bart/bin/train.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_bart/bin/train.src-tgt.tgt.bin
preprocess/english_error_coded_with_syntax_bart/bin/train.src-tgt.tgt.idx
preprocess/english_error_coded_with_syntax_bart/bin/valid.conll.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_bart/bin/valid.conll.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_bart/bin/valid.dpd.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_bart/bin/valid.dpd.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_bart/bin/valid.probs.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_bart/bin/valid.probs.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_bart/bin/valid.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_bart/bin/valid.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_bart/bin/valid.src-tgt.tgt.bin
preprocess/english_error_coded_with_syntax_bart/bin/valid.src-tgt.tgt.idx
preprocess/english_error_coded_with_syntax_transformer/
preprocess/english_error_coded_with_syntax_transformer/bin/
preprocess/english_error_coded_with_syntax_transformer/bin/dict.label.txt
preprocess/english_error_coded_with_syntax_transformer/bin/dict.src.txt
preprocess/english_error_coded_with_syntax_transformer/bin/dict.tgt.txt
preprocess/english_error_coded_with_syntax_transformer/bin/preprocess.log
preprocess/english_error_coded_with_syntax_transformer/bin/train.conll.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_transformer/bin/train.conll.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_transformer/bin/train.dpd.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_transformer/bin/train.dpd.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_transformer/bin/train.probs.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_transformer/bin/train.probs.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_transformer/bin/train.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_transformer/bin/train.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_transformer/bin/train.src-tgt.tgt.bin
preprocess/english_error_coded_with_syntax_transformer/bin/train.src-tgt.tgt.idx
preprocess/english_error_coded_with_syntax_transformer/bin/valid.conll.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_transformer/bin/valid.conll.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_transformer/bin/valid.dpd.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_transformer/bin/valid.dpd.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_transformer/bin/valid.probs.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_transformer/bin/valid.probs.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_transformer/bin/valid.src-tgt.src.bin
preprocess/english_error_coded_with_syntax_transformer/bin/valid.src-tgt.src.idx
preprocess/english_error_coded_with_syntax_transformer/bin/valid.src-tgt.tgt.bin
preprocess/english_error_coded_with_syntax_transformer/bin/valid.src-tgt.tgt.idx
preprocess/english_wi_locness_with_syntax_bart/
preprocess/english_wi_locness_with_syntax_bart/bin/
preprocess/english_wi_locness_with_syntax_bart/bin/dict.label.txt
preprocess/english_wi_locness_with_syntax_bart/bin/dict.src.txt
preprocess/english_wi_locness_with_syntax_bart/bin/dict.tgt.txt
preprocess/english_wi_locness_with_syntax_bart/bin/preprocess.log
preprocess/english_wi_locness_with_syntax_bart/bin/train.conll.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_bart/bin/train.conll.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_bart/bin/train.dpd.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_bart/bin/train.dpd.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_bart/bin/train.probs.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_bart/bin/train.probs.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_bart/bin/train.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_bart/bin/train.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_bart/bin/train.src-tgt.tgt.bin
preprocess/english_wi_locness_with_syntax_bart/bin/train.src-tgt.tgt.idx
preprocess/english_wi_locness_with_syntax_bart/bin/valid.conll.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_bart/bin/valid.conll.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_bart/bin/valid.dpd.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_bart/bin/valid.dpd.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_bart/bin/valid.probs.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_bart/bin/valid.probs.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_bart/bin/valid.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_bart/bin/valid.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_bart/bin/valid.src-tgt.tgt.bin
preprocess/english_wi_locness_with_syntax_bart/bin/valid.src-tgt.tgt.idx
preprocess/english_wi_locness_with_syntax_transformer/
preprocess/english_wi_locness_with_syntax_transformer/bin/
preprocess/english_wi_locness_with_syntax_transformer/bin/dict.label.txt
preprocess/english_wi_locness_with_syntax_transformer/bin/dict.src.txt
preprocess/english_wi_locness_with_syntax_transformer/bin/dict.tgt.txt
preprocess/english_wi_locness_with_syntax_transformer/bin/preprocess.log
preprocess/english_wi_locness_with_syntax_transformer/bin/train.conll.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_transformer/bin/train.conll.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_transformer/bin/train.dpd.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_transformer/bin/train.dpd.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_transformer/bin/train.probs.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_transformer/bin/train.probs.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_transformer/bin/train.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_transformer/bin/train.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_transformer/bin/train.src-tgt.tgt.bin
preprocess/english_wi_locness_with_syntax_transformer/bin/train.src-tgt.tgt.idx
preprocess/english_wi_locness_with_syntax_transformer/bin/valid.conll.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_transformer/bin/valid.conll.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_transformer/bin/valid.dpd.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_transformer/bin/valid.dpd.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_transformer/bin/valid.probs.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_transformer/bin/valid.probs.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_transformer/bin/valid.src-tgt.src.bin
preprocess/english_wi_locness_with_syntax_transformer/bin/valid.src-tgt.src.idx
preprocess/english_wi_locness_with_syntax_transformer/bin/valid.src-tgt.tgt.bin
preprocess/english_wi_locness_with_syntax_transformer/bin/valid.src-tgt.tgt.idx

and I run the bash file:

root@309e7fc0781e:/mnt/ssd_mnt/pyj/SynGEC/data# cd /mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/
root@309e7fc0781e:/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp# ls
generate_syngec_bart.sh preprocess_syngec_bart.sh
generate_syngec_transformer.sh preprocess_syngec_transformer.sh
nohup.out train_syngec_bart.sh
pipeline_gopar.sh train_syngec_transformer.sh
root@309e7fc0781e:/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp# ./pipeline_gopar.sh
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../src/src_gopar/parse.py", line 17, in
input_sentences = load(sys.argv[1])
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../src/src_gopar/parse.py", line 9, in load
with open(filename, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/wi_locness_train/tgt.txt'
Loading resources...
Processing parallel files...
Traceback (most recent call last):
File "/opt/conda/bin/errant_parallel", line 8, in
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/errant/commands/parallel_to_m2.py", line 16, in main
in_files = [stack.enter_context(open(i)) for i in [args.orig]+args.cor]
File "/opt/conda/lib/python3.10/site-packages/errant/commands/parallel_to_m2.py", line 16, in
in_files = [stack.enter_context(open(i)) for i in [args.orig]+args.cor]
FileNotFoundError: [Errno 2] No such file or directory: '../../data/wi_locness_train/tgt.txt'
Traceback (most recent call last):
File "/mnt/ssd_mnt/pyj/SynGEC/bash/english_exp/../../src/src_gopar/convert_gec_data_to_parsing_data_english.py", line 153, in
with open(conll_file, "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: '../../data/wi_locness_train/tgt.txt.conll_predict'
/opt/conda/lib/python3.10/site-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use-env is set by default in torchrun.
If your script expects --local-rank argument to be set, please
change it to read from os.environ['LOCAL_RANK'] instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions

warnings.warn(
WARNING:torch.distributed.run:

---

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
/opt/conda/bin/python: No module named supar.cmds.biaffine_dep
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 34107) of binary: /opt/conda/bin/python
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launch.py", line 196, in
main()
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launch.py", line 192, in main
launch(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launch.py", line 177, in launch
run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

supar.cmds.biaffine_dep FAILED

Failures:
[1]:
time : 2023-08-25_12:30:09
host : 309e7fc0781e
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 34108)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2023-08-25_12:30:09
host : 309e7fc0781e
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 34109)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2023-08-25_12:30:09
host : 309e7fc0781e
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 34110)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
time : 2023-08-25_12:30:09
host : 309e7fc0781e
rank : 4 (local_rank: 4)
exitcode : 1 (pid: 34111)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
time : 2023-08-25_12:30:09
host : 309e7fc0781e
rank : 5 (local_rank: 5)
exitcode : 1 (pid: 34112)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
time : 2023-08-25_12:30:09
host : 309e7fc0781e
rank : 6 (local_rank: 6)
exitcode : 1 (pid: 34113)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
time : 2023-08-25_12:30:09
host : 309e7fc0781e
rank : 7 (local_rank: 7)
exitcode : 1 (pid: 34114)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Root Cause (first observed failure):
[0]:
time : 2023-08-25_12:30:09
host : 309e7fc0781e
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 34107)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

nohup: appending output to 'nohup.out'
nohup: appending output to 'nohup.out'
nohup: appending output to 'nohup.out'
nohup: appending output to 'nohup.out'
nohup: appending output to 'nohup.out'
nohup: appending output to 'nohup.out'

what's the problem and how can I fix it?

[HillZhang1999]

If you don't want to re-train the parser, you can directly skip the data preprocess step. The preprocessed file can be directly downloaded from our Google Drive.
If you want to re-train the parser, you must download the required datasets from their official websites, and put them into the corresponding director (src.txt, tgt.txt, one sentence one line).

[hwlys]

您好，请问您解决了这个问题吗，我解压缩后也没有明确的src.txt和tgt.txt，解压缩的文件是这样的，请问该如何做呢

[HillZhang1999]

您好，请问您解决了这个问题吗，我解压缩后也没有明确的src.txt和tgt.txt，解压缩的文件是这样的，请问该如何做呢

由于版权问题，我们没有提供文本文件，只有处理好的二进制文件，可以直接拿来训练