freeze_support() error

[sara-wennersten]

### Code that I run:

mkdir -p /Users/sarawennersten/imputation_accuracy_calculator-master
cd /Users/sarawennersten/imputation_accuracy_calculator-master
python3 Compare_imputation_to_WGS.py --ga chr1_merged.vcf.gz --imputed chr1.dose.vcf.gz --wgs chr1_genotypes.vcf.gz

### Files that I used (in both vcf.gz and vcf.gz.tbi formats)

• pre-imputation file: chr1_merged
• post-imputation file: chr1.dose
• whole genome sequencing data (1000GP_Phase3): chr1_genotypes

### Error that shows up: (my apologies for the length - this series repeats over and over and doesn't stop until I exit the terminal...perhaps an issue with iteration?):

Namespace(ga=['chr1_merged.vcf.gz'], wgs=['chr1_genotypes.vcf.gz'], imputed=['chr1.dose.vcf.gz'], ref='', max_total_rows=10000, max_per_core=1000, min_per_core=100, sout='', vout='', xmode='False')
10000 1000 100
Processing  264 imputed samples
Processing chunk: 1 Max rows per chunk: 10000
Namespace(ga=['chr1_merged.vcf.gz'], wgs=['chr1_genotypes.vcf.gz'], imputed=['chr1.dose.vcf.gz'], ref='', max_total_rows=10000, max_per_core=1000, min_per_core=100, sout='', vout='', xmode='False')
10000 1000 100
Namespace(ga=['chr1_merged.vcf.gz'], wgs=['chr1_genotypes.vcf.gz'], imputed=['chr1.dose.vcf.gz'], ref='', max_total_rows=10000, max_per_core=1000, min_per_core=100, sout='', vout='', xmode='False')
10000 1000 100
Processing  264 imputed samples
Processing  264 imputed samples
Namespace(ga=['chr1_merged.vcf.gz'], wgs=['chr1_genotypes.vcf.gz'], imputed=['chr1.dose.vcf.gz'], ref='', max_total_rows=10000, max_per_core=1000, min_per_core=100, sout='', vout='', xmode='False')
10000 1000 100
Processing  264 imputed samples
Namespace(ga=['chr1_merged.vcf.gz'], wgs=['chr1_genotypes.vcf.gz'], imputed=['chr1.dose.vcf.gz'], ref='', max_total_rows=10000, max_per_core=1000, min_per_core=100, sout='', vout='', xmode='False')
10000 1000 100
Processing  264 imputed samples
Namespace(ga=['chr1_merged.vcf.gz'], wgs=['chr1_genotypes.vcf.gz'], imputed=['chr1.dose.vcf.gz'], ref='', max_total_rows=10000, max_per_core=1000, min_per_core=100, sout='', vout='', xmode='False')
10000 1000 100
Processing  264 imputed samples
Namespace(ga=['chr1_merged.vcf.gz'], wgs=['chr1_genotypes.vcf.gz'], imputed=['chr1.dose.vcf.gz'], ref='', max_total_rows=10000, max_per_core=1000, min_per_core=100, sout='', vout='', xmode='False')
10000 1000 100
Namespace(ga=['chr1_merged.vcf.gz'], wgs=['chr1_genotypes.vcf.gz'], imputed=['chr1.dose.vcf.gz'], ref='', max_total_rows=10000, max_per_core=1000, min_per_core=100, sout='', vout='', xmode='False')
10000 1000 100
Processing  264 imputed samples
Processing  264 imputed samples
Namespace(ga=['chr1_merged.vcf.gz'], wgs=['chr1_genotypes.vcf.gz'], imputed=['chr1.dose.vcf.gz'], ref='', max_total_rows=10000, max_per_core=1000, min_per_core=100, sout='', vout='', xmode='False')
10000 1000 100
Processing  264 imputed samples
Processing chunk: 1 Max rows per chunk: 10000
Processing chunk: 1 Max rows per chunk: 10000
Processing chunk: 1 Max rows per chunk: 10000
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/sarawennersten/imputation_accuracy_calculator-master/Compare_imputation_to_WGS.py", line 1233, in <module>
    main()
  File "/Users/sarawennersten/imputation_accuracy_calculator-master/Compare_imputation_to_WGS.py", line 1224, in main
    results = load_file_chunks(my_ncores)
  File "/Users/sarawennersten/imputation_accuracy_calculator-master/Compare_imputation_to_WGS.py", line 727, in load_file_chunks
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare
Processing chunk: 1 Max rows per chunk: 10000
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/sarawennersten/imputation_accuracy_calculator-master/Compare_imputation_to_WGS.py", line 1233, in <module>
    main()
  File "/Users/sarawennersten/imputation_accuracy_calculator-master/Compare_imputation_to_WGS.py", line 1224, in main
    results = load_file_chunks(my_ncores)
  File "/Users/sarawennersten/imputation_accuracy_calculator-master/Compare_imputation_to_WGS.py", line 727, in load_file_chunks
    pool = mp.Pool(ncores)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 119, in Pool
    pool = mp.Pool(ncores)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 212, in __init__
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 303, in _repopulate_pool
    self._repopulate_pool()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    self._popen = self._Popen(self)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    return Popen(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    self._popen = self._Popen(self)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    super().__init__(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    return Popen(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
    self._launch(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 154, in get_preparation_data
    self._launch(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    _check_not_importing_main()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 154, in get_preparation_data
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
    _check_not_importing_main()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

[raqueldias]

Thank you for rising this concern! We realized that the difference between the --wgs argument and --ref can be confusing. --wgs is the empirically determined high coverage genotypes (e.g. whole genome sequencing or ground truth) for the same samples provided in the sparse genotype array input (provided through --ga), whereas --ref is just an optional argument to provide the reference panel used during imputation, just for MAF calculation purposes.
To clarify this question, I uploaded a sample_data folder including sample files that you can use as example for testing and learning how to run this tool. I also updated the README.md file so now it includes instructions on how to test the script on the sample data provided.
This more explicit example of how to run the tool should help to explain things a bit better. Thanks again for your feedback. This helped me to improve the documentation. Let me know if you need any help!

[sara-wennersten]

So, I believe the error was with my computer (see an explanation of a part of the error with some similar issues here). I was also using the incorrect file for wgs. To fix this, I ended up "masking" the pre-imputed vcf files, then imputed them again, and am using the un-masked, pre-imputed files as the wgs file.

Because there seems to be an issue with my computer and I wanted to work out the bugs on files that were certain to work, I switched over to our cluster and am using your sample data files. After setting up an environment with the correct packages/working through some errors, I was able to eliminate most of the error messages and am left with the following:

Namespace(ga=['input_genotype_array_100samples.vcf.gz'], imputed=['imputed_file_100samples.vcf.gz'], max_per_core=1000, max_total_rows=10000, min_per_core=100, ref='', sout='', vout='', wgs=['whole_genome_file_100samples.vcf.gz'], xmode='False')
10000 1000 100
Traceback (most recent call last):
  File "/gpfs/gpfs2/home/swennersten/imputation_accuracy_calculator-master/Compare_imputation_to_WGS.py", line 1232, in <module>
    main()
  File "/gpfs/gpfs2/home/swennersten/imputation_accuracy_calculator-master/Compare_imputation_to_WGS.py", line 1223, in main
    results = load_file_chunks(my_ncores)
  File "/gpfs/gpfs2/home/swennersten/imputation_accuracy_calculator-master/Compare_imputation_to_WGS.py", line 681, in load_file_chunks
    imputed_sample_ids=extract_sample_ids_from_vcf(IMPUTED_file)
  File "/gpfs/gpfs2/home/swennersten/imputation_accuracy_calculator-master/Compare_imputation_to_WGS.py", line 440, in extract_sample_ids_from_vcf
    file = gzip.open(input_file,'rb')
  File "/gpfs/gpfs2/home/swennersten/.conda/envs/cyvcf2/lib/python3.7/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/gpfs/gpfs2/home/swennersten/.conda/envs/cyvcf2/lib/python3.7/gzip.py", line 168, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'imputed_file_100samples.vcf.gz'

The final error saying there is no such file or directory is strange because that file is definitely in the sample_data folder.

Could it be due to using python 3.9 instead of 3.6?

Thanks for your help!

[raqueldias]

Thanks for the updated info! Hopefully python3.9 isn't a problem. I tested up to python3.8.3 so far. I will try to test on python 3.9 this weekend.
About the "No such file or directory" error, did you cd into the sample_data folder before running the "python3 ../Compare_imputation_to_WGS.py...." command? What specific command are you running and what directory are you running it from when you got that error?
If you did not cd into the folder, or keep seeing that error, an alternative is to specify the full path to the input files. That should solve the problem no matter where you are running the command from.

[sara-wennersten]

Thanks for your response! Yes, I did cd into the sample_data folder but I went ahead and specified the full path just in case and I still got the same error. All of the files are in the folder.

cd /gpfs/gpfs2/home/swennersten/imputation_accuracy_calculator-master/sample_data

The code (long version with file paths) I am running is:

python3 /gpfs/gpfs2/home/swennersten/imputation_accuracy_calculator-master/Compare_imputation_to_WGS.py --ga /gpfs/gpfs2/home/swennersten/imputation_accuracy_calculator-master/input_genotype_array_100samples.vcf.gz --wgs /gpfs/gpfs2/home/swennersten/imputation_accuracy_calculator-master/sample_data/whole_genome_file_100samples.vcf.gz --imputed /gpfs/gpfs2/home/swennersten/imputation_accuracy_calculator-master/sample_data/sample_data/imputed_file_100samples.vcf.gz

Thanks!

[raqueldias]

Your imputed file path:
/gpfs/gpfs2/home/swennersten/imputation_accuracy_calculator-master/sample_data/sample_data/imputed_file_100samples.vcf.gz

Shouldn't it be like this?
/gpfs/gpfs2/home/swennersten/imputation_accuracy_calculator-master/sample_data/imputed_file_100samples.vcf.gz

There is an extra "sample_data" there

Also, your --ga input file path is missing the "sample_data" subdirectory in its path.

Let me know if those edits work.