RateMap.read_hapmap error about Position values not strictly increasing

[lakishadavid]

(newbie question)
When using RateMap.read_hapmap on concatenated recombination map sorted on the position value, I get ValueError: Position values not strictly increasing at indexes [ 286356 509716.... How do I fix the file so that I can use it for msprime?

import msprime
import glob
import pandas as pd

# recombination_map_source = https://ftp.ncbi.nlm.nih.gov/hapmap/recombination/2011-01_phaseII_B37/
genetic_map_dir = ".../references/nih_hapmap37_map/genetic_map_GRCh37_chr[0-9]*.txt"
genetic_maps = [filepath for filepath in glob.glob(genetic_map_dir)]
combined_map = pd.concat([pd.read_csv(filename, delimiter='\t') for filename in genetic_maps])

combined_map['Chromosome'] = pd.Categorical(
    combined_map['Chromosome'], 
    ["chr1", "chr2", "chr3", "chr4", "chr5", "chr6", "chr7", "chr8", "chr9", "chr10", 
    "chr11", "chr12", "chr13", "chr14", "chr15", "chr16", "chr17", "chr18", "chr19", 
    "chr20", "chr21", "chr22"])
combined_map.sort_values(by=['Chromosome', 'Position(bp)'], inplace=True)

full_map_path = ".../results/full_genetic_map.csv"
combined_map.to_csv(full_map_path, sep=' ', index=False, encoding='utf-8-sig')

rate_map = msprime.RateMap.read_hapmap(fileobj=full_map_path)
print(type(rate_map))

Output

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
...
File ~/.local/share/virtualenvs/python_workspace-Okc1r0v5/lib/python3.8/site-packages/msprime/intervals.py:77, in RateMap.__init__(self, position, rate)
     [75](file:///~/.local/share/virtualenvs/python_workspace-Okc1r0v5/lib/python3.8/site-packages/msprime/intervals.py?line=74) if np.any(span <= 0):
     [76](file:///~/.local/share/virtualenvs/python_workspace-Okc1r0v5/lib/python3.8/site-packages/msprime/intervals.py?line=75)     bad_pos = np.where(span <= 0)[0] + 1
---> [77](file:///~/.local/share/virtualenvs/python_workspace-Okc1r0v5/lib/python3.8/site-packages/msprime/intervals.py?line=76)     raise ValueError(
     [78](file:///~/.local/share/virtualenvs/python_workspace-Okc1r0v5/lib/python3.8/site-packages/msprime/intervals.py?line=77)         f"Position values not strictly increasing at indexes {bad_pos}"
     [79](file:///~/.local/share/virtualenvs/python_workspace-Okc1r0v5/lib/python3.8/site-packages/msprime/intervals.py?line=78)     )
     [80](file:///~/.local/share/virtualenvs/python_workspace-Okc1r0v5/lib/python3.8/site-packages/msprime/intervals.py?line=79) if np.any(self._rate < 0):
     [81](file:///~/.local/share/virtualenvs/python_workspace-Okc1r0v5/lib/python3.8/site-packages/msprime/intervals.py?line=80)     bad_rates = np.where(self._rate < 0)[0]

ValueError: Position values not strictly increasing at indexes [ 256896  543251  766611  977726 1193140 1427562 1611792 1802854 1960854
 2141602 2310466 2475823 2612477 2720436 2813545 2906539 2981213 3083926
 3130890 3213852 3258572]

[jeromekelleher]

It looks like you are trying to combine the genetic maps to simulate all human chromosomes at once here @lakishadavid? If so, this documentation should help. You will need to do some manual calculations with the coordinates, and I don't think you can just concatenate HapMap files since the main value used is the genetic distance (so, create the RateMap directly rather than use read_hapmap for the final step).

This is going to be a very big simulation though, so I'd recommend having a close look at the "Do I really need this" section!

[lakishadavid]

Thanks, @jeromekelleher. From my read of the documentation, I think I need to model multiple chromosomes. I'm working on a pedigree reconstruction project where the TMRCA would be 1 to 25 generations ago (with some ancestors being much more distant than that). In my real sample, I will have two populations. Relatedness within population would be closer (e.g., extended family members) whereas across populations would be more distant. I'm not interested in relatedness between populations but rather relatedness between pairs where one individual is in one population and the other individual is in the other population, to produce a pedigree. I think I need to model multiple chromosomes, but I could simulate one chromosome for the more distantly related pairs.