exactly how to conert UKB data, using example_ukb_to_bin.ipynb ❓

[jielab]

Hi, there:

Below are the first few lines of example_ukb_to_bin.ipynb.

labels_file = 'labels.csv'
ukb_field_to_icd10_map_file = 'icd10_codes_mod.tsv'
ubk_basket_tab_file = 'ukb_basket.tab' # file path to ukb basket download .tab format

The first 14 lines for the labels.csv are as below. I wrote them in one line to save space.
Padding, No event, Female, Male, BMI_low, BMI_mid, BMI_high, Smoking_low, Smoking_mid, Smoking_high, Alcohol_low, Alcohol_mid, Alcohol_high, A00 (cholera), A01 (typhoid and paratyphoid fevers).

My own ukb_basket.tab file looks like below. It has all variables including age, sex, bmi, ICD10, etc.
The previous version uses f. as variable names while the current version uses p as prefix.

I did not find the icd10_codes_mod.tsv file.

So, can you please let me know how to run your example_ukb_to_bin.ipynb with my own ukb_basket.tab file?

Thank you & best regards,
Jie

[j262byuu]

I believe the paper authors were using a local version of UKB, so all the variable names use the old f. format.

What I did:

1. I replaced all f.*.0.0 with p*_i0_a0 in the hardcoding part since only the baseline data is needed.

2. Edit some of the code

import csv # since some of the descriptions had quotes
import re
icdict ={}
icdcodes = []
with open(ukb_field_to_icd10_map_file,'r') as f:
    reader = csv.reader(f, delimiter='\t') # tsv
    for row in reader:
        if len(row) >= 2:
            field_code = row[0]        # old f code
            description = row[1]       # description
            
            icd_match = re.search(r'([A-Z]\d{2})', description)
            if icd_match:
                icd_code = icd_match.group(1)  
                icdict[field_code] = icd_code
                
                # convert to dnanexus format
                if field_code.startswith('f.'):
                    parts = field_code.split('.')
                    p_field = f"p{parts[1]}_i{parts[2]}_a{parts[3]}"
                    icdict[p_field] = icd_code
                
                icdcodes.append(icd_code)

[jielab]

Hi,

Thank you very much for sharing the code.

I still don't know where the ukb_field_to_icd10_map_file is and how it is supposed to look like.

Let's say that the actual UKB data is ukb_basket.tab, shown as the screenshot that I previously posted.
Can you please post your full python script to do the conversion?

Best regards,
Jie

[robomotic]

Yes please I have the same issue.

[tf2]

Hi both,

Yes i can see where some potential problems might come in here with "f." changed to "p_" etc.

• i presume you are working inside the UKB RAP correct?

You can find the mapping file here:
data/ukb_simulated_data/icd10_codes_mod.tsv

Let me know if you still have any issues and i will look into it further.

Cheers,

Tom

[jielab]

Dear Tom:

The phenotype file was downloaded from UKB RAP. I am not working inside RAP.
This new file has data field names such as p53 instead of f.53.0.0.

I managed to make the majority part of your .ipynb script work for the new pheno file,
Here is my version: https://github.com/jielab/001/blob/master/scripts/ukb2del.py.

Below is an error message that I got.
Once you helped to fix this error, I hope it will be useful to others.

Best regards,
Jie

File ~/anaconda3/envs/ai/lib/python3.11/site-packages/pandas/core/internals/managers.py:430, in BaseBlockManager.astype(self, dtype, copy, errors)
...
    132     # Explicit copy, or required since NumPy can't view from / to object.
--> [133](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/scripts/ukb/~/anaconda3/envs/ai/lib/python3.11/site-packages/pandas/core/dtypes/astype.py:133)     return arr.astype(dtype, copy=True)
    135 return arr.astype(dtype, copy=copy)

ValueError: invalid literal for int() with base 10: '2019-10-31'
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?51dee751-5824-454c-9a66-065e7fb40dea) or open in a [text editor](command:workbench.action.openLargeOutput?51dee751-5824-454c-9a66-065e7fb40dea). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...

[jielab]

Hi, guys:

Below is the python script that works for my local dataset.
https://github.com/jielab/001/blob/master/scripts/phe2delphi.py

I hope you find it useful.

Best regards,
Jie

[yozhikoff]

Hi @jielab

Thanks a lot for sharing - this is very useful!
I want to merge this script into the repo.
Could you please briefly describe how you obtained your local dataset and what format it is in?
That way I can provide more context to the users about when it should be used.

Best regards,
Artem

[jielab]

Dear Artem:

It is great that you find it useful. Regarding "how you obtained your local dataset and what format it is in", maybe you could simply write something as below:

For users who obtained phenotype file from UKB RAP, and the header of the tab separated input file [pheno.tab.gz] has column names such as "eid pX_i0" instead of "f.eid f.X.0.0", they can use the following .py script to convert data.

Best regards,
Jie

[HelloWorldLTY]

Hi, the .py file seems not working. Would you prefer sharing it again?

[jielab]

sure, i now put it at https://github.com/jielab/hello/tree/main/scripts

[HelloWorldLTY]

Thank you, yesterday I figured out the transformation by my self, but thanks for your help!

I am also wondering if you meet this error or not:

#27

It seems that we must shift 2 tokens for each phenotypes and make prediction. Do you think that is because we actually -2 token when processing the data? Thanks.

The author's comments resolved my issue. Thanks