Error messages running DEMETER2

[ADvorkinGheva]

Dear Sir/Madam,

While attempting to run DEMETER2 on our shRNA data, I have encountered the following error:

Eliminating 0 promiscuous hairpins
Traceback (most recent call last):
File "/demeter2/modules/fit_demeter2.py", line 161, in
data = preprocess.make_demeter2_data(dsets, sh_targets)
File "/demeter2/modules/preprocess.py", line 83, in make_demeter2_data
target_df = target_df[~target_df['Gene_ID'].str.contains('NO_CURRENT')]
File "/opt/conda/lib/python3.5/site-packages/pandas/core/generic.py", line 2966, in getattr
return object.getattribute(self, name)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/base.py", line 243, in get
return self.construct_accessor(instance)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/strings.py", line 1909, in make_str_accessor
raise AttributeError("Can only use .str accessor with string "
AttributeError: Can only use .str accessor with string values, which use np.object dtype in pandas

Please, advise on how to solve this issue.

Thank you very much.

Sincerely,
Anna Dvorkin-Gheva

[jmmcfarl]

Hi Anna,

I suspect that the problem is with the shRNA-to-gene mapping file you're using. The Gene_ID column needs to be a string. Perhaps you're mapping file has all numeric IDs without any non-numeric characters, so it's getting cast as a numeric type. If you don't have any non-targeting shRNAs (which we gave the geneID 'NO_CURRENT') you might just add a line in the code to force gene_ID column to be a string.

Hope that helps!

[ADvorkinGheva]

Dear Dr. McFarland,

Thank you very much for your quick answer!
Yes, I have no "NO_CURRENT" gene IDs. This explains the problem with the string...
I apologize for the next question, but can you, please, provide more details on how to add a line to the code to force this column to be a string?
Or would it help if I enclose all the numeric IDs in that column in " or ' ?

Thank you very much!

Regards,
Anna

[jmmcfarl]

Hi Anna, As a work-around, you might be able to just add a fake shRNA with NO_CURRENT as the gene_ID (might need to be the right number of nucleotides, but just a random sequence). This shouldn't affect the end-results, but I think would trigger it to treat that column as a string. Otherwise, you should be able to insert a line to cast this column as a string, see e.g. https://stackoverflow.com/questions/22005911/convert-columns-to-string-in-pandas Best, James

…

On Mon, Jun 15, 2020 at 11:03 AM ADvorkinGheva ***@***.***> wrote: Dear Dr. McFarland, Thank you very much for your quick answer! Yes, I have no "NO_CURRENT" gene IDs. This explains the problem with the string... I apologize for the next question, but can you, please, provide more details on how to add a line to the code to force this column to be a string? Or would it help if I enclose all the numeric IDs in that column in " or ' ? Thank you very much! Regards, Anna — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPPU3FZG46YAUHWRFMTSJLRWYZ4LANCNFSM4N6H43UQ> .

[ADvorkinGheva]

Hi James,

Thank you very much for the advice. I have inserted a fake shRNA (I verified that its sequence does not appear as a real shRNA). I was not sure whether I should just add it to the targets file or to the data file as well. Since I didn't want to introduce fake data, I first tried to add the fake shRNA only to the targets file. Running DEMETER2 led to the following error message:

Loading dataset from: /data/All_files_T2-to-T0_FC_forDEMETER2.csv
Read 1 data files
Loading positive/negative control sets
Using 589 positive and 927 negative control genes for training
Making processed data
Eliminating 0 promiscuous hairpins
Eliminated 0/78432 non-targeting hairpins from map
Identified 0 gene families
Creating dataset with:
78204 hairpins
27 CLs
15982 genes
14985 seeds
Initializing model
2020-06-16 03:41:42.342879: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2020-06-16 03:41:42.342940: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2020-06-16 03:41:42.342950: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2020-06-16 03:41:42.342964: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2020-06-16 03:41:42.342974: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Model initialized
Traceback (most recent call last):
File "/opt/conda/lib/python3.5/site-packages/pandas/core/indexes/base.py", line 2393, in get_loc
return self._engine.get_loc(key)
File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5239)
File "pandas/_libs/index.pyx", line 154, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5085)
File "pandas/_libs/hashtable_class_helper.pxi", line 1207, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20405)
File "pandas/_libs/hashtable_class_helper.pxi", line 1215, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20359)
KeyError: 'index'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/demeter2/modules/fit_demeter2.py", line 227, in
mod.init_slopes(data['LFC_mats'])
File "/demeter2/modules/demeter2.py", line 692, in init_slopes
cur_d = cur_d.reset_index().drop_duplicates(subset='index', keep='last').set_index('index')
File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3196, in drop_duplicates
duplicated = self.duplicated(subset, keep=keep)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3242, in duplicated
labels, shape = map(list, zip(*map(f, vals)))
File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3241, in
vals = (self[col].values for col in subset)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 2062, in getitem
return self._getitem_column(key)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 2069, in _getitem_column
return self._get_item_cache(key)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/generic.py", line 1534, in _get_item_cache
values = self._data.get(item)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/internals.py", line 3590, in get
loc = self.items.get_loc(item)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/indexes/base.py", line 2395, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5239)
File "pandas/_libs/index.pyx", line 154, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5085)
File "pandas/_libs/hashtable_class_helper.pxi", line 1207, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20405)
File "pandas/_libs/hashtable_class_helper.pxi", line 1215, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20359)
KeyError: 'index'
Exception ignored in: <bound method BaseSession.del of <tensorflow.python.client.session.Session object at 0x7f9a0e14bf98>>
Traceback (most recent call last):
File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 595, in del
TypeError: 'NoneType' object is not callable

Then I added fake shRNA data to the data file as well and ran it again. I received the following error message:

Loading dataset from: /data/All_files_T2-to-T0_FC_forDEMETER2_lastArtificial.csv
Read 1 data files
Loading positive/negative control sets
Using 589 positive and 927 negative control genes for training
Making processed data
Eliminating 0 promiscuous hairpins
Eliminated 1/78433 non-targeting hairpins from map
Identified 0 gene families
Creating dataset with:
78205 hairpins
27 CLs
15982 genes
14986 seeds
Initializing model
2020-06-16 03:45:05.231831: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2020-06-16 03:45:05.231864: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2020-06-16 03:45:05.231874: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2020-06-16 03:45:05.231887: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2020-06-16 03:45:05.231900: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Model initialized
Traceback (most recent call last):
File "/opt/conda/lib/python3.5/site-packages/pandas/core/indexes/base.py", line 2393, in get_loc
return self._engine.get_loc(key)
File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5239)
File "pandas/_libs/index.pyx", line 154, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5085)
File "pandas/_libs/hashtable_class_helper.pxi", line 1207, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20405)
File "pandas/_libs/hashtable_class_helper.pxi", line 1215, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20359)
KeyError: 'index'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/demeter2/modules/fit_demeter2.py", line 227, in
mod.init_slopes(data['LFC_mats'])
File "/demeter2/modules/demeter2.py", line 692, in init_slopes
cur_d = cur_d.reset_index().drop_duplicates(subset='index', keep='last').set_index('index')
File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3196, in drop_duplicates
duplicated = self.duplicated(subset, keep=keep)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3242, in duplicated
labels, shape = map(list, zip(*map(f, vals)))
File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3241, in
vals = (self[col].values for col in subset)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 2062, in getitem
return self._getitem_column(key)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 2069, in _getitem_column
return self._get_item_cache(key)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/generic.py", line 1534, in _get_item_cache
values = self._data.get(item)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/internals.py", line 3590, in get
loc = self.items.get_loc(item)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/indexes/base.py", line 2395, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5239)
File "pandas/_libs/index.pyx", line 154, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5085)
File "pandas/_libs/hashtable_class_helper.pxi", line 1207, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20405)
File "pandas/_libs/hashtable_class_helper.pxi", line 1215, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20359)
KeyError: 'index'

Can you, please, help me to understand what to do to solve this? Is it due to the complications created because of the fake shRNA? If it is, I will try to insert a line to cast this column as string in the fit_demeter2.py file. Please, let me know if I'm supposed to do it in another file, not this one.

Thank you very much for all your help!

Regards,
Anna

[jmmcfarl]

Hi Anna, Hmm, I'm not sure. There might be another issue. I wish I could take a look directly, but don't have time at the moment. One other thing I'd suggest to try, is to insert this line: sh_targets['Gene_ID'] = sh_targets['Gene_ID'].astype(str) in the fit_demeter2.py file on line 130. That should force Gene_ID to be a string. THen you wouldn't have to add the fake shRNA to your mapping file which might create it's own issues. Hope that helps, and again sorry I can't be more helpful directly right now.

…

On Tue, Jun 16, 2020 at 11:44 AM ADvorkinGheva ***@***.***> wrote: Hi James, Thank you very much for the advice. I have inserted a fake shRNA (I verified that its sequence does not appear as a real shRNA). I was not sure whether I should just add it to the targets file or to the data file as well. Since I didn't want to introduce fake data, I first tried to add the fake shRNA only to the targets file. Running DEMETER2 led to the following error message: Loading dataset from: /data/All_files_T2-to-T0_FC_forDEMETER2.csv Read 1 data files Loading positive/negative control sets Using 589 positive and 927 negative control genes for training Making processed data Eliminating 0 promiscuous hairpins Eliminated 0/78432 non-targeting hairpins from map Identified 0 gene families Creating dataset with: 78204 hairpins 27 CLs 15982 genes 14985 seeds Initializing model 2020-06-16 03:41:42.342879: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2020-06-16 03:41:42.342940: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2020-06-16 03:41:42.342950: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2020-06-16 03:41:42.342964: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2020-06-16 03:41:42.342974: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. Model initialized Traceback (most recent call last): File "/opt/conda/lib/python3.5/site-packages/pandas/core/indexes/base.py", line 2393, in get_loc return self._engine.get_loc(key) File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5239) File "pandas/_libs/index.pyx", line 154, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5085) File "pandas/_libs/hashtable_class_helper.pxi", line 1207, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20405) File "pandas/_libs/hashtable_class_helper.pxi", line 1215, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20359) KeyError: 'index' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/demeter2/modules/fit_demeter2.py", line 227, in mod.init_slopes(data['LFC_mats']) File "/demeter2/modules/demeter2.py", line 692, in init_slopes cur_d = cur_d.reset_index().drop_duplicates(subset='index', keep='last').set_index('index') File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3196, in drop_duplicates duplicated = self.duplicated(subset, keep=keep) File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3242, in duplicated labels, shape = map(list, zip(*map(f, vals))) File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3241, in vals = (self[col].values for col in subset) File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 2062, in *getitem* return self._getitem_column(key) File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 2069, in _getitem_column return self._get_item_cache(key) File "/opt/conda/lib/python3.5/site-packages/pandas/core/generic.py", line 1534, in _get_item_cache values = self._data.get(item) File "/opt/conda/lib/python3.5/site-packages/pandas/core/internals.py", line 3590, in get loc = self.items.get_loc(item) File "/opt/conda/lib/python3.5/site-packages/pandas/core/indexes/base.py", line 2395, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key)) File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5239) File "pandas/_libs/index.pyx", line 154, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5085) File "pandas/_libs/hashtable_class_helper.pxi", line 1207, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20405) File "pandas/_libs/hashtable_class_helper.pxi", line 1215, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20359) KeyError: 'index' Exception ignored in: <bound method BaseSession.*del* of <tensorflow.python.client.session.Session object at 0x7f9a0e14bf98>> Traceback (most recent call last): File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 595, in *del* TypeError: 'NoneType' object is not callable Then I added fake shRNA data to the data file as well and ran it again. I received the following error message: Loading dataset from: /data/All_files_T2-to-T0_FC_forDEMETER2_lastArtificial.csv Read 1 data files Loading positive/negative control sets Using 589 positive and 927 negative control genes for training Making processed data Eliminating 0 promiscuous hairpins Eliminated 1/78433 non-targeting hairpins from map Identified 0 gene families Creating dataset with: 78205 hairpins 27 CLs 15982 genes 14986 seeds Initializing model 2020-06-16 03:45:05.231831: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2020-06-16 03:45:05.231864: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2020-06-16 03:45:05.231874: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2020-06-16 03:45:05.231887: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2020-06-16 03:45:05.231900: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. Model initialized Traceback (most recent call last): File "/opt/conda/lib/python3.5/site-packages/pandas/core/indexes/base.py", line 2393, in get_loc return self._engine.get_loc(key) File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5239) File "pandas/_libs/index.pyx", line 154, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5085) File "pandas/_libs/hashtable_class_helper.pxi", line 1207, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20405) File "pandas/_libs/hashtable_class_helper.pxi", line 1215, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20359) KeyError: 'index' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/demeter2/modules/fit_demeter2.py", line 227, in mod.init_slopes(data['LFC_mats']) File "/demeter2/modules/demeter2.py", line 692, in init_slopes cur_d = cur_d.reset_index().drop_duplicates(subset='index', keep='last').set_index('index') File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3196, in drop_duplicates duplicated = self.duplicated(subset, keep=keep) File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3242, in duplicated labels, shape = map(list, zip(*map(f, vals))) File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 3241, in vals = (self[col].values for col in subset) File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 2062, in *getitem* return self._getitem_column(key) File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 2069, in _getitem_column return self._get_item_cache(key) File "/opt/conda/lib/python3.5/site-packages/pandas/core/generic.py", line 1534, in _get_item_cache values = self._data.get(item) File "/opt/conda/lib/python3.5/site-packages/pandas/core/internals.py", line 3590, in get loc = self.items.get_loc(item) File "/opt/conda/lib/python3.5/site-packages/pandas/core/indexes/base.py", line 2395, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key)) File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5239) File "pandas/_libs/index.pyx", line 154, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5085) File "pandas/_libs/hashtable_class_helper.pxi", line 1207, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20405) File "pandas/_libs/hashtable_class_helper.pxi", line 1215, in pandas._libs.hashtable.PyObjectHashTable.get_item (pandas/_libs/hashtable.c:20359) KeyError: 'index' Can you, please, help me to understand what to do to solve this? Is it due to the complications created because of the fake shRNA? If it is, I will try to insert a line to cast this column as string in the fit_demeter2.py file. Please, let me know if I'm supposed to do it in another file, not this one. Thank you very much for all your help! Regards, Anna — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPPU3EDS6BUF4FGN7ZVTQDRW6HM3ANCNFSM4N6H43UQ> .

[ADvorkinGheva]

Hi James,

Thank you very much for the advice. I have added the code you suggested to the fit_demeter2.py file, but unfortunately, I still get the same error after running it.

I understand that you are currently very busy; therefore, I was wondering if it would be possible for you to look at it in the future when you have time?
Please, let me know, so I would know if I should wait.

In case you will be able to look at this issue in the future, I have saved the exact error message and I will be able to provide it.

Thank you very much for your explanations and for all your help!

Regards,
Anna

[ADvorkinGheva]

Hi James,

How are you?
I apologize for writing to you again, but we were really hoping to be able to use DEMETER2 on our data. You mentioned that you didn't have time to check the issue I was having, and I was wondering if you could let me know if that had changed or if there was anyone else I could ask about that bug?

Thank you very much for all your help!

I apologize for bothering you with this.

Regards,
Anna

[jmmcfarl]

Hi Anna, Thanks for checking in again. I still don't have the time to really dig into this, and I don't think there's anyone else who'd be able to work on this now either. So, I might recommend looking into an alternative strategy. Indeed, the advantages of using DEMETER2 really depend on having a significant number of cell lines screened with the same library, so if your data is only for a smallish number of cell lines (e.g. < 25) it might make sense to try using a simpler method that doesn't try to learn the seed-sequence specific effects explicitly. (e.g. RIGER, RSA). If you think DEMETER2 will be particularly well suited to your application I can try to take a quick look at the issue, but can't make any promises (and I wouldn't want to hold you up anymore). Let me know if you'd like to pursue this and I can send some thoughts on next steps. Thanks, James

…

On Mon, Aug 24, 2020 at 1:34 PM ADvorkinGheva ***@***.***> wrote: Hi James, How are you? I apologize for writing to you again, but we were really hoping to be able to use DEMETER2 on our data. You mentioned that you didn't have time to check the issue I was having, and I was wondering if you could let me know if that had changed or if there was anyone else I could ask about that bug? Thank you very much for all your help! I apologize for bothering you with this. Regards, Anna — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPPU3ENFO3O2XKZ7S6NWMDSCKQAZANCNFSM4N6H43UQ> .

[ADvorkinGheva]

Hi James,

Thank you very much for your quick reply and for the advice!

Our case seems to be borderline - we have 27 cell lines... The reason we became interested in DEMETER2 originally was that we were planning to compare the results we were going to get from our shRNA screens to the data presented in DepMap, and we wanted to use the same method. In addition, we were also very impressed by the strengths of DEMETER2.

I understand that you are very busy and I do not want to take much of your time, so I was wondering if it would be ok if we try to pursue this by following your thoughts for the next few steps, and if these steps do not solve the problems, we will move to a method alternative to DEMETER2.

Please, let me know if it is ok with you.

Thank you very much with your help!

Regards,
Anna

[jmmcfarl]

Hi Anna, I see. That indeed may be borderline. You may need to do some optimization of the hyperparameters, which were tuned to application of the model to hundreds of screens. I haven't tested it in that regime so I don't know. In terms of next steps: -Can you get the method to work on the published data? -If so, can you send me your shRNA mapping file and a piece of your data (just enough so I can check if there are formatting issues)?

…

On Mon, Aug 24, 2020 at 9:55 PM ADvorkinGheva ***@***.***> wrote: Hi James, Thank you very much for your quick reply and for the advice! Our case seems to be borderline - we have 27 cell lines... The reason we became interested in DEMETER2 originally was that we were planning to compare the results we were going to get from our shRNA screens to the data presented in DepMap, and we wanted to use the same method. In addition, we were also very impressed by the strengths of DEMETER2. I understand that you are very busy and I do not want to take much of your time, so I was wondering if it would be ok if we try to pursue this by following your thoughts for the next few steps, and if these steps do not solve the problems, we will move to a method alternative to DEMETER2. Please, let me know if it is ok with you. Thank you very much with your help! Regards, Anna — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPPU3E2EX5KAV2HWVM57T3SCMKYBANCNFSM4N6H43UQ> .

[ADvorkinGheva]

Hi James,

Thank you very much for the advice and for offering to check the formatting of our data! I will try to get and run DEMETER2 on some published data to see that it runs. I will let you know how it goes, and then, depending on the outcome, we will see if any additional steps (like checking the formatting) are required.

Regards,
Anna

[ADvorkinGheva]

Hi James,

I've ran DEMETER2 on Marcotte's data I've downloaded and it runs without any errors. Following this, I understand that the problem is with formatting most probably. I have been trying to compare the file formats and used the same template for formatting my data files (unix (LF), UTF-8) etc. I searched for hidden symbols using Notepad++, but there were none.
Unfortunately, I'm still getting error messages (I'm including the loading information, but I've removed the warnings):

Read 1 data files
Loading positive/negative control sets
Using 589 positive and 904 negative control genes for training
Making processed data
Eliminating 0 promiscuous hairpins
Eliminated 2474/78432 non-targeting hairpins from map
Identified 0 gene families
Creating dataset with:
78204 hairpins
27 CLs
15466 genes
14985 seeds
Initializing model

... (warning messages)

Traceback (most recent call last):
File "/demeter2/modules/fit_demeter2.py", line 221, in
optim_params = optim_params, test_inds = test_inds, log_file = log_file)
File "/demeter2/modules/demeter2.py", line 205, in init
LFC_mats, gene_matrix, data_names['hps'], data_names['CLs'], data_names['genes'])
File "/demeter2/modules/demeter2.py", line 106, in compute_hairpins_per_gene_CL
cur_hps_per_gene = sparse_mat[cur_hp_set,:].transpose().dot(~np.isnan(LFC_mat))
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Please, let me know if these messages tell you which changes I should introduce to the files I'm using or if you would like to see a part of each of the files I'm using so you'll be able to see the reason for the errors, if it doesn't take too much of your time).

Thank you very much, I greatly appreciate all your help!

Regards,
Anna

[jmmcfarl]

Hi Anna, I suspect it is something to do with formatting, or maybe some edge case that occurs in your data which hadn't come up in the datasets we'd used. It looks like the error you're getting now is different than before. Is that right? Did you change anything on your end? If you can send post chunks of the data you're using I can take a look. Thanks, James

…

On Sun, Aug 30, 2020 at 11:12 PM ADvorkinGheva ***@***.***> wrote: Hi James, I've ran DEMETER2 on Marcotte's data I've downloaded and it runs without any errors. Following this, I understand that the problem is with formatting most probably. I have been trying to compare the file formats and used the same template for formatting my data files (unix (LF), UTF-8) etc. I searched for hidden symbols using Notepad++, but there were none. Unfortunately, I'm still getting error messages (I'm including the loading information, but I've removed the warnings): Read 1 data files Loading positive/negative control sets Using 589 positive and 904 negative control genes for training Making processed data Eliminating 0 promiscuous hairpins Eliminated 2474/78432 non-targeting hairpins from map Identified 0 gene families Creating dataset with: 78204 hairpins 27 CLs 15466 genes 14985 seeds Initializing model ... (warning messages) Traceback (most recent call last): File "/demeter2/modules/fit_demeter2.py", line 221, in optim_params = optim_params, test_inds = test_inds, log_file = log_file) File "/demeter2/modules/demeter2.py", line 205, in *init* LFC_mats, gene_matrix, data_names['hps'], data_names['CLs'], data_names['genes']) File "/demeter2/modules/demeter2.py", line 106, in compute_hairpins_per_gene_CL cur_hps_per_gene = sparse_mat[cur_hp_set,:].transpose().dot(~np.isnan(LFC_mat)) TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'' Please, let me know if these messages tell you which changes I should introduce to the files I'm using or if you would like to see a part of each of the files I'm using so you'll be able to see the reason for the errors, if it doesn't take too much of your time). Thank you very much, I greatly appreciate all your help! Regards, Anna — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPPU3BWTFC2WMJM65FTIDDSDMIJRANCNFSM4N6H43UQ> .

[ADvorkinGheva]

Hi James,

Yes, the error is different now. It has changed after I changed the formatting, trying to get the files to be formatted in the exactly same way as the Marcotte's files, which ran well.

Thank you very much for offering to look at my files. I will select some chunks from each of the files for you to look at. Would that be ok with you if I email them to you instead of posting them here? If yes, please, let me know your email.

Thank you very much!

Regards,
Anna

[jmmcfarl]

Hi Anna, Sure, please contact me directly at jmmcfarl@broadinstitute.org

…

On Fri, Sep 4, 2020 at 8:38 PM ADvorkinGheva ***@***.***> wrote: Hi James, Yes, the error is different now. It has changed after I changed the formatting, trying to get the files to be formatted in the exactly same way as the Marcotte's files, which ran well. Thank you very much for offering to look at my files. I will select some chunks from each of the files for you to look at. Would that be ok with you if I email them to you instead of posting them here? If yes, please, let me know your email. Thank you very much! Regards, Anna — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPPU3E5BA2OU6VZDTBMA4LSEGB7LANCNFSM4N6H43UQ> .