Limit of 30 characters in filename prefix

[ryanjameskennedy]

When attempting to update from v3.3.2 to v3.4.1, I ran basic validation tests to confirm that nothing drastic had changed before implementation into our pipeline. I ran into the problem created by v3.3.5, whereby our hospital's sample ID prefixes are 46 characters in length (greatly surpassing that of the 30 character limit). I see that this is due to blast's cutoff of 50 characters, but I'm wondering if it is at all possible to set the limit to 50 instead because it would be incredibly difficult to cut our sample ID length down, and ChewBBACA is pivotal to all of our clinical analyses. Thank you in advance!

[ramirma]

We will look into this and will get back to you.

[samuell]

Perhaps one solution, if there are specific tools like Blast that need a limit, is to create a temporary shortened versions for those cases, replacing the latter part of the ID with a hash based on the full ID, and then mapping back to the old ID afterwards?

So, long Original ID:

ACDEF012345-ACDEF012345-ACDEF012345-ACDEF012345-ACDEF012345-ACDEF012345

... would temporarily be converted to, something like:

ACDEF012345-ACDEF012345-ACDEF012345-ACDEF012345-99a144
                                                  ^
                                    hash part, based on orig ID

... storing a mapping between those, to be able to map back afterwards.

[rfm-targa]

Thank you for your suggestion @samuell. We are certainly considering something similar to what you've suggested. Using some kind of shortened version of the original name or assigning unique and short values should be simple and work seamlessly. It is easier to implement in the AlleleCall module, but may be a little tricky in the CreateSchema module as that module uses the IDs of the inputs to define the loci names, forcing them to be short and unique. We will need to change the method used to define the loci names when creating a schema from scratch with the CreateSchema module. Perhaps by determining the shortest unique prefix for each input, using that as locus name, but mapping to something else internally to avoid issues if it exceeds 50 chars. This would allow for greater flexibility when defining the loci names. I think we can start working on this soon.

[Emkago]

I'm following up on this issue to see if there have been any recent developments. We're planning the next steps for a project that relies on this functionality, and knowing the current situation would be very helpful for our scheduling. Thank you again for maintining this project!

[rfm-targa]

Hello @Emkago and everyone else participating in this issue! Sorry for the delay relating this matter. Just to let you know about our schedule, we plan to implement the changes into the AlleleCall module before the end of the year. We are currently working on a new module to compute MSAs from allele calling results or from a folder containing FASTA files, which is in the testing phase before release (ComputeMSA branch). After that, we will implement the necessary changes to accept longer/any input IDs. We will create a dictionary with the correspondence between the full input IDs and short unique identifiers that chewBBACA will use internally. That dictionary will be used every time it is necessary to know the original input ID, such as when writing the output files. This is simple to implement and will allow using input IDs of any size. Everything else about the allele calling will work the same. We will keep you posted about our progress. The CreateSchema module will still trim the input IDs and expect them to be shorter, but we do not think that is an issue as it only applies when creating a schema from scratch and defining the IDs of the loci included in the schema.

[rfm-targa]

Greetings @ryanjameskennedy, @samuell and @Emkago,

We just release chewBBACA v3.5.1 to add support for longer input and loci IDs. The changes are listed in the README and the CHANGELOG. Briefly, chewBBACA now accepts input files with basenames of any length, using each basename as unique identifier (the basename is just the filename without the file extension, and different input files cannot have the same basename). Internally, chewBBACA is trying to avoid issues during the creation of BLAST databases by converting sequence IDs to a shorter format to create the BLAST database and then back to the original IDs after alignment with BLASTp. We are converting all sequence IDs to the lcl|SEQN format, where N is an integer assigned sequentially. We are using this format to try to tell BLAST that the sequence IDs are local and should not be modified. This approach should allow using longer IDs without issues, although we still recommend using short IDs for conciseness, and to definitely not use blank spaces or some special characters (!@#?$^*()+) in the basenames.
Please let us know if these changes allow you to use longer IDs without issues.

Cheers,

Rafael

[rfm-targa]

A note on the latest version of Pyrodigal, used for gene prediction. In our tests, the gene prediction step just hangs if we use the latest version of Pyrodigal (v3.7.0) We managed to solve this issue by downgrading to Pyrodigal v3.6.3. This may be system-specific, but we will open an issue on Pyrodigal's repo to discuss this and try to find a solution. In the meantime, try to use Pyrodigal v3.6.3 to avoid potential issues during gene prediction.

[Emkago]

@rfm-targa thank you for the update!

I have tested a new version of Chewbbaca, v3.5.1, and it works well even with longer sample names when I ran it with a schema prepared with v3.4.0. However, when I prepare schema with v3.5.1 the preparation of schema is successful, but it seems it is missing prefixes. Chewbbaca exites with an error when running AlleleCall and complaines about duplicate keys.

This is how beginning of S. aureus file (SACOL0413.fasta in the output folder of chewie PrepExternalSchema process) looks like:

Prepared with v3.4.0:

>SACOL0413_1
ATGTTTGGAATGAAAGTGAATGAACAAATAACATTAAAAATTTTAGAAGCTCATGACACAGAAGCGCTTTTCAATTTAGTCAATCGTTCAAGAAATTCACTTAGGGAATGGTTACCTTGGGTAGATGCAACTGAGCAACCATCAGATACGCGTGCATTTATTAAAAGAGGACTTTTGCAATTTGCTGATGGTAATGGATTTCAGTGTGGCATTTGGTATGAAGGAACGCTAGTTGGTGTCATCGGTTTACATGAAATTAATCACATGCACAGAAAAACTTCATTAGGGTACTATTTAGATAAAGAATTTGAGGGTCATGGGATTATGACACAAGCAGTTGAGGCATTGATAAAGTATTGTTTCGAAGAGCTTGACTTAAACCGAATTGAGATTAGTGCCGCAGTTAATAATGAAAAAAGCCGGGCTATTCCTGAAAGGCTGGGATTTACTAGAGAAGGTATGTTACGTGACAATGAATTACTAAATGGTATTTATTCATCGAGTTACATCTATAGTTTATTAAAATCAGAATACGACCAAAAATGA
>SACOL0413_2
ATGTTTGGAATGAAAGTGAATGAACAAATAAGATTAAAAATTTTAGAAGCCCATGACACAGAAGCGCTTTTCAATTTAGTCAATCGTTCAAGAAATTCACTTAGGGAATGGTTACCTTGGGTAGATGCAACCGAGCAACCATCAGATACGCGTGCATTTATTAAAAGTGGACTTTTGCAATTTGCTGATGGTAATGGATTTCAGTGTGGCATTTGGTATGAAGGAACGCTAGTTGGTGTCATCGGTTTACATGAAATTAATCACATGCACAGAAAAACTTCATTAGGGTATTATTTAGATAAACAATTTGAGGGTCATGGGATTATGACACAAGCAGTTGAGGCATTGATAAAGTATTGCTTCGATGAGATTGAATTAAATCGAATTGAGATCAGTGCCGCAGTTAATAATGAAAAAAGCCAAGCTATTCCTGAAAGGCTGGGATTTACTAGGGAAGGTATGTTACGTGACAATGAATTACTAAATGGTATTTATTCATCGAGTTACATCTATAGTTTATTAAAATCAGAATATAATGAAGGATGA

Prepared with v3.5.1:

>1
ATGTTTGGAATGAAAGTGAATGAACAAATAACATTAAAAATTTTAGAAGCTCATGACACAGAAGCGCTTTTCAATTTAGTCAATCGTTCAAGAAATTCACTTAGGGAATGGTTACCTTGGGTAGATGCAACTGAGCAACCATCAGATACGCGTGCATTTATTAAAAGAGGACTTTTGCAATTTGCTGATGGTAATGGATTTCAGTGTGGCATTTGGTATGAAGGAACGCTAGTTGGTGTCATCGGTTTACATGAAATTAATCACATGCACAGAAAAACTTCATTAGGGTACTATTTAGATAAAGAATTTGAGGGTCATGGGATTATGACACAAGCAGTTGAGGCATTGATAAAGTATTGTTTCGAAGAGCTTGACTTAAACCGAATTGAGATTAGTGCCGCAGTTAATAATGAAAAAAGCCGGGCTATTCCTGAAAGGCTGGGATTTACTAGAGAAGGTATGTTACGTGACAATGAATTACTAAATGGTATTTATTCATCGAGTTACATCTATAGTTTATTAAAATCAGAATACGACCAAAAATGA
>2
ATGTTTGGAATGAAAGTGAATGAACAAATAAGATTAAAAATTTTAGAAGCCCATGACACAGAAGCGCTTTTCAATTTAGTCAATCGTTCAAGAAATTCACTTAGGGAATGGTTACCTTGGGTAGATGCAACCGAGCAACCATCAGATACGCGTGCATTTATTAAAAGTGGACTTTTGCAATTTGCTGATGGTAATGGATTTCAGTGTGGCATTTGGTATGAAGGAACGCTAGTTGGTGTCATCGGTTTACATGAAATTAATCACATGCACAGAAAAACTTCATTAGGGTATTATTTAGATAAACAATTTGAGGGTCATGGGATTATGACACAAGCAGTTGAGGCATTGATAAAGTATTGCTTCGATGAGATTGAATTAAATCGAATTGAGATCAGTGCCGCAGTTAATAATGAAAAAAGCCAAGCTATTCCTGAAAGGCTGGGATTTACTAGGGAAGGTATGTTACGTGACAATGAATTACTAAATGGTATTTATTCATCGAGTTACATCTATAGTTTATTAAAATCAGAATATAATGAAGGATGA

Sequences in all the SACOLxxxx.fasta files have the same names, numbers starting with 1.

Did you encounter the same problem when creating schema with v3.5.1?