Compression parameter and coverage inconsistencies

[MicroBTM]

Hi,

I was wondering if you think it would be worth adding an option that makes databases which scale a compression parameter to each input genome size? Using -c 100 for virus genomes seems inconsistent, as viruses have a massive range in genome size (spanning around 4 orders of magnitude, with some larger than bacterial genomes). If users picked a minimum and maximum compression value for the scaling approach, they could use the minimum value as a compression parameter to profile the resulting database.

Currently, it looks like small viruses will have much lower sensitivity than large ones with what I've gathered from sylph inspect and metadata on my database genomes:

[bluenote-1577]

@MicroBTM this is a nice idea and could be done in theory. The issue, though, is that the -c parameter for the reads has to be less than the -c parameter for the genomes. Furthermore, I've found that a lower limit of ~30 for -c is recommended, otherwise the statistical model gets weird.

Perhaps in a later version, we could use a special parameter scheme that sets -c between 30 and 200 in the database, and forces -c to be 30 for the reads. This could increase sensitivity, at the cost of ~7x larger sketches and ~7x more RAM usage.

[MicroBTM]

@bluenote-1577 Thanks for that info! I have a couple more questions that might be on the border of enhancements and fixes.

1. Why does the Eff_cov column report 1 when Eff_lambda is LOW? If users are expecting Eff_cov to report the approximate depth of coverage, do you think it would make more sense to report LOW instead of 1? If you delay the Poisson statistics by increasing --min-count-correct, Eff_cov appears to report 1 + the approximate depth of coverage when the delay is in effect, which is also pretty odd for result interpretation.
2. Would it be possible to delay the Poisson statistics based on an approximate depth of coverage threshold? For example, if you could delay the model until coverage ≥ 0.1, I think it would apply more consistently to both large and small genomes with varying numbers of sketched reference genome kmers (as opposed to just using --min-count-correct).