evals

Evals

There are 3 separate evals related to the paper:

• Section 3: synthetic results of Sassy vs Edlib
• Section 3: searching a 23bp pattern in the human genome with Sassy vs Parasail vs Ish.
• Section 4: comparison with crispr off-target tools SWOffinder and CHOPOFF.

Section 3: Sassy vs Edlib on synthetic data

First make sure to build the evals executable:

cargo build --release -p evals

This creates the benchmarks executable in /target/release/. This is what the python scripts use to run the benchmarks.

Running the benchmark. Run the command below inside this directory.

python3 run_tool_bench.py

This creates benchmarks/ that contains some configuration, and data/*.csv with benchmark results.

Throughput plot (scaling with pattern length).

python3 plot_throughput.py

Outputs figs/throughput.{svg,pdf}.

Throughput statistics. Prints numeric throughput (GB/s) statistics corresponding to the plot above.

python3 throughput_stats.py results_*.csv

Throughput plot (scaling with text length).

python3 plot_throughput_text.py

Outputs figs/throughput_text.{svg,pdf}.

Trace plot.

python3 plot_trace.py

Outputs figs/trace.{svg,pdf}.

Section 3: Comparison with Parasail and Ish

To compare with parasail, clone the repo and follow the cmake build instructions. make parasail_aligner is sufficient, but will still take a while. Then, create patterns.fa and download a human genome (chm13v2.0.fa, direct download link):

>pattern
ACTCGACTTCAGCTACGCACATA

and to run with cost parameters matching Ish:

# Default cost params: 53-69s
time ./parasail_aligner -a sg_dx_striped_sse41_128_8  -x -d -t 1 -q patterns.fa -f chm13v2.0.fa -v -V
time ./parasail_aligner -a sg_dx_striped_sse41_128_16 -x -d -t 1 -q patterns.fa -f chm13v2.0.fa -v -V
time ./parasail_aligner -a sg_dx_striped_avx2_256_8   -x -d -t 1 -q patterns.fa -f chm13v2.0.fa -v -V
time ./parasail_aligner -a sg_dx_striped_avx2_256_16  -x -d -t 1 -q patterns.fa -f chm13v2.0.fa -v -V
# Ish' cost params: 81s-198s
time ./parasail_aligner -a sg_dx_striped_sse41_128_8  -x -d -t 1 -X 2 -M 2 -o 3 -e 1 -q patterns.fa -f chm13v2.0.fa -v -V
time ./parasail_aligner -a sg_dx_striped_sse41_128_16 -x -d -t 1 -X 2 -M 2 -o 3 -e 1 -q patterns.fa -f chm13v2.0.fa -v -V
time ./parasail_aligner -a sg_dx_striped_avx2_256_8   -x -d -t 1 -X 2 -M 2 -o 3 -e 1 -q patterns.fa -f chm13v2.0.fa -v -V
time ./parasail_aligner -a sg_dx_striped_avx2_256_16  -x -d -t 1 -X 2 -M 2 -o 3 -e 1 -q patterns.fa -f chm13v2.0.fa -v -V

For Ish (commit), follow the build instructions. To build with sse as the target instead of default avx (because it's faster in our benchmark with quite short queries):

pixi run build -D ISH_SIMD_TARGET=sse

Then run:

time ./ish --scoring-matrix actgn --record-type fastx --threads 1 ACTCGACTTCAGCTACGCACATA chm13v2.0.fa

For me, this takes 69s for SSE, and 110s for AVX2.

Section 4: CIRSPR off-traget

We reused the benchmark of the ChopOff paper from their gitlab with our fork here:

This workflows uses conda/mamba so you need to install that if you don't have it already. Run the following in the root of the fork:

mamba env create --name sassy-benchmark --file environment.yaml
mamba activate sassy-benchmark
snakemake --cores 16

Then the output of the tools is in out_dir and the timings in summary.txt.

Modifications we made to the Chopoff benchmark code:

• Force serial execution as the code did not use threads: in each rule it would execute multiple tools simultaneously that all use the maximum CPU's thereby competing with each other.
• Time indexes, the Chopoff index construction was not timed, we added timings for the construction time for all edit cut-offs.
• Added Sassy
• Remove older tools that were shown to perform worse (CRISPRITz, Cas-OFFinder)
• Used a genome without N characters as tools might deal with that different (chm13)