recent sequences assigned lineage "A"

[7erenceLee]

Hi Pangolin Team,
I've recently installed and ran pangolin on a bunch of sequences downloaded from GISAID with collection dates ~July 2025 and they all returned usher placements: A(1/1) but have scorpio calls Omicron(BA.2-like) (see below lineage report).

Just wondering if I'm getting something wrong (versions or commands etc) or is there an issue with the Usher placements?

pangolin --update

pangolin already latest release (v4.3.1)
pangolin-data already latest release (v1.34)
constellations already latest release (v0.1.12)
scorpio already latest release (v0.3.19)

command I used was simply:
pangolin test.fasta

taxon,lineage,conflict,ambiguity_score,scorpio_call,scorpio_support,scorpio_conflict,scorpio_notes,version,pangolin_version,scorpio_version,constellation_version,is_designated,qc_status,qc_notes,note
hCoV-19/Wuhan/WIV04/2019|EPI_ISL_402124,B,0.0,,,,,,PANGO-v1.34,4.3.1,0.3.19,v0.1.12,True,pass,Ambiguous_content:0.02,Assigned from designation hash.
hCoV-19/Singapore/Y25R28MSC192/2025|EPI_ISL_20063647|2025-07-01,A,0.0,,Omicron (BA.2-like),0.92,0.0,scorpio call: Alt alleles 57; Ref alleles 0; Amb alleles 0; Oth alleles 5,PUSHER-v1.34,4.3.1,0.3.19,v0.1.12,False,pass,Ambiguous_content:0.02,Usher placements: A(1/1); scorpio lineage BA.2 conflicts with inference lineage A
hCoV-19/Singapore/Y25R28MSC132/2025|EPI_ISL_20063697|2025-07-01,A,0.0,,Omicron (BA.2-like),0.92,0.02,scorpio call: Alt alleles 57; Ref alleles 1; Amb alleles 0; Oth alleles 4,PUSHER-v1.34,4.3.1,0.3.19,v0.1.12,False,pass,Ambiguous_content:0.02,Usher placements: A(1/1); scorpio lineage BA.2 conflicts with inference lineage A
hCoV-19/Singapore/Y25R28MSC133/2025|EPI_ISL_20063661|2025-07-01,A,0.0,,Omicron (BA.2-like),0.92,0.02,scorpio call: Alt alleles 57; Ref alleles 1; Amb alleles 0; Oth alleles 4,PUSHER-v1.34,4.3.1,0.3.19,v0.1.12,False,pass,Ambiguous_content:0.02,Usher placements: A(1/1); scorpio lineage BA.2 conflicts with inference lineage A
hCoV-19/Spain/CT-HUVH-E632652/2025|EPI_ISL_20064114|2025-07-01,A,0.0,,Omicron (BA.2-like),0.92,0.02,scorpio call: Alt alleles 57; Ref alleles 1; Amb alleles 0; Oth alleles 4,PUSHER-v1.34,4.3.1,0.3.19,v0.1.12,False,pass,Ambiguous_content:0.02,Usher placements: A(1/1); scorpio lineage BA.2 conflicts with inference lineage A
hCoV-19/Spain/CT-HUJT-E328489/2025|EPI_ISL_20064121|2025-07-01,A,0.0,,Omicron (BA.2-like),0.84,0.02,scorpio call: Alt alleles 52; Ref alleles 1; Amb alleles 4; Oth alleles 5,PUSHER-v1.34,4.3.1,0.3.19,v0.1.12,False,pass,Ambiguous_content:0.11,Usher placements: A(1/1); scorpio lineage BA.2 conflicts with inference lineage A

[AngieHinrichs]

Hi @7erenceLee, sorry about the slow response! Can you run these commands and then upload/attach the resulting file test.tar.gz file to this issue?

pangolin --no-temp --outdir test.out test.fasta >& test.log
tar cvzf test.tar.gz test.out test.log

Can you also send the output of the command uname -a?

[7erenceLee]

Hi @AngieHinrichs, my turn to apologise for the delay.

I've tried on a few fresh installations on different machines and got the same results.

output for uname -a :
Linux mgap 6.8.0-79-generic #79-Ubuntu SMP PREEMPT_DYNAMIC Tue Aug 12 14:42:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

test.tar.gz

[AngieHinrichs]

Thanks! The files are looking as expected right up until usher-sampled runs, and usher-sampled really shouldn't assign A but I see it does. Assuming you used conda to install pangolin, can you send the output of these commands in the conda environment in which you installed pangolin?

ls -l $CONDA_PREFIX/lib/python*/site-packages/pangolin_data/data/lineageTree.pb

md5sum $CONDA_PREFIX/lib/python*/site-packages/pangolin_data/data/lineageTree.pb

usher-sampled --version

[7erenceLee]

Thanks for the suggestion, seems like an issue with usher-sampled based on the outputs.
I'll see if I can troubleshoot this a little more later today.

[AngieHinrichs]

Thanks, that confirms that you have all the correct inputs for usher-sampled. It's very concerning that usher-sampled is running to completion while producing incorrect results... there must be somewhere that it's not checking for an error return, I guess. Maybe if you can get openmpi installed on your system in such a way that there's no warning about the missing component, then it would work? Or perhaps you could try the StaPHB Docker image? https://hub.docker.com/r/staphb/pangolin

[AngieHinrichs]

@7erenceLee can you try this in your conda environment for pangolin?:

conda install usher==0.6.5

and then try running pangolin again?

The bioconda version of usher 0.6.6 was built with a massive patch file that enables it to be built on Mac/ARM which is good, but it might have broken something. When I made a new conda environment with a clean install of pangolin, it assigned all A's with usher 0.6.6, but when I downgraded to 0.6.5 it assigned the correct lineages. (Previously I was using a local built-from-source version of usher 0.6.6 so I didn't suspect that usher 0.6.6 could be the problem, but then I noticed the patching business in bioconda.)

[7erenceLee]

Thanks for looking into this Angie. Can confirm rolling back usher to 0.6.5 resolved the issue when installing from conda.

[AngieHinrichs]

Great, thank you for confirming! I will ping the bioconda maintainers about 0.6.6.

[donutbrew]

Hi Angie! Currently, we're unable to make anything past usher 0.6.3 from bioconda work for us. There are various incompatibilities. When I pin usher 0.6.4 or 0.6.5 in a debian environment and run usher --version I get:

usher: error while loading shared libraries: libboost_program_options.so.1.85.0: cannot open shared object file: No such file or directory

Simply adding libboost=1.85.0 to my conda env I can successfully run usher --version but get a different usher error running pangolin test.fa:

Using UShER as inference engine.
/usr/bin/bash: line 23:   679 Illegal instruction     (core dumped) usher-sampled --help >&/dev/null

*** usher-sampled is not installed -- please upgrade usher to at least v0.6.1 ***
*** If you used conda to install usher, run 'conda update --no-pin usher'     ***
*** Alternatively if mamba is installed, run 'mamba update --no-pin usher'    ***

/usr/bin/bash: line 23:   688 Illegal instruction     (core dumped) $usher -n -D -i /opt/conda/lib/python3.8/site-packages/pangolin_data/data/lineageTree.pb -v /tmp/tmp_t0quvo2/sequences.aln.vcf -T $threads -d '/tmp/tmp_t0quvo2' &> /tmp/tmp_t0quvo2/logs/usher.log
[Mon Oct 20 17:46:28 2025]
Error in rule usher_inference:
    jobid: 3
    input: /tmp/tmp_t0quvo2/not_cache_assigned.fasta, /opt/conda/lib/python3.8/site-packages/pangolin/data/reference.fasta, /opt/conda/lib/python3.8/site-packages/pangolin_data/data/lineageTree.pb
    output: /tmp/tmp_t0quvo2/clades.txt
    log: /tmp/tmp_t0quvo2/logs/usher.log (check log file(s) for error details)
    shell:

        echo "Using UShER as inference engine."
        if [ -s /tmp/tmp_t0quvo2/not_cache_assigned.fasta ]; then
            cat /opt/conda/lib/python3.8/site-packages/pangolin/data/reference.fasta > /tmp/tmp_t0quvo2/sequences.withref.fa
            echo >> /tmp/tmp_t0quvo2/sequences.withref.fa
            usher=usher
            threads=1
            if usher-sampled --help >& /dev/null; then
                usher="usher-sampled --optimization_radius 0"
            else
                echo ""
                echo "*** usher-sampled is not installed -- please upgrade usher to at least v0.6.1 ***"
                echo "*** If you used conda to install usher, run 'conda update --no-pin usher'     ***"
                echo "*** Alternatively if mamba is installed, run 'mamba update --no-pin usher'    ***"
                echo ""
            fi
            cat /tmp/tmp_t0quvo2/not_cache_assigned.fasta >> /tmp/tmp_t0quvo2/sequences.withref.fa
            faToVcf -includeNoAltN /tmp/tmp_t0quvo2/sequences.withref.fa /tmp/tmp_t0quvo2/sequences.aln.vcf
            $usher -n -D -i /opt/conda/lib/python3.8/site-packages/pangolin_data/data/lineageTree.pb -v /tmp/tmp_t0quvo2/sequences.aln.vcf -T $threads -d '/tmp/tmp_t0quvo2' &> /tmp/tmp_t0quvo2/logs/usher.log
        else
            rm -f /tmp/tmp_t0quvo2/clades.txt
            touch /tmp/tmp_t0quvo2/clades.txt
        fi

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)


Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2025-10-20T174626.492813.snakemake.log

This may deserve a separate issue as we're getting into the weeds, but the "A" assignment problem is what led me down this path.

[AngieHinrichs]

Thanks for reporting that @donutbrew and sorry for the inconvenience. Just to confirm, if you conda install pangolin and then conda install usher==0.6.3, it does work OK? What platform are you running on and is it cloud, HPC cluster, local machines, ...?

To me this sounds like a bioconda issue more than a pangolin issue strictly speaking. pangolin has its own environment.yml for dev installs, but the bioconda packages are built using https://github.com/bioconda/bioconda-recipes/blob/master/recipes/pangolin/meta.yaml. I think the solution to this would go something like this:

1. the bioconda package for pangolin should pin usher==0.6.3 until problems with 0.6.4 and beyond are resolved
2. the bioconda package(s) for usher versions 0.6.4, 0.6.5 and still 0.6.6 need to be fixed

Does that sound right?

It's been years since I did any work in bioconda and I recall there was a learning curve (fork bioconda-recipes, clone on local machine, get bioconda-like build working on local machine, make pull request whose CI tests probably fail because it builds in a different environment from local machine, keep making changes to PR until PR's CI passes, use some bioconda bot keyword to request a review, wait for next release?). I'll go get re-familiarized with it, but if anyone listening is super-comfortable with the bioconda process and could volunteer to help me when I get confused by CI failures or bioconda practices, that would be most helpful. IIRC the folks on the bioconda gitter channel are usually very responsive and helpful.

[donutbrew]

Yeah, sorry to complicate the issue a bit. We build containers on the micromamba docker image (derived from Debian linux/amd64). We clone the latest pangolin tag, then build the environment using mamba and environment.yml, and finally install pangolin using pip install . This workflow has served us well for a long time, but as far as I can remember, this is the first real depenency issue we've come across. And to be clear, it still works fine as long as we specify usher==0.6.3.

As an alternative, we could build Usher separately, but using your environment.yml file and conda/mamba has been so nice...

So I think yes your two action items would serve us well. (And update the pangolin/environment.yml to pin usher 0.6.3 and tag a minor version update to pangolin if possible/logical)

[AngieHinrichs]

Ah OK, so you are using environment.yml not the bioconda pangolin package, thanks. I will update the pangolin/environment.yml first, that's easy. 🙂 Yes, I can tag a minor (patch a la https://semver.org/) update.

Honestly I would avoid building Usher separately, it's non-trivial because of its lib dependencies; just stick with bioconda usher==0.6.3. The changes made to the usher code base since 0.6.3 should not affect the way pangolin uses usher-sampled, they're mostly to the matUtils program.