Fitness Logic Overhaul and Immunity Output Restructuring

[zorian15]

Fitness Logic Overhaul and Immunity Output Restructuring Plan

Overview

This plan outlines the removal of fitness calculations during simulation runtime and the transition from individual host immunity histories to population-level centroid tracking in CSV format, with proper handling of naive hosts.

Goals

1. Remove all fitness calculations from the simulation runtime
2. Remove fitness storage from Virus objects
3. Remove fitness from output files
4. Replace individual host immunity history output with population immunity centroids
5. Output immunity data in CSV format for easier analysis
6. Include global population immunity centroid alongside individual deme centroids
7. Properly handle naive hosts (empty immune histories) without contaminating centroid calculations

Part 1: Fitness Logic Removal

1.1 Remove Fitness Field from Virus Class

File: src/main/java/org/antigen/virus/Virus.java

• Remove line 18: private double fitness;
• Remove lines 101-106: getFitness() and setFitness() methods
• Keep the other fields and methods (averageInfectionRisk, probSusceptible, demeSeasonality) as they may be useful for other analyses

1.2 Remove Fitness Calculations from HostPopulation

File: src/main/java/org/antigen/host/HostPopulation.java

• Remove lines 375-385: Fitness calculation in distributeContacts()
• Remove lines 488-496: Fitness calculation in mutate()
• Remove lines 532-540: Fitness calculation in sample()
• Remove lines 502-513: getAverageRisk() method (no longer needed)

1.3 Remove Duplicate Method from Simulation

File: src/main/java/org/antigen/core/Simulation.java

• Remove lines 169-180: getAverageRisk() method

1.4 Remove Fitness from Output Files

File: src/main/java/org/antigen/virus/VirusTree.java

• Lines 645-650: Remove "fitness" from CSV header for geometricSeq phenotype
• Lines 648-649: Remove "fitness" from CSV header for other phenotypes
• Lines 668, 671-672: Remove v.getFitness() calls from tip output
• Lines 693, 695: Remove fitness values from branch output
• Line 728: Remove v.getFitness() from FASTA header

1.5 Remove Fitness Parameter

File: src/main/java/org/antigen/core/Parameters.java

• Remove line 41: public static int fitnessSampleSize = 10000;
• Remove lines 407-409: Loading of fitnessSampleSize from YAML

File: src/main/resources/parameters.yml

• Remove line 10: fitnessSampleSize: 100

Part 2: Immunity Output Restructuring

2.1 Create ImmunitySummary Helper Class

File: src/main/java/org/antigen/host/ImmunitySummary.java

Create new helper class to track immunity data:

package org.antigen.host;

public class ImmunitySummary {
    private final double[] centroid;
    private final double naiveFraction;
    private final int totalSampled;
    private final int experiencedHosts;
    
    public ImmunitySummary(double[] centroid, double naiveFraction, int totalSampled, int experiencedHosts) {
        this.centroid = centroid;
        this.naiveFraction = naiveFraction;
        this.totalSampled = totalSampled;
        this.experiencedHosts = experiencedHosts;
    }
    
    public double[] getCentroid() { return centroid; }
    public double getNaiveFraction() { return naiveFraction; }
    public int getTotalSampled() { return totalSampled; }
    public int getExperiencedHosts() { return experiencedHosts; }
    public boolean hasValidCentroid() { 
        return !Double.isNaN(centroid[0]) && !Double.isNaN(centroid[1]); 
    }
}

2.2 Add Centroid Calculation to Host

File: src/main/java/org/antigen/host/Host.java

Add new method that returns null for naive hosts:

public double[] getImmunityCoordinatesCentroid() {
    if (immuneHistory.length == 0) {
        return null; // Naive host - no immune experience
    }
    
    double sumTraitA = 0.0;
    double sumTraitB = 0.0;
    
    for (Phenotype p : immuneHistory) {
        if (p instanceof GeometricPhenotype) {
            GeometricPhenotype gp = (GeometricPhenotype) p;
            sumTraitA += gp.getTraitA();
            sumTraitB += gp.getTraitB();
        }
    }
    
    int count = immuneHistory.length;
    return new double[]{sumTraitA / count, sumTraitB / count};
}

2.3 Modify HostPopulation Immunity Output

File: src/main/java/org/antigen/host/HostPopulation.java

Replace printHostImmuneHistories() method (lines 652-661) with:

public ImmunitySummary getPopulationImmunitySummary(int n) {
    double sumTraitA = 0.0;
    double sumTraitB = 0.0;
    int experiencedHosts = 0;
    int naiveHosts = 0;
    
    for (int i = 0; i < n; i++) {
        Host h = getRandomHost();
        double[] centroid = h.getImmunityCoordinatesCentroid();
        
        if (centroid == null) {
            naiveHosts++;
        } else {
            sumTraitA += centroid[0];
            sumTraitB += centroid[1];
            experiencedHosts++;
        }
    }
    
    double[] avgCentroid = experiencedHosts > 0 ? 
        new double[]{sumTraitA / experiencedHosts, sumTraitB / experiencedHosts} : 
        new double[]{Double.NaN, Double.NaN};
        
    double naiveFraction = (double) naiveHosts / n;
    
    return new ImmunitySummary(avgCentroid, naiveFraction, n, experiencedHosts);
}

2.4 Modify Simulation Immunity Output

File: src/main/java/org/antigen/core/Simulation.java

Replace printHostImmuneHistories() method (lines 229-238) with CSV output:

private void printPopulationImmunityCentroids(PrintStream historyStream, int day) {
    // Write CSV header on first call
    if (day == 0) {
        historyStream.println("year,deme,ag1,ag2,naive_fraction,experienced_hosts");
    }
    
    double year = (double) day / 365.0;
    
    // Calculate and output individual deme centroids
    double globalSumAg1 = 0.0;
    double globalSumAg2 = 0.0;
    int globalExperiencedHosts = 0;
    int globalTotalSamples = 0;
    
    for (int i = 0; i < Parameters.demeCount; i++) {
        int nSamples = Parameters.hostImmunitySamplesPerDeme[i];
        ImmunitySummary summary = demes.get(i).getPopulationImmunitySummary(nSamples);
        
        // Output deme-specific data
        if (summary.hasValidCentroid()) {
            historyStream.printf("%.4f,%s,%.6f,%.6f,%.4f,%d%n", 
                year, 
                Parameters.demeNames[i], 
                summary.getCentroid()[0], 
                summary.getCentroid()[1],
                summary.getNaiveFraction(),
                summary.getExperiencedHosts());
        } else {
            // All hosts are naive in this deme
            historyStream.printf("%.4f,%s,NaN,NaN,%.4f,%d%n", 
                year, 
                Parameters.demeNames[i], 
                summary.getNaiveFraction(),
                summary.getExperiencedHosts());
        }
        
        // Accumulate for global centroid (only if valid)
        if (summary.hasValidCentroid()) {
            globalSumAg1 += summary.getCentroid()[0] * summary.getExperiencedHosts();
            globalSumAg2 += summary.getCentroid()[1] * summary.getExperiencedHosts();
        }
        globalExperiencedHosts += summary.getExperiencedHosts();
        globalTotalSamples += summary.getTotalSampled();
    }
    
    // Calculate and output global centroid
    if (globalExperiencedHosts > 0) {
        double globalAg1 = globalSumAg1 / globalExperiencedHosts;
        double globalAg2 = globalSumAg2 / globalExperiencedHosts;
        double globalNaiveFraction = (double) (globalTotalSamples - globalExperiencedHosts) / globalTotalSamples;
        
        historyStream.printf("%.4f,total,%.6f,%.6f,%.4f,%d%n", 
            year, 
            globalAg1, 
            globalAg2,
            globalNaiveFraction,
            globalExperiencedHosts);
    } else {
        // All sampled hosts are naive globally
        historyStream.printf("%.4f,total,NaN,NaN,1.0000,%d%n", 
            year, 
            globalExperiencedHosts);
    }
}

2.5 Update Output File Extension

File: src/main/java/org/antigen/core/Simulation.java

• Line ~409: Change output file from out.histories to out.histories.csv

2.6 Update Host Class

File: src/main/java/org/antigen/host/Host.java

• Remove or deprecate printHistoryCoordinates() method (lines 123-132)

Part 3: Unit Testing

3.1 Create Centroid Calculation Tests

File: src/test/java/org/antigen/host/TestHostImmunityCentroid.java

Create comprehensive unit tests:

package org.antigen.host;

import org.antigen.phenotype.GeometricPhenotype;
import org.antigen.phenotype.GeometricSeqPhenotype;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

public class TestHostImmunityCentroid {
    
    @Test
    public void testNaiveHostCentroid() {
        Host host = new Host();
        double[] centroid = host.getImmunityCoordinatesCentroid();
        assertNull(centroid, "Naive host should return null centroid");
    }
    
    @Test
    public void testSingleInfectionCentroid() {
        Host host = new Host();
        GeometricPhenotype phenotype = new GeometricPhenotype();
        phenotype.setTraitA(2.0);
        phenotype.setTraitB(3.0);
        host.addToHistory(phenotype);
        
        double[] centroid = host.getImmunityCoordinatesCentroid();
        assertNotNull(centroid);
        assertEquals(2.0, centroid[0], 1e-10);
        assertEquals(3.0, centroid[1], 1e-10);
    }
    
    @Test
    public void testMultipleInfectionsCentroid() {
        Host host = new Host();
        
        // Add first infection
        GeometricPhenotype p1 = new GeometricPhenotype();
        p1.setTraitA(1.0);
        p1.setTraitB(2.0);
        host.addToHistory(p1);
        
        // Add second infection
        GeometricPhenotype p2 = new GeometricPhenotype();
        p2.setTraitA(3.0);
        p2.setTraitB(4.0);
        host.addToHistory(p2);
        
        double[] centroid = host.getImmunityCoordinatesCentroid();
        assertNotNull(centroid);
        assertEquals(2.0, centroid[0], 1e-10); // (1.0 + 3.0) / 2
        assertEquals(3.0, centroid[1], 1e-10); // (2.0 + 4.0) / 2
    }
    
    @Test
    public void testGeometricSeqPhenotypeCentroid() {
        Host host = new Host();
        GeometricSeqPhenotype phenotype = new GeometricSeqPhenotype();
        phenotype.setTraitA(5.0);
        phenotype.setTraitB(7.0);
        host.addToHistory(phenotype);
        
        double[] centroid = host.getImmunityCoordinatesCentroid();
        assertNotNull(centroid);
        assertEquals(5.0, centroid[0], 1e-10);
        assertEquals(7.0, centroid[1], 1e-10);
    }
    
    @Test
    public void testMixedPhenotypesCentroid() {
        Host host = new Host();
        
        // Add GeometricPhenotype
        GeometricPhenotype gp = new GeometricPhenotype();
        gp.setTraitA(2.0);
        gp.setTraitB(4.0);
        host.addToHistory(gp);
        
        // Add GeometricSeqPhenotype
        GeometricSeqPhenotype gsp = new GeometricSeqPhenotype();
        gsp.setTraitA(6.0);
        gsp.setTraitB(8.0);
        host.addToHistory(gsp);
        
        double[] centroid = host.getImmunityCoordinatesCentroid();
        assertNotNull(centroid);
        assertEquals(4.0, centroid[0], 1e-10); // (2.0 + 6.0) / 2
        assertEquals(6.0, centroid[1], 1e-10); // (4.0 + 8.0) / 2
    }
}

3.2 Create Population Summary Tests

File: src/test/java/org/antigen/host/TestHostPopulationSummary.java

Create tests for population-level immunity summaries:

package org.antigen.host;

import org.antigen.core.Parameters;
import org.antigen.phenotype.GeometricPhenotype;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

public class TestHostPopulationSummary {
    
    private HostPopulation population;
    
    @BeforeEach
    public void setUp() {
        Parameters.demeCount = 1;
        Parameters.initialNs = new int[]{100};
        population = new HostPopulation(0);
    }
    
    @Test
    public void testAllNaivePopulation() {
        ImmunitySummary summary = population.getPopulationImmunitySummary(10);
        
        assertEquals(1.0, summary.getNaiveFraction(), 1e-10);
        assertEquals(0, summary.getExperiencedHosts());
        assertEquals(10, summary.getTotalSampled());
        assertFalse(summary.hasValidCentroid());
        assertTrue(Double.isNaN(summary.getCentroid()[0]));
        assertTrue(Double.isNaN(summary.getCentroid()[1]));
    }
    
    @Test
    public void testMixedPopulation() {
        // Add some infected hosts with known immunity centroids
        for (int i = 0; i < 5; i++) {
            Host host = population.getRandomHost();
            GeometricPhenotype p = new GeometricPhenotype();
            p.setTraitA(i * 1.0);
            p.setTraitB(i * 2.0);
            host.addToHistory(p);
        }
        
        // Sample the population
        ImmunitySummary summary = population.getPopulationImmunitySummary(20);
        
        // Should have some naive and some experienced hosts
        assertTrue(summary.getNaiveFraction() > 0.0);
        assertTrue(summary.getNaiveFraction() < 1.0);
        assertTrue(summary.getExperiencedHosts() > 0);
        assertEquals(20, summary.getTotalSampled());
        
        if (summary.hasValidCentroid()) {
            assertFalse(Double.isNaN(summary.getCentroid()[0]));
            assertFalse(Double.isNaN(summary.getCentroid()[1]));
        }
    }
    
    @Test
    public void testImmunitySummaryProperties() {
        ImmunitySummary summary = new ImmunitySummary(
            new double[]{1.0, 2.0}, 
            0.3, 
            100, 
            70
        );
        
        assertEquals(1.0, summary.getCentroid()[0], 1e-10);
        assertEquals(2.0, summary.getCentroid()[1], 1e-10);
        assertEquals(0.3, summary.getNaiveFraction(), 1e-10);
        assertEquals(100, summary.getTotalSampled());
        assertEquals(70, summary.getExperiencedHosts());
        assertTrue(summary.hasValidCentroid());
    }
    
    @Test
    public void testNaNHandling() {
        ImmunitySummary summary = new ImmunitySummary(
            new double[]{Double.NaN, Double.NaN}, 
            1.0, 
            50, 
            0
        );
        
        assertFalse(summary.hasValidCentroid());
        assertTrue(Double.isNaN(summary.getCentroid()[0]));
        assertTrue(Double.isNaN(summary.getCentroid()[1]));
    }
}

Part 4: Testing and Validation

4.1 Verify Removal

• Ensure simulation runs without fitness calculations
• Verify output files no longer contain fitness columns
• Check that virus objects don't store fitness values

4.2 Validate Centroid Output

• Confirm CSV format includes naive fraction and experienced host count
• Verify centroid calculations exclude naive hosts properly
• Test that global centroid is weighted by experienced hosts across demes
• Verify NaN handling for all-naive populations

4.3 Test Edge Cases

• All hosts naive (early simulation)
• All hosts experienced (late simulation)
• Mixed populations with varying naive fractions

Part 5: Documentation Updates

5.1 Update README

• Remove references to fitness calculations during simulation
• Document new CSV immunity output format including naive fraction
• Explain post-simulation fitness calculation approach
• Document naive host handling approach

5.2 Update Parameter Documentation

• Remove fitnessSampleSize parameter documentation
• Update immunity output parameter descriptions

Benefits

1. Performance: Eliminating fitness calculations should improve simulation speed
2. Storage: CSV centroid format drastically reduces output file size
3. Flexibility: Fitness can be calculated post-hoc with different methods
4. Analysis: CSV format with global and deme-specific centroids plus naive fractions provides comprehensive immunity tracking
5. Accuracy: Proper naive host handling prevents contamination of immunity centroids
6. Epidemiological Insight: Naive fraction provides valuable information about population susceptibility

Migration Path

1. Create feature branch following naming convention
2. Implement ImmunitySummary helper class
3. Update Host centroid calculation with naive handling
4. Update HostPopulation methods
5. Update Simulation output methods
6. Run unit tests to verify all scenarios
7. Test with both Geometric and GeometricSeq phenotypes
8. Update documentation
9. Create pull request with comprehensive description

Future Considerations

• Consider adding confidence intervals for centroids based on sample size
• Implement post-simulation fitness calculation scripts that properly handle naive hosts
• Add visualization tools for immunity centroid trajectories and naive fraction trends
• Explore additional immunity summary statistics beyond centroids
• Consider age-structured naive fraction reporting

[huddlej]

@zorian15 I can't tell from this (AI-generated?) write-up what the broader motivation is for these changes other than speeding up the simulations and saving disk space. As a potential future user of this framework for my own projects, I wanted to vote for keeping "individual host immunity history output". One super power of this simulator is that ability to track the full immune history of individual people over the course of many years along side the fitnesses of viruses that could infect those people.

That sort of infection history coupled with the known antigenic phenotypes of each virus at a given time could be used to test new models that try to account for individual immune histories when inferring antigenic effects of specific genetic changes. This sort of model is likely already being developed by multiple groups, so even if we didn't end up using these simulations to test our own models, other groups would likely benefit from this information.

If storage is a major concern, other file formats like parquet or HDF5 could serialize these same data to disk more efficiently than plain text CSV files.

[zorian15]

@huddlej thanks for your input John! (yes this was a crappy writeup from CC that I should have proof read better).

But I've opted to keep both outputs: the individual out.histories, and histories.csv (which I'll rename) that reports the average host immunities for ad-hoc fitness calculations in a sort of new part of the project that's spun up in the last two weeks after chatting with Marlin. The file sizes of the histories aren't an issue unless you're running a bunch of sweeps (which I was doing to test this out), so I'll keep them around. I'll look into those serialization methods though, as the history files are still kind of large. Thanks!