Move Plotting Module from Examples to Main fpsim Package

[menriquez-IDM]

Goal: Move Plotting Module from Examples to Main fpsim Package

Overview

The plots.py module currently lives in the examples/ directory (Release 3.5) but provides core visualization functionality that should be part of the main fpsim package. This issue outlines the goals and implementation approach for moving this module into the package proper.

Current Status

• Location: /examples/plots.py
• Dependencies: numpy, matplotlib, plotly, sciris, starsim, fpsim
• Exports: 14 public plotting functions + 1 utility function

---

Plot Goals & Descriptions

1. plot_method_mix_simple

Goal: Provide a quick, clean comparison of method mix between two simulations

• Creates side-by-side bar chart and pie chart
• Ideal for presentations and reports
• Customizable colors and chart titles
• Option to exclude 'none' method

2. plot_comparison_full

Goal: Comprehensive visualization of method mix matrix changes

• Multi-panel figure (6 subplots) showing:
  • Side-by-side method comparison bars
  • Difference analysis (horizontal bars)
  • Baseline and intervention pie charts
  • Heatmap comparing baseline vs intervention vs changes
  • Summary statistics panel
• Professional publication-quality output

3. plot_method_bincount

Goal: Efficient visualization of method distribution using bincount approach

• Single or dual simulation comparison
• Efficient for integer method indices
• Shows percentage distribution
• Option to exclude 'none' method
• Customizable colors per simulation

4. plot_method_mix_by_type_per_year

Goal: Track method mix evolution by method type (not individual methods)

• Groups methods by type (pill, injectable, IUD, etc.)
• Time series visualization
• Annotations for method introduction events
• Useful for understanding method category trends

5. plot_pregnancies_per_year

Goal: Compare pregnancy incidence between two simulations

• Aggregates monthly data to yearly counts
• Area fill for visual emphasis
• Direct impact assessment of interventions
• Grid and legends for clarity

6. iterable_plots

Goal: Interactive exploration of method mix over time

• Uses Plotly for interactivity
• Hover details and zoom capabilities
• Method introduction annotations
• Ideal for exploratory data analysis

7. plot_injectable_methods_comparison

Goal: Detailed analysis of injectable methods specifically

• Two panels:
  • Individual injectable trends (baseline vs intervention)
  • Combined injectable share
• Highlights new injectable methods
• Shows intervention year marker
• Quantifies total increase in injectable share

8. plot_method_mix_evolution

Goal: Visualize complete method mix composition over time

• Stacked area chart showing all methods
• Color-coded method bands
• Intervention year marker
• Notes about new method introduction
• Excludes 'none' method for clarity

9. plot_new_method_adoption

Goal: Focus specifically on adoption rates of newly introduced methods

• Filled area + trend line for each new method
• Dynamic detection of new methods
• Final adoption rate annotations
• Method introduction marker
• Supports multiple new methods

10. plot_method_comparison_bar

Goal: Compare final method usage between baseline and intervention

• Bar chart at end of simulation period
• Highlights new methods with distinct color/border
• Shows absolute user counts
• Value labels on bars
• Useful for summary comparisons

11. plot_cpr_comparison

Goal: Track Contraceptive Prevalence Rate (CPR) and modern CPR (mCPR)

• Dual panel: CPR and mCPR side-by-side
• Baseline vs intervention comparison
• Intervention year marker
• Key metrics for policy evaluation
• Standard family planning indicators

12. plot_births_comparison

Goal: Compare births over time between scenarios

• Line plot with markers
• Calculates cumulative births after intervention
• Shows births averted
• Text box with summary statistics
• Direct impact metric for interventions

13. create_summary_figure

Goal: Comprehensive dashboard-style visualization

• 6-panel figure including:
  1. New method adoption over time
  2. Injectable & new methods individual trends
  3. Total injectable share
  4. Method substitution effects
  5. Top 6 methods ranking
  6. All methods comparison bar chart
• Publication-ready comprehensive summary
• Dynamic method detection
• Highlights new methods consistently

14. print_summary_statistics

Goal: Text-based summary statistics complement to visualizations

• Final prevalence rates (CPR, mCPR)
• Births averted calculation
• New method adoption percentages
• Number of users per new method
• Console output for reports

15. generate_figure_prefix (Utility)

Goal: Generate unique prefixes for saved figures to avoid overwrites

• Three modes: 'method', 'timestamp', 'count'
• Converts method names to safe filenames
• Handles multiple new methods
• Ensures unique file naming

---

Helper Functions

• _get_results_dir(): Create/get results directory for saving figures
• _get_new_method_names(): Extract dynamically added method names
• _get_new_method_labels_from_sims(): Get new method labels from simulations
• _process_colors(): Handle color specifications (colormap names, lists, single colors)
• _get_default_colors(): Default color scheme for consistency
• _process_plot_colors(): Merge user colors with defaults
• _get_data(): Extract method counts, proportions, and labels from simulations
• _add_annotations(): Add intervention event annotations to plotly figures
• _adjust_bar_spacing_for_methods(): Adjust bar spacing based on number of methods

---

Suggested Implementation Approach

Phase 1: Module Structure

fpsim/
├── plotting.py (existing, basic plots)
└── plotting/
    ├── __init__.py
    ├── method_mix.py      # Method mix comparison plots
    ├── impact.py          # Impact analysis plots (CPR, births, pregnancies)
    ├── adoption.py        # New method adoption plots
    ├── interactive.py     # Plotly-based interactive plots
    ├── summary.py         # Comprehensive summary figures
    └── utils.py           # Helper functions

Alternative Simpler Structure (Recommended for initial migration):

fpsim/
├── plotting.py (existing, basic plots)
└── plotting_extended.py   # All advanced plots from examples/plots.py

Phase 2: Integration Steps

1. Copy with Refactoring

   • Move examples/plots.py → fpsim/plotting_extended.py
   • Update imports to use absolute paths
   • Ensure no circular dependencies
   • Maintain backward compatibility

2. Update init.py

   # In fpsim/__init__.py, add:
   from . import plotting_extended as plotting_advanced
   # or expose key functions directly:
   from .plotting_extended import (
       plot_method_mix_simple,
       plot_comparison_full,
       create_summary_figure,
       # ... etc
   )

3. Documentation

   • Add docstrings to all public functions (most already have good docs)
   • Create tutorial notebook: docs/tutorials/advanced_plotting.ipynb
   • Update API documentation
   • Add plotting gallery to documentation

4. Testing

   • Create unit tests for each plotting function
   • Test with various simulation outputs
   • Test color customization
   • Test save/show options
   • Test dynamic method detection

5. Deprecation Path

   • Keep examples/plots.py for one release cycle
   • Add deprecation warning pointing to new location
   • Update all examples to use new import path
   • Remove old file in next major version

Phase 3: API Design Considerations

Consistent Function Signatures

• All plot functions should accept consistent parameters
• Use show_figure=True, save_figure=False, filename=None pattern
• Support colors dict for customization
• Support exclude_none where applicable
• Return figure objects for further customization

Color System

• Standardize color palettes
• Provide preset themes: 'default', 'colorblind', 'print', 'presentation'
• Allow full customization via colors dict

Naming Convention

• plot_* for single visualizations
• create_*_figure for multi-panel figures
• print_* for text output

Phase 4: Enhancements (Future)

1. Style Presets

   fp.plotting.set_style('publication')  # seaborn-like style switching
   fp.plotting.set_style('presentation')
   fp.plotting.set_style('default')

2. Plotting Context Manager

   with fp.plotting.figure_context(save_dir='figures/', prefix='sc_dmpa_'):
       fp.plotting.plot_cpr_comparison(baseline, intervention)
       fp.plotting.plot_births_comparison(baseline, intervention)

3. Batch Plotting

   fp.plotting.create_all_comparison_plots(
       baseline_sim, 
       intervention_sim, 
       save_dir='figures/',
       prefix='auto'  # auto-generates from method names
   )

4. Export Options

   • Support multiple formats: PNG, PDF, SVG
   • Resolution settings
   • DPI presets for different uses

---

Benefits of Moving to Package

1. Better Discoverability: Users can find plotting functions via import fpsim as fp; fp.plotting.*
2. Consistent API: Standardized function signatures and behavior
3. Testing & Quality: Plots become part of test suite
4. Documentation: Integrated into package documentation
5. Version Control: Plotting functionality versioned with package
6. Import Simplicity: No need to copy files from examples
7. Maintenance: Centralized location for bug fixes and enhancements

---

Migration Checklist

• Create fpsim/plotting_extended.py or submodule structure
• Copy and refactor plotting functions
• Update imports and dependencies
• Add/improve docstrings
• Update fpsim/__init__.py to expose plotting functions
• Create unit tests
• Create plotting tutorial notebook
• Update API documentation
• Add plotting gallery to docs
• Update examples to use new import path
• Add deprecation warning to examples/plots.py
• Update CHANGELOG
• Test with real simulation outputs
• Review by core team

---

Open Questions

1. Should we keep both plotting.py and plotting_extended.py, or merge them?
2. Should we create a submodule (fpsim/plotting/) or keep as single file?
3. What should be the default save directory? (current: figures/ in CWD)
4. Should we add a figure manager/registry for batch operations?
5. Should we support different backends (matplotlib, plotly, bokeh)?

---

Related Files

• /examples/plots.py - Current implementation
• /fpsim/plotting.py - Existing basic plotting module
• /examples/example_add_new_method.py - Usage examples
• /docs/tutorials/ - Documentation location

---

Estimated Effort

• Refactoring & Move: 4-6 hours
• Documentation: 4-6 hours
• Testing: 6-8 hours
• Review & Integration: 2-4 hours
• Total: ~16-24 hours

---

Priority

Medium-High - This functionality is actively used and would benefit many users, but existing workaround (copying from examples) functions adequately for now.

---

Labels

• enhancement
• plotting
• api-improvement
• documentation
• refactoring

[robynstuart]

@menriquez-IDM I believe this is no longer needed since the plots were moved - can we close it?

[menriquez-IDM]

@robynstuart mmm I'm not sure, the plot that I added recently to the plotting.py file was a very small tiny implementation, just to show a basic plot.

[robynstuart]

Ok maybe it's just out of date - I just looked at the overview section which says "The plots.py module currently lives in the examples/ directory (Release 3.5)", but there's no plots module in the examples directory. I think we can keep this open, but we might need to iterate on the approach. Perhaps the first task would be to figure out what plots we need to keep as core methods vs which ones are custom for particular analyses, which might be part of the UI design work