Tasks to address for next build

[jackjt8]

Tasks for next build

• Address Memory Consumption
• Explore potential issues with satellites and Van Allen Belts

Details

Addressing Memory Consumption

Preface

As it currently stands, when loading and prepping 16 years worth of data for SVN41 memory usage climbs to around 12GB and during actual processing this can exceed 16GB. This is far from ideal and without a pagefile can lead to system crashes. Some attempts have been made in order to manage the applications memory usage but so far, nothing has made a huge impact.

Proposed solutions

1. Prep data on a capable system and then write the prepped data to a file. This should solve the memory consumption issues. However any changes to the data prepping process will require access to a system capable of re-prepping data.

2. Per-datafile prep. This method will require a significant rewrite of how the application works but will offer the best compatibility and potentially the best performance. Though, IO interaction may become a bottleneck in this situation.

3. Chunk based datafile prep. Can be seen as a hybrid of solutions 1 & 2. Rather than working on a single datafile, or all the datafiles for that matter, it works on a handful at a time. It has many of the benefits of solution 2 just without the potential IO interaction limitations.

Performance gains from solutions 2 & 3 are due to smaller particle burst - earthquake time comparisons, mainly useful for satellites that do not have data for the full 16 years.

Exploration of satellite - Van Allen Belt

Unlike work previously done, our satellites seem to reside within the outer Van Allen Belt and some, given their orbital inclination, may exit this belt. In order to explore this, I feel the best approach is to plot all the particle count rates for a period of time along with positional datasets, in order to note any pattern.