Memory Optimized HPC

[hongyuchen1030]

Memory Optimized HPC

Overview

This PR optimizes memory usage in our HPC code by updating the Blob exchange operators to use a single unified container with ghost cells rather than maintaining separate containers for the original core data and the exchanged (ghost-expanded) data. In our 1-processor tests, the total allocated memory dropped from approximately 15.63 GB down to 14.53 GB—yielding about a 1 GB improvement. In a 4-processor run, memory usage in BlobExchangeOP dropped nearly by half.

Changes

1. BlobsExchangeOP Updates

• Unified Container with Ghost Cells:
  The operator now uses a single container (_vecAllBlobsWithGhosts) that is initialized with the core data. Only even-numbered processors (the “prime” processors) lazily expand this container to include extra ghost columns before the exchange.
• Lazy Resizing:
  Instead of preallocating ghost cells in the constructor, the container is expanded only in StartExchange() on even-numbered processors. In EndExchange(), these processors merge the received ghost data into the expanded container. Odd-numbered processors (which send data) keep their container unchanged.

2. BlobBoxesDegExchangeOP Updates

• Unified Container Approach:
  Similarly, BlobBoxesDegExchangeOP now maintains a single unified container (_vecAllBlobBoxesDegWithGhosts).

3. TagExchangeOP Design

• No Ghost Cell Implementation:
  TagExchangeOP still maintains separate “original” versus “exchanged” Tag data. This decision is based on:
  1. GlobalTime Updates: Tag exchange requires updating the GlobalTime for each Tag. Keeping a clean copy of the core data simplifies this update process and minimizes potential errors.
  2. Small Memory Footprint: The memory overhead for Tag data is minimal compared to Blob data, so the benefits of a ghost cell approach are limited.

Testing

• The generated results are the same with the serial run (with MPI disabled)
• Memory usage improvements were confirmed through detailed annotated reports.