Skip to content

Gitc 8361: Memory use optimization#59

Merged
jackiryan merged 2 commits intomainfrom
GITC-8361
Jan 14, 2026
Merged

Gitc 8361: Memory use optimization#59
jackiryan merged 2 commits intomainfrom
GITC-8361

Conversation

@jackiryan
Copy link
Copy Markdown
Collaborator

@jackiryan jackiryan commented Jan 9, 2026

Description

This is quite a large change to the codebase, essentially a rewrite:

  • Refactored code be more memory efficient, reducing peak memory usage when processing large granules by about 90% at no performance hit. This is accomplished by fundamentally changing the way data is read, doing away with rioxarray and instead using rasterio directly. The DatasetReader allows for windowed reads into the source data, at the expense of having to reimplement the mask_and_scale functionality that is provided by rioxarray.
  • Minor changes related to resolving tech debt and code quality improvements throughout.

Jira Issue ID

GITC-8361

Local Test Steps

Ran updated unit tests. The data input/outputs of the tests do not change at a raster level, so this change is intended to be transparent from a data perspective. I also ran end-to-end tests with real data from TEMPO and GHRSST MUR in a number of configurations using a test script to check that there were no regressions and to verify memory usage is within acceptable parameters. Regression test notebook run is pending.

  • For TEMPO, generating a 4096x4096 PNG now uses about 200MB RAM (was 2GB)
  • For GHRSST MUR, the tiled 36000x18000 output now peaks at 1.9GB RAM (was 19GB)
  • Wall clock run times are about the same, a little faster (I don't have numbers offhand)

PR Acceptance Checklist

  • Jira ticket acceptance criteria met.
  • CHANGELOG.md updated to include high level summary of PR changes.
  • docker/service_version.txt updated if publishing a release.
  • Tests added/updated and passing.
  • Documentation updated (if needed).

@jackiryan jackiryan merged commit fd90b02 into main Jan 14, 2026
7 checks passed
@jackiryan jackiryan deleted the GITC-8361 branch January 14, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants