Skip to content

Migrate near-field table cache from HDF5 to SQLite#18

Merged
xywei merged 2 commits intomainfrom
sqlite-cache-migration
Mar 17, 2026
Merged

Migrate near-field table cache from HDF5 to SQLite#18
xywei merged 2 commits intomainfrom
sqlite-cache-migration

Conversation

@xywei
Copy link
Copy Markdown
Owner

@xywei xywei commented Mar 17, 2026

Summary

  • replace NearFieldInteractionTableManager persistence in volumential/table_manager.py with sqlite-backed storage (nearfield_cache, nearfield_cache_kwargs, and nearfield_cache_meta) while keeping existing .hdf5 cache filenames
  • drop the h5py dependency from setup.py, .test-conda-env-py3.yml, and .test-conda-env-py3-macos.yml
  • add test/test_table_manager_sqlite_migration.py and raise a clear runtime error when a legacy HDF5 cache file is opened with the new sqlite backend

Testing

  • on ipa: python -m pytest test/test_table_manager.py test/test_table_manager_sqlite_migration.py test/test_nearfield_potential_table.py test/test_nearfield_interaction_completeness.py -q (31 passed, 36 skipped)
  • on ipa: python -m pytest test/test_volume_fmm.py::test_volume_fmm_laplace -vv -x passes for NVIDIA CUDA when run with the companion sumpy fix in fix: correct NVIDIA CUDA queue guard in run_opencl_fft inducer/sumpy#271

Replace h5py-based cache I/O with sqlite3 storage, remove h5py from dependencies, and add a regression test with a clearer error for legacy HDF5 cache files.
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 52e4f9cdf6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Normalize NumPy scalar serialization for sqlite-backed cache kwargs and fail fast on cache miss or force_recompute in read-only mode to avoid unnecessary recomputation.
@xywei
Copy link
Copy Markdown
Owner Author

xywei commented Mar 17, 2026

Addressed both review comments in commit 27f1270 and pushed to this PR.

Changes made:

  • normalize NumPy float/complex scalar serialization in table_manager (repr(float(value)) / repr(complex(value))) so round-tripping through sqlite kwargs works.
  • fail fast in read-only mode for cache miss and force_recompute=True, and avoid recompute fallback on load corruption when read-only.
  • added regression tests in test/test_table_manager_sqlite_migration.py for NumPy scalar roundtrip and read-only fail-fast behavior.

Validation on ipa:

  • python -m pytest test/test_table_manager_sqlite_migration.py -q -> 5 passed
  • python -m pytest test/test_table_manager.py test/test_table_manager_sqlite_migration.py test/test_nearfield_potential_table.py test/test_nearfield_interaction_completeness.py -q -> 35 passed, 36 skipped

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant