Merge branch 'main' of github.com:Blosc/python-blosc2 into fancyIndex

Author: Luke Shaw

.github/workflows/build.yml

@@ -20,7 +20,7 @@ jobs:
         python-version: ["3.12"]
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v5
 
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v5

.github/workflows/cibuildwheels.yml

@@ -60,7 +60,7 @@ jobs:
             artifact_name: "macos-universal2"
     steps:
       - name: Checkout repo
-        uses: actions/checkout@v4
+        uses: actions/checkout@v5
 
       - name: Set up Python
         uses: actions/setup-python@v5
@@ -124,7 +124,7 @@ jobs:
     # Only upload wheels when tagging (typically a release)
     if: startsWith(github.event.ref, 'refs/tags')
     steps:
-      - uses: actions/download-artifact@v4
+      - uses: actions/download-artifact@v5
         with:
           path: ./wheelhouse
           merge-multiple: true  # Merge all the wheels artifacts into one directory

.github/workflows/wasm.yml

@@ -30,7 +30,7 @@ jobs:
 
     steps:
       - name: Checkout repo
-        uses: actions/checkout@v4
+        uses: actions/checkout@v5
 
       - name: Set up Python
         uses: actions/setup-python@v5

ANNOUNCE.rst

@@ -1,19 +1,15 @@
-Announcing Python-Blosc2 3.7.0
+Announcing Python-Blosc2 3.7.2
 ==============================
 
-In this release:
+This is a maintenance release where:
 
-✅ Overhaul of documentation (API reference and Tutorials)
-✅ Improvements to lazy expression indexing and in particular much more efficient
-memory usage when applying non-unit steps
-✅ Extended functionality of ``expand_dims`` to match that of NumPy
-✅ 3(!) new data storage classes (``EmbedStore``, ``DictStore`` and ``TreeStore``)
-which allow for the efficient storage of heterogeneous array data
+✅ We have updated the Blosc2 C library to 2.21.1, which fixes a regression
+  in the build system detected in Fedora and Gentoo.
+✅ We reverted signature of ``TreeStore.__init__(()`` for making benchmarks
+  to get back to normal performance.
 
-See [here](https://github.com/Blosc/python-blosc2/pull/451#issuecomment-3178828765)
-for plots for the new data storage classes. And
-[here](https://github.com/Blosc/python-blosc2/pull/446#issuecomment-3167060686) for the improved performance
-of lazy expression slicing.
+Check our new blog post about ``TreeStore`` usage and performance at:
+https://www.blosc.org/posts/new-treestore-blosc2
 
 You can think of Python-Blosc2 3.x as an extension of NumPy/numexpr that:
 

CMakeLists.txt

@@ -50,7 +50,7 @@ else()
     include(FetchContent)
     FetchContent_Declare(blosc2
         GIT_REPOSITORY https://github.com/Blosc/c-blosc2
-        GIT_TAG d75993535461aaf2ded996f0a625cbec8df9655c  # v2.20.0
+        GIT_TAG 96bce728dcdbf41fd86f142ebef9e513f87a7afb  # v2.21.1
     )
     FetchContent_MakeAvailable(blosc2)
     include_directories("${blosc2_SOURCE_DIR}/include")

RELEASE_NOTES.md

@@ -1,8 +1,28 @@
 # Release notes
-## Changes from 3.7.0 to 3.7.1
+
+## Changes from 3.7.2 to 3.7.3
 
 XXX version-specific blurb XXX
 
+## Changes from 3.7.1 to 3.7.2
+
+* C-Blosc2 internal library updated to latest 2.21.1.
+
+* Revert signature of `TreeStore.__init__` for making benchmarks to get back
+  to normal performance.
+
+## Changes from 3.7.0 to 3.7.1
+
+* Added `C2Array.slice()` method and `C2Array.nbytes`, `C2Array.cbytes`, `C2Array.cratio`, `C2Array.vlmeta` and `C2Array.info` properties (PR #455).
+
+* Many usability improvements to the `TreeStore` class and friends.
+
+* New section about `TreeStore` in basics NDArray tutorial.
+
+* New blog post about `TreeStore` usage and performance at: https://www.blosc.org/posts/new-treestore-blosc2
+
+* C-Blosc2 internal library updated to latest 2.21.0.
+
 ## Changes from 3.6.1 to 3.7.0
 
 * Overhaul of documentation (API reference and Tutorials)

ROADMAP-TO-4.0.md

@@ -0,0 +1,23 @@
+List of desired features for a 4.0 release
+------------------------------------------
+
+* First and foremost, we would like to have at least of basic implementation of the [array API](https://data-apis.org/array-api).  This will require a lot of low-level work on the basic NDArray container to make indexing to work as close as the standard.
+
+* Have a completely specified format for the `TreeStore` and `DictStore`.  The format should allow to have containers either in memory or on disk.  Also, it should allow a sparse or contiguous storage.  The user will be able to specify these properties by following the same conventions than for NDArray objects (alas, `urlpath` and `contiguous` params).
+
+    * New `.save()` and `.to_cframe()` methods should be implemented to convert from in-memory representations to on disk and viceversa.
+    * The format for `TreeStore` and `DictStore` will initially be defined at Python level, and documented only in the Python-Blosc2 repository.  An implementation in the C library is desirable, but not mandatory at this time.
+
+* A new `Table` object should be implemented based on the `TreeStore` class (a subclass?), with a label ('table'?) in metalayers indicating that the contents of the tree can be interpreted as regular table.  As `TreeStore` is hierarchical, a subtree can also be interpreted as a `Table` if there a label in the metalayer of the subtree (or group in HDF5 parlance); that can lead to tables than can have different subtables embedded.  It is not clear yet if should impose the same number of rows for all the columns.
+
+The constructor for the `Table` object should take some parameters to specify properties:
+
+    * `columnar`: True or False. If True, every column will be stored in a different NDArray object.  If False, the columns will be stored in the same NDArray object, with a compound dtype.  In principle, one should be able to create tables that are hybrid between column and row wise, but at this point it is not clear what is the best way to do that.
+
+`Table` should support at least these methods:
+
+    * `.__getitem__()` and `.__setitem__()` so that values can be get and set.
+    * `.append()` for appending (multi-) rows of data for all columns in one go.
+    * `.__iter__()` for easy and fast iteration over rows.
+    * `.where()`: an iterator for queying with conditions that are evaluated with the internal compute engine.
+    * `.index()` for indexing a column and getting better performance in queries (desirable, but optional for 4.0).

bench/b2zip-linspace.py

@@ -0,0 +1,57 @@
+#######################################################################
+# Copyright (c) 2019-present, Blosc Development Team <[email protected]>
+# All rights reserved.
+#
+# This source code is licensed under a BSD-style license (found in the
+# LICENSE file in the root directory of this source tree)
+#######################################################################
+
+# This compares performance of creating and reading a NumPy array in different ways:
+# 1) memory
+# 2) disk
+# 3) disk with b2zip format
+
+import blosc2
+
+from time import time
+
+# Number of elements in array
+N = 2**27
+
+def b2_native(urlpath=None):
+    t0 = time()
+    a = blosc2.linspace(0., 1., N, urlpath=urlpath, mode="w")
+    # a = blosc2.linspace(0., 1., 2**27, cparams=blosc2.CParams(codec=blosc2.Codec.LZ4))
+    # a = blosc2.linspace(0., 1., 2**27, dparams=blosc2.DParams(nthreads=1))
+    t1 = time()
+    print(f"Time to create a linspace array: {t1 - t0:.2f}s, bandwidth: {a.nbytes / (t1 - t0) / 1e9:.2f} GB/s")
+    #print(a.info)
+
+    t0 = time()
+    b = a[:]
+    t1 = time()
+    print(f"Time to read the array: {t1 - t0:.2f}s, bandwidth: {b.nbytes / (t1 - t0) / 1e9:.2f} GB/s")
+
+def b2_b2zip(urlpath):
+    t0 = time()
+    with blosc2.TreeStore(localpath=urlpath, mode="w") as tstore:
+        a = blosc2.linspace(0., 1., N)
+        # a = blosc2.linspace(0., 1., 2**27, cparams=blosc2.CParams(codec=blosc2.Codec.LZ4))
+        tstore["/b"] = a
+    t1 = time()
+    print(f"Time to store a linspace array: {t1 - t0:.2f}s, bandwidth: {a.nbytes / (t1 - t0) / 1e9:.2f} GB/s")
+
+    t0 = time()
+    with blosc2.TreeStore(localpath=urlpath, mode="r") as tstore_read:
+        b = tstore_read["/b"][:]
+    t1 = time()
+    print(f"Time to read the array: {t1 - t0:.2f}s, bandwidth: {b.nbytes / (t1 - t0) / 1e9:.2f} GB/s")
+
+
+if __name__ == "__main__":
+    print("Blosc2 in-memory")
+    b2_native()
+    print("Blosc2 on disk")
+    b2_native("linspace.b2nd")
+    print("Blosc2 on disk with b2zip format")
+    b2_b2zip("my_tstore.b2z")