WIP: Complete reimplementation of getindex and setindex (#146)

* Move _DiskArray out of tests

* First commit of bikeshedding everything

* Start new indexing scheme

* More tests passing

* Tests pass for first time

* setindex tests pass as well

* delete most of batchgetindex

* start work on batch optimization

* cont

* batch reading for vector is done in slices

* All tests passing for chunkwise access

* Fix merge errors

* Remove Manifest

* more batch options

* range-based approach

* clean-up work

* Improve Readme

* some cleanup

* remove include

* Fix some type instabilities

* Fix some type instabilities and test them

* Fix a special case

* Clean up and fix vec

* Update README.md

Co-authored-by: Felix Cremer <[email protected]>

* SMall readme fix

---------

Co-authored-by: Felix Cremer <[email protected]>

Author: meggart

.github/workflows/ci.yml

@@ -15,7 +15,6 @@ jobs:
       fail-fast: false
       matrix:
         version:
-          - '1.6' # Replace this with the minimum Julia version that your package supports. E.g. if your package requires Julia 1.5 or higher, change this to '1.5'.
           - '1' # Leave this line unchanged. '1' will automatically expand to the latest stable 1.x release of Julia.
           - 'nightly'
         os:

.gitignore

@@ -3,3 +3,4 @@
 *.jl.mem
 /deps/deps.jl
 /docs/build
+Manifest.toml

Project.toml

@@ -1,7 +1,7 @@
 name = "DiskArrays"
 uuid = "3c3547ce-8d99-4f5e-a174-61eb10b00ae3"
 authors = ["Fabian Gans <[email protected]>"]
-version = "0.3.23"
+version = "0.4.0"
 
 [deps]
 LRUCache = "8ac3fa9e-de4c-5943-b1dc-09c6b5f20637"
@@ -10,10 +10,10 @@ OffsetArrays = "6fe1bfb0-de20-5000-8ca7-80f57d26f881"
 [compat]
 Aqua = "0.8"
 LRUCache = "1"
-OffsetArrays = "1"
-Statistics = "1.6"
-Test = "1.6"
 julia = "1.6"
+OffsetArrays = "1"
+Statistics = "1.9"
+Test = "1.9"
 
 [extras]
 Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"

README.md

@@ -6,12 +6,13 @@
 [![CI](https://github.com/meggart/DiskArrays.jl/actions/workflows/ci.yml/badge.svg)](https://github.com/meggart/DiskArrays.jl/actions/workflows/ci.yml)
 [![Codecov](https://codecov.io/gh/meggart/DiskArrays.jl/branch/main/graph/badge.svg)](https://codecov.io/gh/meggart/DiskArrays.jl/tree/main)
 
-This package is an attempt to collect utilities for working with n-dimensional array-like data
-structures that do not have considerable overhead for single read operations. Most important
-examples are arrays that represent data on hard disk that are accessed through a C
-library or that are compressed in chunks. It can be inadvisable to make these arrays a subtype
-of `AbstractArray` many functions working with AbstractArrays assume fast random access into single
-values (including basic things like `getindex`, `show`, `reduce`, etc...). Currently supported features are:
+This package provides a collection of utilities for working with n-dimensional array-like data
+structures that do have considerable overhead for single read operations. 
+Most important examples are arrays that represent data on hard disk that are accessed through a C
+library or that are compressed in chunks. 
+It can be inadvisable to make these arrays a direct subtype of `AbstractArray` many functions working with AbstractArrays assume fast random access into single values (including basic things like `getindex`, `show`, `reduce`, etc...). 
+
+Currently supported features are:
 
   - `getindex`/`setindex` with the same rules as base (trailing or singleton dimensions etc)
   - views into `DiskArrays`
@@ -23,13 +24,44 @@ values (including basic things like `getindex`, `show`, `reduce`, etc...). Curre
   - customization of `broadcast` when there is a `DiskArray` on the LHS. This at least makes things
   like `a.=5` possible and relatively fast
 
-There are basically two ways to use this package.
-Either one makes the abstraction directly a subtype of `AbstractDiskArray` which requires
-to implement a single `readblock!` method that reads a Cartesian range of data points.
-The remaining `getindex` methods will 
-come for free then. The second way is to use
-the `interpret_indices_disk` function to get a translation of the user-supplied indices
-into a set of ranges and then use these to read the data from disk.
+
+## AbstractDiskArray Interface definition
+
+Package authors who want to use this library to make their disk-based array an `AbstractDiskArray` should at least
+implement methods for the following functions:
+
+````julia
+Base.size(A::CustomDiskArray)
+readblock!(A::CustomDiskArray{T,N},aout,r::Vargarg{AbstractUnitRange,N})
+writeblock!(A::CustomDiskArray{T,N},ain,r::Vargarg{AbstractUnitRange,N})
+```` 
+
+Here `readblock!` will read a subset of array `A` in a hyper-rectangle defined by the unit ranges `r`. The results shall be written into `aout`. `writeblock!` should write the data given by `ain` into the (hyper-)rectangle of A defined by `r`
+When defining the functions it can be safely assumed that `length(r) == ndims(A)` as well as `size(ain) == length.(r)`.
+However, bounds checking is *not* performed by the DiskArray machinery and currently should be done by the implementation. 
+
+If the data on disk has rectangular chunks as underlying storage units, you should addtionally implement the following
+methods to optimize some operations like broadcast, reductions and sparse indexing:
+
+````julia
+DiskArrays.haschunks(A::CustomDiskArray) = DiskArrays.Chunked()
+DiskArrays.eachchunk(A::CustomDiskArray) = DiskArrays.GridChunks(A, chunksize)
+````
+
+where `chunksize` is a int-tuple of chunk lengths. If the array does not have an internal chunking structure, one should
+define
+
+````julia
+DiskArrays.haschunks(A::CustomDiskArray) = DiskArrays.Unchunked()
+````
+
+Implementing only these methods makes all kinds of strange indexing patterns work (Colons, StepRanges, Integer vectors,
+Boolean masks, CartesianIndices, Arrays of CartesianIndex, and mixtures of all these) while making sure that as few
+`readblock!` or `writeblock!` calls as possible are performed by reading a rectangular bounding box of the required
+array values and re-arranging the resulting values into the output array. 
+
+In addition, DiskArrays.jl provides a few optimizations for sparse indexing patterns to avoid reading and discarding 
+too much unnecessary data from disk, for example for indices like `A[:,:,[1,1500]]`. 
 
 # Example
 

src/DiskArrays.jl

@@ -9,7 +9,7 @@ using LRUCache: LRUCache, LRU
     read(path, String)
 end DiskArrays
 
-export AbstractDiskArray, interpret_indices_disk, eachchunk, ChunkIndex, ChunkIndices
+export AbstractDiskArray, eachchunk, ChunkIndex, ChunkIndices
 
 include("scalar.jl")
 include("diskarray.jl")
@@ -42,7 +42,6 @@ macro implement_diskarray(t)
         @implement_array_methods $t
         @implement_permutedims $t
         @implement_subarray $t
-        @implement_batchgetindex $t
         @implement_cat $t
         @implement_zip $t
         @implement_show $t
@@ -60,12 +59,12 @@ end
 @implement_array_methods AbstractDiskArray
 @implement_permutedims AbstractDiskArray
 @implement_subarray AbstractDiskArray
-@implement_batchgetindex AbstractDiskArray
 @implement_cat AbstractDiskArray
 @implement_generator AbstractDiskArray
 @implement_show AbstractDiskArray
 
 #And we define the test types
 include("util/testtypes.jl")
 
+
 end # module