Skip to content

Commit ba834f3

Browse files
authored
use nbytes instead of length in the index description
1 parent c232b05 commit ba834f3

File tree

1 file changed

+9
-9
lines changed
  • docs/storage_transformers/sharding

1 file changed

+9
-9
lines changed

docs/storage_transformers/sharding/v1.0.rst

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -143,20 +143,20 @@ configuration key. Other binary formats might be added in future versions.
143143

144144
In the indexed binary format chunks are written successively in a shard, where
145145
unused space between them is allowed, followed by an index referencing them.
146-
The index is placed at the end of the file and has a length of 16 bytes multiplied by the number of chunks
146+
The index is placed at the end of the file and has a size of 16 bytes multiplied by the number of chunks
147147
in a shard, for example ``16 bytes * 64 = 1014 bytes`` for ``chunks_per_shard=[32, 2]``.
148-
The index holds an `offset, length` pair of little-endian uint64 per chunk,
148+
The index holds an `offset, nbytes` pair of little-endian uint64 per chunk,
149149
the chunks-order in the index is row-major (C) order, for example for
150150
``chunks_per_shard=[2, 2]`` an index would look like:
151151

152152
.. code-block::
153153
154154
| chunk (0, 0) | chunk (0, 1) | chunk (1, 0) | chunk (1, 1) |
155-
| offset | length | offset | length | offset | length | offset | length |
155+
| offset | nbytes | offset | nbytes | offset | nbytes | offset | nbytes |
156156
| uint64 | uint64 | uint64 | uint64 | uint64 | uint64 | uint64 | uint64 |
157157
158158
159-
Empty chunks are denoted by setting both offset and length to ``2^64 - 1``.
159+
Empty chunks are denoted by setting both offset and nbytes to ``2^64 - 1``.
160160
The index always has the full shape of all possible chunks per shard,
161161
even if they are outside of the array size.
162162

@@ -191,7 +191,7 @@ relayed to the underlying store and not described explicitly.
191191
* ``get_partial_values(key_ranges) -> values``:
192192
For each referenced key, request the indices from the underlying store using
193193
``get_partial_values``. For each `key`, `range` pair in in `key_ranges`,
194-
check if the chunk exists by checking if the index offset and length
194+
check if the chunk exists by checking if the index offset and nbytes
195195
are both ``2^64 - 1``. For existing keys, request the actual chunks by
196196
their ranges as read from the index using ``get_partial_values``.
197197
This operation should be implemented using two ``get_partial_values``
@@ -204,7 +204,7 @@ relayed to the underlying store and not described explicitly.
204204
`Binary shard format`_ directly.
205205
For all other keys, request the indices from the underlying store using
206206
``get_partial_values``. All chunks that are not updated completely and
207-
exist according to the index (index offset and length are both
207+
exist according to the index (index offset and nbytes are both
208208
``2^64 - 1``) need to be read via ``get_partial_values`` from the
209209
underlying store. For simplification purposes a shard may also be read
210210
completely, combining the previous two `get` operations into one.
@@ -222,7 +222,7 @@ relayed to the underlying store and not described explicitly.
222222
on the underlying store.
223223
For all other keys, request the indices from the underlying
224224
store using ``get_partial_values``. Update the index using and offset and
225-
length of ``2^64 - 1`` to mark missing chunks. The updated index may be
225+
nbytes of ``2^64 - 1`` to mark missing chunks. The updated index may be
226226
be written in-place using ``set_partial_values(key_start_values)``,
227227
or a larger rewrite of the shard may be done including the index update,
228228
but also removing value ranges corresponding to the erased chunks.
@@ -243,7 +243,7 @@ relayed to the underlying store and not described explicitly.
243243
Then, ``list_prefix`` is called on the underlying store with the translated
244244
prefix. For all listed shards request the indices from the underlying store
245245
using ``get_partial_values``. Existing chunks, where the index offset or
246-
length are not ``2^64 - 1`` are then listed by their original key.
246+
nbytes are not ``2^64 - 1`` are then listed by their original key.
247247

248248
* ``list_dir(prefix)`` : If the prefix contains a part of the chunk-grid
249249
key, this part is translated to the referenced shard and contained chunks.
@@ -252,7 +252,7 @@ relayed to the underlying store and not described explicitly.
252252
the corresponding original prefixes covering all possible chunks in the shard
253253
are listed. For *retrieved full keys* the the indices from the underlying store
254254
are requested using ``get_partial_values``. Existing chunks, where the index
255-
offset or length are not ``2^64 - 1`` are then listed by their original key.
255+
offset or nbytes are not ``2^64 - 1`` are then listed by their original key.
256256

257257
.. note::
258258
Not all listed prefixes must necessarily contain keys, as shard prefixes with

0 commit comments

Comments
 (0)