You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Memory-mapped files allow accessing small segments of large disks stored on disk, without reading the entire file into memory. Not only can this be advantageous for memory performance, but it also facilitates shared memory between processes (e.g., operating on the same array in both Node.js and Python running in two separate processes).
1701
+
1702
+
The goal of this project is to add support for working with typed arrays backed by memory-mapped files. Memory-mapped-backed typed arrays should support all the APIs of built-in typed arrays, with the exceptions that the constructors will need to support `mmap`-related arguments (e.g., filename, mode, offset) and indexing will require accessors, not square bracket syntax. The project is well-prepared to support accessors (see `array/bool`, `array/complex128`, etc), such that, provided a memory-mapped typed array supports the accessor protocol, passing to downstream utilities should just work.
1703
+
1704
+
Similar to how we've approached fixed-endian typed arrays (see [`array/fixed-endian-factory`](https://github.com/stdlib-js/stdlib/tree/develop/lib/node_modules/%40stdlib/array/fixed-endian-factory)), we can likely create a package exposing a constructor factory and then create lightweight wrappers for type-specific constructors (E.g., [`array/little-endian-float64`](https://github.com/stdlib-js/stdlib/tree/develop/lib/node_modules/%40stdlib/array/little-endian-float64).
1705
+
1706
+
This project may require figuring out a strategy for C-JS iterop which can be used across constructors.
1707
+
1708
+
### Expected outcomes
1709
+
1710
+
Ideally, we would have the following constructors:
1711
+
1712
+
-`Float64ArrayMMap`
1713
+
-`Float32ArrayMMap`
1714
+
-`Int32ArrayMMap`
1715
+
-`Int16ArrayMMap`
1716
+
-`Int8ArrayMMap`
1717
+
-`Uint32ArrayMMap`
1718
+
-`Uint16ArrayMMap`
1719
+
-`Uint8ArrayMMap`
1720
+
-`Uint8ClampedArrayMMap`
1721
+
-`BooleanArrayMMap`
1722
+
-`Complex128ArrayMMap`
1723
+
-`Complex64ArrayMMap`
1724
+
1725
+
Additionally, the following constructors would also be useful:
1726
+
1727
+
-`DataViewMMap`
1728
+
1729
+
### Status
1730
+
1731
+
None.
1732
+
1733
+
### Involved software
1734
+
1735
+
C compiler such as GCC or Clang.
1736
+
1737
+
### Technology
1738
+
1739
+
C, JavaScript, nodejs, native addons
1740
+
1741
+
### Other technology
1742
+
1743
+
None
1744
+
1745
+
### Difficulty
1746
+
1747
+
5
1748
+
1749
+
### Difficulty justification
1750
+
1751
+
Figuring out an effective bridge between JavaScript and C for working with memory-mapped files will likely require some R&D. It is not clear whether we'd need to first develop separate dedicated `mmap(2)`-like functionality in JavaScript or whether we can directly interface into C. Once the lower-level details are determined, the next steps will be implementing all the user-facing APIs expected from typed arrays. This should be straightforward; however, there may be some unexpected challenges and constraints surrounding read-only access, etc.
1752
+
1753
+
### Prerequisite knowledge
1754
+
1755
+
C, JavaScript, and Node.js experience will be useful.
0 commit comments