|
| 1 | +module System.FS.BlockIO ( |
| 2 | + -- * Description |
| 3 | + -- $description |
| 4 | + |
| 5 | + -- * Re-exports |
| 6 | + module System.FS.BlockIO.API |
| 7 | + , module System.FS.BlockIO.IO |
| 8 | + |
| 9 | + -- * Example |
| 10 | + -- $example |
| 11 | +) where |
| 12 | + |
| 13 | +import System.FS.BlockIO.API |
| 14 | +import System.FS.BlockIO.IO |
| 15 | + |
| 16 | +{------------------------------------------------------------------------------- |
| 17 | + Examples |
| 18 | +-------------------------------------------------------------------------------} |
| 19 | + |
| 20 | +{- $description |
| 21 | +
|
| 22 | + The 'HasBlockIO' record type defines an /abstract interface/. A value of a |
| 23 | + 'HasBlockIO' type is what we call an /instance/ of the abstract interface, and |
| 24 | + an instance is produced by a function that we call an /implementation/. In |
| 25 | + principle, we can have multiple instances of the same implementation. |
| 26 | +
|
| 27 | + There are currently two known implementations of the interface: |
| 28 | +
|
| 29 | + * An implementation using the real file system, which can be found in the |
| 30 | + "System.FS.BlockIO.IO" module. This implementation is platform-dependent, |
| 31 | + but has largely similar observable behaviour. |
| 32 | +
|
| 33 | + * An implementation using a simulated file system, which can be found in the |
| 34 | + @System.FS.BlockIO.Sim@ module of the @blockio:sim@ sublibrary. This |
| 35 | + implementation is uniform across platforms. |
| 36 | +
|
| 37 | + The 'HasBlockIO' abstract interface is an extension of the 'HasFS' abstract |
| 38 | + interface that is provided by the |
| 39 | + [@fs-api@](https://hackage.haskell.org/package/fs-api) package. Whereas |
| 40 | + 'HasFS' defines many primitive functions, for example for opening a file, the |
| 41 | + main feature of 'HasBlockIO' is to define a function for performing batched |
| 42 | + I\/O. As such, users of @blockio@ will more often than not need to pass both a |
| 43 | + 'HasFS' and a 'HasBlockIO' instance to their functions. |
| 44 | +-} |
| 45 | + |
| 46 | +{- $example |
| 47 | +
|
| 48 | + >>> import Control.Monad |
| 49 | + >>> import Control.Monad.Primitive |
| 50 | + >>> import Data.Primitive.ByteArray |
| 51 | + >>> import Data.Vector qualified as V |
| 52 | + >>> import Data.Word |
| 53 | + >>> import Debug.Trace |
| 54 | + >>> import System.FS.API as FS |
| 55 | + >>> import System.FS.BlockIO.IO |
| 56 | + >>> import System.FS.BlockIO.API |
| 57 | + >>> import System.FS.IO |
| 58 | +
|
| 59 | + The main feature of the 'HasBlockIO' abstract interface is that it provides a |
| 60 | + function for performing batched I\/O using 'submitIO'. Depending on the |
| 61 | + implementation of the interface, performing I\/O in batches concurrently using |
| 62 | + 'submitIO' can be much faster than performing each I\/O operation in a |
| 63 | + sequential order. We will not go into detail about this performance |
| 64 | + consideration here, but more information can be found in the |
| 65 | + "System.FS.BlockIO.IO" module. Instead, we will show an example of how |
| 66 | + 'submitIO' can be used in your own projects. |
| 67 | +
|
| 68 | + We aim to build an example that writes some contents to a file using |
| 69 | + 'submitIO', and then reads the contents out again using 'submitIO'. The file |
| 70 | + contents will simply be bytes. |
| 71 | +
|
| 72 | + >>> type Byte = Word8 |
| 73 | +
|
| 74 | + The first part of the example is to write out bytes to a given file path using |
| 75 | + 'submitIO'. We define a @writeFile@ function that does just that. The file is |
| 76 | + assumed to not exist already. |
| 77 | +
|
| 78 | + The bytes, which are provided as separate bytes, are written into a buffer (a |
| 79 | + mutable byte array). Note that the buffer should be /pinned/ memory to prevent |
| 80 | + pointer aliasing. In the case of write operations, this buffer is used to |
| 81 | + communicate to the backend what the bytes are that should be written to disk. |
| 82 | + For simplicity, we create a separate 'IOOpWrite' instruction for each byte. |
| 83 | + This instruction requires information about the write operation. In order of |
| 84 | + appearence these are: the file handle to write bytes to, the offset into that |
| 85 | + file, the buffer, the offset into that buffer, and the number of bytes to |
| 86 | + write. Finally, all instructions are batched together and submitted in one go |
| 87 | + using 'submitIO'. For each instruction, an 'IOResult' is returned, which |
| 88 | + describes in this case the number of written bytes. If any of the instructions |
| 89 | + failed to be performed, an error is thrown. We print the 'IOResult's to |
| 90 | + standard output. |
| 91 | +
|
| 92 | + Note that in real scenarios it would be much more performant to aggregate the |
| 93 | + bytes into larger chunks, and to create an instruction for each of those |
| 94 | + chunks. A sensible size for those chunks would be the disk page size (4Kb for |
| 95 | + example), or a multiple of that disk page size. The disk page size is |
| 96 | + typically the smallest chunk of memory that can be written to or read from the |
| 97 | + disk. In some cases it is also desirable or even required that the buffers are |
| 98 | + aligned to the disk page size. For example, alignment is required when using |
| 99 | + direct I\/O. |
| 100 | +
|
| 101 | + >>> :{ |
| 102 | + writeFile :: |
| 103 | + HasFS IO HandleIO |
| 104 | + -> HasBlockIO IO HandleIO |
| 105 | + -> FsPath |
| 106 | + -> [Byte] |
| 107 | + -> IO () |
| 108 | + writeFile hasFS hasBlockIO file bytes = do |
| 109 | + let numBytes = length bytes |
| 110 | + FS.withFile hasFS file (FS.WriteMode FS.MustBeNew) $ \h -> do |
| 111 | + buffer <- newPinnedByteArray numBytes |
| 112 | + forM_ (zip [0..] bytes) $ \(i, byte) -> |
| 113 | + let bufferOffset = fromIntegral i |
| 114 | + in writeByteArray @Byte buffer bufferOffset byte |
| 115 | + results <- submitIO hasBlockIO $ V.fromList [ |
| 116 | + IOOpWrite h fileOffset buffer bufferOffset 1 |
| 117 | + | i <- take numBytes [0..] |
| 118 | + , let fileOffset = fromIntegral i |
| 119 | + bufferOffset = FS.BufferOffset i |
| 120 | + ] |
| 121 | + print results |
| 122 | + :} |
| 123 | +
|
| 124 | + The second part of the example is to read a given number of bytes from a given |
| 125 | + file path using 'submitIO'. We define a @readFile@ function that follows the |
| 126 | + same general structure and behaviour as @writeFile@, but @readFile@ is of |
| 127 | + course reading bytes instead of writing bytes. |
| 128 | +
|
| 129 | + >>> :{ |
| 130 | + readFile :: |
| 131 | + HasFS IO HandleIO |
| 132 | + -> HasBlockIO IO HandleIO |
| 133 | + -> FsPath |
| 134 | + -> Int |
| 135 | + -> IO [Byte] |
| 136 | + readFile hasFS hasBlockIO file numBytes = do |
| 137 | + FS.withFile hasFS file FS.ReadMode $ \h -> do |
| 138 | + buffer <- newPinnedByteArray numBytes |
| 139 | + results <- submitIO hasBlockIO $ V.fromList [ |
| 140 | + IOOpRead h fileOffset buffer bufferOffset numBytes |
| 141 | + | i <- [0..3] |
| 142 | + , let fileOffset = fromIntegral i |
| 143 | + bufferOffset = FS.BufferOffset i |
| 144 | + numBytes = 1 |
| 145 | + ] |
| 146 | + print results |
| 147 | + forM (take numBytes [0..]) $ \i -> |
| 148 | + let bufferOffset = i |
| 149 | + in readByteArray @Byte buffer i |
| 150 | + :} |
| 151 | +
|
| 152 | + Now we can combine @writeFile@ and @readFile@ into a very small example called |
| 153 | + @writeReadFile@, which does what we set out to do: write a few bytes to a |
| 154 | + (temporary) file and read them out again using 'submitIO'. We also print the |
| 155 | + bytes that were written and the bytes that were read, so that the user can |
| 156 | + check by hand whether the bytes match. |
| 157 | +
|
| 158 | + >>> :{ |
| 159 | + writeReadFile :: HasFS IO HandleIO -> HasBlockIO IO HandleIO -> IO () |
| 160 | + writeReadFile hasFS hasBlockIO = do |
| 161 | + let file = FS.mkFsPath ["simple_example.txt"] |
| 162 | + let bytesIn = [1,2,3,4] |
| 163 | + print bytesIn |
| 164 | + writeFile hasFS hasBlockIO file bytesIn |
| 165 | + bytesOut <- readFile hasFS hasBlockIO file 4 |
| 166 | + print bytesOut |
| 167 | + FS.removeFile hasFS file |
| 168 | + :} |
| 169 | +
|
| 170 | + In order to run @writeReadFile@, we will need 'HasFS' and 'HasBlockIO' |
| 171 | + instances. This is where the separation between interface and implementation |
| 172 | + shines: @writeReadFile@ is agnostic to the implementations of the the abstract |
| 173 | + interfaces, so we could pick any implementations and slot them in. For this |
| 174 | + example we will use the /real/ implementation from "System.FS.BlockIO.IO", but |
| 175 | + we could have used the /simulated/ implementation from the @blockio:sim@ |
| 176 | + sub-library just as well. We define the @example@ function, which uses |
| 177 | + 'withIOHasBlockIO' to instantiate both a 'HasFS' and 'HasBlockIO' instance, |
| 178 | + which we pass to 'writeReadFile'. |
| 179 | +
|
| 180 | + >>> :{ |
| 181 | + example :: IO () |
| 182 | + example = |
| 183 | + withIOHasBlockIO (MountPoint "") defaultIOCtxParams $ \hasFS hasBlockIO -> |
| 184 | + writeReadFile hasFS hasBlockIO |
| 185 | + :} |
| 186 | +
|
| 187 | + Finally, we can run the example to produce some output. As we can see, the |
| 188 | + input bytes match the output bytes. |
| 189 | +
|
| 190 | + >>> example |
| 191 | + [1,2,3,4] |
| 192 | + [IOResult 1,IOResult 1,IOResult 1,IOResult 1] |
| 193 | + [IOResult 1,IOResult 1,IOResult 1,IOResult 1] |
| 194 | + [1,2,3,4] |
| 195 | +-} |
0 commit comments