Simple Fragmented Plot Files

[adraffy]

I would like to propose the idea of fragmented plots to utilize the unused (0, 108GiB) of free space with a platform-independent solution.

I believe the only necessary modification is making DiskProver support this new meta format.

---

Splitting / Joining

This can be accomplished by a very simple CLI program. (I can easily supply the code for this utility.)

• You take a plot (eg. A.plot 1000 bytes) and split it into A1 (600 bytes) and A2 (400 bytes):
  eg. A.plot 600 => A1 (first 600 bytes) A2 (remaining bytes)
• There could be more than one split:
  eg. A.plot 400 400 => A1 (first 400 bytes) A2 (next 400 bytes) A3 (remaining bytes)

This splitter program would also generate a new plot meta file (preferably clear-text so it's easy to edit the paths to the corresponding fragments.)

• eg. { files: ["/mnt/disk1/A1", "/mnt/disk2/A2"], sizes: [600]}
• eg. { files: ["/mnt/disk1/A1", "/mnt/disk2/A2", "/mnt/disk3/A3"], sizes: [400, 400]}
• Various other formats could work, eg. <path1>\n<len>\n<path2>...

This program could also:

• reconstruct a plot from a meta file, by joining the fragments together
• validate that the fragments of a meta file are readable and sized correctly
• move the fragments (if provided destinations) and delete the source plot when splitting (and vice-versa for joining)

Note: the paths would be any reachable file that normally can be accessed by chiapos, so you could have one fragment on a local disk and another on a samba share, or whatever.

The fancier version of this program would:

• estimate how much free space you have across your farm, tell you how many extra K32's you should make
• given a directory of plots, automatically split them across your farm (creating the appropriate meta and fragment files)
• undo this process (reassemble all your fragmented plots to standard plots)

---

File Extensions

I've purposely dropped the file extensions above as there's a bunch of options possible depending on how many code modifications are required.

The meta plot file could retain the .plot extension and just become .meta.plot, so chia-blockchain/chia/plotting/util.py doesn't need modified, or new extension could be used, eg. .plot.meta (requiring modification.)

The fragmented plot files should get a new extension, so they are not considered as plots, eg. .plot.frag$i.

---

DiskProver

The only real modification necessary is to DiskProver in chiapos/src/prover_disk.hpp.

• Already, the IO is abstracted to Safe{Read|Seek}, so the only necessary change is abstracting over ifstream, such that (depending on the choice of extension above), if the file is a normal plot, it's simply a single fragment, and if the file is a meta plot, then read in the fragment files and corresponding sizes.
• Instead of ifstream, you'd have a wrapper struct, that when provided to Safe{Read|Seek} react accordingly.
• SafeRead would just need extra logic when it reads across a boundary.
• To map from offset =>[fragment_file, fragment_offset], since N is typically 2 or 3, you can just linearly search through the corresponding offsets computed from FoldList[Plus, 0, sizes].

---

I'd probably also change chia-blockchain/chia/plotting/manager.py so stat_info = file_path.stat() becomes prover.get_file_size() which maps to DiskProver::GetLength() which stats the fragments.

[adraffy]

Here is a working patch for chiapos/src/disk_prover.hpp:
https://gist.github.com/adraffy/109837ffc30bee509b68d70613914050

This allows maximal utilization of your storage. For example, my farm has 25TB of fragmented space.

I chose the extension .meta.plot. The format is simply a text file with paths to the corresponding fragments. I decided to compute the sizes dynamically so they don't need to be hard-coded.

I took a K=32 plot, split it in half, and added .1 and .2 to their file names. I made a file Test.meta.plot which has 2 lines corresponding to the paths to those fragments. I ran ./ProofOfSpace check -f Test.meta.plot and got the same results as the original plot.

I guess I will submit a PR this weekend.

---

I also discovered a bug with the parallel_plots logic. The current code is opening the plot file 2x as many times as necessary. When spawning the left-right tree in GetInputs, one of the branches can reuse the previously opened file.

[cheezburga]

Great minds think alike! I started a similar discussion at #7829 . Having fragmented plot files really simplify the use of fragmented hdd spaces. Many people use RAID0 on multiple HDD but I don't like the idea of having to replot if one of the drives fail.

[adraffy]

Sorry, I haven’t had time to submit this as a PR. Yeah, I looked at the general file-system level solutions and they all sucked and seem fragile, whereas my implementation above works simply if the paths in the meta.plot are reachable, regardless of their underlying systems.