Sector Storage (worker) groups #7642

jennijuju · 2021-11-18T05:02:05Z

jennijuju
Nov 18, 2021
Maintainer

A new feature, creating sector storage/ worker groups is introduced in #7453.

Feature

Reference: PR description of #7453

🆕 Two new optional fields was added to sectorstore.json:

Groups []string - list of group names the storage path belongs to.
AllowTo []string - list of group names to which sectors can be fetched to from this storage path.

Those two should allow for creating worker groups, and avoiding unnecesarily moving data between multi-purpose workers. For example in the following setup:

Storage miner
- Path with Groups: ["example-storage-group-1"]
Worker 1 (PC1, PC2):
- Path with Groups: ["example-seal-group-1"], AllowTo: ["example-seal-group-1"]
Worker 2 (PC1, PC2):
- Path with Groups: ["example-seal-group-2"], AllowTo: ["example-seal-group-2"]
Worker 3 (PC1):
- Path with AllowTo: ["example-seal-group-1""]

Without storage groups, PC2 tasks on workers 1/2 could be scheduled with sector data from other workers, which would often waste bandwidth and occasionally block processing on fetching data.
With storage groups configured as above, sectors which had PC1 done on worker1 / worker2 will always execute PC2 on the same worker. Sectors from worker3 will only go to worker1 for PC2

🆕 Groups can be setup in two ways:

For new storage paths, set with lotus-miner storage attach --init --groups <["group-a", "group-b" ... ]>, --allow-to=<["group-a", "group-b" ... ]> or lotus-worker storage attach --init --groups <["group-a", "group-b" ... ]>, --allow-to <["group-a", "group-b" ... ]>

For existing storage paths, modify [path]/sectorstore.json, then restarting lotus-miner/worker

{
 "ID": "74e1d667-7bc9-49bc-a9a6-0c30afd8684c",
 "Weight": 10,
 "CanSeal": false,
 "CanStore": true,
 "MaxStorage": 0,
 "Groups": ["group-a", "group-b"]
  "AllowTo": ["storage0"]
}

Note: if --allow-to or AllowTo is not specified, all groups will be accessible.

Steps

Give this post a 🥇 if you are trying it out.
Checkout show-you-the-magik branch on your lotus-miner and lotus-worker.
Be creative but also practical with your worker groups based on your needs.
Restart your miner and workers!
Making deals, and seal deal/cc sectors. Confirm the job assignments to individual workers are restricted as you set up the groups!
Once it works, copy Ozzy's Storage System to share your group setups, sectorstore.json as examples for other community members!

⚠️ Note

While this feature has been tested internally, mostly by @magik6k, we are definitely missing more testing on different SP system set up. So please:
- leave a comment here(recommneded) or in #minerx3 channel if you have any questions.
- help us making better user documentation by sharing your setup and how you configured your sector storage paths with groups and allow-to!

jennijuju · 2021-11-18T05:03:59Z

jennijuju
Nov 18, 2021
Maintainer Author

Ozzy's Storage System

Storage Miner

Groups:
AllowTo:
sectorstore.json:

Worker 1

Jobs:
Groups:
AllowTo:
sectorstore.json:

Worker 2

Jobs:
Groups:
AllowTo:
sectorstore.json:

0 replies

neondragon · 2021-11-18T15:02:34Z

neondragon
Nov 18, 2021

Alex Fox's (NeonixAF) Storage System (64GiB)

Workers

storage-miner
Tasks:

ap-1
Tasks: AP

pc1-1
Tasks: PC1, UNS

pc1-2
Tasks: PC1, UNS

pc2-c2-1
Tasks: PC2, C2

pc2-c2-2
Tasks: PC2, C2

Storage Locations

Sector Storage
Groups: storage0
AllowTo: seal-64-2, seal-64-3
Workers: storage-miner, pc1-1, pc1-2, pc2-c2-1, pc2-c2-2

Seal Storage 1
Groups: seal-64-1
AllowTo: seal-64-2, seal-64-3
Workers: ap-1

Seal Storage 2
Groups: seal-64-2
AllowTo: storage0
Workers: pc1-1, pc2-c2-1, pc2-c2-2

Seal Storage 3
Groups: seal-64-3
AllowTo: storage0
Workers: pc1-2, pc2-c2-1, pc2-c2-2

0 replies

jennijuju · 2021-11-18T16:53:34Z

jennijuju
Nov 18, 2021
Maintainer Author

Documentation feedbacks

3 replies

jennijuju Nov 18, 2021
Maintainer Author

from @ f8-ptrk: maybe make clear that workers/the miner PULL stuff and do not push stuff. so AllowTo defines where from the sector can be pulled

jennijuju Nov 18, 2021
Maintainer Author

@neondragon I think we need to be clear in docs that storage paths are distinct from workers. Many people have one storage path per worker and so moving a task between workers always means moving data between storage paths. I have multiple workers that can see the same storage paths and so tasks could run on a different worker but on the same path. PC2 can run on any worker that has access to the storage path the PC1 is stored on, and it will successfully prevent an unwanted GET.

followup by @f8-ptrk:

maybe we shouldn't say
Worker 1 (PC1, PC2):
Path with Groups: ["example-seal-group-1"], AllowTo: ["example-seal-group-1"]
but
Worker 1 (PC1, PC2):
connected to/using/... storage Path1 with Groups: ["example-seal-group-1"], AllowTo: ["example-seal-group-1"]

kernelogic Nov 22, 2021

@jennijuju when you say "Many people have one storage path per worker", do you mean the LOTUS_WORKER_PATH or a sectorstore with CanStore: true?

I guess I am a little confused about how to set the "AllowTo" part. Should it points to a storage group or a seal group?

tmyuu · 2021-11-19T01:03:44Z

tmyuu
Nov 19, 2021

Livspect Storage System (32GiB)

Version:

Daemon:  1.13.1-rc2+mainnet+git.160323f15+api1.2.0
Local: lotus-miner version 1.13.1-rc2+mainnet+git.160323f15

Storage Miner

Groups: storage_1
AllowTo: storage_1
sectorstore.json:

{ 
  "ID": "",
  "Weight": 10,
  "CanSeal": false,
  "CanStore": true,
  "MaxStorage": 0,
  "Groups": ["storage_1"],
  "AllowTo": ["storage_1"]
}

Worker 1

Jobs: AP PC1 PC2 C1 C2
Groups: sealing_1
AllowTo: sealing_1
sectorstore.json:

{
  "ID": "",
  "Weight": 10,
  "CanSeal": true,
  "CanStore": false,
  "MaxStorage": 0,
  "Groups": ["sealing_1"],
  "AllowTo": ["sealing_1"]
}

Worker 2

Jobs: AP PC1 PC2 C1 C2
Groups: sealing_2
AllowTo: sealing_2
sectorstore.json:

{
  "ID": "",
  "Weight": 10,
  "CanSeal": true,
  "CanStore": false,
  "MaxStorage": 1099511627776,
  "Groups": ["sealing_2"],
  "AllowTo": ["sealing_2"]
}

1 reply

tmyuu Dec 3, 2021

Changed from underscore to hyphen.

Livspect Storage System (32GiB)

Version:

Daemon:  1.13.1-rc2+mainnet+git.160323f15+api1.2.0
Local: lotus-miner version 1.13.1-rc2+mainnet+git.160323f15

Storage Miner

Groups: storage_1
AllowTo: storage_1
sectorstore.json:

{ 
  "ID": "",
  "Weight": 10,
  "CanSeal": false,
  "CanStore": true,
  "MaxStorage": 0,
  "Groups": ["storage-1"],
  "AllowTo": ["storage-1"]
}

Worker 1

Jobs: AP PC1 PC2 C1 C2
Groups: sealing-1
AllowTo: sealing-1
sectorstore.json:

{
  "ID": "",
  "Weight": 10,
  "CanSeal": true,
  "CanStore": false,
  "MaxStorage": 0,
  "Groups": ["sealing-1"],
  "AllowTo": ["sealing-1"]
}

Worker 2

Jobs: AP PC1 PC2 C1 C2
Groups: sealing-2
AllowTo: sealing-2
sectorstore.json:

{
  "ID": "",
  "Weight": 10,
  "CanSeal": true,
  "CanStore": false,
  "MaxStorage": 1099511627776,
  "Groups": ["sealing-2"],
  "AllowTo": ["sealing-2"]
}

jennijuju · 2021-12-01T00:26:43Z

jennijuju
Dec 1, 2021
Maintainer Author

This is now in v1.13.2-rc1!

6 replies

tmyuu Dec 3, 2021

Using a hyphen could work it correctly.

hcm609 Dec 3, 2021

Hi again.
I have made a new test and used simple names like seal1 and seal2.
They still move files between each other.
This is not fixed yet.

tmyuu Dec 3, 2021

You mean did one worker move the cache file to the other after the PC1 job?

hcm609 Dec 3, 2021

exactly.

tmyuu Dec 3, 2021

Do you have a log like the following?

move sector data {"from": "~/cache/fetching/s-<minerId>-<sectorNum>", "to": "~/cache/fetching/s-<minerId>-<sectorNum>"}

Or have you tried rebooting miner and workers after the change?

hcm609 · 2021-12-03T16:26:33Z

hcm609
Dec 3, 2021

Sure. Here are some examples;
2021-12-03T12:34:02.971Z INFO stores stores/remote.go:374 Delete http://192.168.10.12:10012/remote/unsealed/s-t0141XXXX-1422
2021-12-03T12:34:04.422Z INFO stores stores/http_handler.go:87 SERVE GET /remote/sealed/s-t0141XXXX-1422
2021-12-03T12:38:42.326Z INFO stores stores/http_handler.go:159 SERVE DELETE /remote/sealed/s-t0141XXXX-1422?keep=dcb954f0-1111-1111-9709-782e5c7286ed
2021-12-03T12:38:42.327Z INFO stores stores/local.go:639 remove /home2/sealed/s-t0141XXXX-1422
2021-12-03T12:38:44.886Z INFO stores stores/http_handler.go:87 SERVE GET /remote/cache/s-t0141XXXX-1422
2021-12-03T12:38:45.022Z INFO stores stores/http_handler.go:159 SERVE DELETE /remote/cache/s-t0141XXXX-1422?keep=dcb954f0-1111-1111-9709-782e5c7286ed
2021-12-03T12:38:45.023Z INFO stores stores/local.go:639 remove /home2/cache/s-t0141XXXX-1422

== and ==

2021-12-03T15:59:23.372Z INFO stores stores/local.go:639 remove /home6/unsealed/s-t01410000-1133
2021-12-03T15:59:23.466Z INFO stores stores/remote.go:374 Delete http://192.168.168.15:3456/remote/unsealed/s-t01410000-1133
2021-12-03T15:59:27.338Z INFO stores stores/http_handler.go:87 SERVE GET /remote/sealed/s-t01410000-1133
2021-12-03T16:04:05.025Z INFO stores stores/http_handler.go:159 SERVE DELETE /remote/sealed/s-t01410000-1133?keep=dcb954f0-2222-2222-9709-782e5c7286ed
2021-12-03T16:04:05.047Z INFO stores stores/local.go:639 remove /home6/sealed/s-t01410000-1133
2021-12-03T16:04:07.187Z INFO stores stores/http_handler.go:87 SERVE GET /remote/cache/s-t01410000-1133
2021-12-03T16:04:07.601Z INFO stores stores/http_handler.go:159 SERVE DELETE /remote/cache/s-t01410000-1133?keep=dcb954f0-2222-2222-9709-782e5c7286ed
2021-12-03T16:04:07.624Z INFO stores stores/local.go:639 remove /home6/cache/s-t01410000-1133

1 reply

tmyuu Dec 3, 2021

Please check your storage ID.
I think it doesn't correspond to your worker's ID.

"ID": "1762559e-8000-4ddd-a000-6f4b6b000000"
"ID": "87777d9e-8000-4ddd-a000-6f1987000000"

The transfer between workers is such a log.

move sector data {"from": "~/cache/fetching/s-<minerId>-<sectorNum>", "to": "~/cache/fetching/s-<minerId>-<sectorNum>"}

Not a SERVE GET.

Sector Storage (worker) groups #7642

Uh oh!

Uh oh!

jennijuju Nov 18, 2021 Maintainer

Feature

Steps

⚠️ Note

Replies: 6 comments · 11 replies

Uh oh!

Uh oh!

jennijuju Nov 18, 2021 Maintainer Author

Ozzy's Storage System

Storage Miner

Worker 1

Worker 2

Uh oh!

Alex Fox's (NeonixAF) Storage System (64GiB)

Workers

Storage Locations

Uh oh!

jennijuju Nov 18, 2021 Maintainer Author

Documentation feedbacks

Uh oh!

jennijuju Nov 18, 2021 Maintainer Author

Uh oh!

jennijuju Nov 18, 2021 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

Livspect Storage System (32GiB)

Storage Miner

Worker 1

Worker 2

Uh oh!

Livspect Storage System (32GiB)

Storage Miner

Worker 1

Worker 2

Uh oh!

jennijuju Dec 1, 2021 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jennijuju
Nov 18, 2021
Maintainer

Replies: 6 comments 11 replies

jennijuju
Nov 18, 2021
Maintainer Author

jennijuju
Nov 18, 2021
Maintainer Author

jennijuju Nov 18, 2021
Maintainer Author

jennijuju Nov 18, 2021
Maintainer Author

jennijuju
Dec 1, 2021
Maintainer Author