LVM: join different regions into single volume group

[pohly]

It may be useful to combine all PMEM in a system into a single volume for apps which need one very large memory-mapped address range or managing more than one volume is too cumbersome.

This cannot be done across CPU sockets by ipmctl (https://docs.pmem.io/ipmctl-user-guide/provisioning/concepts, "Regions cannot be created across CPU sockets."). It can be done by LVM, which is how PMEM-CSI could support this. The downside is that region affinity gets lost.

[pohly]

I can see two ways of solving this (not mutually exclusive, both make sense depending on usage patterns):

• add a parameter that influences how PMEM-CSI sets up volume group(s) when it starts on a node: one volume group per region (current behavior, default) vs. merge into one volume group
• during node provisioning, set up volume group(s) and then let PMEM-CSI manage those which have a certain configurable tag

The first option may be simpler to use (just create regions) while the second may also be useful for other use cases (managing non-PMEM storage).

[tigerhu2008]

Hi Patrick, one customer is asking us whether PMem-CSI can support cross-region LVM, because they want a bigger volume. other customer may have the same requirement. I think this is a useful feature.

[pohly]

@tigerhu2008: which of the two proposed solutions will be easier to use for your customer?

[tigerhu2008]

Will check with customer.

I have a question about implementation, saying there are two CPUs, each has 1TB PMem, how will the resource be reported to resource manager? only 2TB, or 2 * 1TB, or both? When asks for a >1TB PMem, scheduler needs to know the available PMem capacity.

[pohly]

Once the two 1TB regions are joined via LVM into one volume group (regardless how that is set up), PMEM-CSI will manage that volume group, so the reported size will be 2TB.

Another, more complicated alternative would be to manage individual physical volumes (one per region), and create one volume group per Kubernetes PVC as desired. That would be a way to specify that a PVC may span different regions or must not span different regions (when NUMA is important).

[pohly]

I've been told that a future Linux kernel feature might support namespaces that cross regions.

[tigerhu2008]

Sounds great! Do you have more info about the schedule? Thanks.

…

On Thu, Mar 10, 2022 at 3:26 PM Patrick Ohly ***@***.***> wrote: I've been told that a future Linux kernel feature might support namespaces that cross regions. — Reply to this email directly, view it on GitHub <#1045 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVNZVFJDANY5PPKMCBG5UBDU7GP4BANCNFSM5GUWAG2Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[pohly]

Sorry, I don't know if or when that enhanced Linux kernel support might get included.