Adding PVC's annotation to CreateVolumeRequest.Parameters

[orainxiong]

As the title says, there is a great possibility that the application-specific Operator would leverage PVC's Annotation for storing custom data. The custom data will tell storage provisioner how CSI drive create volume correctly.

Here is already existing logic for provision volume :

	// Create a CSI CreateVolumeRequest and Response
	req := csi.CreateVolumeRequest{

		Name:       pvName,
		Parameters: options.Parameters,
		VolumeCapabilities: []*csi.VolumeCapability{
			{
				AccessType: accessType,
				AccessMode: accessMode,
			},
		},
		CapacityRange: &csi.CapacityRange{
			RequiredBytes: int64(volSizeBytes),
		},
	}

Note : more details

CSI Plugin specifies that CreateVolumeRequest's field Parameters is OPTIONAL and opaque.
What I am suggesting is that we could pass in PVC's Annotation in addition to Storageclass.Parameter.

[clintonk]

I second this request, and we discussed this idea with @saad-ali at this week's Storage SIG F2F. The chief concern is name collisions between storage class parameters and PVC annotations. Saad suggested this capability be gated via a startup flag. @orainxiong, are you planning to submit a PR for this?

[orainxiong]

@clintonk Thx for your replay.
In order to solve this issue immediately, I handle this issue in a temporary way.

• A new function, called appendPVCAnnotation, is introduced to encode PVC annotation to JSON string and modify options.parameter.

func appendPVCAnnotation(options controller.VolumeOptions) error {
	annotationPVC, err := json.Marshal(options.PVC.Annotations)
	if err != nil {
		return err
	}

	parameters := map[string]string{
		"annotationsPVC": string(annotation),
	}
	return nil
}

Note : Reverse key name annotationsPVC to avoid name conflict.

• A new flag, called annotationPVCNeeded, is added to determine whether PVC annotation should be appended into options.parameter.

The advantage of implementation hardly modifies existing external-provisioner logic. Honestly, it is a not a preferable way.
Looking forward to your suggestions.

[jsafrane]

While I agree that there should be a way how to pass parameters from user to CreateVolume, I don't like suggested solution:

• It makes the CSI driver Kubernetes aware. Whole point of CSI is to have independent plugins that can be used in Kubernetes, Mesos, Cloud Foundry, Docker Swarm and others. The plugins should not know anything about PVCs, these are pure Kubernetes objects.

• Kubernetes admin needs a way how to restrict what parameters can be set by users. Kubernetes is (or can be) multi tenant system and not all users can use all parameters that a CSI plugin exposes.

• User should be allowed to list possible parameters together with some description so they can choose the right value. Note that there may be GUI between users and Kubernetes, so the GUI must be able at least to enumerate these parameters. We tried in Proposal for user configuration of dynamic provisioning kubernetes/community#543, however, nobody has stepped in to finish the proposal and implemented it.

[fatih]

After creating the csi-digitalocean driver, I also received a feedback, that needs this feature: digitalocean/csi-digitalocean#30

I'm using the name of the request ot create the volumes, which ends up being the PV name. So it's in the form of kubernetes-dynamic-pv-<randomid>. However on the UI (at DigitalOcean.com), this is not useful because it doesn't give the person an easy way to track back to which entity it belongs to.

I thought maybe I can use the StorageClass parameters and add a new field called volumePrefix, such as:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: do-block-storage
  namespace: kube-system
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: com.digitalocean.csi.dobs
parameters:
  volumePrefix: "csi-digitalocean"

and then in the Controller plugin, I'll change the name in CreateVolume according to this information:

volumeName := req.Name
// change the name if the storage-class has a prefix option
prefix, ok := req.Parameters["volumePrefix"]
if ok {
	volumeName = prefix + "-" + volumeName
}

This allows me to create a volume based on this users CSI deployment. However, this is not scalable because:

• you only create a single StorageClass, so it's not easy to update it every single time
• you're not allows to update the parameters field of a StorageClass
• StorageClass is global, whereas we need a local information from a local resource

A better idea would be to use the annotations of a PVC. This would allow us to create volume's directly based on the information bound to a specific PVC.

We're already passing down the information from a Kubernetes resource (i.e StorageClass) down to the plugin, I think there is not much that prevents us to pass the information fo another resource (i.e PersistentVolumeClaim) to the plugin.

[sbezverk]

check this out:
https://github.com/kubernetes-csi/external-provisioner/blob/master/cmd/csi-provisioner/csi-provisioner.go#L46
I think you can leverage this parameter.

[fatih]

@sbezverk thanks for the tip, I didn't know it.

Though, the prefix was just an example, it could be also something else. Such tagging the resource on the cloud provider side based on the created PVC labels.

[orainxiong]

@jsafrane
Thx for your reply.

If I understand correctly, I notice that external-provisioner has exposed StorageClass parameters to end storage drivers through VolumeOptions.

	options := VolumeOptions{
		PersistentVolumeReclaimPolicy: reclaimPolicy,
		PVName:     pvName,
		PVC:        claim,
		Parameters: parameters,  -- Store `StorageClass` parameters here
	}

	volume, err = ctrl.provisioner.Provision(options)

End storage drivers can get StorageClass parameters depending on their own logic. However, PVC's annotation is still missing in VolumeOptions.

I very agree with your opinion. It's a temporary solution which makes storage driver Kubernetes aware. It is a very time consuming job to figure out a COs neutral solution.
As far as I think of parameters and description, it is very likely to be exhaust to create a common body of naming language across COs.

Looking forward to your suggestions.

[jsafrane]

@fatih, for tagging we could perhaps extend CSI's CreateVolume call with optional metadata such as Kubernetes PVC namespace + name, PV name, storage class name (?) or PVC labels (which one? all of them?). It would be used only to tag volumes in the storage backend so users can map these volumes to CO objects. These cannot be used to configure provisioned volume (see #86 (comment) above).

CO would provide generic map[string]string, opaque to CSI driver and CSI driver would blindly "attach" it to the volume via AWS tags or GCE labels or OpenStack metadata or whatever the storage supports (or throw them away if the storage does not support tagging).

However, PVC's annotation is still missing in VolumeOptions.

@orainxiong, that's intentional.

I could imagine our CSI provisioner passes content of some alpha annotations from PVC as CreateVolumeRequest.parameters in CreateVolume call, but these should be really alpha. To get the to GA, we need policy and discovery and whatnot.

[fatih]

for tagging we could perhaps extend CSI's CreateVolume call with optional metadata such as Kubernetes PVC namespace + name, PV name, storage class name (?) or PVC labels (which one? all of them?).

I think passing the PVC metadata would be great. Because it already includes the following important bits:

• name
• namespace
• labels
• annotations

I'm not sure how to pass them via CO though. a map[string]string is the easiest way. But then how are we going to represent nested maps? We could pass it as a JSON maybe:

map[string]string{
 "pvc" : "{\"name\": ...}",
 "storageclass": " { ... }",
}

I know that Kubernetes annotations mostly store data like this and it's a common thing there.

CO would provide generic map[string]string, opaque to CSI driver and CSI driver would blindly "attach" it to the volume via AWS tags or GCE labels or OpenStack metadata or whatever the storage supports (or throw them away if the storage does not support tagging).

Yeap. Attaching is only one of the things the CSI driver can accomplish. Changing the name is another action it should be allowed to do.

[orainxiong]

@jsafrane Thanks. I got it. I will close this issue.

[fatih]

Why is this closed? Did we have a PR that fixes this issue or a solution that is applicable right now?

[orainxiong]

@fatih

At least, in my opinion, there is a temporary way to handle this issue. But, it isn't applicable for all COs that support CSI.

As your suggestions, a new function injectCOParameter is introduced into external-provisioner :

func injectCOParameter(options controller.VolumeOptions) (map[string]string, error) {

	annotationsPVC, err := json.Marshal(options.PVC.Annotations)
	if err != nil {
		return nil, err
	}

	labelsPVC, err := json.Marshal(options.PVC.Labels)
	if err != nil {
		return nil, err
	}

	storageClass, err := json.Marshal(options.Parameters)
	if err != nil {
		return nil, err
	}

	parameters := map[string]string{
		"annotationsPVC": string(annotationsPVC),
		"labelsPVC":     string(labelsPVC),
		"storageClass":  string(storageClass),
	}
	return parameters, nil
}

injectCOParameter is in the charge of encoding labels and annotations and storageclass into
CreateVolumeRequest.Parameters. Specifically, CreateVolumeRequest.Parameters is defined by CSI to store opaque parameter and COs will ignore it.

On the storage provisioner side, they can decode parameters from CreateVolumeRequest.Parameters according to their own operational logic.

[fatih]

@orainxiong Thanks for the reply. The title says Adding PVC's annotation to CreateVolumeRequest.Parameters and unless I'm missing something, there is no such feature and no one contributed to the provisioner/attacher to add this feature. Hence my question why we have closed this issue. Nothing is fixed yet and it's not possible to pass any PVC/PV data to the plugin from the CO. This issue should be re-opened, and closed when there is an action task we can work together.

[orainxiong]

@fatih ok, you're right. : - )