Allow vddk to be dynamically added without rebuilding container images

[bdunne]

The current VDDK installation in pods involves rebuilding the container images, but that is not convenient for users. As discussed, we need an easier solution.

Redistribute the VDDK

• Need to be a Select level member of the VMware TAP program^[ref]

ConfigMap

• Limited to 1MiB, file is ~20MiB

Keep the current solution of rebuilding the images

• High maintenance for upgrades

Single PersistentVolume mounted to all pods

• limits the backing storage types that we can run on
• More things to worry about for backup and restore
• Convenient that we only need to upload to a single place

Multiple PersistentVolumes (one per worker pod)

• Orchestrator will have to set them up copy file(s) in
• More things to worry about for backup and restore
• Don't need to worry about storage types, but requires the cluster to have multiple volumes available

BinaryBlobs

• Allows a single place to upload files
• Don't need to worry about backup / restore
• Don't need to worry about available volumes or their backing storage
• May be tricky to coordinate the initial upload and restarting of workers to make use of the new blob
• Every pod start would require a download and install

Download on demand

• Allow the user to specify a location to download the VDDK, like an FTP server or something
• User would have to permanently store the VDDK somwhere.
• Adding on the fly would require a worker restart on podified
• Every pod start would require a download and install

Thoughts @Fryguy & @agrare ?

volume access modes

[Fryguy]

I'd also like @jrafanie and @chessbyte to add their thoughts here.

---

ConfigMap is right out, because of the size limitations, so I crossed it out in the OP.

I'd prefer whatever solution we use to be consistent across both appliances and podified, so that we can reduce any speciality. That's another upside to BinaryBlobs, because we can document it as the preferred way for both appliances and podified.

Slight downside to BinaryBlobs is we will need to have a specific UI for uploading the file, which hopefully we can do in a pluggable way. We probably should block that to super-admins only or introduce a new RBAC setting or something for that.

[Fryguy]

In the private discussion, @agrare mentioned that we really only need the VDDK on the operation worker, which, I think, there can only be one replica of, however there is one per VMware EMS instance.

[bdunne]

Another advantage for BinaryBlobs option is that the admin doesn't have to install the package on every appliance that could run a Vmware operations worker, single upload and they're done. This could also simplify updates to the package.

Slight downside to BinaryBlobs is we will need to have a specific UI for uploading the file, which hopefully we can do in a pluggable way. We probably should block that to super-admins only or introduce a new RBAC setting or something for that.

Yeah, I was thinking a simple HTTP PUT to the API, but UI would be nice too.
I agree, it should be super-admin only

[Fryguy]

If we do BinaryBlobs, we have to make sure that the blob record is attached to something or it will get purged...the docs would have to tell the user to attach it to the enterprise or something.

[Fryguy]

HTTP PUT is a cool idea. I'm concerned they may try to upload multiple and we'd have to either disallow that or figure out the "right" one. Perhaps, instead of directly on a /binary_blobs endpoint, we could have an API on the EMS or perhaps the Enterprise? That API could do the right thing under the hood.

We'll probably need an API for the UI anyway, so that's a good start.

[chessbyte]

If we do BinaryBlobs, we have to make sure that the blob record is attached to something or it will get purged...the docs would have to tell the user to attach it to the enterprise or something.

Why not attach it to the VMware Provider EMS?

[Fryguy]

Why not attach it to the VMware Provider EMS?

The user would be required to upload it multiple times per VMware EMS instance, if they have multiple VMwares.

[agrare]

Why not attach it to the VMware Provider EMS?

Because we only need one instance of it

[Fryguy]

Just thought of another option, which I believe is along the lines of what the v2v team did.

Allow the user to specify a location to download the VDDK, like an FTP server or something. This could be set up as an extra tab in the vmware DDF form adding an endpoint/authentication for the vddk location. Then, when the operations worker starts, if there's a vddk defined, it can download and install it in the right place.

[agrare]

If we have the tgz file in BinaryBlobs then we will have to pull it down and extract it to the target dir. On an appliance this will only have to be done once but on podified it will have to be done everytime the pod starts up. Not a dealbreaker but not ideal

[Fryguy]

@agrare Same would be true for the "download the VDDK" option.

[agrare]

Yes I was comparing to the "mount a volume' option (unless that's already been NAK'd?)

[Fryguy]

On an appliance this will only have to be done once

On an appliance this will only have to be done once... per server