Investigate Data Foundation/Data Fabric/CP4D options

[triceam]

The current Data Foundation modules have been updated to use the gitops provider and gitea, however there is still work to bring cp4d modules where they need to be. Additional work includes:

• updating remaining cp4d software modules to use the gitops provider and gitea (data fabric modules, db2, etc...)
• update all cp4d modules for cp4d 4.5 (current modules are for cp4d 4.0)

We need to decide the approach for finishing up Data Foundation/Data Fabric modules.

Options:

1. ❌ Update remaining cp4d modules for gitea & gitops provider, targeting cp4d 4.0.
   • Shortest timeline to update remaining data fabric modules.
   • Approximately 1 week's worth of effort to finish remaining modules
   • Modules will still target a back-level version of CP4D, not the latest
   • Does not re-use automation from the product team
2. ❌ Update all cp4d modules for gitea & gitops provider, targeting cp4d 4.5.
   • Approximately 2-3 week's worth of effort to update all modules
   • Modules will target latest version of CP4D
   • Does not re-use automation from the product team, so we maintain our own deployment artifacts (not desirable)
   • We can leverage cpd-cli to generate kube yaml (in preview.sh files), and we extract correct yaml by hand.
3. ✅ Update all cp4d modules to leverage Cloud Pak Deployer
   • Rewrite all modules to leverage https://ibm.github.io/cloud-pak-deployer/
   • May require an operator to manage Cloud Pak Deployer interactions (like MAS)
   • Modules can target latest version of CP4D
   • Is not supported by the product team
   • Would be abstraction on top of abstraction (This leverages cpd-cli under the hood)
   • At least 1+ month (optimistically)
4. ✅ Update all cp4d modules to leverage cpd-cli
   • Rewrite all modules to leverage https://github.com/IBM/cpd-cli/
   • May require an operator to manage cpd-cli interactions (like MAS)
   • Modules can target latest version of CP4D
   • Is supported by the product team
   • At least 1+ month (optimistically)
   • Duplication of effort (basically we'd be doing the same as the Cloud Pak Deployer, but in terraform instead of Ansible)

[triceam]

Unfortunately, it doesn't look like there's an easy way for our modules to generate yaml using cpd-cli. cpd-cli will generate a preview shell script that contains the yaml that needs to be applied to the cluster, but it isn't in a format that we could dump directly into gitops repos. For having a near-term solution, my preference is option 2: to update all modules to use cpd 4.5 by hand (by generating preview yaml using cpd-cli), and manually updatingn our modules.

Longer term, I think writing an operator that wraps cpd-cli might be a better solution, but it is a much higher level of effort. I don't think using Cloud Pak Deployer is the right option at all b/c it would be abstraction on top of (unsupported) abstraction.

[triceam]

Outcome of discussion:

• Options 1 and 2 are off the table. There's no sense in creating more technical debt
• Additional research will be done for Cloud Pak Deployer and cpd-cli
• Will setup discussion with Cloud Pak Deployer (Deepak) and India team.

[triceam]

The prototype effort to deploy Cloud Pak for Data using the Cloud Pak Deployer has proven successful.

I've got a prototype module in PR here: cloud-native-toolkit/terraform-gitops-cp4d-deployer#1

I built a Cloud Pak Deployer image and pushed it to quay.io/repository/cloudnativetoolkit/cloud-pak-deployer, based on changes in this PR: IBM/cloud-pak-deployer#264

So far I've been able to successfully create a job that executes the Cloud Pak Deployer installation using a basic configuration that gets stored in a ConfigMap. That job is able to execute to completion, and deploys CP4D 4.5.0 foundation successfully. I have not yet tried to deploy any additional components, but in theory, that should work fine b/c we're just wrapping the Cloud Pak Deployer.

This is not yet ready for production use, but provides a solid approach for us to go forward, without having to maintain our own installation modules for CP4D, and other offerings as they're supported by the deployer.

What this still needs:

• a mechanism to select which products should be deployed (simple booleans vars on this module would be easiest)
• a way to cleanup the deployment on destroy (currently the deployer's installation remains in a cpd-instance namespace created by the deployer after the module/gitops are destroyed)
• correctly apply service account/permission through the module, without manual intervention
• a lot more testing
• need to determine how this works with a true gitops workflow

Each deployment iteration takes over an hour just to deploy (not including destroy/cleanup).

Screenshots:

Job successfully running:

Operator installation:

Running CPD UI: