OWLS-109755 - Changes for Operator Dedicated Mode to suppress readCRD call failed message and clarify webhook configuration.

Author: ankedia

common/src/main/java/oracle/kubernetes/common/CommonConstants.java

@@ -5,6 +5,8 @@
 
 public class CommonConstants {
 
+  public static final String CRD = "CRD";
+
   private CommonConstants() {
     //not called
   }

documentation/site/content/managing-operators/conversion-webhook.md

@@ -46,6 +46,10 @@ Beginning with  operator version 4.0, when you [install the operator]({{<relref
 The `helm install` step requires cluster-level permissions for listing and reading all Namespaces and Deployments to search for existing conversion webhook deployments. If you cannot grant the cluster-level permissions and have multiple operators deployed, then install the conversion webhook separately and set the Helm configuration value `operatorOnly` to `true` in the `helm install` command to prevent multiple conversion webhook deployments. In addition, the webhook uses a service account that is usually the same service account as an operator running in the same namespace. This service account requires permissions to create and read events in the conversion webhook namespace. For more information, see [RBAC]({{<relref "/managing-operators/rbac.md" >}}).
 {{% /notice %}}
 
+{{% notice note %}}
+The Operator version 4.x requires a conversion webhook installation. The `operatorOnly` Helm configuration value is an advanced setting and should be used only when a conversion webhook is already installed.
+{{% /notice %}}
+
 
 If you want to install _only_ the conversion webhook (and not the operator) in the given namespace, set the Helm configuration value `webhookOnly` to `true` in the `helm install` command. After meeting the [prerequisite requirements]({{<relref "/managing-operators/preparation.md">}}), call:
 ```

documentation/site/content/managing-operators/installation.md

@@ -24,8 +24,8 @@ See [Prepare for installation]({{<relref "/managing-operators/preparation.md">}}
 {{% notice note %}}
 By default, installing the operator also configures a deployment and supporting resources for the
 [conversion webhook]({{<relref "/managing-operators/conversion-webhook">}})
-and deploys the conversion webhook.
-To skip the conversion webhook installation, set helm configuration value `operatorOnly` to `true`
+and deploys the conversion webhook. The conversion webhook deployment is required for Operator version 4.x.
+To skip the conversion webhook installation when a conversion webhook is already installed, set the helm configuration value `operatorOnly` to `true`
 in the `helm install` command.
 For more details, see [install the conversion webhook]({{<relref "/managing-operators/conversion-webhook#install-the-conversion-webhook">}}).
 {{% /notice %}}

documentation/site/content/managing-operators/troubleshooting.md

@@ -324,6 +324,38 @@ weblogic-operator-webhook-svc   ClusterIP   10.106.89.198   <none>        8084/T
 ```
 If the conversion webhook Deployment status is not ready, then [check the conversion webhook log]({{<relref "/managing-operators/troubleshooting#check-the-conversion-webhook-log">}}) and the [conversion webhook events]({{<relref "/managing-operators/troubleshooting#check-for-conversion-webhook-events">}}) in the conversion webhook namespace. If the conversion webhook service doesn't exist, make sure that the conversion webhook was installed correctly and reinstall the conversion webhook to see if it resolves the issue.
 
+#### X509: Certificate signed by unknown authority error from the webhook
+The following `x509: certificate signed by unknown authority` error from the conversion webhook can be due to the incorrect proxy configuration of the Kubernetes API server in your environment or incorrect self-signed certificate in the conversion webhook configuration in the Domain CustomResourceDefinition (CRD).
+
+```
+Error from server (InternalError): error when creating "./weblogic-domains/sample-domain1/domain.yaml": Internal error occurred: conversion webhook for weblogic.oracle/v8, Kind=Domain failed: Post "https://weblogic-operator-webhook-svc.sample-weblogic-operator-ns.svc:8084/webhook?timeout=30s": x509: certificate signed by unknown authority
+```
+- If your environment uses a PROXY server, then ensure that the NO_PROXY settings of the Kubernetes API server include `.svc` value. The Kubernetes API server makes a REST request to the conversion webhook REST end-point using the hostname `weblogic-operator-webhook-svc.${NAMESPACE}.svc` in the POST URL. If the REST request is routed through a PROXY server, then you will see an "x509: certificate signed by unknown authority" error. As this REST request is internal to your Kubernetes cluster, ensure that it doesn't get routed through a PROXY server by adding `.svc` to the `NO_PROXY` settings.
+- If your CRD conversion webhook configuration has an incorrect self-signed certificate for some reason, then you can patch the CRD to remove the existing conversion webhook configuration. The operator will re-create the CRD conversion webhook configuration using the correct self-signed certificate. Use the below patch command to remove the conversion webhook configuration in the CRD to see if it resolves the error.
+
+    ```
+    kubectl patch crd domains.weblogic.oracle --type=merge --patch '{"spec": {"conversion": {"strategy": "None", "webhook": null}}}'
+    ```
+
+#### Webhook errors in older operator versions
+When you install Operator version 4.x or upgrade to Operator 4.x, a conversion webhook configuration is added to your Domain CRD. If you downgrade or switch back to the operator version 3.x, the conversion webhook configuration is not removed from the CRD. This is to support the environments with multiple Operator installations potentially with different versions. For environments having a single Operator installation, use the below patch command to manually remove the conversion webhook configuration from Domain CRD. 
+
+```
+kubectl patch crd domains.weblogic.oracle --type=merge --patch '{"spec": {"conversion": {"strategy": "None", "webhook": null}}}'
+```
+
+#### Webhook errors in operator Dedicated Mode
+If the Operator is running in the `Dedicated` mode, the operator's service account will not have the permission to read or update the CRD. If you need to convert the domain resources with `weblogic.oracle/v8` schema to `weblogic.oracle/v9` schema using conversion webhook in `Dedicated` mode, then you can manually add the conversion webhook configuration to the Domain CRD. Use the below patch command to add the conversion webhook configuration to the Domain CRD.
+
+**Note**: Substitute `YOUR_OPERATOR_NS` in the below command with the namespace where the operator is installed.
+
+```
+export OPERATOR_NS=YOUR_OPERATOR_NS
+```
+```
+kubectl patch crd domains.weblogic.oracle --type=merge --patch '{"spec": {"conversion": {"strategy": "Webhook", "webhook": {"clientConfig": { "caBundle": "'$(kubectl get secret weblogic-webhook-secrets -n ${OPERATOR_NS} -o=jsonpath="{.data.webhookCert}"| base64 --decode)'", "service": {"name": "weblogic-operator-webhook-svc", "namespace": "'${OPERATOR_NS}'", "path": "/webhook", "port": 8084}}, "conversionReviewVersions": ["v1"]}}}}'
+```
+
 #### Check for runtime errors during conversion
 If you see a `WebLogic Domain custom resource conversion webhook failed` error when creating a Domain with a `weblogic.oracle/v8` schema domain resource, then [check the conversion webhook runtime Pod logs]({{<relref "/managing-operators/troubleshooting#check-the-conversion-webhook-log">}}) and [check for the generated events]({{<relref "/managing-operators/troubleshooting#check-for-conversion-webhook-events">}}) in the conversion webhook namespace. Assuming that the conversion webhook is deployed in the `sample-weblogic-operator-ns` namespace, run the following commands to check for logs and events.
 

documentation/site/content/managing-operators/using-helm.md

@@ -265,6 +265,8 @@ If set to `true`, the `helm install` will install _only_ the conversion webhook
 Defaults to `false`.
 
 ##### `operatorOnly`
+**Note**: This is an advanced setting and should be used only in environments where a conversion webhook is already installed. The operator version 4.x requires a conversion webhook to be installed.
+
 Specifies whether only the operator should be installed during the `helm install` and that the conversion webhook installation should be skipped. By default, the `helm install` command installs both the operator and the conversion webhook.
 If set to `true`, the `helm install` will install _only_ the operator (and not the conversion webhook).
 

kubernetes/charts/weblogic-operator/templates/_operator-dep.tpl

@@ -224,6 +224,12 @@ spec:
       namespace: {{ .Release.Namespace | quote }}
     data:
       serviceaccount: {{ .serviceAccount | quote }}
+      {{- if .featureGates }}
+      featureGates: {{ .featureGates | quote }}
+      {{- end }}
+      {{- if .domainNamespaceSelectionStrategy }}
+      domainNamespaceSelectionStrategy: {{ .domainNamespaceSelectionStrategy | quote }}
+      {{- end }}
 ---
     # webhook does not exist or chart version is newer, create a new webhook
     apiVersion: "apps/v1"

operator/src/main/java/oracle/kubernetes/operator/Namespaces.java

@@ -47,7 +47,7 @@
 public class Namespaces {
   private static final LoggingFacade LOGGER = LoggingFactory.getLogger("Operator", "Operator");
 
-  static final String SELECTION_STRATEGY_KEY = "domainNamespaceSelectionStrategy";
+  public static final String SELECTION_STRATEGY_KEY = "domainNamespaceSelectionStrategy";
   /**
    * The key in a Packet of the collection of existing namespaces that are designated as domain namespaces.
    */

operator/src/main/java/oracle/kubernetes/operator/helpers/ResponseStep.java

@@ -12,6 +12,7 @@
 import io.kubernetes.client.common.KubernetesObject;
 import io.kubernetes.client.openapi.ApiException;
 import oracle.kubernetes.common.logging.MessageKeys;
+import oracle.kubernetes.operator.OperatorMain;
 import oracle.kubernetes.operator.builders.CallParams;
 import oracle.kubernetes.operator.calls.AsyncRequestStep;
 import oracle.kubernetes.operator.calls.CallResponse;
@@ -26,6 +27,7 @@
 import oracle.kubernetes.weblogic.domain.model.DomainCondition;
 import oracle.kubernetes.weblogic.domain.model.DomainResource;
 
+import static oracle.kubernetes.common.CommonConstants.CRD;
 import static oracle.kubernetes.operator.KubernetesConstants.HTTP_CONFLICT;
 import static oracle.kubernetes.operator.KubernetesConstants.HTTP_FORBIDDEN;
 import static oracle.kubernetes.operator.KubernetesConstants.HTTP_NOT_FOUND;
@@ -178,9 +180,7 @@ private NextAction doPotentialRetry(Step conflictStep, Packet packet, CallRespon
   }
 
   private NextAction logNoRetry(Packet packet, CallResponse<T> callResponse) {
-    if ((callResponse != null)
-        && (callResponse.getStatusCode() != HTTP_NOT_FOUND)
-        && (callResponse.getStatusCode() != HTTP_CONFLICT)) {
+    if (shouldLogNoRetryWarning(callResponse)) {
       addDomainFailureStatus(packet, callResponse.getRequestParams(), callResponse.getE());
       if (LOGGER.isWarningEnabled()) {
         LOGGER.warning(
@@ -206,6 +206,25 @@ private NextAction logNoRetry(Packet packet, CallResponse<T> callResponse) {
     return null;
   }
 
+  private boolean shouldLogNoRetryWarning(CallResponse<T> callResponse) {
+    return (callResponse != null)
+        && (callResponse.getStatusCode() != HTTP_NOT_FOUND)
+        && (callResponse.getStatusCode() != HTTP_CONFLICT)
+        && (isNotCrdForbiddenWhenDedicated(callResponse));
+  }
+
+  private boolean isNotCrdForbiddenWhenDedicated(CallResponse<T> callResponse) {
+    if (OperatorMain.isDedicated() && isCRDCall(callResponse)) {
+      return callResponse.getStatusCode() != HTTP_FORBIDDEN && callResponse.getStatusCode() != HTTP_UNAUTHORIZED;
+    }
+    return true;
+  }
+
+  private boolean isCRDCall(CallResponse<T> callResponse) {
+    return Optional.ofNullable(callResponse.getRequestParams()).map(r -> r.call)
+        .map(c -> c.toUpperCase().contains(CRD)).orElse(false);
+  }
+
   private void addDomainFailureStatus(Packet packet, RequestParams requestParams, ApiException apiException) {
     DomainPresenceInfo.fromPacket(packet)
         .map(DomainPresenceInfo::getDomain)

operator/src/test/java/oracle/kubernetes/operator/helpers/CrdHelperTest.java

@@ -21,6 +21,7 @@
 import io.kubernetes.client.openapi.models.V1ObjectMeta;
 import oracle.kubernetes.operator.KubernetesConstants;
 import oracle.kubernetes.operator.LabelConstants;
+import oracle.kubernetes.operator.Namespaces;
 import oracle.kubernetes.operator.logging.LoggingFactory;
 import oracle.kubernetes.operator.tuning.TuningParametersStub;
 import oracle.kubernetes.operator.utils.Certificates;
@@ -37,6 +38,7 @@
 import org.junit.jupiter.params.provider.EnumSource;
 
 import static com.meterware.simplestub.Stub.createStrictStub;
+import static java.net.HttpURLConnection.HTTP_FORBIDDEN;
 import static java.net.HttpURLConnection.HTTP_UNAUTHORIZED;
 import static oracle.kubernetes.common.logging.MessageKeys.ASYNC_NO_RETRY;
 import static oracle.kubernetes.common.logging.MessageKeys.CREATE_CRD_FAILED;
@@ -233,6 +235,37 @@ void whenNotAuthorizedToReadCrd_retryOnFailureAndLogWarningMessageInOnFailureNoR
     assertThat(logRecords, containsWarning(ASYNC_NO_RETRY));
   }
 
+  @ParameterizedTest
+  @EnumSource(value = TestSubject.class)
+  void whenNotAuthorizedToReadCrdDedicatedMode_dontLogWarningMessageInOnFailureNoRetry(TestSubject testSubject) {
+    consoleHandlerMemento.collectLogMessages(logRecords, ASYNC_NO_RETRY);
+    defineSelectionStrategy(Namespaces.SelectionStrategy.DEDICATED);
+    testSupport.addRetryStrategy(retryStrategy);
+    testSupport.failOnResource(CUSTOM_RESOURCE_DEFINITION, null, null, HTTP_UNAUTHORIZED);
+
+    Step scriptCrdStep = testSubject.createCrdStep(PRODUCT_VERSION);
+    testSupport.runSteps(scriptCrdStep);
+    assertThat(logRecords, not(containsWarning(ASYNC_NO_RETRY)));
+  }
+
+  private void defineSelectionStrategy(Namespaces.SelectionStrategy selectionStrategy) {
+    TuningParametersStub.setParameter(Namespaces.SELECTION_STRATEGY_KEY, selectionStrategy.toString());
+  }
+
+  @ParameterizedTest
+  @EnumSource(value = TestSubject.class)
+  void whenForbiddenToReadCrdDedicatedMode_dontLogWarningMessageInOnFailureNoRetry(TestSubject testSubject) {
+    consoleHandlerMemento.collectLogMessages(logRecords, ASYNC_NO_RETRY);
+    defineSelectionStrategy(Namespaces.SelectionStrategy.DEDICATED);
+    testSupport.addRetryStrategy(retryStrategy);
+    testSupport.failOnResource(CUSTOM_RESOURCE_DEFINITION, null, null, HTTP_FORBIDDEN);
+
+    Step scriptCrdStep = testSubject.createCrdStep(PRODUCT_VERSION);
+    testSupport.runSteps(scriptCrdStep);
+    assertThat(logRecords, not(containsWarning(ASYNC_NO_RETRY)));
+  }
+
+
   @ParameterizedTest
   @EnumSource(value = TestSubject.class)
   void whenNotAuthorizedToReadCrdAndWarningNotEnabled_DontLogWarningInOnFailureNoRetry(TestSubject testSubject) {