Deploy with new scheduler (#729)

Author: hunhoffe

.travis.yml

@@ -26,6 +26,7 @@ env:
     - TRAVIS_KUBE_VERSION=v1.19 OW_INCLUDE_SYSTEM_TESTS=false OW_CONTAINER_FACTORY=kubernetes OW_LEAN_MODE=true
     - TRAVIS_KUBE_VERSION=v1.20 OW_INCLUDE_SYSTEM_TESTS=false OW_CONTAINER_FACTORY=kubernetes
     - TRAVIS_KUBE_VERSION=v1.21 OW_INCLUDE_SYSTEM_TESTS=false OW_CONTAINER_FACTORY=kubernetes
+    - TRAVIS_KUBE_VERSION=v1.21 OW_INCLUDE_SYSTEM_TESTS=false OW_CONTAINER_FACTORY=kubernetes OW_SCHEDULER_ENABLED=true
 
 services:
   - docker

docs/configurationChoices.md

@@ -45,6 +45,16 @@ components is not currently supported:
   better management of the database and decouples its lifecycle from that of the OpenWhisk deployment.
 - The event providers: alarmprovider and kafkaprovider.
 
+### Openwhisk Scheduler
+
+By default, the scheduler is disabled. To enable the scheduler, add the following
+to your `mycluster.yaml`
+
+```yaml
+scheduler:
+  enabled: true
+```
+
 ### Using an external database
 
 You may want to use an external CouchDB or Cloudant instance instead
@@ -180,6 +190,8 @@ k8s:
     enabled: false
 ```
 
+Currently, etcd persistence is not supported.
+
 ### Selectively Deploying Event Providers
 
 The default settings of the Helm chart will deploy OpenWhisk's alarm

docs/k8s-custom-build-cluster-scaleup.md

@@ -40,7 +40,7 @@ Modifying the above mentioned parameters, one can easily increase the concurrenc
 In order to further increase the scale-up beyond `Small Scale`, one needs to modify the following additional configurations appropriately (on top of the above mentioned):
 * `invoker:jvmHeapMB`: jvmHeap memory available to each invoker instance. May or may not require increase based on running functions. For more information check `troubleshooting` below.
 * `invoker:containerFactory:_:replicaCount`: number of invoker instances that will be used to handle the incoming workload. By default, there is only one invoker instance which can become overwhelmed if workload goes beyond a certain threshold.
-* `controller:replicaCount`: number of controller instances that will be used to handle the incoming workload. Same as invoker instances.
+* `controller:replicaCount`: number of controller instances that will be used to handle the incoming workload. Same as invoker and scheduler instances.
 * `invoker:options`: Log processing at the invoker can become a bottleneck for the KubernetesContainerFactory. One might try disabling invoker log processing by setting it to `-Dwhisk.spi.LogStoreProvider=org.apache.openwhisk.core.containerpool.logging.LogDriverLogStoreProvider`. In general, one needs to offload log processing from the invoker to a node-level log store provider if one is trying to push a large load through the system.
 
 ## Troubleshooting

docs/k8s-kind.md

@@ -94,8 +94,8 @@ OpenWhisk apihost property to be set to localhost:31001
 ## Hints and Tips
 
 If you are working on the core OpenWhisk system and want
-to use a locally built controller or invoker image to test
-your changes, you need to push the image to the docker image
+to use a locally built controller, invoker, or scheduler image
+to test your changes, you need to push the image to the docker image
 repository inside the `kind` cluster.
 
 For example, suppose I had a local change to the controller

helm/openwhisk/templates/_helpers.tpl

@@ -42,6 +42,11 @@ app: {{ template "openwhisk.fullname" . }}
 {{ .Release.Name }}-controller.{{ .Release.Namespace }}.svc.{{ .Values.k8s.domain }}
 {{- end -}}
 
+{{/* hostname for scheduler */}}
+{{- define "openwhisk.scheduler_host" -}}
+{{ .Release.Name }}-scheduler.{{ .Release.Namespace }}.svc.{{ .Values.k8s.domain }}
+{{- end -}}
+
 {{/* hostname for database */}}
 {{- define "openwhisk.db_host" -}}
 {{- if .Values.db.external -}}
@@ -68,6 +73,15 @@ app: {{ template "openwhisk.fullname" . }}
 {{- end -}}
 {{- end -}}
 
+{{/* hostname for etcd */}}
+{{- define "openwhisk.etcd_host" -}}
+{{- if .Values.etcd.external -}}
+{{ .Values.etcd.host }}
+{{- else -}}
+{{ .Release.Name }}-etcd.{{ .Release.Namespace }}.svc.{{ .Values.k8s.domain }}
+{{- end -}}
+{{- end -}}
+
 {{/* client connection string for zookeeper cluster (server1:port,server2:port, ... serverN:port)*/}}
 {{- define "openwhisk.zookeeper_connect" -}}
 {{- if .Values.zookeeper.external -}}
@@ -196,10 +210,24 @@ app: {{ template "openwhisk.fullname" . }}
   value: {{ .Values.whisk.limits.activation.payload.max | quote }}
 {{- end -}}
 
+{{/* Environment variables for configuring etcd */}}
+{{- define "openwhisk.etcdConfigEnvVars" -}}
+- name: "CONFIG_whisk_cluster_name"
+  value: {{ .Values.etcd.clusterName | quote }}
+- name: "CONFIG_whisk_etcd_hosts"
+  value: {{ include "openwhisk.etcd_host" . }}:{{ .Values.etcd.port }}
+- name: "CONFIG_whisk_etcd_lease_timeout"
+  value: {{ .Values.etcd.leaseTimeout | quote }}
+- name: "CONFIG_whisk_etcd_pool_threads"
+  value: {{ .Values.etcd.poolThreads | quote }}
+{{- end -}}
+
 {{/* Environment variables for configuring kafka topics */}}
 {{- define "openwhisk.kafkaConfigEnvVars" -}}
 - name: "CONFIG_whisk_kafka_replicationFactor"
   value: {{ .Values.whisk.kafka.replicationFactor | quote }}
+- name: "CONFIG_whisk_kafka_topics_prefix"
+  value: {{ .Values.whisk.kafka.topics.prefix | quote }}
 - name: "CONFIG_whisk_kafka_topics_cacheInvalidation_retentionBytes"
   value: {{ .Values.whisk.kafka.topics.cacheInvalidation.retentionBytes | quote }}
 - name: "CONFIG_whisk_kafka_topics_cacheInvalidation_retentionMs"
@@ -224,12 +252,27 @@ app: {{ template "openwhisk.fullname" . }}
   value: {{ .Values.whisk.kafka.topics.health.retentionMs | quote }}
 - name: "CONFIG_whisk_kafka_topics_health_segmentBytes"
   value: {{ .Values.whisk.kafka.topics.health.segmentBytes | quote }}
+
 - name: "CONFIG_whisk_kafka_topics_invoker_retentionBytes"
   value: {{ .Values.whisk.kafka.topics.invoker.retentionBytes | quote }}
 - name: "CONFIG_whisk_kafka_topics_invoker_retentionMs"
   value: {{ .Values.whisk.kafka.topics.invoker.retentionMs | quote }}
 - name: "CONFIG_whisk_kafka_topics_invoker_segmentBytes"
   value: {{ .Values.whisk.kafka.topics.invoker.segmentBytes | quote }}
+
+- name: "CONFIG_whisk_kafka_topics_scheduler_retentionBytes"
+  value: {{ .Values.whisk.kafka.topics.scheduler.retentionBytes | quote }}
+- name: "CONFIG_whisk_kafka_topics_scheduler_retentionMs"
+  value: {{ .Values.whisk.kafka.topics.scheduler.retentionMs | quote }}
+- name: "CONFIG_whisk_kafka_topics_scheduler_segmentBytes"
+  value: {{ .Values.whisk.kafka.topics.scheduler.segmentBytes | quote }}
+
+- name: "CONFIG_whisk_kafka_topics_creationAck_retentionBytes"
+  value: {{ .Values.whisk.kafka.topics.creationAck.retentionBytes | quote }}
+- name: "CONFIG_whisk_kafka_topics_creationAck_retentionMs"
+  value: {{ .Values.whisk.kafka.topics.creationAck.retentionMs | quote }}
+- name: "CONFIG_whisk_kafka_topics_creationAck_segmentBytes"
+  value: {{ .Values.whisk.kafka.topics.creationAck.segmentBytes | quote }}
 {{- end -}}
 
 {{/* tlssecretname for ingress */}}

helm/openwhisk/templates/_readiness.tpl

@@ -38,6 +38,17 @@
   command: ["sh", "-c", 'cacert="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"; token="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"; while true; do rc=$(curl -sS --cacert $cacert --header "Authorization: Bearer $token" https://kubernetes.default.svc/api/v1/namespaces/{{ .Release.Namespace }}/endpoints/{{ .Release.Name }}-kafka | jq -r ".subsets[].addresses | length"); echo "num ready kafka endpoints is $rc"; if [ $rc -gt 0 ]; then echo "Success: ready kafka endpoint!"; break; fi; echo "kafka not ready yet; sleeping for 3 seconds"; sleep 3; done;']
 {{- end -}}
 
+{{/* Init container that waits for etcd to be ready */}}
+{{- define "openwhisk.readiness.waitForEtcd" -}}
+- name: "wait-for-etcd"
+  image: "{{- .Values.docker.registry.name -}}{{- .Values.utility.imageName -}}:{{- .Values.utility.imageTag -}}"
+  imagePullPolicy: "IfNotPresent"
+  env:
+  - name: "READINESS_URL"
+    value: http://{{ include "openwhisk.etcd_host" . }}:{{ .Values.etcd.port }}/health
+  command: ["sh", "-c", "while true; do echo 'checking etcd readiness'; health_result=$(curl -m 5 $READINESS_URL) && echo $health_result | jq -e '. | select(.health==\"true\")'; result=$?; if [ $result -eq 0 ]; then echo 'Success: etcd is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done;"]
+{{- end -}}
+
 {{/* Init container that waits for zookeeper to be ready */}}
 {{- define "openwhisk.readiness.waitForZookeeper" -}}
 - name: "wait-for-zookeeper"
@@ -57,6 +68,17 @@
   command: ["sh", "-c", "result=1; until [ $result -eq 0 ]; do echo 'Checking controller readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1; done; echo 'Success: controller is ready'"]
 {{- end -}}
 
+{{/* Init container that waits for scheduler to be ready */}}
+{{- define "openwhisk.readiness.waitForScheduler" -}}
+- name: "wait-for-scheduler"
+  image: "{{- .Values.docker.registry.name -}}{{- .Values.busybox.imageName -}}:{{- .Values.busybox.imageTag -}}"
+  imagePullPolicy: "IfNotPresent"
+  env:
+  - name: "READINESS_URL"
+    value: http://{{ include "openwhisk.scheduler_host" . }}:{{ .Values.scheduler.endpoints.port }}/ping
+  command: ["sh", "-c", "result=1; until [ $result -eq 0 ]; do echo 'Checking scheduler readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1; done; echo 'Success: scheduler is ready'"]
+{{- end -}}
+
 {{/* Init container that waits for at least 1 healthy invoker */}}
 {{- define "openwhisk.readiness.waitForHealthyInvoker" -}}
 - name: "wait-for-healthy-invoker"

helm/openwhisk/templates/controller-pod.yaml

@@ -60,6 +60,9 @@ spec:
 {{- if not .Values.controller.lean }}
       # The controller must wait for kafka and/or couchdb to be ready before it starts
 {{ include "openwhisk.readiness.waitForKafka" . | indent 6 }}
+{{- if .Values.scheduler.enabled }}
+{{ include "openwhisk.readiness.waitForEtcd" . | indent 6 }}
+{{- end }}
 {{- end }}
 {{ include "openwhisk.readiness.waitForCouchDB" . | indent 6 }}
 {{- if eq .Values.activationStoreBackend "ElasticSearch" }}
@@ -85,7 +88,7 @@ spec:
         - name: controller
           containerPort: {{ .Values.controller.port }}
         - name: akka-remoting
-          containerPort: 2552
+          containerPort: 25520
         - name: akka-mgmt-http
           containerPort: 19999
 {{- if .Values.controller.lean }}
@@ -114,6 +117,11 @@ spec:
         - name: "TZ"
           value: {{ .Values.docker.timezone | quote }}
 
+        - name: "POD_IP"
+          valueFrom:
+            fieldRef:
+              fieldPath: status.podIP
+
         - name:  "CONFIG_whisk_info_date"
           valueFrom:
             configMapKeyRef:
@@ -137,6 +145,15 @@ spec:
         - name: "RUNTIMES_MANIFEST"
           value: {{ template "openwhisk.runtimes_manifest" . }}
 
+        # scheduler settings
+{{ if .Values.scheduler.enabled }}
+        - name: "CONFIG_whisk_spi_LoadBalancerProvider"
+          value: "org.apache.openwhisk.core.loadBalancer.FPCPoolBalancer"
+
+        - name: "CONFIG_whisk_spi_EntitlementSpiProvider"
+          value: "org.apache.openwhisk.core.entitlement.FPCEntitlementProvider"
+{{ end }}
+
         # Action limits
 {{ include "openwhisk.limitsEnvVars" . | indent 8 }}
 
@@ -151,11 +168,17 @@ spec:
           value: "{{ include "openwhisk.kafka_connect" . }}"
 {{ include "openwhisk.kafkaConfigEnvVars" . | indent 8 }}
 
+        # etcd properties
+{{- if .Values.scheduler.enabled }}
+{{ include "openwhisk.etcdConfigEnvVars" . | indent 8 }}
+{{- end }}
+
         # properties for DB connection
 {{ include "openwhisk.dbEnvVars" . | indent 8 }}
 
         - name: "CONTROLLER_INSTANCES"
           value: {{ .Values.controller.replicaCount | quote }}
+
 {{- if gt (int .Values.controller.replicaCount) 1 }}
         - name: "CONFIG_whisk_cluster_useClusterBootstrap"
           value: "true"
@@ -169,7 +192,11 @@ spec:
           value: "name={{ .Release.Name }}-controller"
         - name: "CONFIG_akka_discovery_kubernetesApi_podPortName"
           value: "akka-mgmt-http"
+{{- else }}
+        - name: "CONFIG_akka_cluster_seedNodes_0"
+          value: "akka://controller-actor-system@$(POD_IP):25520"
 {{- end }}
+
 {{- if .Values.metrics.prometheusEnabled }}
         - name: "OPENWHISK_ENCODED_CONFIG"
           value: {{ template "openwhisk.whiskconfig" . }}

helm/openwhisk/templates/etcd-pod.yaml

@@ -0,0 +1,114 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{ if not .Values.etcd.external }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ .Release.Name }}-etcd
+  labels:
+    name: {{ .Release.Name }}-etcd
+{{ include "openwhisk.label_boilerplate" . | indent 4 }}
+spec:
+  replicas: {{ .Values.etcd.replicaCount }}
+  selector:
+    matchLabels:
+      name: {{ .Release.Name }}-etcd
+  {{- if .Values.k8s.persistence.enabled }}
+  strategy:
+    type: "Recreate"
+  {{- end }}
+  template:
+    metadata:
+      labels:
+        name: {{ .Release.Name }}-etcd
+{{ include "openwhisk.label_boilerplate" . | indent 8 }}
+    spec:
+      restartPolicy: {{ .Values.etcd.restartPolicy }}
+
+      {{- if .Values.affinity.enabled }}
+      affinity:
+{{ include "openwhisk.affinity.core" . | indent 8 }}
+{{ include "openwhisk.affinity.selfAntiAffinity" ( printf "%s-etcd" .Release.Name | quote ) | indent 8 }}
+      {{- end }}
+
+      {{- if .Values.toleration.enabled }}
+      tolerations:
+{{ include "openwhisk.toleration.core" . | indent 8 }}
+      {{- end }}
+
+{{- if .Values.k8s.persistence.enabled }}
+      volumes:
+      - name: etcd-data
+        persistentVolumeClaim:
+          claimName: {{ .Release.Name }}-etcd-pvc
+{{- end }}
+
+{{- if .Values.k8s.persistence.enabled }}
+      initContainers:
+      - name: etcd-init
+        image: "{{- .Values.docker.registry.name -}}{{- .Values.busybox.imageName -}}:{{- .Values.busybox.imageTag -}}"
+        command:
+          - chown
+          - -v
+          - -R
+          - 999:999
+          - /data
+        volumeMounts:
+        - mountPath: /data
+          name: etcd-data
+          readOnly: false
+{{- end }}
+{{ include "openwhisk.docker.imagePullSecrets" . | indent 6 }}
+      # current command will always restart from scratch (no persistence)
+      containers:
+        - name: etcd
+          image: "{{- .Values.docker.registry.name -}}{{- .Values.etcd.imageName -}}:{{- .Values.etcd.imageTag -}}"
+          command:
+            - /usr/local/bin/etcd
+            - --data-dir=/data
+            - --name
+            - etcd0
+            - --initial-advertise-peer-urls
+            - http://127.0.0.1:2480
+            - --advertise-client-urls
+            - http://0.0.0.0:{{ .Values.etcd.port }}
+            - --listen-peer-urls
+            - http://127.0.0.1:2480
+            - --listen-client-urls
+            - http://0.0.0.0:{{ .Values.etcd.port }}
+            - --initial-cluster
+            - etcd0=http://127.0.0.1:2480
+            - --initial-cluster-state
+            - new
+            - --initial-cluster-token
+            - openwhisk-etcd-token
+            - --quota-backend-bytes
+            - "0"
+            - --snapshot-count
+            - "100000"
+            - --auto-compaction-retention
+            - "1"
+            - --auto-compaction-mode
+            - periodic
+            - --log-level
+            - info
+          imagePullPolicy: {{ .Values.etcd.imagePullPolicy | quote }}
+          ports:
+          - name: etcd
+            containerPort: {{ .Values.etcd.port }}
+{{ end }}

helm/openwhisk/templates/etcd-pvc.yaml

@@ -0,0 +1,34 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{- if and (not .Values.etcd.external) .Values.k8s.persistence.enabled }}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: {{ .Release.Name }}-etcd-pvc
+  labels:
+{{ include "openwhisk.label_boilerplate" . | indent 4 }}
+spec:
+{{- if not .Values.k8s.persistence.hasDefaultStorageClass }}
+  storageClassName: {{ .Values.k8s.persistence.explicitStorageClass }}
+{{- end }}
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: {{ .Values.etcd.persistence.size }}
+{{- end }}