Fix Alluxio environment variable syntax, image reference format, and ZooKeeper classpath in QA deployment

[Copilot]

Problem

The QA deployment was failing with three critical issues:

1. Alluxio master startup failure with the error:

Exception in thread "main" java.lang.RuntimeException: Invalid property key POD_IP
	at alluxio.conf.InstancedConfiguration.lookupRecursively(InstancedConfiguration.java:442)
	at alluxio.conf.InstancedConfiguration.lookup(InstancedConfiguration.java:413)
	at alluxio.conf.InstancedConfiguration.isResolvable(InstancedConfiguration.java:152)

2. Pod initialization failure with the error:

Failed to apply default image tag "/accumulo/accumulo:4.0.0-SNAPSHOT": couldn't parse image name "/accumulo/accumulo:4.0.0-SNAPSHOT": invalid reference format

3. Accumulo initialization failure with the error:

find: '/opt/zookeeper/lib/': No such file or directory
java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException

These errors prevented the Alluxio master and Accumulo components from starting, blocking the entire QA deployment process.

Root Causes

Issue 1: Environment Variable Syntax

The Alluxio configuration in configmap.yaml was using incorrect syntax for environment variable references:

alluxio.master.hostname=${POD_IP}
alluxio.worker.hostname=${ALLUXIO_WORKER_HOSTNAME}

Alluxio's property resolution system interprets ${VARIABLE_NAME} as a property key reference, not an environment variable. When Alluxio tried to resolve ${POD_IP}, it looked for an Alluxio property named "POD_IP" instead of reading the environment variable, resulting in the "Invalid property key POD_IP" error.

Issue 2: Invalid Image Reference Format

The image helper templates in _helpers.tpl were producing invalid image references when the registry was set to an empty string. The templates concatenated an empty registry with the repository name, resulting in /accumulo/accumulo:4.0.0-SNAPSHOT instead of accumulo/accumulo:4.0.0-SNAPSHOT. Container runtimes reject image names with leading slashes.

Issue 3: ZooKeeper Classpath Mismatch

The accumulo-env.sh configuration was looking for ZooKeeper jars in ${ZOOKEEPER_HOME}/lib/, but the ZooKeeper binary distribution places jar files in the root directory. This caused ClassNotFoundException errors when Accumulo tried to load ZooKeeper classes.

Solution

Fix 1: Corrected Environment Variable Syntax

Updated the Alluxio configuration to use the correct syntax for environment variable references: ${env.VARIABLE_NAME}

Changes in charts/accumulo/templates/configmap.yaml:

• Line 201: alluxio.master.hostname=${POD_IP} → alluxio.master.hostname=${env.POD_IP}
• Line 256: alluxio.worker.hostname=${ALLUXIO_WORKER_HOSTNAME} → alluxio.worker.hostname=${env.ALLUXIO_WORKER_HOSTNAME}

Fix 2: Corrected Image Helper Templates

Updated both accumulo.image and alluxio.image helper templates to conditionally include the registry prefix only when it's not empty.

Changes in charts/accumulo/templates/_helpers.tpl:

• Added conditional logic to check if registry is set before including it in the image name
• When registry is empty: produces repository:tag
• When registry is set: produces registry/repository:tag

Fix 3: Corrected ZooKeeper Classpath

Updated the ZooKeeper jar discovery to match the actual directory structure of the ZooKeeper binary distribution.

Changes in charts/accumulo/templates/configmap.yaml:

• Line 93: Changed find "${ZOOKEEPER_HOME}/lib/" to find "${ZOOKEEPER_HOME}" to match the ZooKeeper binary distribution structure

Documentation Update

Changes in charts/accumulo/README.md:
Added a troubleshooting entry with concrete example to help users identify and resolve this issue in the future:

Alluxio "Invalid property key POD_IP": This error occurs when Alluxio configuration uses incorrect environment variable syntax. Ensure all environment variables in alluxio-site.properties use the ${env.VARIABLE_NAME} format, not ${VARIABLE_NAME}. For example, use alluxio.master.hostname=${env.POD_IP} instead of alluxio.master.hostname=${POD_IP}

Impact

These changes fix the Alluxio master startup failure, the pod initialization failure, and the ZooKeeper classpath issue, allowing the QA deployment to proceed successfully. The environment variables POD_IP and ALLUXIO_WORKER_HOSTNAME are correctly set in the deployment manifests and will now be properly resolved by Alluxio. Container images will be correctly referenced without invalid leading slashes, and Accumulo will be able to load ZooKeeper classes from the correct location.

Testing

• ✅ Verified Helm chart syntax is valid
• ✅ Confirmed environment variables are properly set in deployment manifests
• ✅ Verified image helper templates produce valid image references with and without registry values
• ✅ Verified ZooKeeper classpath matches binary distribution structure
• ✅ Changes are surgical and only affect Alluxio configuration syntax, image reference formatting, and ZooKeeper classpath
• ✅ Code review completed with no issues

Fixes #14

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Issues in QA build and deploying</issue_title>
<issue_description>Formatting Alluxio Master @ accumulo-alluxio-master-5cb6b9cf65-dn24h
2025-10-10 21:04:00,049 INFO main - Formatting master journal: /opt/alluxio/journal/
2025-10-10 21:04:00,152 INFO main - Loading core jars from /opt/alluxio-2.9.4/lib
2025-10-10 21:04:00,250 INFO main - Loading extension jars from /opt/alluxio-2.9.4/extensions
2025-10-10 21:04:00,264 INFO main - Formatting /opt/alluxio/journal/BlockMaster/v1
2025-10-10 21:04:00,270 INFO main - Formatting /opt/alluxio/journal/JournalMaster/v1
2025-10-10 21:04:00,271 INFO main - Formatting /opt/alluxio/journal/FileSystemMaster/v1
2025-10-10 21:04:00,272 INFO main - Formatting /opt/alluxio/journal/ThrottleMaster/v1
2025-10-10 21:04:00,273 INFO main - Formatting /opt/alluxio/journal/MetaMaster/v1
2025-10-10 21:04:00,274 INFO main - Formatting /opt/alluxio/journal/MetricsMaster/v1
2025-10-10 21:04:00,275 INFO main - Formatting complete
Exception in thread "main" java.lang.RuntimeException: Invalid property key POD_IP
at alluxio.conf.InstancedConfiguration.lookupRecursively(InstancedConfiguration.java:442)
at alluxio.conf.InstancedConfiguration.lookup(InstancedConfiguration.java:413)
at alluxio.conf.InstancedConfiguration.isResolvable(InstancedConfiguration.java:152)
at alluxio.conf.InstancedConfiguration.isSet(InstancedConfiguration.java:162)
at alluxio.conf.AlluxioConfiguration.getOrDefault(AlluxioConfiguration.java:65)
at alluxio.cli.GetConf.getConfImpl(GetConf.java:189)
at alluxio.cli.GetConf.getConf(GetConf.java:146)
at alluxio.cli.GetConf.main(GetConf.java:267)
Starting master @ accumulo-alluxio-master-5cb6b9cf65-dn24h. Logging to /opt/alluxio-2.9.4/logs
Exception in thread "main" java.lang.RuntimeException: Invalid property key POD_IP
at alluxio.conf.InstancedConfiguration.lookupRecursively(InstancedConfiguration.java:442)
at alluxio.conf.InstancedConfiguration.lookup(InstancedConfiguration.java:413)
at alluxio.conf.InstancedConfiguration.isResolvable(InstancedConfiguration.java:152)
at alluxio.conf.InstancedConfiguration.isSet(InstancedConfiguration.java:162)
at alluxio.util.network.NetworkAddressUtils.getConnectHost(NetworkAddressUtils.java:332)
at alluxio.util.network.NetworkAddressUtils.getConnectAddress(NetworkAddressUtils.java:273)
at alluxio.util.ConfigurationUtils.getMasterEmbeddedJournalAddresses(ConfigurationUtils.java:78)
at alluxio.util.ConfigurationUtils.getEmbeddedJournalAddresses(ConfigurationUtils.java:62)
at alluxio.util.ConfigurationUtils.getMasterRpcAddresses(ConfigurationUtils.java:115)
at alluxio.util.ConfigurationUtils.isHaMode(ConfigurationUtils.java:242)
at alluxio.master.AlluxioMasterMonitor.main(AlluxioMasterMonitor.java:51)
--- [ FAILED ] The master @ accumulo-alluxio-master-5cb6b9cf65-dn24h is not serving requests after 120s. Please check if the process is running and the logs/ if necessary.
--- Printing the log tail for /opt/alluxio-2.9.4/logs/master.log

BEGIN
2025-10-10 21:04:02,653 ERROR main - Fatal error: Failed to create master process
java.lang.RuntimeException: Invalid property key POD_IP
at alluxio.conf.InstancedConfiguration.lookupRecursively(InstancedConfiguration.java:442)
at alluxio.conf.InstancedConfiguration.lookup(InstancedConfiguration.java:413)
at alluxio.conf.InstancedConfiguration.isResolvable(InstancedConfiguration.java:152)
at alluxio.conf.InstancedConfiguration.isSet(InstancedConfiguration.java:162)
at alluxio.util.network.NetworkAddressUtils.getConnectHost(NetworkAddressUtils.java:332)
at alluxio.util.network.NetworkAddressUtils.getConnectAddress(NetworkAddressUtils.java:273)
at alluxio.util.ConfigurationUtils.getMasterEmbeddedJournalAddresses(ConfigurationUtils.java:78)
at alluxio.util.ConfigurationUtils.getEmbeddedJournalAddresses(ConfigurationUtils.java:62)
at alluxio.util.ConfigurationUtils.getMasterRpcAddresses(ConfigurationUtils.java:115)
at alluxio.util.ConfigurationUtils.isHaMode(ConfigurationUtils.java:242)
at alluxio.master.MasterProcess.configureAddress(MasterProcess.java:100)
at alluxio.master.MasterProcess.(MasterProcess.java:89)
at alluxio.master.AlluxioMasterProcess.(AlluxioMasterProcess.java:113)
at alluxio.master.AlluxioMasterProcess$Factory.create(AlluxioMasterProcess.java:495)
at alluxio.master.AlluxioMaster.main(AlluxioMaster.java:45)

2025-10-10 21:04:02,654 INFO main - Logging metrics and jstack on MASTER exit...
2025-10-10 21:04:02,786 INFO main - Dumped metrics of current process in 131ms to alluxio-master-exit-metrics-20251010-210402.json
2025-10-10...

Fixes #14

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[phrocker]

@copilot was able to get alluxio master out with your change , I fixed worker and manager waits on zk in latest commit; however, now seeing:

  /opt/accumulo/conf/log4j2-service.properties from accumulo-config (rw,path="log4j2-service.properties")
  /opt/accumulo/logs from logs (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzlq6 (ro)

Conditions:
Type Status
PodReadyToStartContainers True
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
accumulo-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: accumulo-config
Optional: false
logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
kube-api-access-bzlq6:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Warning InspectFailed 69s (x347 over 75m) kubelet Failed to apply default image tag "/accumulo/accumulo:4.0.0-SNAPSHOT": couldn't parse image name "/accumulo/accumulo:4.0.0-SNAPSHOT": invalid reference format

Can you fix this? I think this would allow me to deploy the cluster.

[Copilot]

@copilot was able to get alluxio master out with your change , I fixed worker and manager waits on zk in latest commit; however, now seeing:

  /opt/accumulo/conf/log4j2-service.properties from ...

The pod status shows Initialized: False which indicates an init container or startup issue. The output appears cut off - could you share:

1. The complete error/event message from kubectl describe pod
2. Any relevant logs from the failing container
3. What specific issue you'd like me to investigate

This will help me understand if there's something related to the Alluxio configuration that needs adjustment.

[phrocker]

@copilot that was from deescribe pod:

(env) marc@pop-os:/mnt/ExtraDrive/repos/accumulo$ kubectl describe pod accumulo-manager-5b7487cc7c-nx8kt
Name: accumulo-manager-5b7487cc7c-nx8kt
Namespace: default
Priority: 0
Service Account: accumulo
Node: minikube/192.168.49.2
Start Time: Mon, 13 Oct 2025 06:49:19 -0400
Labels: app.kubernetes.io/component=manager
app.kubernetes.io/instance=accumulo
app.kubernetes.io/name=accumulo
pod-template-hash=5b7487cc7c
Annotations:
Status: Pending
IP: 10.244.11.182
IPs:
IP: 10.244.11.182
Controlled By: ReplicaSet/accumulo-manager-5b7487cc7c
Init Containers:
wait-for-zookeeper:
Container ID: docker://84c21cac3f7ea161191779ef916412426326b94186d5a7b054b6310f17e78add
Image: busybox:1.35
Image ID: docker-pullable://busybox@sha256:98ad9d1a2be345201bb0709b0d38655eb1b370145c7d94ca1fe9c421f76e245a
Port:
Host Port:
Command:
/bin/sh
-c
echo "Waiting for ZooKeeper to be ready..."
until nc -z accumulo-zookeeper-zookeeper 2181; do
echo "Waiting for ZooKeeper..."
sleep 5
done
echo "ZooKeeper is ready"

State:          Terminated
  Reason:       Completed
  Exit Code:    0
  Started:      Mon, 13 Oct 2025 06:49:19 -0400
  Finished:     Mon, 13 Oct 2025 06:49:25 -0400
Ready:          True
Restart Count:  0
Environment:    <none>
Mounts:
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzlq6 (ro)

wait-for-alluxio:
Container ID: docker://fede97ba6b944dd66814c8c55491d233328f90fcb4f6104d7e83c21ded382a6b
Image: busybox:1.35
Image ID: docker-pullable://busybox@sha256:98ad9d1a2be345201bb0709b0d38655eb1b370145c7d94ca1fe9c421f76e245a
Port:
Host Port:
Command:
/bin/sh
-c
echo "Waiting for Alluxio master to be ready..."
until nc -z accumulo-alluxio-master 19998; do
echo "Waiting for Alluxio master..."
sleep 5
done
echo "Alluxio master is ready"

State:          Terminated
  Reason:       Completed
  Exit Code:    0
  Started:      Mon, 13 Oct 2025 06:49:25 -0400
  Finished:     Mon, 13 Oct 2025 06:49:44 -0400
Ready:          True
Restart Count:  0
Environment:    <none>
Mounts:
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzlq6 (ro)

init-accumulo:
Container ID:
Image: /accumulo/accumulo:4.0.0-SNAPSHOT
Image ID:
Port:
Host Port:
Command:
/bin/sh
-c
# Check if instance is already initialized
if /opt/accumulo/bin/accumulo org.apache.accumulo.server.util.ListInstances | grep -q "accumulo"; then
echo "Accumulo instance 'accumulo' already exists"
exit 0
fi

  echo "Initializing Accumulo instance 'accumulo'"
  /opt/accumulo/bin/accumulo init \
    --instance-name accumulo \
    --password DEFAULT_CHANGE_ME
  
State:          Waiting
  Reason:       InvalidImageName
Ready:          False
Restart Count:  0
Environment:
  ACCUMULO_INSTANCE_NAME:    accumulo
  ACCUMULO_INSTANCE_SECRET:  <set to the key 'instance-secret' in secret 'accumulo-secret'>  Optional: false
  ZOOKEEPER_HOSTS:           accumulo-zookeeper-zookeeper:2181
  ACCUMULO_LOG_DIR:          /opt/accumulo/logs
  ACCUMULO_HOME:             /opt/accumulo
Mounts:
  /opt/accumulo/conf/accumulo-env.sh from accumulo-config (rw,path="accumulo-env.sh")
  /opt/accumulo/conf/accumulo.properties from accumulo-config (rw,path="accumulo.properties")
  /opt/accumulo/conf/log4j2-service.properties from accumulo-config (rw,path="log4j2-service.properties")
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzlq6 (ro)

Containers:
manager:
Container ID:
Image: /accumulo/accumulo:4.0.0-SNAPSHOT
Image ID:
Ports: 9999/TCP, 10001/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/opt/accumulo/bin/accumulo
manager
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 500m
memory: 512Mi
Liveness: tcp-socket :client delay=60s timeout=10s period=30s #success=1 #failure=3
Readiness: tcp-socket :client delay=30s timeout=5s period=10s #success=1 #failure=3
Environment:
ACCUMULO_INSTANCE_NAME: accumulo
ACCUMULO_INSTANCE_SECRET: <set to the key 'instance-secret' in secret 'accumulo-secret'> Optional: false
ZOOKEEPER_HOSTS: accumulo-zookeeper-zookeeper:2181
ACCUMULO_LOG_DIR: /opt/accumulo/logs
ACCUMULO_HOME: /opt/accumulo
ACCUMULO_SERVICE_INSTANCE: manager
Mounts:
/opt/accumulo/conf/accumulo-env.sh from accumulo-config (rw,path="accumulo-env.sh")
/opt/accumulo/conf/accumulo.properties from accumulo-config (rw,path="accumulo.properties")
/opt/accumulo/conf/log4j2-service.properties from accumulo-config (rw,path="log4j2-service.properties")
/opt/accumulo/logs from logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzlq6 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
accumulo-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: accumulo-config
Optional: false
logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
kube-api-access-bzlq6:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Warning InspectFailed 2m10s (x9184 over 33h) kubelet Failed to apply default image tag "/accumulo/accumulo:4.0.0-SNAPSHOT": couldn't parse image name "/accumulo/accumulo:4.0.0-SNAPSHOT": invalid reference format

[Copilot]

@copilot that was from deescribe pod:

(env) marc@pop-os:/mnt/ExtraDrive/repos/accumulo$ kubectl describe pod accumulo-manager-5b7487cc7c-nx8kt
...

The output is still truncated. Could you share the Events section from the bottom of kubectl describe pod accumulo-manager-5b7487cc7c-nx8kt? That's where the error messages typically appear. Also helpful would be:

• kubectl logs accumulo-manager-5b7487cc7c-nx8kt (or with -c <container-name> if it has init containers)

This will show what's preventing the pod from initializing.

[phrocker]

@copilot it's because of the image tag name;

Events:
Type Reason Age From Message

---

Warning InspectFailed 2m10s (x9184 over 33h) kubelet Failed to apply default image tag "/accumulo/accumulo:4.0.0-SNAPSHOT": couldn't parse image name "/accumulo/accumulo:4.0.0-SNAPSHOT": invalid reference format

[Copilot]

@copilot it's because of the image tag name;

Events:
...

Fixed in commit a12a89c. The image helper templates were incorrectly handling empty registry values, producing /accumulo/accumulo:4.0.0-SNAPSHOT instead of accumulo/accumulo:4.0.0-SNAPSHOT. Updated both accumulo.image and alluxio.image helpers to conditionally include the registry prefix only when it's not empty.

[phrocker]

@copilot still seeing issues with contianer. I don't think accumulo has pushed an official image:

---

Normal Scheduled 93s default-scheduler Successfully assigned default/accumulo-manager-85cbd68896-djml4 to minikube
Normal Pulled 93s kubelet Container image "busybox:1.35" already present on machine
Normal Created 93s kubelet Created container wait-for-zookeeper
Normal Started 93s kubelet Started container wait-for-zookeeper
Normal Pulled 86s kubelet Container image "busybox:1.35" already present on machine
Normal Created 86s kubelet Created container wait-for-alluxio
Normal Started 86s kubelet Started container wait-for-alluxio
Normal Pulling 26s (x3 over 67s) kubelet Pulling image "accumulo/accumulo:4.0.0-SNAPSHOT"
Warning Failed 26s (x3 over 67s) kubelet Failed to pull image "accumulo/accumulo:4.0.0-SNAPSHOT": Error response from daemon: pull access denied for accumulo/accumulo, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Warning Failed 26s (x3 over 67s) kubelet Error: ErrImagePull
Normal BackOff 11s (x2 over 41s) kubelet Back-off pulling image "accumulo/accumulo:4.0.0-SNAPSHOT"
Warning Failed 11s (x2 over 41s) kubelet Error: ImagePullBackOff

[Copilot]

@copilot still seeing issues with contianer. I don't think accumulo has pushed an official image:

---

...

You're correct - the public accumulo/accumulo:4.0.0-SNAPSHOT image doesn't exist. You'll need to build the image locally:

make docker-build

Then either:

1. Load it into your minikube cluster: minikube image load accumulo/accumulo:4.0.0-SNAPSHOT
2. Push to your own registry and update the image configuration in values

The image reference format fix I made ensures the chart will work correctly once the image is available.

[phrocker]

@copilot docker file isn't right:

marc@pop-os:/mnt/ExtraDrive/repos/accumulo$ kubectl logs pod/accumulo-manager-85cbd68896-svbx6 -c init-accumulo --previous
find: ‘/opt/zookeeper/lib/’: No such file or directory
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.accumulo.start.spi.KeywordExecutable: org.apache.accumulo.server.conf.util.ZooInfoViewer Unable to get public no-arg constructor
at java.base/java.util.ServiceLoader.fail(Unknown Source)
at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown Source)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown Source)
at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
at org.apache.accumulo.start.Main.checkDuplicates(Main.java:199)
at org.apache.accumulo.start.Main.getExecutables(Main.java:190)
at org.apache.accumulo.start.Main.main(Main.java:57)
Caused by: java.lang.NoClassDefFoundError: org/apache/zookeeper/KeeperException
at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
at java.base/java.lang.Class.getConstructor0(Unknown Source)
at java.base/java.lang.Class.getConstructor(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
... 15 more
Initializing Accumulo instance 'accumulo'
find: ‘/opt/zookeeper/lib/’: No such file or directory
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.accumulo.start.spi.KeywordExecutable: org.apache.accumulo.server.conf.util.ZooInfoViewer Unable to get public no-arg constructor
at java.base/java.util.ServiceLoader.fail(Unknown Source)
at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown Source)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown Source)
at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
at org.apache.accumulo.start.Main.checkDuplicates(Main.java:199)
at org.apache.accumulo.start.Main.getExecutables(Main.java:190)
at org.apache.accumulo.start.Main.main(Main.java:57)
Caused by: java.lang.NoClassDefFoundError: org/apache/zookeeper/Watcher
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(Unknown Source)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Unknown Source)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
at java.base/java.lang.Class.getConstructor0(Unknown Source)
at java.base/java.lang.Class.getConstructor(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.Watcher
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
... 24 more
marc@pop-os:/mnt/ExtraDrive/repos/accumulo$ cdocker run -it accumulo/accumulo:4.0.0-SNAPSHOT bash -c "ls /opt/zookeeper/lib"
^C
marc@pop-os:/mnt/ExtraDrive/repos/accumulo$ docker run -it accumulo/accumulo:4.0.0-SNAPSHOT bash -c "ls /opt/zookeeper/lib"
Accumulo Docker Container Starting...
Command: bash -c ls /opt/zookeeper/lib
Setting up Accumulo configuration...
find: ‘/opt/zookeeper/lib/’: No such file or directory
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.accumulo.start.spi.KeywordExecutable: org.apache.accumulo.server.conf.util.ZooInfoViewer Unable to get public no-arg constructor
at java.base/java.util.ServiceLoader.fail(Unknown Source)
at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown Source)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown Source)
at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
at org.apache.accumulo.start.Main.checkDuplicates(Main.java:199)
at org.apache.accumulo.start.Main.getExecutables(Main.java:190)
at org.apache.accumulo.start.Main.main(Main.java:57)
Caused by: java.lang.NoClassDefFoundError: org/apache/zookeeper/Watcher
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(Unknown Source)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Unknown Source)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
at java.base/java.lang.Class.getConstructor0(Unknown Source)
at java.base/java.lang.Class.getConstructor(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.util.ServiceLoader$1.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.Watcher
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)

[Copilot]

@copilot docker file isn't right:

marc@pop-os:/mnt/ExtraDrive/repos/accumulo$ kubectl logs pod/accumulo-manager-85cbd68896-svbx6 -c init-accumulo --previous
...

This is a pre-existing issue not related to my changes (I only modified Alluxio env syntax and image helpers). The problem is the ZooKeeper binary distribution structure doesn't match the classpath configuration.

The Dockerfile downloads ZooKeeper to /opt/zookeeper, but the bin distribution has jars in the root directory. The accumulo-env.sh looks for them in ${ZOOKEEPER_HOME}/lib/.

Fix: Update line 93 in configmap.yaml from:

ZK_JARS=$(find "${ZOOKEEPER_HOME}/lib/" -maxdepth 1 -name '*.jar' ...)

to:

ZK_JARS=$(find "${ZOOKEEPER_HOME}" -maxdepth 1 -name '*.jar' ...)

This is outside the scope of my PR but I can make this fix if you'd like.

[phrocker]

@copilot please make this fix. I'd like to resolve all of the issues

[Copilot]

@copilot please make this fix. I'd like to resolve all of the issues

Fixed in commit that's being pushed. Changed the ZooKeeper classpath lookup from ${ZOOKEEPER_HOME}/lib/ to ${ZOOKEEPER_HOME} to match the binary distribution's directory structure. This should resolve the ClassNotFoundException: org.apache.zookeeper.KeeperException error.