Travis build fails in s390x architecture

[Rohanjos97]

Currently, our Dockerfile looks like this

FROM icr.io/appcafe/websphere-liberty:24.0.0.6-kernel-java11-openj9-ubi

...

COPY --chown=1001:0 server.xml /opt/ibm/wlp/usr/servers/defaultServer/server.xml
RUN configure.sh

...

And the build passes for amd64 archtecture. But when the same code executes for s390x architecture builds, we are seeing some error that is originating from populate_scc.sh file, specifically travis build fails after server stop command with this output:

310.5 Stopping server defaultServer.
314.2 The command stop failed because of a communication error with the server.
314.2 Server defaultServer stop failed. Check server logs for details.

Tried to debug it using sed command by printing the console.log file after the server start and server stop command:

RUN sed -i "s|\/opt\/ibm\/wlp\/bin\/server start|\/opt\/ibm\/wlp\/bin\/server start \&\& ls \/logs \&\& \-al \&\& cat \/logs\/console.log|g" /opt/ibm/helpers/build/populate_scc.sh
RUN sed -i "s|\/opt\/ibm\/wlp\/bin\/server stop|\/opt\/ibm\/wlp\/bin\/server stop \&\& ls \-al \/logs \&\& cat \/logs\/console.log|g" /opt/ibm/helpers/build/populate_scc.sh

This was the console.log output after server start command:

#9 304.7 Launching defaultServer (WebSphere Application Server 24.0.0.6/wlp-1.0.90.cl240620240603-2001) on Eclipse OpenJ9 VM, version 11.0.24+8 (en_US)
#9 304.7 [AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
#9 304.7 [AUDIT   ] CWWKE0100I: This product is licensed for development, and limited production use. The full license terms can be viewed here: https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/license/base_ilan/ilan/24.0.0.6/lafiles/en.html
#9 304.7 [AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ibm/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
#9 304.7 [AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
#9 304.7 [ERROR   ] CWSJY0035E: Unable to load classes from the WebSphere MQ resource adapter, because the wmqJmsClient.rar.location variable has not been set. Set this variable to point to the location of the MQ resource adapter
#9 304.7 [ERROR   ] CWWKE0701E: bundle com.ibm.ws.jca:1.0.90.cl240620240603-2001 (98)[com.ibm.ws.jca.internal.ResourceAdapterService(219)] : The activate method has thrown an exception java.lang.IllegalArgumentException: CWSJY0035E: Unable to load classes from the WebSphere MQ resource adapter, because the wmqJmsClient.rar.location variable has not been set. Set this variable to point to the location of the MQ resource adapter
#9 304.7 	at com.ibm.ws.wmq.WMQService.setClassLoaderID(WMQService.java:205)
#9 304.7 	at [internal classes]
#9 304.7   
#9 304.7 [AUDIT   ] CWWKS4104A: LTPA keys created in 7.658 seconds. LTPA key file: /opt/ibm/wlp/output/defaultServer/resources/security/ltpa.keys
#9 304.7 [AUDIT   ] CWPKI0803A: SSL certificate created in 23.795 seconds. SSL key file: /opt/ibm/wlp/output/defaultServer/resources/security/key.p12
#9 304.7 [AUDIT   ] CWWKI0001I: The CORBA name server is now available at corbaloc:iiop:localhost:2809/NameService.
#9 304.7 [ERROR   ] CWSJY0035E: Unable to load classes from the WebSphere MQ resource adapter, because the wmqJmsClient.rar.location variable has not been set. Set this variable to point to the location of the MQ resource adapter
#9 304.7 [ERROR   ] CWWKE0701E: bundle com.ibm.ws.jca:1.0.90.cl240620240603-2001 (98)[com.ibm.ws.jca.internal.ResourceAdapterService(219)] : The activate method has thrown an exception java.lang.IllegalArgumentException: CWSJY0035E: Unable to load classes from the WebSphere MQ resource adapter, because the wmqJmsClient.rar.location variable has not been set. Set this variable to point to the location of the MQ resource adapter
#9 304.7 	at com.ibm.ws.wmq.WMQService.setClassLoaderID(WMQService.java:205)
#9 304.7 	at [internal classes]
#9 304.7   
#9 304.7 [AUDIT   ] CWWKT0016I: Web application available (default_host): http://localhost:9080/jwt/
#9 304.7 [AUDIT   ] CWWKT0016I: Web application available (default_host): http://localhost:9080/oidcclient/
#9 304.7 [AUDIT   ] CWWKT0016I: Web application available (default_host): http://localhost:9080/oauth2/
#9 304.7 [AUDIT   ] CWWKF0012I: The server installed the following features: [appSecurity-2.0, appSecurity-3.0, beanValidation-2.0, cdi-2.0, distributedMap-1.0, ejbHome-3.2, ejbLite-3.2, ejbRemote-3.2, el-3.0, federatedRegistry-1.0, jaspic-1.1, javaMail-1.6, jaxb-2.2, jaxrs-2.1, jaxrsClient-2.1, jaxws-2.2, jca-1.7, jdbc-4.2, jmsMdb-3.2, jndi-1.0, jpa-2.2, jpaContainer-2.2, jsf-2.3, json-1.0, jsonb-1.0, jsonp-1.1, jsp-2.3, jwt-1.0, jwtSso-1.0, ldapRegistry-3.0, managedBeans-1.0, mdb-3.2, monitor-1.0, oauth-2.0, openidConnectClient-1.0, servlet-4.0, ssl-1.0, transportSecurity-1.0, wasJmsClient-2.0, wasJmsServer-1.0, webProfile-8.0, websocket-1.1, wmqJmsClient-2.0].
#9 304.7 [AUDIT   ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 95.428 seconds.

I noticed that the [ERROR] message was being printed on the amd64 side as well (not the probable cause of failure for our build)...

The output of console.log after server stop command:

#9 304.6 + cat /logs/console.log
#9 304.7 Launching defaultServer (WebSphere Application Server 24.0.0.6/wlp-1.0.90.cl240620240603-2001) on Eclipse OpenJ9 VM, version 11.0.24+8 (en_US)
#9 304.7 [AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
#9 304.7 [AUDIT   ] CWWKE0100I: This product is licensed for development, and limited production use. The full license terms can be viewed here: https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/license/base_ilan/ilan/24.0.0.6/lafiles/en.html
#9 304.7 [AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ibm/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
#9 304.7 [AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
#9 304.7 [ERROR   ] CWSJY0035E: Unable to load classes from the WebSphere MQ resource adapter, because the wmqJmsClient.rar.location variable has not been set. Set this variable to point to the location of the MQ resource adapter
#9 304.7 [ERROR   ] CWWKE0701E: bundle com.ibm.ws.jca:1.0.90.cl240620240603-2001 (98)[com.ibm.ws.jca.internal.ResourceAdapterService(219)] : The activate method has thrown an exception java.lang.IllegalArgumentException: CWSJY0035E: Unable to load classes from the WebSphere MQ resource adapter, because the wmqJmsClient.rar.location variable has not been set. Set this variable to point to the location of the MQ resource adapter
#9 304.7 	at com.ibm.ws.wmq.WMQService.setClassLoaderID(WMQService.java:205)
#9 304.7 	at [internal classes]
#9 304.7   
#9 304.7 [AUDIT   ] CWWKS4104A: LTPA keys created in 7.658 seconds. LTPA key file: /opt/ibm/wlp/output/defaultServer/resources/security/ltpa.keys
#9 304.7 [AUDIT   ] CWPKI0803A: SSL certificate created in 23.795 seconds. SSL key file: /opt/ibm/wlp/output/defaultServer/resources/security/key.p12
#9 304.7 [AUDIT   ] CWWKI0001I: The CORBA name server is now available at corbaloc:iiop:localhost:2809/NameService.
#9 304.7 [ERROR   ] CWSJY0035E: Unable to load classes from the WebSphere MQ resource adapter, because the wmqJmsClient.rar.location variable has not been set. Set this variable to point to the location of the MQ resource adapter
#9 304.7 [ERROR   ] CWWKE0701E: bundle com.ibm.ws.jca:1.0.90.cl240620240603-2001 (98)[com.ibm.ws.jca.internal.ResourceAdapterService(219)] : The activate method has thrown an exception java.lang.IllegalArgumentException: CWSJY0035E: Unable to load classes from the WebSphere MQ resource adapter, because the wmqJmsClient.rar.location variable has not been set. Set this variable to point to the location of the MQ resource adapter
#9 304.7 	at com.ibm.ws.wmq.WMQService.setClassLoaderID(WMQService.java:205)
#9 304.7 	at [internal classes]
#9 304.7   
#9 304.7 [AUDIT   ] CWWKT0016I: Web application available (default_host): http://localhost:9080/jwt/
#9 304.7 [AUDIT   ] CWWKT0016I: Web application available (default_host): http://localhost:9080/oidcclient/
#9 304.7 [AUDIT   ] CWWKT0016I: Web application available (default_host): http://localhost:9080/oauth2/
#9 304.7 [AUDIT   ] CWWKF0012I: The server installed the following features: [appSecurity-2.0, appSecurity-3.0, beanValidation-2.0, cdi-2.0, distributedMap-1.0, ejbHome-3.2, ejbLite-3.2, ejbRemote-3.2, el-3.0, federatedRegistry-1.0, jaspic-1.1, javaMail-1.6, jaxb-2.2, jaxrs-2.1, jaxrsClient-2.1, jaxws-2.2, jca-1.7, jdbc-4.2, jmsMdb-3.2, jndi-1.0, jpa-2.2, jpaContainer-2.2, jsf-2.3, json-1.0, jsonb-1.0, jsonp-1.1, jsp-2.3, jwt-1.0, jwtSso-1.0, ldapRegistry-3.0, managedBeans-1.0, mdb-3.2, monitor-1.0, oauth-2.0, openidConnectClient-1.0, servlet-4.0, ssl-1.0, transportSecurity-1.0, wasJmsClient-2.0, wasJmsServer-1.0, webProfile-8.0, websocket-1.1, wmqJmsClient-2.0].
#9 304.7 [AUDIT   ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 95.428 seconds.
#9 304.7 + '[' true == true ']'
#9 304.7 + curl --silent --output /dev/null --show-error --fail --max-time 5 localhost:9080/
#9 307.0 + '[' true == true ']'
#9 307.0 + curl --silent --output /dev/null --show-error --fail --max-time 5 localhost:9080/openapi
#9 307.2 curl: (22) The requested URL returned error: 404 Not Found
#9 307.2 + echo 'localhost:9080/openapi call failed, continuing'
#9 307.2 + /opt/ibm/wlp/bin/server stop
#9 307.2 localhost:9080/openapi call failed, continuing
#9 307.7 
#9 310.5 Stopping server defaultServer.
#9 314.2 The command stop failed because of a communication error with the server.
#9 314.2 Server defaultServer stop failed. Check server logs for details.

We then checked if the server is stopping successfully or is there's an issue during stop, so we printed server stop || sever status
and the output was:

#9 340.8 + /opt/ibm/wlp/bin/server status
1057#9 341.3 
1058#9 344.5 Server defaultServer is not running.

Even though server stop command returned non zero code, the server stopped anyway according to the console log above. So, for a workaround for now, we are skipping the error for s390x build by adding || true after server stop command, which looks something like this:

RUN if [ "$ARCHITECTURE" != "amd64" ] ; then \
        sed -i "s|\/opt\/ibm\/wlp\/bin\/server stop|\/opt\/ibm\/wlp\/bin\/server stop \|\| true|g" /opt/ibm/helpers/build/populate_scc.sh; \
    fi

I was hoping to get some more insights on why the build is failing and if we can add similar logic to the code for s390x builds, or better, if we can figure out what is causing the issue to fix at the root source.

[leochr]

Hi @Rohanjos97 Thanks for reaching out.

Is the failure occurring on s390x consistently? or is it intermittent?
When it fails, does it always fail on the server stop command (in populate_scc.sh)? or does it also fail on server start?

Apart from the 'communication error' message, I don't see anything else to indicate the reason for the failure.

Please print out the start and stop logs to see if they contain any additional information. Thank you

  /opt/ol/wlp/bin/server start  || { ec=$?; echo "==> start failed" >&2; ls -la /logs >&2; cat /logs/messages.log >&2; cat /logs/console.log >&2; cat /logs/start.log >&2; exit $ec; }

...

  /opt/ol/wlp/bin/server stop  || { ec=$?; echo "==> stop failed" >&2; ls -la /logs >&2; cat /logs/messages.log >&2; cat /logs/console.log >&2; cat /logs/stop.log >&2; exit $ec; }

[Rohanjos97]

Hi @leochr,

Yes, this failure is consistent in all our s390x travis builds...

In some of my previous failed builds, I was seeing this error at RUN configure.sh command which was intermittent. I couldn't make much out of it, but maybe it is helpful?

#8 [ 4/28] RUN configure.sh
#8 0.337 WARNING: This is not an optimal build configuration. Although features in server.xml will continue to be installed correctly, the 'RUN features.sh' command should be added to the Dockerfile prior to configure.sh. See https://ibm.biz/wl-app-image-template for a sample application image template.
#8 0.394 + main
#8 0.398 + WLP_INSTALL_DIR=/opt/ibm/wlp
#8 0.401 + SHARED_CONFIG_DIR=/opt/ibm/wlp/usr/shared/config
#8 0.402 + SHARED_RESOURCE_DIR=/opt/ibm/wlp/usr/shared/resources
#8 0.402 + SNIPPETS_SOURCE=/opt/ibm/helpers/build/configuration_snippets
#8 0.402 + SNIPPETS_TARGET=/config/configDropins/overrides
#8 0.402 + SNIPPETS_TARGET_DEFAULTS=/config/configDropins/defaults
#8 0.404 + mkdir -p /config/configDropins/overrides
#8 0.470 + mkdir -p /config/configDropins/defaults
#8 0.543 + '[' false == false ']'
#8 0.544 + '[' '' == true ']'
#8 0.545 + '[' false == true ']'
#8 0.545 + '[' '' == true ']'
#8 0.545 + '[' false == true ']'
#8 0.546 + '[' false == true ']'
#8 0.546 + '[' '' == true ']'
#8 0.547 + '[' false == true ']'
#8 0.547 + '[' false == true ']'
#8 0.549 + [[ -n '' ]]
#8 0.550 + '[' false == client ']'
#8 0.551 + '[' false == embedded ']'
#8 0.551 + [[ -n '' ]]
#8 0.551 + '[' '' == true ']'
#8 0.551 + '[' '' == true ']'
#8 0.551 + keystorePath=/config/configDropins/defaults/keystore.xml
#8 0.552 + '[' '' '!=' false ']'
#8 0.552 + '[' '' '!=' false ']'
#8 0.552 + '[' '!' -e /config/configDropins/defaults/keystore.xml ']'
#8 0.562 ++ openssl rand -base64 32
#8 0.643 + export KEYSTOREPWD=sEW5TT/5rW6HMMbCEVGTonGftfCNnGHukGSqomgbguA=
#8 0.645 + KEYSTOREPWD=sEW5TT/5rW6HMMbCEVGTonGftfCNnGHukGSqomgbguA=
#8 0.648 + sed 's|REPLACE|sEW5TT/5rW6HMMbCEVGTonGftfCNnGHukGSqomgbguA=|g' /opt/ibm/helpers/build/configuration_snippets/keystore.xml
#8 0.725 + chmod g+w /config/configDropins/defaults/keystore.xml
#8 0.785 + [[ -n '' ]]
#8 0.785 + '[' '' '!=' true ']'
#8 0.785 + '[' false == false ']'
#8 0.786 + '[' '' ']'
#8 0.786 + installUtility install --acceptLicense defaultServer
#8 6.011 Checking for missing features required by the server ...
#8 6.494 Establishing a connection to the configured repositories ...
#8 6.494 This process might take several minutes to complete.
#8 6.494 
#8 12.49 Successfully connected to all configured repositories.
#8 12.49 
#8 12.50 Preparing assets for installation. This process might take several minutes to complete.
#8 44.66 Unhandled exception
#8 44.66 Type=Floating point error vmState=0x00000000
#8 44.66 J9Generic_Signal_Number=00000088 Signal_Number=00000008 Error_Value=00000000 Signal_Code=00000000
#8 44.66 Handler1=0000004008DE5F50 Handler2=000000400909FA60
#8 44.66 gpr0=0000000000000000 gpr1=000000000D4A85C8 gpr2=000000000DA661A8 gpr3=00000000F4BBBA48
#8 44.66 gpr4=0000000000000000 gpr5=000000000016C1A0 gpr6=0000000000000000 gpr7=0000000000000000
#8 44.66 gpr8=00000001E97775C0 gpr9=000000000DA661A8 gpr10=000000000DA661A8 gpr11=00000001E9777480
#8 44.66 gpr12=00000001E9777480 gpr13=000000000008E200 gpr14=000000401D6951B8 gpr15=0000004000AB5B98
#8 44.66 psw=000000401D695322 mask=0705200180000000 fpc=00000000 bea=0000000000000000
#8 44.67 fpr0=000000400a1c9dc8 (f: 169647552.000000, d: 1.358915e-312)
#8 44.67 fpr1=ff00000000000000 (f: 0.000000, d: -5.486124e+303)
#8 44.67 fpr2=5700000000000000 (f: 0.000000, d: 1.202454e+111)
#8 44.67 fpr3=2020202020202020 (f: 538976256.000000, d: 6.013470e-154)
#8 44.67 fpr4=617ae0f6f8fe0000 (f: 4177395712.000000, d: 3.778914e+161)
#8 44.67 fpr5=a0c0a0c0a0c00000 (f: 2696937472.000000, d: -6.349662e-151)
#8 44.67 fpr6=dfdfb5b5ffff0000 (f: 4294901760.000000, d: -6.643110e+153)
#8 44.67 fpr7=80808080a0a00000 (f: 2694840320.000000, d: -2.937447e-306)
#8 44.67 fpr8=00000000000644c0 (f: 410816.000000, d: 2.029701e-318)
#8 44.67 fpr9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
#8 44.67 fpr10=0000000000064be0 (f: 412640.000000, d: 2.038712e-318)
#8 44.67 fpr11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
#8 44.67 fpr12=0000004000ab7910 (f: 11237648.000000, d: 1.358133e-312)
#8 44.67 fpr13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
#8 44.67 fpr14=0000000000040000 (f: 262144.000000, d: 1.295163e-318)
#8 44.67 fpr15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
#8 44.67 
#8 44.67 Compiled_method=org/glassfish/json/JsonObjectBuilderImpl$JsonObjectImpl.getString(Ljava/lang/String;)Ljava/lang/String;
#8 44.67 Target=2_90_20240802_1032 (Linux 4.19.86)
#8 44.67 CPU=s390x (8 logical CPUs) (0x77c0c1000 RAM)
#8 44.67 ----------- Stack Backtrace -----------
#8 44.67  (0x000000401D695322 [<unknown>+0x0])
#8 44.67 ---------------------------------------
#8 44.68 JVMDUMP039I Processing dump event "gpf", detail "" at 2024/08/29 17:20:27 - please wait.
#8 44.69 JVMDUMP032I JVM requested System dump using '//core.20240829.172027.40.0001.dmp' in response to an event
#8 46.10 JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/lib/systemd/coredump-wrapper %E %P %u %g %s %t %c %h %e" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.78.  Review the manual for the external program to find where the core dump is written and ensure the program does not truncate it.
#8 46.10 
#8 52.02 JVMPORT049I The core file created by child process with pid = 78 was not found. Review the documentation for the /proc/sys/kernel/core_pattern program "|/usr/lib/systemd/coredump-wrapper %E %P %u %g %s %t %c %h %e" to find where the core file is written and ensure that program does not truncate it.
#8 52.02 
#8 52.05 JVMDUMP012E Error in System dump: //core.20240829.172027.40.0001.dmp
#8 52.06 JVMDUMP032I JVM requested Java dump using '//javacore.20240829.172027.40.0002.txt' in response to an event
#8 52.22 /opt/ibm/wlp/bin/installUtility: line 218:    40 Real-time signal 0      "${JAVA_CMD}" "$@"
#8 52.22 + rc=162
#8 52.23 + '[' 162 -ne 22 ']'
#8 52.23 + exit 162
#8 ERROR: process "/dev/.buildkit_qemu_emulator /bin/sh -c configure.sh" did not complete successfully: exit code: 162

There was no /opt/ol/wlp/bin/server directory, so I changed it according to our code /opt/ibm/wlp/bin/server and the output was:

#7 [ 3/30] RUN /opt/ibm/wlp/bin/server start  || { ec=$?; echo "==> start failed" >&2; ls -la /logs >&2; cat /logs/messages.log >&2; cat /logs/console.log >&2; cat /logs/start.log >&2; exit $ec; }
#7 0.625 
#7 4.067 Starting server defaultServer.
#7 12.03 Server defaultServer started with process ID 43.
#7 DONE 12.3s

Here's the output of server stop command:

#8 [ 4/30] RUN /opt/ibm/wlp/bin/server stop  || { ec=$?; echo "==> stop failed" >&2; ls -la /logs >&2; cat /logs/messages.log >&2; cat /logs/console.log >&2; cat /logs/stop.log >&2; exit $ec; }
#8 0.592 
#8 3.624 Stopping server defaultServer.
#8 3.684 Server defaultServer is not running.
#8 3.878 ==> stop failed
#8 3.968 total 112
#8 3.968 drwxrwxr-x 1 default root    22 Sep  5 07:38 .
#8 3.968 drwxr-xr-x 1 root    root    40 Sep  5 07:37 ..
#8 3.968 -rw-r----- 1 root    root  6299 Sep  5 07:37 console.log
#8 3.968 -rw-r----- 1 root    root  9061 Sep  5 07:37 messages.log
#8 3.968 -rw-r----- 1 root    root   372 Sep  5 07:38 stop.log
#8 3.968 -rw-r----- 1 root    root 90080 Sep  5 07:37 verbosegc.001.log
#8 4.029 ********************************************************************************
#8 4.029 product = WebSphere Application Server 24.0.0.6 (wlp-1.0.90.cl240620240603-2001)
#8 4.029 wlp.install.dir = /opt/ibm/wlp/
#8 4.029 server.output.dir = /opt/ibm/wlp/output/defaultServer/
#8 4.029 java.home = /opt/java/openjdk
#8 4.029 java.version = 11.0.24
#8 4.029 java.runtime = IBM Semeru Runtime Open Edition (11.0.24+8)
#8 4.029 os = Linux (4.19.86; s390x) (en_US)
#8 4.029 process = 43@127.0.0.1
#8 4.029 Classpath = /opt/ibm/wlp/bin/tools/ws-server.jar
#8 4.029 Java Library path = /opt/java/openjdk/lib/default:/opt/java/openjdk/lib:/usr/lib64:/usr/lib
#8 4.029 ********************************************************************************
#8 4.029 [9/5/24, 7:37:50:865 UTC] 00000001 com.ibm.ws.kernel.launch.internal.FrameworkManager           A CWWKE0001I: The server defaultServer has been launched.
#8 4.029 [9/5/24, 7:37:50:963 UTC] 00000001 com.ibm.ws.kernel.launch.internal.FrameworkManager           A CWWKE0100I: This product is licensed for development, and limited production use. The full license terms can be viewed here: https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/license/base_ilan/ilan/24.0.0.6/lafiles/en.html
#8 4.029 [9/5/24, 7:37:56:327 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            I CWWKF0007I: Feature update started.
#8 4.029 [9/5/24, 7:37:56:340 UTC] 00000001 com.ibm.ws.kernel.launch.internal.FrameworkManager           I CWWKE0002I: The kernel started after 7.186 seconds
#8 4.029 [9/5/24, 7:37:56:622 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the ssl-1.0 feature.  Try running the command, bin/installUtility install ssl-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:631 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the wmqjmsclient-2.0 feature.  Try running the command, bin/installUtility install wmqjmsclient-2.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:734 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the jndi-1.0 feature.  Try running the command, bin/installUtility install jndi-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:812 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the ejbremote-3.2 feature.  Try running the command, bin/installUtility install ejbremote-3.2, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:819 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the openidconnectclient-1.0 feature.  Try running the command, bin/installUtility install openidconnectclient-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:826 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the jdbc-4.2 feature.  Try running the command, bin/installUtility install jdbc-4.2, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:833 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the jaxws-2.2 feature.  Try running the command, bin/installUtility install jaxws-2.2, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:839 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the jsonp-1.1 feature.  Try running the command, bin/installUtility install jsonp-1.1, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:847 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the javamail-1.6 feature.  Try running the command, bin/installUtility install javamail-1.6, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:853 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the ejbhome-3.2 feature.  Try running the command, bin/installUtility install ejbhome-3.2, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:861 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the wasjmsclient-2.0 feature.  Try running the command, bin/installUtility install wasjmsclient-2.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:867 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the transportsecurity-1.0 feature.  Try running the command, bin/installUtility install transportsecurity-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:874 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the webprofile-8.0 feature.  Try running the command, bin/installUtility install webprofile-8.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:881 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the jwtsso-1.0 feature.  Try running the command, bin/installUtility install jwtsso-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:894 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the monitor-1.0 feature.  Try running the command, bin/installUtility install monitor-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:929 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the wasjmsserver-1.0 feature.  Try running the command, bin/installUtility install wasjmsserver-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:56:942 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0042E: A feature definition cannot be found for the jmsmdb-3.2 feature.  Try running the command, bin/installUtility install jmsmdb-3.2, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.029 [9/5/24, 7:37:57:186 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            A CWWKF0012I: The server installed the following features: [].
#8 4.029 [9/5/24, 7:37:57:191 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            I CWWKF0008I: Feature update completed in 0.885 seconds.
#8 4.029 [9/5/24, 7:37:57:193 UTC] 00000034 com.ibm.ws.kernel.feature.internal.FeatureManager            A CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 8.062 seconds.
#8 4.092 Launching defaultServer (WebSphere Application Server 24.0.0.6/wlp-1.0.90.cl240620240603-2001) on Eclipse OpenJ9 VM, version 11.0.24+8 (en_US)
#8 4.092 [AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
#8 4.092 [AUDIT   ] CWWKE0100I: This product is licensed for development, and limited production use. The full license terms can be viewed here: https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/license/base_ilan/ilan/24.0.0.6/lafiles/en.html
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the ssl-1.0 feature.  Try running the command, bin/installUtility install ssl-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the wmqjmsclient-2.0 feature.  Try running the command, bin/installUtility install wmqjmsclient-2.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the jndi-1.0 feature.  Try running the command, bin/installUtility install jndi-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the ejbremote-3.2 feature.  Try running the command, bin/installUtility install ejbremote-3.2, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the openidconnectclient-1.0 feature.  Try running the command, bin/installUtility install openidconnectclient-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the jdbc-4.2 feature.  Try running the command, bin/installUtility install jdbc-4.2, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the jaxws-2.2 feature.  Try running the command, bin/installUtility install jaxws-2.2, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the jsonp-1.1 feature.  Try running the command, bin/installUtility install jsonp-1.1, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the javamail-1.6 feature.  Try running the command, bin/installUtility install javamail-1.6, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the ejbhome-3.2 feature.  Try running the command, bin/installUtility install ejbhome-3.2, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the wasjmsclient-2.0 feature.  Try running the command, bin/installUtility install wasjmsclient-2.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the transportsecurity-1.0 feature.  Try running the command, bin/installUtility install transportsecurity-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the webprofile-8.0 feature.  Try running the command, bin/installUtility install webprofile-8.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the jwtsso-1.0 feature.  Try running the command, bin/installUtility install jwtsso-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the monitor-1.0 feature.  Try running the command, bin/installUtility install monitor-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the wasjmsserver-1.0 feature.  Try running the command, bin/installUtility install wasjmsserver-1.0, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [ERROR   ] CWWKF0042E: A feature definition cannot be found for the jmsmdb-3.2 feature.  Try running the command, bin/installUtility install jmsmdb-3.2, to install the feature. Alternatively, you can run the command, bin/installUtility install defaultServer, to install all features that are referenced by this configuration.
#8 4.092 [AUDIT   ] CWWKF0012I: The server installed the following features: [].
#8 4.092 [AUDIT   ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 8.062 seconds.
#8 4.159 
#8 4.159 arg0=defaultServer
#8 4.159 arg1=--pid=43
#8 4.159 arg2=--stop
#8 4.159 exit=1
#8 4.159 
#8 4.159                  Java home:  /opt/java/openjdk
#8 4.159               Install root:  /opt/ibm/wlp/
#8 4.159           System libraries:  /opt/ibm/wlp/lib/
#8 4.159                  User root:  /opt/ibm/wlp/usr/
#8 4.159                     Config:  /opt/ibm/wlp/usr/servers/defaultServer/
#8 4.159                     Output:  /opt/ibm/wlp/output/defaultServer/
#8 4.159 
#8 ERROR: process "/dev/.buildkit_qemu_emulator /bin/sh -c /opt/ibm/wlp/bin/server stop  || { ec=$?; echo \"==> stop failed\" >&2; ls -la /logs >&2; cat /logs/messages.log >&2; cat /logs/console.log >&2; cat /logs/stop.log >&2; exit $ec; }" did not complete successfully: exit code: 1
------
 > [ 4/30] RUN /opt/ibm/wlp/bin/server stop  || { ec=$?; echo "==> stop failed" >&2; ls -la /logs >&2; cat /logs/messages.log >&2; cat /logs/console.log >&2; cat /logs/stop.log >&2; exit $ec; }:
4.159 arg2=--stop
4.159 exit=1
4.159 
4.159                  Java home:  /opt/java/openjdk
4.159               Install root:  /opt/ibm/wlp/
4.159           System libraries:  /opt/ibm/wlp/lib/
4.159                  User root:  /opt/ibm/wlp/usr/
4.159                     Config:  /opt/ibm/wlp/usr/servers/defaultServer/
4.159                     Output:  /opt/ibm/wlp/output/defaultServer/
4.159 
------
Dockerfile:52
--------------------
  50 |     RUN /opt/ibm/wlp/bin/server start  || { ec=$?; echo "==> start failed" >&2; ls -la /logs >&2; cat /logs/messages.log >&2; cat /logs/console.log >&2; cat /logs/start.log >&2; exit $ec; }
  51 |     
  52 | >>> RUN /opt/ibm/wlp/bin/server stop  || { ec=$?; echo "==> stop failed" >&2; ls -la /logs >&2; cat /logs/messages.log >&2; cat /logs/console.log >&2; cat /logs/stop.log >&2; exit $ec; }
  53 |     
  54 |     RUN sed -i "s|\/opt\/ibm\/wlp\/bin\/server start|\/opt\/ibm\/wlp\/bin\/server start || { ec\=\$\?\; echo \"==> start failed\" \>\&2\; ls \-la \/logs \>\&2\; cat \/logs\/messages.log \>\&2\; cat \/logs\/console.log \>\&2\; cat \/logs\/start.log \>\&2\; exit \$ec\; }|g" /opt/ibm/helpers/build/populate_scc.sh
--------------------
ERROR: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c /opt/ibm/wlp/bin/server stop  || { ec=$?; echo \"==> stop failed\" >&2; ls -la /logs >&2; cat /logs/messages.log >&2; cat /logs/console.log >&2; cat /logs/stop.log >&2; exit $ec; }" did not complete successfully: exit code: 1

Let me know if anything else is required.
Thanks!

[leochr]

@Rohanjos97 Thank you for providing the logs.

@ymanton Error occurs during SCC generation (populate_scc.sh). Noticed the errors below. Could you please take a look to see if this is same as the the other issue (segmentation error / findSharedClassImpl3)? Thank you.

#8 12.50 Preparing assets for installation. This process might take several minutes to complete.
#8 44.66 Unhandled exception
#8 44.66 Type=Floating point error vmState=0x00000000
#8 44.66 J9Generic_Signal_Number=00000088 Signal_Number=00000008 Error_Value=00000000 Signal_Code=00000000
....
....
----------- Stack Backtrace -----------
#8 44.67  (0x000000401D695322 [<unknown>+0x0])
#8 44.67 ---------------------------------------
#8 44.68 JVMDUMP039I Processing dump event "gpf", detail "" at 2024/08/29 17:20:27 - please wait.
#8 44.69 JVMDUMP032I JVM requested System dump using '//core.20240829.172027.40.0001.dmp' in response to an event
#8 46.10 JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/lib/systemd/coredump-wrapper %E %P %u %g %s %t %c %h %e" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.78.  Review the manual for the external program to find where the core dump is written and ensure the program does not truncate it.
#8 46.10 
#8 52.02 JVMPORT049I The core file created by child process with pid = 78 was not found. Review the documentation for the /proc/sys/kernel/core_pattern program "|/usr/lib/systemd/coredump-wrapper %E %P %u %g %s %t %c %h %e" to find where the core file is written and ensure that program does not truncate it.
#8 52.02 
#8 52.05 JVMDUMP012E Error in System dump: //core.20240829.172027.40.0001.dmp
#8 52.06 JVMDUMP032I JVM requested Java dump using '//javacore.20240829.172027.40.0002.txt' in response to an event

[ymanton]

No, this is a different kind of error. The JVM is crashing while executing JIT compiled code because of a floating point exception. FYI @r30shah.

[leochr]

Thank you for the confirmation Younes.

The floating point exception occurs intermittently, but server stop failure is consistent, at least on s390x. I think we should focus on the server stop failure during SCC generation.

@Rohanjos97 Are the latest logs from starting and stopping the server within populate_scc.sh? Noticed some feature definition missing error (CWWKF0042E), so wanted to clarify. Thank you

[ymanton]

I don't see any output from the JVM in any of the logs after server stop, but the process looks like its no longer running.

That suggests that it was killed forcibly without giving the JVM a chance to handle the error. Typically that happens when the system is low on memory.

Is it possible to check the kernel log after server stop fails? Can you execute the dmesg command once the container image build fails and capture the output? If dmesg doesn't work on Travis something like sudo journalctl might

[r30shah]

Regarding to failures in the JIT compiled code posted in #635 (comment), I see that JVM tries to produce the system dump at the time of the failure, but could not (May be due core pattern is not set inside container / some other limitations on the container), if this is consistently reproducible, would it be possible to get the core-dump ?

I am not sure if this is reproducible using one of the template servers, will try couple to see, but in order to diagnosis the segmentation fault, would need core-dump to see why it failed with floating point error.

[Rohanjos97]

@Rohanjos97 Are the latest logs from starting and stopping the server within populate_scc.sh? Noticed some feature definition missing error (CWWKF0042E), so wanted to clarify. Thank you

Hi @leochr, sorry for the late response, yes the logs are from within populate_scc.sh file as I modified the code from inside my container using this sed command
RUN sed -i "s|\/opt\/ibm\/wlp\/bin\/server stop|\/opt\/ibm\/wlp\/bin\/server stop \|\| cat \/logs\/console.log \&\& \/opt\/ibm\/wlp\/bin\/server status|g" /opt/ibm/helpers/build/populate_scc.sh

[leochr]

@ymanton @r30shah Thank you for taking a look and suggesting the diagnostic materials required for further investigation.

@Rohanjos97 Could you please gather them? Thank you

[Rohanjos97]

Hi all,
I tried to get the logs using dmesg and journalctl command as suggested by @ymanton, but I was not able to fetch the logs of dmesg command

output of sudo dmesg :

#8 329.0 Stopping server defaultServer.
#8 332.6 The command stop failed because of a communication error with the server.
#8 332.6 Server defaultServer stop failed. Check server logs for details.
#8 332.7 + sudo dmesg
#8 332.8 dmesg: read kernel buffer failed: Operation not permitted
#8 332.8 + cat /logs/console.log

I'm running these commands with root user and tried to install sudo, but still seeing the same error for s390x achitecture...

output of journalctl :

#8 301.7 + journalctl
#8 301.8 No journal files were found.
#8 301.8 -- No entries --

I'm trying to print the output of dmesg as suggested by @ymanton once I manage to get past the sudo error, I'll post it here. Meanwhile, lemme know if you want me to try some other commands.

@r30shah Please refer the attached file for the dumpfile that was in the location: opt/ibm/wlp/output/defaultServer/javacore.20240911.071456.259.0002.txt.

Please let me know if I missed something/ more information is required.

dump_file.txt

Thanks!

[Rohanjos97]

Hi All,
I resolved the sudo error that I was getting earlier on usingdmesg command, but it seems it wasn't a sudo error, but something else... been trying to work it out, but couldn't get to print any logs even after fixing sudo.

I tried to install rsyslog and check if I can print logs from /var/log/syslog, but that also doesn't show up.

There are all the files available at the /var/log location, please let me know if you require any of these logs to investigate.

Thanks!

[ymanton]

Try sudo journalctl rather than just journalctl. If that still doesn't produce anything try:

sudo grep -r "Killed process" /var/log/*

[Rohanjos97]

Hi @ymanton,

I tried both commands,

sudo journalctl still returns same output:

#15 330.3 + sudo journalctl
#15 330.6 No journal files were found.
#15 330.6 -- No entries --

And as for this command: sudo grep -r "Killed process" /var/log/*, I got no matching result in those logs.

Anything else you have in mind that I could try?

Thanks!

[ymanton]

Just so we are on the same page, you are checking the journal, dmesg, /var/log, etc on the Travis build node, not inside the build container where server stop fails, right?

The content of your /var/log above shows stuff like dnf.log and rhsm which I would expect on RedHat OSes and inside our UBI-based containers, but not on Travis build nodes which run Ubuntu.

[Rohanjos97]

Just so we are on the same page, you are checking the journal, dmesg, /var/log, etc on the Travis build node, not inside the build container where server stop fails, right?

I am checking it inside the container... Since it is the container which is failing, I assumed that all the logs would be inside the container...

Do I have to run these commands in my travis build after the container fails?