bin/ejbca.sh CLI Issue after EJBCA upgrading

[s0p4LiN]

Hello,

I upgraded my EJBCA single instance.

From:
EJBCA 6.15.2
wildfly-10.1.0.Final
openjdk 8

to:
EJBCA 7.10.0.2
wildfly-10.1.0.Final
openjdk 11

INFO: I did not find how to upgrade Wildfly for now. Maybe the error is due to this.
Wildfly 26 is recommended for version >= EJBCA 7.8

I tried to renew the SuperAdmin certificate password and I have many error/warning:

bin/ejbca.sh ra setendtitystatus superadmin 10
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.xnio.nio.NioXnio$2 (file:/home/pki/EJBCA/ejbca/dist/ejbca-ejb-cli/lib/jboss-client.jar) to constructor sun.nio.ch.EPollSelectorProvider()
WARNING: Please consider reporting this to the maintainers of org.xnio.nio.NioXnio$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" java.util.ServiceConfigurationError: org.ejbca.ui.cli.infrastructure.command.CliCommandPlugin: Provider org.ejbca.ui.cli.ra.AddEndEntityCommand could not be instantiated
        at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582)
        at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:804)
        at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:722)
        at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1395)
        at org.ejbca.ui.cli.infrastructure.library.CommandLibrary.<init>(CommandLibrary.java:52)
        at org.ejbca.ui.cli.infrastructure.library.CommandLibrary.<clinit>(CommandLibrary.java:36)
        at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:37)
Caused by: java.lang.ExceptionInInitializerError
        at org.jboss.marshalling.river.RiverMarshaller.<clinit>(RiverMarshaller.java:1233)
        at org.jboss.marshalling.river.RiverMarshallerFactory.createMarshaller(RiverMarshallerFactory.java:54)
        at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocationInternal(RemotingConnectionEJBReceiver.java:331)
        at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:276)
        at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:184)
        at org.jboss.ejb.client.EJBObjectInterceptor.handleInvocation(EJBObjectInterceptor.java:58)
        at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
        at org.jboss.ejb.client.EJBHomeInterceptor.handleInvocation(EJBHomeInterceptor.java:83)
        at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
        at org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:42)
        at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
        at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:138)
        at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
        at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:255)
        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
        at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
        at com.sun.proxy.$Proxy5.getCachedConfiguration(Unknown Source)
        at org.ejbca.ui.cli.ra.AddEndEntityCommand.<init>(AddEndEntityCommand.java:98)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:780)
        ... 5 more
Caused by: java.lang.IllegalStateException: No standard field found for reverse order comparator!
        at org.jboss.marshalling.river.Protocol.<clinit>(Protocol.java:220)
        ... 29 more

What I need to do to fix it and also how to check if any other commands are working ?

[s0p4LiN]

I tried rollback JAVA to JDK 8 but similar error:

Exception in thread "main" java.util.ServiceConfigurationError: org.ejbca.ui.cli.infrastructure.command.CliCommandPlugin: Provider org.ejbca.ui.cli.ra.AddEndEntityCommand could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:232)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
        at org.ejbca.ui.cli.infrastructure.library.CommandLibrary.<init>(CommandLibrary.java:52)
        at org.ejbca.ui.cli.infrastructure.library.CommandLibrary.<clinit>(CommandLibrary.java:36)
        at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:37)
Caused by: javax.ejb.EJBException: java.io.IOException: Channel Channel ID a43bf7b4 (outbound) of Remoting connection 7e13f4d0 to localhost/127.0.0.1:4447 of endpoint "config-based-ejb-client-endpoint" <6d8c3abc> has been closed
        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:238)
        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
        at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
        at com.sun.proxy.$Proxy5.getCachedConfiguration(Unknown Source)
        at org.ejbca.ui.cli.ra.AddEndEntityCommand.<init>(AddEndEntityCommand.java:98)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
        ... 5 more
Caused by: java.io.IOException: Channel Channel ID a43bf7b4 (outbound) of Remoting connection 7e13f4d0 to localhost/127.0.0.1:4447 of endpoint "config-based-ejb-client-endpoint" <6d8c3abc> has been closed        at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:123)
        at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:115)
        at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:514)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:419)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:290)
        at org.jboss.remoting3.remote.RemoteConnectionChannel.closeReadsAndWrites(RemoteConnectionChannel.java:274)
        at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:534)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:154)
        at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleEnd(ChannelAssociation.java:485)
        at org.jboss.remoting3.remote.RemoteConnectionChannel$2.run(RemoteConnectionChannel.java:287)
        at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

[primetomas]

It looks like your WildFly is not running, the CLI can not connect to it at least. Check the WildFly log, it probably have errors in it.
If you roll back to Java 8 you have to "ant clean deployear" as well to recompile.

[s0p4LiN]

Ah ! I think I get it, I did upgrade from Java 8 to Java 11 after EJBCA update but I did not run 'ant clean deployear'.
What are all the steps to do to have a valid dependancies upgrade of EJBCA (Java and Wildfly) ?

When I run 'bin/ejbca.sh ra setendtitystatus superadmin 10' or any CLI command with bin/ejbca.sh;

The logs tells me:

022-12-07 10:35:46,251 ERROR [org.jboss.as.ejb3.remote] (default task-46) WFLYEJB0148: Exception on channel Channel ID 098f2254 (inbound) of Remoting connection 05c3763a to /127.0.0.1:54907 of endpoint "unknown-host" <15283091> from message org.jboss.remoting3.remote.InboundMessage$3@7fd8af16: java.io.IOException: WFLYEJB0134: Failed to read EJB Locator
        at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:141)
        at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
        at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
        at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
        at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:456)
        at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:731)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.jboss.marshalling.river.ClassDescriptors
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:1142)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:918)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:918)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1255)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
        at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:139)
        ... 8 more

[primetomas]

You have to run the same version of Java and EJBCA both on the server and on the CLI.

[s0p4LiN]

Ok, so when I upgraded EJBCA it took the JAVA 8 version as reference. But then I did upgrade to Java 11 and when I run the CLI it use JAVA 11 and the server does not understant because it still use java 8 ?

Do I just need to run 'ant clean deployear' to fix it or is there any other steps ?

thank you

[s0p4LiN]

I rollback to my previous snapshot, where EJBCA 7.10 with JAVA 8 and Wildfly 10/JBoss 7 is working even with CLI.

I read that the minimum supported application server is JBoss EAP7/Wildfly 10 for EJBCA 7, but I would like upgrade these to the last recommended version (Wildfly 26/JBoss EAP 7.4) to prepare in the next major releases where JBoss EAP7/Wildfly10 will be deprecated.

I can not find any procedure on EJBCA docs showing how to upgrade Wildfly26/JBoss along with Java 11. I thought for Java, the version change will not affect but it seems to.

[primetomas]

If you can, upgrade one component at a time. I.e. java 11 first, then wildfly, or the other way around.
You can find instructions for the different application servers here:
https://doc.primekey.com/ejbca/ejbca-installation/application-servers
The CLI uses "Remoting", so that part is important.

[s0p4LiN]

I decided that I will not try to upgrade. Two times reading the doc for Widfly can't understand what to do just for upgrading.
Same for java.

As I'm leaving the company in the end of december, I will not have the time to search and try the upgrade of the components.

Thank you for the help.

[primetomas]

Nowadays, using containers or software appliance makes things so much easier. Other thinks like Ansible scripts also makes install and upgrades more reproducible. Our Ansible playbooks are made for WildFly 26.
https://github.com/Keyfactor/ansible-ejbca-signserver-playbooks

[s0p4LiN]

Yes I know... I did install EJBCA 6.15.2 on bare metal when I was in apprenticeship and I was not able to do it in container.
Is it possible to run a container with last version and then migrate the PKI, cert and keys to the new updated container ?

I could be easier than updating ?

[primetomas]

It should be possible, you can point the container to the (external) database, which would let EJBCA in the container to read the old database. There's documentation here about keystores, using "simple" TLS setup and things.
https://hub.docker.com/r/keyfactor/ejbca-ce

[s0p4LiN]

thanks !