Replies: 15 comments 14 replies
-
|
From the container you will get logs to the console. These are the logs that tell you what EJBCA is doing. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @primetomas, Thanks for your answer! I have checked the database and it seems indexes are existing:
The database is not large. I only have 1 CA and 1 Intermediate CA. The number of issued certs is roundabout 20. So I don't think it is related to the data itself. The container restrictions are visible here:
CPU Priority is low, the memory limit is set to 2 GB. Do you think this is not sufficient? Bye |
Beta Was this translation helpful? Give feedback.
-
|
Can you check the console output to see if anything is going on? 2GB should be enough for such a small environment. We're using the container is very very large deployments without issues. SO something fishy is going on in that environment. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @primetomas, sorry for the late reply, I was not at home. I have now redirected the output of the console to a file and am tracking when the load will increase. After that I will check the file and provide more information. Thanks for your hints! |
Beta Was this translation helpful? Give feedback.
-
|
After I enabled the logging the issue has not occurred so far. Will continue to monitor and revert back if the issue reappears. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @primetomas, and here we go... In the attached log you can see that ejbca was not able to connect to ldap server (it was offline for some time, backup...) and since then it seems the load increased. Hopefully the log will give you some hints on why this happend. |
Beta Was this translation helpful? Give feedback.
-
|
You have a publisher queue process worker configured. How did you configure this? Does it run very often? I guess there are many retries or something. But the log looks ok, the service worker finishes saying that it fails to connect to the LDAP server. The service should run again and eventually publish to the LDAP server once it is up and running, unless TCP connections hang for some reason, but it doesn't look like that from the log. (Using LDAP is getting quite rare out there...) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @primetomas , I had the worker configured to run every 5 minutes but realized that it does not make sense to call it that often. Now changed it to "once per day", let's see if this make the issue appear again. Thanks so far! |
Beta Was this translation helpful? Give feedback.
-
|
Hi @primetomas, another exception today, this time related to the database server: `2023-03-13 02:00:38,198+0000 WARN [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory] (ConnectionValidator) IJ030027: Destroying connection that is not valid, due to the following exception: org.mariadb.jdbc.Connection@20fa1282: java.sql.SQLNonTransientConnectionException: (conn=103908) Socket error 2023-03-13 02:00:38,203+0000 WARN [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory] (ConnectionValidator) IJ030027: Destroying connection that is not valid, due to the following exception: org.mariadb.jdbc.Connection@118ff1a6: java.sql.SQLNonTransientConnectionException: (conn=103909) Socket error 2023-03-13 02:00:38,205+0000 WARN [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory] (ConnectionValidator) IJ030027: Destroying connection that is not valid, due to the following exception: org.mariadb.jdbc.Connection@18bb9ae0: java.sql.SQLNonTransientConnectionException: (conn=103910) Socket error 2023-03-13 02:00:38,208+0000 WARN [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory] (ConnectionValidator) IJ030027: Destroying connection that is not valid, due to the following exception: org.mariadb.jdbc.Connection@64c79624: java.sql.SQLNonTransientConnectionException: (conn=103911) Socket error 2023-03-13 02:00:38,210+0000 WARN [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory] (ConnectionValidator) IJ030027: Destroying connection that is not valid, due to the following exception: org.mariadb.jdbc.Connection@45d6310e: java.sql.SQLNonTransientConnectionException: (conn=103912) Socket error After that the load again increased. Is there anything you can determine from the logs what goes wrong here? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
" (conn=103910) Socket error" is an error from the MariaDB database driver. Is there an issue with the database? EJBCA tries to make an SQL query but the database connection is closed. Can you check what happens on your database? Does it allow large enough packets to store CRLs for example? The database log may show something about the DB configuration. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @primetomas, yes, MySQL was not available for a few minutes, but this shouldn't cause the load to increase, right? I am aware that from time to time the database might not be available, but the question is why then the load on EJBCA container increases. How can I do more debugging and figure out what causes the load? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Java flight recorder has been used for detailed analysis of runtime performance. And no, if the database is restarted it typically simply reconnects and carries on when the database is up. While the database is down you will see tons of error of course, as the machine can't do anything. It's not advised to use "take database offline" backup procedures, on-line backups work well and EJBCA is used in many "always on" mission critical environments. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I am constantly monitoring my ejbca instance still and facing the same issue over and over again. I have already disabled services like CRL Update and Publishing, but at some point, when I try to access the application, the docker load increases and stays at a high level. From the logs I cannot see anything special, but my monitoring tools are seeing this:
So it is not constantly increasing but rather jumps up suddenly, even without direct access to the application (was sleeping at that time :)) In case you wonder: the load is down at the end because I restarted the ejbca docker container. Here is the server log file from within this time frame: By the way, today there was no real database outage... |
Beta Was this translation helpful? Give feedback.
-
|
Hi, and once again, out of a sudden:
Here is the log from this time frame: Can you see anything that can cause such a behavior? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
As far as I can see nothing is happening in EJBCA. I assume that the database is on a separate container, you don't have any backup jobs or similar that runs by the platform? The container is not RAM constrained, etc? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @primetomas, thanks for your quick answer! In fact the container WAS RAM constrained to 2GB. I have removed this limit now to see if it gets better. Are there any other settings required for the container to run without issues? Anything I should configure? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, so currently I have these env variables enabled for debug logging:
I will remove the 2 "SERVER" variables and only activate "LOG_LEVEL_APP", as mentioned on the docker page. Will revert back with the next logs when the issue occurs :) Thanks for your patience! |
Beta Was this translation helpful? Give feedback.
-
|
Hi @primetomas, Issue occurred again. I cleaned up the log file to contain some blocks before and after the time it occurred. I was notified at 17:14 about it. Do you see anything "strange"? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Except for re-loading 5 certificates in the OCSP signing cache regularly, which is normal, I don't see anything happening. Can you see if there are any queries being run on the database? If there is anything, the free RAM looks pretty low on your system, in the top screenshot it shows only 350MB available on the system? |
Beta Was this translation helpful? Give feedback.
-
|
Is Oberservability being used in the container to capture metrics? I'm wondering if that could help track the resource increases. I've never heard of this happening for anyone else yet. |
Beta Was this translation helpful? Give feedback.
-
|
Hi guys, today I realized something interesting: when accessing the web frontend after a day or so (during which the load didn't increase), the answer from the server takes very long (round about 30 seconds). Sometimes ressources like css or javascript files are not loaded at all and I get a non-formatted page. When reloading the frontend everything is fine again, but: the load increases and stays high. Not sure if such a behavior is something that might shed a little bit light on the issue... I am running the docker container on a Synology NAS, where Docker is limited in features. Not sure what kind of things I can additionally configure, but I haven't found any way to activate additional metrics. Do you have a hint on how to do that? The database is not busy during the increased load, I already checked that. |
Beta Was this translation helpful? Give feedback.
-
|
A question that came up. Do you have something making connection checks, or doing an EJBCA Healthcheck? |
Beta Was this translation helpful? Give feedback.
-
|
No, I am not actively requesting anything from EJBCA regularily. Mostly relying on the push mechanisms. I have an implementation on the servers using a cmp client to update certs. It is executed once a day (in the early morning). It basically checks if the server cert is still valid, which is always the case at the moment. Do you think the cmp endpoint might cause issues? |
Beta Was this translation helpful? Give feedback.
-
|
I don't think the CMP endpoint would cause any issues. It's used to issue billions of certificates in mission critical systems without known issues. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, a week ago I updated to EJBCA Community 8.0 and since then the issue did not occur. So it seems something was changed that prevents the problem from happening. Any information of what changes might have solved it? Was there an active analysis concerning this or was it fixed "accidentially"? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Nope, nothing specific that has been done. Generic upgrades of many components of course, dependencies, java, base-os, etc etc. |
Beta Was this translation helpful? Give feedback.
-
|
When opening the web page now the waiting time is much shorter. I think it was something related to the webserver, but cannot prove it. But luckily for me the issue seems resolved. No problem since almost 2 weeks now. Thanks anyways for your support! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi guys,
I am facing an issue with the official ejbca community docker installation. I am running it on a Synology NAS and the container starts to have a high load after some time (sometimes 1 day, sometimes more days).
I already tried to deactivate my publishers (Active Directory and custom Unix publishers), but to no avail. It seems to me that this is somehow related with the database connection, but I cannot really prove it. I checked the logs but cannot find anything problematic:
ejbca.zip
I am using a MariaDB database as backend.
Maybe somebody has a hint what to do to avoid ejbca exhausting the CPU that way.
Thanks!
Bye
Beta Was this translation helpful? Give feedback.
All reactions