-
|
Our application uses an embedded Jetty server (12.0.22 EE8), which occasionally gets restarted (when configuration changes are applied). I have observed the stack trace at the bottom of this post being logged when restarting Jetty (but not on the initial start). I suspect that the problem is triggered by how /* ensure scratch dir */
File scratch;
if (getInitParameter("scratchdir") == null) {
File tmp = (File) getServletHandler().getServletContext().getAttribute(ServletContext.TEMPDIR);
scratch = new File(tmp, "jsp");
setInitParameter("scratchdir", scratch.getAbsolutePath());
}
scratch = new File(getInitParameter("scratchdir"));
if (!scratch.exists() && !scratch.mkdir())
throw new IllegalStateException("Could not create JSP scratch directory");Our application does not set the 'scratchdir' init parameter. Upon first start, this code creates a scratch dir relative to the I suspect that upon a _re_start, the init parameter now being defined, causes an issue. The This is the stack trace that is logged when restarting the embedded Jetty instance. |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 7 replies
-
|
Initial gut reaction ... the temp directory is being cleaned out between restarts, and that is breaking things. What is the value of your Anyway, the But since Jetty will ensure it is set, we have a problem. We can do some extra checks in |
Beta Was this translation helpful? Give feedback.
-
|
I'm starting to think that your inability to create the JSP scratch directory is something else. So I opened a new PR to switch to |
Beta Was this translation helpful? Give feedback.
-
|
I think it is failing to (re)create that directory, because after a restart, the parent dir is different. The code is using |
Beta Was this translation helpful? Give feedback.
-
I think you're right. From within |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure if this has to do with anything, but while debugging, I noticed that a temp directory gets created, discarded and created under a different name a lot, |
Beta Was this translation helpful? Give feedback.
-
|
That's because of a bug in our code. |
Beta Was this translation helpful? Give feedback.
-
|
Fixed that bug, but it doesn't prevent the 'Could not create JSP scratch directory' error from happening. |
Beta Was this translation helpful? Give feedback.
-
|
@guusdk I can't reproduce the problem with restarting an Looking at your stacktrace, it looks pretty unusual, particularly these lines: It looks like you are in the middle of servicing a JSP request, and then somehow calling Moreover, you only need to restart a single context, if that's all that has changed. The usual ways to do that are either:
If you've changed the configuration of the jetty server itself, then you might need to do a full stop and then restart of the If the above analysis isn't helpful, then you'll need to explain how you are starting/restarting jetty or the webapp. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Jan, thanks for your insights. To explain, I think I need to give a bit of background. With my writing style, 'a bit' is never short. Sorry! We use Jetty for a number of things:
Apart from that, our plugin API allows any third-party plugin to register a Context, thus adding some kind of functionality to the system. Examples of such plugin-provided features include a REST API, various web-based IM clients, servlets that are used to upload and share pictures, etc. There's a question if 'everything' should be hosted by the same Jetty server instance (we actually use two: the admin console is on a different server), but for now, that's how things are. If you're interested, this is how we've tied things together: https://github.com/igniterealtime/Openfire/blob/main/xmppserver/src/main/java/org/jivesoftware/openfire/http/HttpBindManager.java#L308 Whenever a configuration change is detected, we restart the server. This mostly handles updated TLS certificates, but also deals with administrative changes (eg: change the port on which the server hosts the websocket endpoints). To generate the stack trace that you've seen, I manually triggered a configuration change through our admin console, which is a JSP That's why you see that in the stack trace (interestingly, I believe that the JSP that triggers this change is hosted by the other server instance). In any case, we also see the "Could not create JSP scratch directory" warning when the restart is not directly triggered via a JSP. We have a file-based check that restarts the server upon a change applied to the TLS certificates. I will include a stack trace of that event below. Contrary to your belief, we do not see JSP failures when this problem occurs (we pre-compile our JSPs, maybe that is relevant? They may also be hosted by the server that's not restarted). We do see issues related to the BOSH endpoint, which may or may not be related. See #13342 for an example. |
Beta Was this translation helpful? Give feedback.
-
|
@guusdk thanks for that, but I'm still not quite clear on how you're using jetty.
A few things I noticed while looking at your code: I don't see why you are starting the I don't see anywhere where you've set the Finally, I noticed you have some kind of file toucher thread to keep modifying the date on the tmp files. Have you considered using the I haven't looked really deeply into the code, but I would have expected to see a bit more threading coordination - eg, what's to prevent |
Beta Was this translation helpful? Give feedback.
-
|
Hi @janbartel - thanks for the suggestions. I disappeared for a family vacation for a while, which is why it took me some time to respond. What do you mean with 'process'? There's one JVM running in which all of the servers are present. Openfire is starting (at least) two Server instances:
We separated the two, because we consider the former to be a 'public facing' server (with which end-users can be expected to interact) while the latter one is administrative (and isn't expected to be exposed to the general public). Some of our plugins are provided by third parties, over which we do not have control. Although we expect most plugins to hook into the 'extension handlers' that are part of HttpBindManager, there probably are going to be plugins that launch their own server. Our code is explicitly stopping and starting all handlers to work around an issue (logged as OF-1789) that we observed in older versions. Without this, restarting Jetty (sometimes) caused an error, indicating that we're trying to start something that's already started. This seems to happen only when the handler was being used when it was supposed to be stopped (e.g. have an active BOSH connection). Perhaps using We have no need or desire to persist the content of the temp directory between restarts, and are happy to have its content be managed by Jetty, using whatever process it believes is most appropriate. We are using the file toucher to modify the date on tmp files to prevent files from disappearing while the server is running . The problem that this avoids is caused by OS-based tools (like I think you're right that threading coordination with regards to start and stop is poor. The |
Beta Was this translation helpful? Give feedback.
-
|
After upgrading to 12.0.24, this problem hasn't been reproduced yet. I assume that this is caused by the improvements referenced in #13362 (reply in thread) Thanks! |
Beta Was this translation helpful? Give feedback.
After upgrading to 12.0.24, this problem hasn't been reproduced yet. I assume that this is caused by the improvements referenced in #13362 (reply in thread)
Thanks!