Spring Batch Metrics

[janossch]

Dear Spring Batch developers,

We are exposing our metrics via the production ready feature of Spring Boot, described here.

We would like to create a monitoring dashboard based on the provided metrics by Spring Batch framework.
Some of our envisioned dashboard tiles would provide an overview of what errors occurring in our batch steps/jobs.
e.g. number of jobs grouped by error causes, or something like what kind of error (and how many) arise in our steps.

However as I have seen so far, all exceptions are wrapped around a single type of exception and the only error type which appears in the metrics is IllegalStateException independetly of the real nature/type of the problem/exception.

References:

spring-batch/spring-batch-core/src/main/java/org/springframework/batch/core/job/AbstractJob.java

Lines 373 to 376 in fe421d0

	private IllegalStateException mergedThrowables(List<Throwable> throwables) {
	return new IllegalStateException(
	throwables.stream().map(Throwable::toString).collect(Collectors.joining("\n")));
	}

example prometheus data for a job:
spring_batch_job_seconds_count{error="IllegalStateException",instance_id="c69a861e-3e74-4bcb-8069-abead0aa5836",spring_batch_job_name="customjob",spring_batch_job_status="FAILED"} 1

spring-batch/spring-batch-core/src/main/java/org/springframework/batch/core/step/AbstractStep.java

Lines 367 to 370 in fe421d0

	private IllegalStateException mergedThrowables(List<Throwable> throwables) {
	return new IllegalStateException(
	throwables.stream().map(Throwable::toString).collect(Collectors.joining("\n")));
	}

example prometheus data for a step:
spring_batch_step_seconds_count{error="IllegalStateException",instance_id="c69a861e-3e74-4bcb-8069-abead0aa5836",spring_batch_step_job_name="store-job",spring_batch_step_name="job-starter-step",spring_batch_step_status="FAILED"} 1

Could you please point me to a direction how to overcome about the above problem or how to get default metrics around errors?
Or can you confirm is this a defect in the implementation?

🙏

[fmbenhassine]

I see your point, and I think there is a room for improvement in Spring Batch here.

However, I would like to know more about your expectations in terms of error reporting. For example, in the reference to AbstractJob you mentioned, the observation is recorded after the job is finished, and at that point in time, Spring Batch provides an API to get the list of all exceptions that happened during the job execution. How do you suggest to report/publish these to the metrics backend? In this particular case, it is a single metric, tagged with an (aggregated) error.

[janossch]

Dear @fmbenhassine

I really appreciate you taking the time to discuss this.

The first thing I would consider is when only one exception is inside the IllegalStateException. In that case it is obvious what to do with it (no merge of exceptions is needed). But when multiple "reasons" are inside the IllegalStateException that's where things are getting complicated. Even if there is the possibility to get all exceptions via an API the Spring Batch metrics won't reflect the real cause of the error.

How the end user would like to handle the above cases (1 or more errors occur), it may vary on the business needs of that particular job/step. If it is mission critical to know and monitor all errors, then it should be somehow possible to report all occurred exceptions.

So the framework should be configurable to see:

• only one error (maybe by providing a selector interface):
  • only the first or last error reason,
  • pick the most prominent error (e.g. to pick to most frequently occurring one)
• or report all errors in the metrics (which will generate the most data in the metrics database)

By default, I would consider the last error of an aggregated error for the metrics.
This way it would give you some hint of the nature of the occurring errors compared to having IllegalStateException always.

Getting to my specific scenario, in my case (even if retry happens) the last error (or the single error which occurred) would be sufficient.