javax.net.ssl.SSLHandshakeExceptions

[ben-sec]

Hi!

I'm troublshoting my ticket #9612 and found tons of javax.net.ssl.SSLHandshakeExceptions in logstash.log on my manager node:

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
...
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed

I don't now since when I receive all these alerts, but I have not experienced any erors so far, but maybe I'm missing something ... Do I have to worry about these log entries?

Cheers, Ben

[cm-ops]

Are you still seeing those in the logstash.log or did resolving #9612 clear those up?

[ben-sec]

Hmm, not that I'm aware of. Where would I find them?

[cm-ops]

If you did, they would be in /opt/so/saltstack/local/salt/logstash/pipelines/config/custom. I would also validate system time is synchronized across the nodes.

[ben-sec]

Ah, you mean custom pipelines. Yes, I read Windows DNS logs from a mounted file share and write them to a directory on the Manager node:

Input:

1 input {
2         file {
3               path => "/var/log/logstash/dns-share-pdc/dns.log"
4               tags => ["dns","pdc"]
5               mode => "tail"
6               #start_position => "beginning"
7               sincedb_path => "/var/log/logstash/dns-pdc-log"
8              }
9 }
10
11 filter { if "pdc" in [tags] {
12        if [message] =~ /^\r/ {
13        drop {}
14        }
15
16        mutate { gsub => [
17                         "message", "\s\(\d+\)", " ",
18                         "message", "\(\d+\)\r", "\r" ,
19                         "message", "\(\d+\)", "."
20                         ]
21                 add_field => {"dns-server" => "pdc"}
22               }
23        }
24
25 }

Output:

1 output {
2   if "dns" in [tags] {
3       file {
4               path => "/var/log/logstash/dns-import/dns-%{+YYYY-MM-dd}.log"
5       }
6   }
7 }

But that works well.

[cm-ops]

Okay, if time is synchronized as well, do you see any failures in the Logstash pipeline stats? so-logstash-pipeline-stats $pipeline (manager or search).

[ben-sec]

Can't find any failures there (at least no errors are screaming at me) and all nodes are NTP snychronized.

[ben-sec]

I found the root cause: I recently had to re-new certificates on Windows clients for encrypted Winlogbeat data transfer. For an unknown reason this failed for a subset of clients. Winlogbeat still tried to connect to Logstash with an outdated certificate and failed. This caused all the errors in my logstash.log. I restarted the rollout of the new certificate to the clients. Thank you anyway for your support!