Upgrade from .190 to .200

[mbaki]

After upgrading my distributed SO, I do not see any alerts, I get Request failed with status code 502
, plus the storage shows after a so-status

Checking Docker status

Docker ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]

Checking container statuses

so-curator ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-elasticsearch ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]
so-filebeat ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-logstash ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-nginx --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-sensoroni ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]
so-telegraf ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [ OK ]
so-wazuh --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]

[mbaki]

sostorage.log:

[2023-01-24T16:43:48,797][ERROR][org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch[sostorage][generic][T#1]], exiting
java.lang.InternalError: a fault occurred in an unsafe memory access operation
[2023-01-24T16:56:03,983][INFO ][org.elasticsearch.node.Node] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-12785628971579178442, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms21792m, -Xmx21792m, -Des.transport.cname_in_publish_address=true, -Dlog4j2.formatMsgNoLookups=true, -XX:MaxDirectMemorySize=11425284096, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
[2023-01-24T16:59:26,789][ERROR][org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch[sostorage][generic][T#3]], exiting
java.lang.InternalError: a fault occurred in an unsafe memory access operation
[2023-01-24T17:11:04,253][INFO ][org.elasticsearch.node.Node] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-12247674441132057028, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms21792m, -Xmx21792m, -Des.transport.cname_in_publish_address=true, -Dlog4j2.formatMsgNoLookups=true, -XX:MaxDirectMemorySize=11425284096, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
[2023-01-24T17:14:06,058][ERROR][org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch[sostorage][generic][T#10]], exiting
java.lang.InternalError: a fault occurred in an unsafe memory access operation
[2023-01-24T17:26:04,568][INFO ][org.elasticsearch.node.Node] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11537272037832817361, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms21792m, -Xmx21792m, -Des.transport.cname_in_publish_address=true, -Dlog4j2.formatMsgNoLookups=true, -XX:MaxDirectMemorySize=11425284096, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
[2023-01-24T17:28:58,692][ERROR][org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch[sostorage][generic][T#4]], exiting
java.lang.InternalError: a fault occurred in an unsafe memory access operation
[2023-01-24T17:41:03,284][INFO ][org.elasticsearch.node.Node] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-16951348348548173847, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms21792m, -Xmx21792m, -Des.transport.cname_in_publish_address=true, -Dlog4j2.formatMsgNoLookups=true, -XX:MaxDirectMemorySize=11425284096, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]

[mbaki]

Turns out I had tons of indices in "Red" state, closing and deleted them I am finally seeing events.