[CI] ILMHistoryItemTests testTruncateLongError failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #7138 / windows-2019_checkpart2_platform-support-windows
• elasticsearch-periodic-platform-support #7138 / windows-2022_checkpart2_platform-support-windows
• elasticsearch-periodic #7116 / openjdk17_checkpart2_java-fips-matrix
• elasticsearch-periodic #7116 / openjdk17_checkpart2_java-matrix
• elasticsearch-periodic #7116 / graalvm-ce17_checkpart2_java-matrix
• elasticsearch-periodic-platform-support #7124 / windows-2019_checkpart2_platform-support-windows
• elasticsearch-periodic-platform-support #7124 / windows-2022_checkpart2_platform-support-windows
• elasticsearch-periodic #7102 / openjdk17_checkpart2_java-fips-matrix
• elasticsearch-periodic #7102 / openjdk17_checkpart2_java-matrix
• elasticsearch-periodic #7102 / graalvm-ce17_checkpart2_java-matrix

Reproduction Line:

./gradlew ":x-pack:plugin:ilm:test" --tests "org.elasticsearch.xpack.ilm.history.ILMHistoryItemTests.testTruncateLongError" -Dtests.seed=FB314CB68F54F81E -Dtests.locale=asa-TZ -Dtests.timezone=America/Mazatlan -Druntime.java=17

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: "{\"type\":\"illegal_argument_exception\",\"reason\":\"iiKfktKSpRbcAoAQafqyhoSqnuBofOwtTaNuUTXPmRMYIJnHnYNkmkzzdcIJTcKjmdYyAVZmGRJOjRzrAgxPtsZKtcPNEvDyfFUisCLRXdFACTjLTNanxmqFxqJxCddrYamcQbjuNKwthDynxwxoHMpQuqjisKUtAlwBZJlonbWwthsYlodHZZxnBOdlwfMbcSjUVhmFUOZMyPiqhYyDloGICseKyiTMCPpcouRCansfkvsYhIVDFfDOsNHdUiZGiegoBxVWnCqDFSEXlLpcnOrYzyJaknwdsNWuuUlSSAZhVCqJSfukVlOvlmTDnRHoSftYlLLXscJuWKfJoXhGppZfoCLntuOOFobAzwSKOltbwvIbCUrAvYTxmSBXdWrPLuFAvJizITQozThiYZOQkhOUurCnCqoRXRETYxvzYhsgSbyXnIulVToSArDXKObubuDJgBKRzbRgpZFeLjXwaCEquluLfRPdovDydlAWqUUIjgGuhhCHGONxjfSFTBjhQCDHbGrgjkvdVKXoNueXVugUkVSsaWSsbYxvkuCsEhEiwxICpuBdYehWCBgCmXSnOSXOnKHzfoZRGMoWVHnGAeMQrICrgESnNtqPGzkMlxfnNlKXMEpUbstKSRDWpbRzYOZUvTQFBTYpaUstMTfiQGmTAWRpYMIcqLFVjaBEBNnTNacDZZcUohuKqrwRxbSkIeHxxKIJEadqLclkeLTQEjmYGOBdpuXpfpOGrfikLwFcbhrFlmRSIiywjlNyKVvUEdjZoeyuwKAjTzaDzBsKRXoETKFlAhMaHrFGRrDQRlWllMdefcRMRXgQXvCURMFjIGuSRwgLwzgOuawhvjHtyTMPajoMrGOcHxexSCQYxCBZzdBLiUcHVsSHUtsuJxDQUFRLpXWDMYvS
[truncated]

Issue Reasons:

• [8.x] 5 consecutive failures in test testTruncateLongError
• [8.x] 3 consecutive failures in step windows-2019_checkpart2_platform-support-windows
• [8.x] 3 consecutive failures in step windows-2022_checkpart2_platform-support-windows
• [8.x] 3 consecutive failures in step openjdk17_checkpart2_java-fips-matrix
• [8.x] 3 consecutive failures in step openjdk17_checkpart2_java-matrix
• [8.x] 3 consecutive failures in step graalvm-ce17_checkpart2_java-matrix
• [8.x] 15 failures in test testTruncateLongError (14.7% fail rate in 102 executions)
• [8.x] 3 failures in step windows-2019_checkpart2_platform-support-windows (100.0% fail rate in 3 executions)
• [8.x] 3 failures in step windows-2022_checkpart2_platform-support-windows (100.0% fail rate in 3 executions)
• [8.x] 3 failures in step openjdk17_checkpart2_java-fips-matrix (100.0% fail rate in 3 executions)
• [8.x] 3 failures in step openjdk17_checkpart2_java-matrix (100.0% fail rate in 3 executions)
• [8.x] 3 failures in step graalvm-ce17_checkpart2_java-matrix (100.0% fail rate in 3 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic-platform-support (75.0% fail rate in 4 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic (75.0% fail rate in 4 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

java.lang.AssertionError: 	
Expected: "{\"type\":\"illegal_argument_exception\",\"reason\":\"yQSAQIzcPDGQUJZVsRcDPAfRuhXsabemZfhdOBUgCRPPMjAAGhxYnUhOrKxDgJcQlYdVgdUCTwljjrJyWxVONmaFijXlblhpnBIMcSGBFUBsbrmVgrREIvoqCsEjznNQOAXDMirpajDGGbIbUaJcrausOnDjZOGlLwMpwWklsINQjJvGsjvHpkoPfxLtjdKfrtsyyjHKVkJTqMyshNubvgEqwljampMmyYKsBtEDCFlQXVithgSzGiGhkgIRjAiJmmdJBeeTiVWMQKKkxwojwBWafzoTVAbIVzbZIKXVhhLyoPjiPdhpwiluYZgdVnfRCopXOSBLQPTgnoLZyIAZuhZLWkBIIGyKuaGTfrOJYsetiRrFZhnrzMhSurumJfkwwTyjFdOzfeJVREMhEahMkTnzetTWBZIlkBdENFFgooxmUgUcswbdQoVtDtVeofDDaRSpogygrerqAAMQEEAuqZKeeLAODPsPuhrhPUGfoCCELdXdJIpSguZWqhTftXlavDvuzbaTCXDlerfOTjzFBHKpidvaDsHaDEdSEnWDPjrqXPQrMrVVqLfSFbghALnxYyfimGBviXLdjesQItCFVDUArNYNpxPwlsBqzBZCiAsouIOneLHislvYzWfmMjHPWGjbKQfwiJpKsSVpkcvwbZsWAPVbnWzUtjOxgvhZypWgRJqXDpligxyCOzZfhsSWUYVMEqwTJmXgjDnlWmBZlwwARDsuuAYEscNurOLhEOpBVPUrVmNRCpDxEaQqkRoSwxOzZvuqneGZaDyMVyCjgCnkdOaKpbSigtLhaxGGQiqozFKILRxydRxwnuwgnFbyZXwEnfCQZeyhWfYJSYfWfIQwRBbLRUqydkSMHiNJbZSLoeWItTopZoMdoLeTtFyeyIcbMfbFuxhIyVRCAfSHlqpegfEbDwbIMAeAywNZfWdXXtAPCIJgDjARPNYWqDzTFiBvoeHteSOCrKKwB... (5126 chars truncated)\"}"	
     but: was "{\"type\":\"illegal_argument_exception\",\"reason\":\"yQSAQIzcPDGQUJZVsRcDPAfRuhXsabemZfhdOBUgCRPPMjAAGhxYnUhOrKxDgJcQlYdVgdUCTwljjrJyWxVONmaFijXlblhpnBIMcSGBFUBsbrmVgrREIvoqCsEjznNQOAXDMirpajDGGbIbUaJcrausOnDjZOGlLwMpwWklsINQjJvGsjvHpkoPfxLtjdKfrtsyyjHKVkJTqMyshNubvgEqwljampMmyYKsBtEDCFlQXVithgSzGiGhkgIRjAiJmmdJBeeTiVWMQKKkxwojwBWafzoTVAbIVzbZIKXVhhLyoPjiPdhpwiluYZgdVnfRCopXOSBLQPTgnoLZyIAZuhZLWkBIIGyKuaGTfrOJYsetiRrFZhnrzMhSurumJfkwwTyjFdOzfeJVREMhEahMkTnzetTWBZIlkBdENFFgooxmUgUcswbdQoVtDtVeofDDaRSpogygrerqAAMQEEAuqZKeeLAODPsPuhrhPUGfoCCELdXdJIpSguZWqhTftXlavDvuzbaTCXDlerfOTjzFBHKpidvaDsHaDEdSEnWDPjrqXPQrMrVVqLfSFbghALnxYyfimGBviXLdjesQItCFVDUArNYNpxPwlsBqzBZCiAsouIOneLHislvYzWfmMjHPWGjbKQfwiJpKsSVpkcvwbZsWAPVbnWzUtjOxgvhZypWgRJqXDpligxyCOzZfhsSWUYVMEqwTJmXgjDnlWmBZlwwARDsuuAYEscNurOLhEOpBVPUrVmNRCpDxEaQqkRoSwxOzZvuqneGZaDyMVyCjgCnkdOaKpbSigtLhaxGGQiqozFKILRxydRxwnuwgnFbyZXwEnfCQZeyhWfYJSYfWfIQwRBbLRUqydkSMHiNJbZSLoeWItTopZoMdoLeTtFyeyIcbMfbFuxhIyVRCAfSHlqpegfEbDwbIMAeAywNZfWdXXtAPCIJgDjARPNYWqDzTFiBvoeHteSOCrKKwB... (5208 chars truncated)\"}"

Only the number of chars seems to mismatch.

All four failures occurred on Windows, that might be a clue? It doesn't reproduce for me locally.

Assigning to @dakrone as it's caused by #125054

[prdoyle]

Wild guess: base64-encoded line endings?

[prdoyle]

I wonder why this isn't getting muted?

[prdoyle]

I've muted it in #125217.