downloaded_splits directory is not being cleaned up in the indexer

[earlbread]

In the indexer, the downloaded-splits directory is not being cleaned up, causing disk usage to repeatedly spike to 100% and then drop back down.

Below are the disk usage metrics and the actual disk usage details.

• Disk Usage

• Directory Info

I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata$ du -sh *
158G    indexer-split-cache
108G    indexing
16K     lost+found
12K     queues
74M     wal
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata$ cd index
indexer-split-cache/ indexing/
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata$ cd indexing/
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing$ du -sh *
4.0K    cloudfront-analytics%01K46QJFB8TFWFXQG2BHWTX4ZT%_ingest-api-source%01K5QXPJVJJYJ24KB4NJT316J7%yWOHyx
5.0M    cloudfront-analytics%01K46QJFB8TFWFXQG2BHWTX4ZT%cloudfront-analytics%01K5QXTDX291C6K4XH4X0H85XB%Z8nMgZ
4.0K    log.common.access_log_v2_quickwit%01JWA8T4D80XNZ0JXVFJBBZGDP%_ingest-api-source%01K5QXPJVJ8PAPG6QKXDG0FC4Z%3yotTw
4.0K    log.common.accesslog_v1_1%01JNN5K4NT0MX73J1W62HNPBF3%_ingest-api-source%01K5QXPJVJ7ARDY3T6WZSJHXZN%UMjIUL
4.0K    log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%_ingest-api-source%01K5QXPJVJMA4DBW1792YPS8TE%DnBA3h
108G    log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn
27M     log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX2562N8QFK96Q0YNFJ%H8bor1
33M     log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX2MQ502KPRGRVVERH1%xSZOTa
26M     log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX2P2VPGW0B82CDHDH8%fFyBAF
4.0K    log.common.karrot_audit_authorization_log_v1%01K1ARAG165KH1BK7118FCSSHR%_ingest-api-source%01K5QXPJVJ0FYDGGBHN5FF4Y2F%DvB40E
4.0K    log.common.karrot_audit_authorization_log_v1%01K1ARAG165KH1BK7118FCSSHR%kafka_to_karrot_audit_authorization_log_v1_ca%01K5QXTDX2PRK0JXBBCHSEEJB6%1JFoxJ
4.0K    log.common.karrot_audit_log_v1%01K1ARAC9ZVJ5AA28D9K6575TY%_ingest-api-source%01K5QXPJVJ88QP0712KMTHXRWY%YP8bBZ
4.0K    log.karrot-gateway.access_log_v2_quickwit%01K4Y6016Q0QWFT41GC5ZVEM4X%_ingest-api-source%01K5QXPJVJFYRW8AV3SKFD142Y%7ho0S8
4.0K    log.karrotgateway.accesslog_v1_1%01JPAG00RG5TRD0F3X0Z1ZGPTC%_ingest-api-source%01K5QXPJVJS8MCS7VBGWHB404X%prNb4Y
4.0K    otel-logs-v0_7%01JE5QBDXD8GPEAQR2GZKB5RRR%_ingest-api-source%01K5QXPJVJGJ4QS9ZHVGNY8465%UmuipQ
4.0K    otel-logs-v0_9%01JY22ACBKBTMT2SFSDS23ZT6Y%_ingest-api-source%01K5QXPJVJFJE1H87WK18HR248%zyxdJz
4.0K    otel-traces-v0_7%01JNG4MET50159AFDBMJ9A4EJ3%_ingest-api-source%01K5QXPJVJF2H9QZFWZ93DBXVB%CwwVk3
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-api-source%01K5QXPJVJFMGGDXGZFAFG4GDB%gAUQYX
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5RJZSJ542K53K84DVM58B61%hfrZpG
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5RK6FJ0X9AEFRAT2J1YC83V%jwYe7Q
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5RMNTX1VB9SXD1942DDC4Q7%MOCBOH
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5RN3BS6NCWX78MP7A2SFZSZ%OQOQM3
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5T1AWJZH974XYB52MH2YDCQ%5hmdMy
4.0K    otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5T2SWW5CH4457XAY40ENF5G%2ptknm
20K     otel-traces-v0_9%01JY22ACBSZ1WNH0STQYB8285X%_ingest-source%01K5T3C4WY0W8J9BZ4CFPJ8Y6Q%zgliAv
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing$ cd log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing/log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn$ du -sh *
108G    merge%MKKSPc
504K    split-01K5T5FM1G8YN36DVMCQBBFGMH-UHdGlE
34M     split-01K5T5FM2WTS7NY81CGTK7801K-jyDT2u
12K     split-01K5T5FM53GEZ9JYENXSSJEW1Z-zO2jBt
12K     split-01K5T5FNZT8HSD4NA9JW4YJZWZ-xmx8YX
12K     split-01K5T5FSSYVYWHNV3SFY6SREVS-rBhVvP
12K     split-01K5T5FVERCJH32XYEMTF5V1AQ-BV1cL9
12K     split-01K5T5GQBKASBZB518JXEJFSKC-TPApuk
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing/log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn$ cd merge%MKKSPc/
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing/log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn/merge%MKKSPc$ du -sh *
108G    downloaded-splits%u5FqAB
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing/log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn/merge%MKKSPc$ cd downloaded-splits%u5FqAB/
I have no name!@quickwit-prod-kr-a-indexer-1:/quickwit/qwdata/indexing/log.common.application_log_v1_quickwit%01K180MYSKWTD0ZEANKNFRBMVD%kafka_to_application_log_v1_quickwit%01K5QXTDX22SWVX87D3TFZ21CF%E9Gchn/merge%MKKSPc/downloaded-splits%u5FqAB$ du -sh *
15G     01K5R3ZGEAYSG1APWW1TFPJNXV.split
27G     01K5R6WJS6BCZ83MTT0QQVD215.split
23G     01K5R9NEZE01B3RQM2Z3CVP05R.split
22G     01K5RCHDHXHXHKMYQXRG9GRKZV.split
24G     01K5RFH9T91R08DAE8B5RXEB1G.split