build index 把集群整挂了，咋办

[zong400]

集群是1fe 2be 2.0.2版本，表数据2000条，同时build了2个倒排索引，然后show查看一直卡住没完成，看fe疯狂刷warn log， be无限重启

fe warn

2023-11-30 15:34:21,948 WARN (schema change|15) [ThreadPoolManager$LogDiscardPolicy.rejectedExecution():289] Task java.util.concurrent.FutureTask@2338b173 rejected from schema-change-pool java.util.concurrent.ThreadPoolExecutor@5fd5dafd[Running, pool size = 10, active threads = 1, queued tasks = 0, completed tasks = 69435]
2023-11-30 15:34:21,948 WARN (schema change|15) [ThreadPoolManager$LogDiscardPolicy.rejectedExecution():289] Task java.util.concurrent.FutureTask@72b7667f rejected from schema-change-pool java.util.concurrent.ThreadPoolExecutor@5fd5dafd[Running, pool size = 10, active threads = 1, queued tasks = 0, completed tasks = 69438]
2023-11-30 15:34:21,949 WARN (schema change|15) [ThreadPoolManager$LogDiscardPolicy.rejectedExecution():289] Task java.util.concurrent.FutureTask@51e0327d rejected from schema-change-pool java.util.concurrent.ThreadPoolExecutor@5fd5dafd[Running, pool size = 10, active threads = 1, queued tasks = 0, completed tasks = 69440]
2023-11-30 15:34:21,949 WARN (schema change|15) [ThreadPoolManager$LogDiscardPolicy.rejectedExecution():289] Task java.util.concurrent.FutureTask@2bde7b69 rejected from schema-change-pool java.util.concurrent.ThreadPoolExecutor@5fd5dafd[Running, pool size = 10, active threads = 1, queued tasks = 0, completed tasks = 69443]
2023-11-30 15:34:21,949 WARN (schema change|15) [ThreadPoolManager$LogDiscardPolicy.rejectedExecution():289] Task java.util.concurrent.FutureTask@4d8be448 rejected from schema-change-pool java.util.concurrent.ThreadPoolExecutor@5fd5dafd[Running, pool size = 10, active threads = 0, queued tasks = 0, completed tasks = 69446]

be warn

W1130 16:33:32.717034 3557839 olap_server.cpp:669] Have not get FE Master heartbeat yet
W1130 16:33:42.518539 3557930 index_builder.cpp:171] referenced column was missing. [column=app_code referenced_column=-1]
W1130 16:33:42.520421 3557929 index_builder.cpp:171] referenced column was missing. [column=app_code referenced_column=-1]
W1130 16:33:42.518554 3557928 index_builder.cpp:171] referenced column was missing. [column=app_code referenced_column=-1]
E1130 16:33:46.447531 3559156 variable.cpp:179] Already exposed `doris_cache_obj_lrucache' whose value is `0'
E1130 16:33:46.447896 3559156 variable.cpp:179] Already exposed `doris_cache_obj_lrucache_persecond' whose value is `0'
E1130 16:33:46.458463 3559156 variable.cpp:179] Already exposed `doris_cache_data_page_cache' whose value is `0'
E1130 16:33:46.458482 3559156 variable.cpp:179] Already exposed `doris_cache_data_page_cache_persecond' whose value is `0'
E1130 16:33:46.458565 3559156 variable.cpp:179] Already exposed `doris_cache_index_page_cache' whose value is `0'
E1130 16:33:46.458570 3559156 variable.cpp:179] Already exposed `doris_cache_index_page_cache_persecond' whose value is `0'
E1130 16:33:46.458643 3559156 variable.cpp:179] Already exposed `doris_cache_pkindex_page_cache' whose value is `0'
E1130 16:33:46.458647 3559156 variable.cpp:179] Already exposed `doris_cache_pkindex_page_cache_persecond' whose value is `0'
E1130 16:33:46.458699 3559156 variable.cpp:179] Already exposed `doris_cache_row_cache' whose value is `0'
E1130 16:33:46.458704 3559156 variable.cpp:179] Already exposed `doris_cache_row_cache_persecond' whose value is `0'
E1130 16:33:46.458753 3559156 variable.cpp:179] Already exposed `doris_cache_segment_cache' whose value is `0'
E1130 16:33:46.458760 3559156 variable.cpp:179] Already exposed `doris_cache_segment_cache_persecond' whose value is `0'
E1130 16:33:46.458777 3559156 variable.cpp:179] Already exposed `doris_cache_schema_cache' whose value is `0'
E1130 16:33:46.458782 3559156 variable.cpp:179] Already exposed `doris_cache_schema_cache_persecond' whose value is `0'
E1130 16:33:46.458801 3559156 variable.cpp:179] Already exposed `doris_cache_lookup_connection_cache' whose value is `0'
E1130 16:33:46.458806 3559156 variable.cpp:179] Already exposed `doris_cache_lookup_connection_cache_persecond' whose value is `0'
E1130 16:33:46.458842 3559156 variable.cpp:179] Already exposed `doris_cache_inverted_index_searcher_cache' whose value is `0'
E1130 16:33:46.458846 3559156 variable.cpp:179] Already exposed `doris_cache_inverted_index_searcher_cache_persecond' whose value is `0'
E1130 16:33:46.458869 3559156 variable.cpp:179] Already exposed `doris_cache_inverted_index_query_cache' whose value is `0'
E1130 16:33:46.458873 3559156 variable.cpp:179] Already exposed `doris_cache_inverted_index_query_cache_persecond' whose value is `0'
E1130 16:33:46.458986 3559156 variable.cpp:179] Already exposed `doris_cache_lastest_success_channel_cache' whose value is `0'
E1130 16:33:46.458994 3559156 variable.cpp:179] Already exposed `doris_cache_lastest_success_channel_cache_persecond' whose value is `0'
E1130 16:33:46.637516 3559156 variable.cpp:179] Already exposed `doris_cache_tablet_version_cache' whose value is `0'
E1130 16:33:46.637531 3559156 variable.cpp:179] Already exposed `doris_cache_tablet_version_cache_persecond' whose value is `0'
E1130 16:33:46.671253 3559713 variable.cpp:179] Already exposed `doris_cache_delete_bitmap_agg_cache' whose value is `0'
E1130 16:33:46.671267 3559713 variable.cpp:179] Already exposed `doris_cache_delete_bitmap_agg_cache_persecond' whose value is `0'

试过重启fe，drop index

[zong400]

dorp index 然后关闭fe，等be重启好之后在启动fe集群能稳下来
但是下次build index还是挂，是bug？

[LemonLiTree]

Take a look at be.out

[zong400]

*** Query id: 0-0 ***
*** tablet id: 0 ***
*** Aborted at 1701336663 (unix time) try "date -d @1701336663" if you are using GNU date ***
*** Current BE git commitID: ae923f7 ***
*** SIGSEGV address not mapped to object (@0x7f76ffffffff) received by PID 4065711 (TID 4066409 OR 0x7f7566cbb700) from PID 18446744073709551615; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/src/doris-2.0/be/src/common/signal_handler.h:417
 1# 0x00007F76F0CFD577 in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
 3# 0x00007F76F0CF641C in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
 4# 0x00007F76F538C0C0 in /lib/x86_64-linux-gnu/libc.so.6
 5# doris::vectorized::OlapBlockDataConvertor::convert_column_data(unsigned long) at /root/src/doris-2.0/be/src/vec/olap/olap_data_convertor.cpp:222
 6# doris::IndexBuilder::_write_inverted_index_data(std::shared_ptr<doris::TabletSchema>, int, doris::vectorized::Block*) at /root/src/doris-2.0/be/src/olap/task/index_builder.cpp:267
 7# doris::IndexBuilder::handle_single_rowset(std::shared_ptr<doris::RowsetMeta>, std::vector<std::shared_ptr<doris::segment_v2::Segment>, std::allocator<std::shared_ptr<doris::segment_v2::Segment> > >&) at /root/src/doris-2.0/be/src/olap/task/index_builder.cpp:230
 8# doris::IndexBuilder::handle_inverted_index_data() at /root/src/doris-2.0/be/src/olap/task/index_builder.cpp:397
 9# doris::IndexBuilder::do_build_inverted_index() in /data/doris/be/lib/doris_be
10# doris::StorageEngine::_handle_index_change(std::shared_ptr<doris::IndexBuilder>) at /root/src/doris-2.0/be/src/olap/olap_server.cpp:1039
11# doris::StorageEngine::process_index_change_task(doris::TAlterInvertedIndexReq const&) at /root/src/doris-2.0/be/src/olap/olap_server.cpp:1033
12# doris::EngineIndexChangeTask::execute() at /root/src/doris-2.0/be/src/olap/task/engine_index_change_task.cpp:43
13# doris::StorageEngine::execute_task(doris::EngineTask*) at /root/src/doris-2.0/be/src/olap/storage_engine.cpp:1168
14# doris::TaskWorkerPool::_alter_inverted_index_worker_thread_callback() at /root/src/doris-2.0/be/src/agent/task_worker_pool.cpp:354
15# doris::ThreadPool::dispatch_thread() in /data/doris/be/lib/doris_be
16# doris::Thread::supervise_thread(void*) at /root/src/doris-2.0/be/src/util/thread.cpp:470
17# start_thread in /lib/x86_64-linux-gnu/libpthread.so.0
18# __clone in /lib/x86_64-linux-gnu/libc.so.6

[LemonLiTree]

#27276
fixed

[zong400]

Thanks ,but this is not a new column.
Are u sure is the same bug?

[LemonLiTree]

2.0.3 will be released soon, please test it after the release.