failed to fetch internal transactions for blocks - batch too large

[budisetiawan760]

i get multiple this error log, i already set INDEXER_INTERNAL_TRANSACTIONS_BATCH_SIZE from 1 to 1200, still get same error
when i check the block usually it have 800 - 3000 tx, what should i do to get this log away? or maybe i get wrong env to set for this?

{"time":"2024-07-22T05:43:59.411Z","severity":"error","message":"failed to fetch internal transactions for blocks [2789018]: ** (ErlangError) Erlang error: [%{code: -32600, data: %{block_number: 2789018, transaction_index: 529, transaction_hash: \"0x8c5cc1e9d359652eabef66a9eff395ec564f6883bf8d7650d8b21cc4b98bc5ac\"}, message: \"batch too large\"}]","metadata":{"count":1,"fetcher":"internal_transaction","error_count":1}}

[sl1depengwyn]

This error comes from your rpc node, make sure it's in archive state

[budisetiawan760]

i think its already in archive state, i already run with --gcmode archive and already connect read node with old historical data

[ekawibisono]

Hello @budisetiawan760 Can you try this ?

INDEXER_DISABLE_INTERNAL_TRANSACTIONS_FETCHER=true

[budisetiawan760]

but we need to fetch internal transaction, what is the unit to count from this batch? number of internal tx or size byte in internal tx?

[ekawibisono]

Can you try this again

[budisetiawan760]

can i finish block fetch first and then sync internal transaction? how can i speed up the block indexing sync? is there any way to do this because i already run testnet sync a week still not sync total block only 3 million with 32 transaction but seems take long time to fetch, i already use highest instance i can afford (8 cpu and 32 ram) but still need a week and today still 62%, is this because failed internal fetch?

i was use default env only change connection and seems it take very long time, my read node always 100% so i thought it already do maximum effort but it seems a week only take 62%

now i set
INDEXER_DISABLE_INTERNAL_TRANSACTIONS_FETCHER=true
INDEXER_CATCHUP_BLOCKS_BATCH_SIZE=1500
INDEXER_MEMORY_LIMIT=25Gb

with postgres connection 200

any idea to faster it up, if there is a good idea i can upgrade max 128 gb ram 32cpu but seems need good config from the env

and some info i think highest internal transaction is around 1.5 MB per transaction, and highest transaction per block is around 2000-3000 transaction (we doing load test few times i think, around 5-6 times last month )