local messages in mpool could not be cleanup correctly, which cause message blocked seriously

[narkissos]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

Daemon:  1.15.1+mainnet+git.731da455d.dirty+api1.5.0
Local: lotus version 1.15.1+mainnet+git.731da455d.dirty

Describe the Bug

I want to send fil from one address to another, but I found message was blocked for too many days. And I tried to increase/replace the --gas-feecap/--gas-premium in order to enable the message could be sent, but it didn't work.
Then I tried to run lotus-shed to clear the local messages in mpool, it didn't work as well, everytime I restart the lotus daemon, the deleted messages were added back by lotus. It's weried.

In the log file below, we can see that lotus added back the previous messages like it is crazy

Logging Information

2022-04-23T22:07:51.543+0800    ERROR   messagepool     messagepool/messagepool.go:1521 adding local message: message from f3rv7tkoy2raum2qov7icg4il23636fexncd35bsfa37icuuj5xr2pre3a2axp4bjlaq6irlpfg45jnmhispqa with nonce 2405 already in mpool, increase GasPremium to 7008554 from 3588379 to trigger replace by fee:
    github.com/filecoin-project/lotus/chain/messagepool.(*msgSet).add
        /root/go/src/lotus/chain/messagepool/messagepool.go:275
  - replace by fee has too low GasPremium
2022-04-23T22:07:51.871+0800    ERROR   messagepool     messagepool/messagepool.go:1521 adding local message: message from f3rv7tkoy2raum2qov7icg4il23636fexncd35bsfa37icuuj5xr2pre3a2axp4bjlaq6irlpfg45jnmhispqa with nonce 2405 already in mpool, increase GasPremium to 7008554 from 197271 to trigger replace by fee:
    github.com/filecoin-project/lotus/chain/messagepool.(*msgSet).add
        /root/go/src/lotus/chain/messagepool/messagepool.go:275
  - replace by fee has too low GasPremium
2022-04-23T22:07:54.058+0800    ERROR   messagepool     messagepool/messagepool.go:1521 adding local message: message from f3rv7tkoy2raum2qov7icg4il23636fexncd35bsfa37icuuj5xr2pre3a2axp4bjlaq6irlpfg45jnmhispqa with nonce 2405 already in mpool, increase GasPremium to 7008554 from 4485474 to trigger replace by fee:
    github.com/filecoin-project/lotus/chain/messagepool.(*msgSet).add
        /root/go/src/lotus/chain/messagepool/messagepool.go:275
  - replace by fee has too low GasPremium
2022-04-23T22:07:54.412+0800    ERROR   messagepool     messagepool/messagepool.go:1521 adding local message: message from f3rv7tkoy2raum2qov7icg4il23636fexncd35bsfa37icuuj5xr2pre3a2axp4bjlaq6irlpfg45jnmhispqa with nonce 2405 already in mpool, increase GasPremium to 7008554 from 602028 to trigger replace by fee:
    github.com/filecoin-project/lotus/chain/messagepool.(*msgSet).add
        /root/go/src/lotus/chain/messagepool/messagepool.go:275

Repo Steps

1. run lotus-shed mpool clear --local --really-do-it to clean up local message.
2. run lotus mpool pending --local to confirm that all local messages were removed.
3. restart lotus deamon
4. the removed local messages were added back by lotus.
5. In the log, we can see that lotus adds back the previous messages like it is crazy, and the gas is still very low, which causes the message to be blocked seriously and cannot be deleted.

[Reiers]

Hi @narkissos

Thanks for the report. Looks like you are running Lotus with custom code or adjustment .dirty
Please rebuild to stock lotus, make clean all !

1. lotus-shed should only be used as a last resort.

2. the error message above states the exact number you need to increase the GasPremium too - increase GasPremium to 7008554 from 602028 - if you replace the message with 7009000 as GasPremium, Im pretty sure the the message will go away/land.

I will have to move this to Discussion since I cant repro the issue.

Moving this issue to Lotus Discussions for help and troubleshooting.

[Reiers]

Let me know how it goes. If it doesnt work - lets troubleshoot some more!

[narkissos]

Hi @Reiers, thanks for your kindly reply. I didn't modify the lotus code, maybe when I did git pull, it caused the merge. To be on the safe side, I downloaded a clean code and recompiled it.
I tried to exec lotus mpool replace --gas-premium 260684300 --gas-feecap 9079496880 --gas-limit 59533500 --fee-limit 8 bafy2bzaceacznxpzxzxiolnejoo5dgsbt5euw4kafq4mojcyie4wz7s3soj6e. I guess gas fee/limit is enough now.... but the issue still there...
btw, I can't find any log in lotus.log when I exec gas replace cmd, is it correct?