Lotus Node Running out of Space corrupts datastore(level DB)

[Meatball13]

Describe the bug
Drive hosting lotus daemon and lotus-miner ran out of space and the miner / daemon had errors. lotus-miner will not longer start with the following error.

ERROR: starting node: could not build arguments for function "github.com/filecoin-project/lotus/node/modules/lp2p".StartListening.func1 (/home/tbtminer/lotus/node/modules/lp2p/addrs.go:98): failed to build host.Host: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build lp2p.BaseIpfsRouting: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.MetadataDS: received non-nil error from function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): opening datastore /metadata: leveldb/journal: block/chunk corrupted: chunk length overflows block (3280 bytes) [file=000145.log]

To Reproduce
Steps to reproduce the behavior:

1. Run lotus daemon/lotus-miner.
2. Run out of space on drive.
3. Daemon/miner will start throwing errors. Kill them and then restart.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Version (run lotus version):
lotus version 0.7.0+git.232cc320

Additional context
Log file referenced is attached.
000145.log

[Meatball13]

Quick update. On a hunch I renamed that 00145.log file and restarted lotus-miner and it appears to have restarted without a problem. Log file must get corrupted and then lotus-miner can't restart. So, still a bug there, but seems to have a workaround.

[modnarwolf]

renaming didnt work,, I just deleted it 😜

[RobQuistNL]

Related: #1958

[RobQuistNL]

More: #3837

[jennijuju]

You can find some level db recovery via google, for example

$ python
…
>>> import leveldb
>>> leveldb.RepairDB(“)
``

[jennijuju]

Moving this to discussion for folks to share solutions

[s0nik42]

SAme problem to me, I'm going to try @Meatball13 workaround

[s0nik42]

Meatball method in my case solve the issue.

[jennijuju]

Whaaat! That’s amazing!! Glad you are able to proceed now!

[jennijuju]

Are you able to pledge new sectors afterwards?

[kikakkz]

miner的重启对于metadata和journal是不安全的，迄今为止我遇到过两种corrupt会导致miner无法重新运行。
类型一：
重启miner，miner将会报告某个步骤的replay失败，这意味着最后的日志文件写入并不完整，此时可以停止miner，将$LOTUS_STORAGE_PATH/journal中时间最新的文件删除，再次重启miner。考虑到journal在每个已经split的文件是完备的，这一操作只会丢失一些重放步骤，应该是安全的。
类型二：
重启miner，miner汇报

这意味着miner在重启的时候，由于不完整的写入，丢失了‘miner-address‘相关的leveldb内容。这个地方最正确的做法，是在python里面查询leveldb的对应key，仿照其他正在运行的矿工的正确格式，重新设置正确后继续运行miner，并一路修复，缺什么key修复什么key。但是这里我想分享一点我走的弯路：
我最开始也感觉一脸懵逼，因为我不了解filecoin的leveldb结构，所以我把原来的storage mv了，重新init了一个storage，然后miner正常运行了，但是pledge的时候，sector id编程了21，在corrupt之前已经是1000+了，这意味着如果我不把start sector id设置到1000+，我会做很长一段时间杨白劳，并且会丢算力。以下是设置sector id的步骤：
1 安装python和pip
2 用pip安装leveldb
3 执行python进入python console
4 执行 import leveldb
5 执行 db = leveldb.LevelDB('/opt/data/lotusstorage/datastore/metadata')，其中的路径应该修改成你的矿工metadata路径
6 执行 db.Get(bytes('/storage/nextid', encoding="utf-8")) 可以得到下一个pledge的sector id，为什么我知道是这个key，来自于代码和错误日志，O(∩_∩)O哈哈~
7 执行 db.Put(b'/storage/nextid', '\xc6\x02'.encode())，注意第二个参数的值，我是从一个已经pledge到30000+的miner查询过来的，设置到我的miner即可。我目前没有深研第二个参数的编码与存储方式。
8 执行miner。
上面这些步骤意味着我删除掉metadata的尝试是一条弯路，最开始我发现差miner-address的时候，应该直接通过leveldb的接口查询miner-address，并且重新设置即可。
以上~

[kikakkz]

补充一下：如果是删除了重新设置sector id，lotus-miner sectors xxxx命令将会看不到旧扇区的任何信息，但是新的pledging扇区是正常的。

[jennijuju]

To folks here: I think #4133 may help with the situation!

[funktafide]

I had the same issue. Used the workaround from @Meatball13 and all is good.