SIGILL: illegal instruction

[skyqhxiao]

My system is working well with lotus 0.5. After upgrading with 0.6, 0.7 and 0.71, it keeps failing with illgal instruction issue. It is a dual Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz Dell R730 server.
I tried to recompile lotus and tried different go and rustup version. No go.
Here is the log,
[bafy2bzacea4j4zdyi7uw5qongenik43yeuq4pq6cn7h3x47mhxkp2xi5v3xxe bafy2bzacecw6tw75bbb6bploiz7cxruhihvxkryntotk4pywyszutjkrkvhm2] from 12D3KooWGw128i8A9fPXy2uTJxcR3UkKk8hoPKBor95gq472153a
2020-09-18T14:57:30.352+0800 �[34mINFO�[0m hello hello/hello.go:180 time offset {"offset": -0.038513225, "peerid": "12D3KooWFPqr7BZPFDQ1aeCQJWt5XQXtd6nRRKg7xhcn4RYHQxUp"}
SIGILL: illegal instruction
PC=0x17f516e m=27 sigcode=2
instruction bytes: 0xf3 0x4c 0xf 0x38 0xf6 0xcf 0x66 0x4c 0xf 0x38 0xf6 0xd5 0xc4 0xe2 0xc3 0xf6

goroutine 0 [idle]:
runtime: unknown pc 0x17f516e
stack: frame={sp:0x7ff0afffe8c8, fp:0x0} stack=[0x7ff0af7feec8,0x7ff0afffeac8)
00007ff0afffe7c8: 000000c08270d510 000000c08270e000
00007ff0afffe7d8: 000000c08270d000 000000c08270d000
00007ff0afffe7e8: 000000c08270e000 fffffffffed82000
00007ff0afffe7f8: 000000000177c0c5 <github.com/filecoin-project/lotus/chain/sub.(*MessageValidator).Validate+869> 00000007009d32bd
00007ff0afffe808: 0000000000cbb6d7 <github.com/filecoin-project/lotus/chain/types.(*Message).ToStorageBlock+119> 000000020044f9a5
00007ff0afffe818: 0000000000000000 0000000000000000
00007ff0afffe828: 0000000000cbc6b4 <github.com/filecoin-project/lotus/chain/types.(*SignedMessage).Cid+52> 00000001004546c8
00007ff0afffe838: 00000000010e9c66 <github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).checkMessage+614> 000000010065c09f
00007ff0afffe848: 0000000000000000 0000000000000000
00007ff0afffe858: 0000000000000000 0000000000000000
00007ff0afffe868: 000000000074bdf1 <github.com/filecoin-project/go-state-types/abi.cidBuilder.Sum+113> 00000001001c596a
00007ff0afffe878: 0000000000c20e58 <github.com/libp2p/go-libp2p-pubsub.(*validation).validate.func1+120> 000000010041b7cb
00007ff0afffe888: 00000000010ea30e <github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).VerifyMsgSig+46> 000000000065c422 <text/tabwriter.(*Writer).Write+34>
00007ff0afffe898: 00000000006cba38 <github.com/multiformats/go-multihash.Sum+216> 0000000100184849
00007ff0afffe8a8: 0000000000000000 0000000000000000
00007ff0afffe8b8: 0000000000c1cb7b <github.com/libp2p/go-libp2p-pubsub.(*topicVal).validateMsg+251> 0000000100419535
00007ff0afffe8c8: <00000000017f50eb 89f3fffcfffcfffd
00007ff0afffe8d8: 2c15144bf9f604e8 000000000065c306 <text/tabwriter.(*Writer).flushNoDefers+198>
00007ff0afffe8e8: 00007ff0afffe920 00007ff0afffe950
00007ff0afffe8f8: 0000000002f28c80 0000000002f28f20
00007ff0afffe908: 000000c07c638300 00000000000000b6
00007ff0afffe918: 00000000017ea14a aaaa380493c31afc
00007ff0afffe928: caca09e6539cb9f8 61eb38d123b4a787
00007ff0afffe938: 06062bccb7fcbb25 61d57dc4b24d3965
00007ff0afffe948: 0c5e8cead54503de aaaa380493c31afc
00007ff0afffe958: caca09e6539cb9f8 61eb38d123b4a787
00007ff0afffe968: 06062bccb7fcbb25 61d57dc4b24d3965
00007ff0afffe978: 0c5e8cead54503de c2c8f21affb1544d
00007ff0afffe988: 81399d55ed5b82f7 c9403a7c6227c0a0
00007ff0afffe998: bb4a127c1c3ae6e7 82c5546f87f37274
00007ff0afffe9a8: 160abd531b912e9a 0000000000000049
00007ff0afffe9b8: 0000000004859ac8 0000000004859ac8
runtime: unknown pc 0x17f516e
stack: frame={sp:0x7ff0afffe8c8, fp:0x0} stack=[0x7ff0af7feec8,0x7ff0afffeac8)
00007ff0afffe7c8: 000000c08270d510 000000c08270e000
00007ff0afffe7d8: 000000c08270d000 000000c08270d000
00007ff0afffe7e8: 000000c08270e000 fffffffffed82000
00007ff0afffe7f8: 000000000177c0c5 <github.com/filecoin-project/lotus/chain/sub.(*MessageValidator).Validate+869> 00000007009d32bd
00007ff0afffe808: 0000000000cbb6d7 <github.com/filecoin-project/lotus/chain/types.(*Message).ToStorageBlock+119> 000000020044f9a5
00007ff0afffe818: 0000000000000000 0000000000000000
00007ff0afffe828: 0000000000cbc6b4 <github.com/filecoin-project/lotus/chain/types.(*SignedMessage).Cid+52> 00000001004546c8
00007ff0afffe838: 00000000010e9c66 <github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).checkMessage+614> 000000010065c09f
00007ff0afffe848: 0000000000000000 0000000000000000
00007ff0afffe858: 0000000000000000 0000000000000000
00007ff0afffe868: 000000000074bdf1 <github.com/filecoin-project/go-state-types/abi.cidBuilder.Sum+113> 00000001001c596a
00007ff0afffe878: 0000000000c20e58 <github.com/libp2p/go-libp2p-pubsub.(*validation).validate.func1+120> 000000010041b7cb
00007ff0afffe888: 00000000010ea30e <github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).VerifyMsgSig+46> 000000000065c422 <text/tabwriter.(*Writer).Write+34>
00007ff0afffe898: 00000000006cba38 <github.com/multiformats/go-multihash.Sum+216> 0000000100184849
00007ff0afffe8a8: 0000000000000000 0000000000000000
00007ff0afffe8b8: 0000000000c1cb7b <github.com/libp2p/go-libp2p-pubsub.(*topicVal).validateMsg+251> 0000000100419535
00007ff0afffe8c8: <00000000017f50eb 89f3fffcfffcfffd
00007ff0afffe8d8: 2c15144bf9f604e8 000000000065c306 <text/tabwriter.(*Writer).flushNoDefers+198>
00007ff0afffe8e8: 00007ff0afffe920 00007ff0afffe950
00007ff0afffe8f8: 0000000002f28c80 0000000002f28f20
00007ff0afffe908: 000000c07c638300 00000000000000b6
00007ff0afffe918: 00000000017ea14a aaaa380493c31afc
00007ff0afffe928: caca09e6539cb9f8 61eb38d123b4a787
00007ff0afffe938: 06062bccb7fcbb25 61d57dc4b24d3965
00007ff0afffe948: 0c5e8cead54503de aaaa380493c31afc
00007ff0afffe958: caca09e6539cb9f8 61eb38d123b4a787
00007ff0afffe968: 06062bccb7fcbb25 61d57dc4b24d3965
00007ff0afffe978: 0c5e8cead54503de c2c8f21affb1544d
00007ff0afffe988: 81399d55ed5b82f7 c9403a7c6227c0a0
00007ff0afffe998: bb4a127c1c3ae6e7 82c5546f87f37274
00007ff0afffe9a8: 160abd531b912e9a 0000000000000049
00007ff0afffe9b8: 0000000004859ac8 0000000004859ac8

goroutine 4175 [syscall]:
runtime.cgocall(0x17e0900, 0xc08148f4b8, 0xc07e30f200)
/usr/local/go/src/runtime/cgocall.go:133 +0x5b fp=0xc08148f488 sp=0xc08148f450 pc=0x51bbbb
github.com/supranational/blst/bindings/go._Cfunc_blst_p2_uncompress(0xc07c638300, 0xc07c63a541, 0xc000000000)
_cgo_gotypes.go:695 +0x4d fp=0xc08148f4b8 sp=0xc08148f488 pc=0x113374d
github.com/supranational/blst/bindings/go.(_Ctype_struct___5).Uncompress(0xc07c638300, 0xc07c63a541, 0x60, 0x60, 0x28f3620)
/home/gogo/lotus/extern/fil-blst/blst/bindings/go/blst.go:1598 +0x48 fp=0xc08148f4e0 sp=0xc08148f4b8 pc=0x1139d68
github.com/supranational/blst/bindings/go.(_Ctype_struct___5).AggregateVerifyCompressed.func1(0x28f3620)
/home/gogo/lotus/extern/fil-blst/blst/bindings/go/blst.go:357 +0xc5 fp=0xc08148f528 sp=0xc08148f4e0 pc=0x113bae5
github.com/supranational/blst/bindings/go.coreAggregateVerifyPkInG1(0xc08148f6b0, 0xc07e30ddd0, 0xc080ca7400, 0x1, 0x1, 0xc081486660, 0x2b, 0x2b, 0xc08148f6af, 0x1, ...)
/home/gogo/lotus/extern/fil-blst/blst/bindings/go/blst.go:461 +0x2a2 fp=0xc08148f648 sp=0xc08148f528 pc=0x1135d42
github.com/supranational/blst/bindings/go.(_Ctype_struct___5).AggregateVerifyCompressed(0xc08148f818, 0xc07c63a541, 0x60, 0x60, 0xc080ca73e0, 0x1, 0x1, 0xc080ca7400, 0x1, 0x1, ...)
/home/gogo/lotus/extern/fil-blst/blst/bindings/go/blst.go:383 +0x1a5 fp=0xc08148f6e0 sp=0xc08148f648 pc=0x1135a25
github.com/supranational/blst/bindings/go.(_Ctype_struct___5).VerifyCompressed(0xc08148f818, 0xc07c63a541, 0x60, 0x60, 0xc081486630, 0x30, 0x30, 0xc081486600, 0x26, 0x30, ...)
/home/gogo/lotus/extern/fil-blst/blst/bindings/go/blst.go:296 +0x225 fp=0xc08148f780 sp=0xc08148f6e0 pc=0x11354e5
github.com/filecoin-project/lotus/lib/sigs/bls.blsSigner.Verify(0xc07c63a541, 0x60, 0x60, 0xc080cb9540, 0x31, 0xc081486600, 0x26, 0x30, 0x0, 0xc0000f80a0)
/home/gogo/lotus/lib/sigs/bls/init.go:53 +0x1db fp=0xc08148f8f0 sp=0xc08148f780 pc=0x113e57b
github.com/filecoin-project/lotus/lib/sigs/bls.(*blsSigner).Verify(0x52f1e30, 0xc07c63a541, 0x60, 0x60, 0xc080cb9540, 0x31, 0xc081486600, 0x26, 0x30, 0x30, ...)
:1 +0x9d fp=0xc08148f950 sp=0xc08148f8f0 pc=0x113ea1d
github.com/filecoin-project/lotus/lib/sigs.Verify(0xc07c631d20, 0xc080cb9540, 0x31, 0xc081486600, 0x26, 0x30, 0xc0003ffe88, 0x1)
/home/gogo/lotus/lib/sigs/sigs.go:48 +0xf9 fp=0xc08148f9c8 sp=0xc08148f950 pc=0xd856d9
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).VerifyMsgSig(0xc077f302c0, 0xc07c631cb0, 0x1, 0x0)
/home/gogo/lotus/chain/messagepool/messagepool.go:587 +0xd6 fp=0xc08148fa28 sp=0xc08148f9c8 pc=0x10ea3b6
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).checkMessage(0xc077f302c0, 0xc07c631cb0, 0xc080ce0d78, 0xc08270db01)
/home/gogo/lotus/chain/messagepool/messagepool.go:533 +0x267 fp=0xc08148fad8 sp=0xc08148fa28 pc=0x10e9c67
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).Add(0xc077f302c0, 0xc07c631cb0, 0x0, 0x0)
/home/gogo/lotus/chain/messagepool/messagepool.go:542 +0x59 fp=0xc08148fb50 sp=0xc08148fad8 pc=0x10e9e79
github.com/filecoin-project/lotus/chain/sub.(*MessageValidator).Validate(0xc077f41640, 0x2eca260, 0xc076143700, 0xc07d9c9740, 0x26, 0xc085969680, 0x4a6f876465ed)
/home/gogo/lotus/chain/sub/incoming.go:543 +0x366 fp=0xc08148fc78 sp=0xc08148fb50 pc=0x177c0c6
github.com/filecoin-project/lotus/chain/sub.(*MessageValidator).Validate-fm(0x2eca260, 0xc076143700, 0xc07d9c9740, 0x26, 0xc085969680, 0xc078422298)
/home/gogo/lotus/chain/sub/incoming.go:529 +0x5c fp=0xc08148fcc0 sp=0xc08148fc78 pc=0x178c35c
github.com/libp2p/go-libp2p-pubsub.(*topicVal).validateMsg(0xc0784195f0, 0x2eca260, 0xc076143700, 0xc07d9c9740, 0x26, 0xc085969680, 0x0)
/usr/local/go/path/pkg/mod/github.com/libp2p/[email protected]/validation.go:430 +0xfc fp=0xc08148fd68 sp=0xc08148fcc0 pc=0xc1cb7c
github.com/libp2p/go-libp2p-pubsub.(*validation).validateSingleTopic(0xc0005ddfb0, 0xc0784195f0, 0xc07d9c9740, 0x26, 0xc085969680, 0x0)
/usr/local/go/path/pkg/mod/github.com/libp2p/[email protected]/validation.go:408 +0xab fp=0xc08148fde8 sp=0xc08148fd68 pc=0xc1c96b
github.com/libp2p/go-libp2p-pubsub.(*validation).validateTopic(0xc0005ddfb0, 0xc07c3a6238, 0x1, 0x1, 0xc07d9c9740, 0x26, 0xc085969680, 0x0)
/usr/local/go/path/pkg/mod/github.com/libp2p/[email protected]/validation.go:357 +0x3aa fp=0xc08148feb0 sp=0xc08148fde8 pc=0xc1c86a
github.com/libp2p/go-libp2p-pubsub.(*validation).doValidateTopic(0xc0005ddfb0, 0xc07c3a6238, 0x1, 0x1, 0xc07d9c9740, 0x26, 0xc085969680, 0x0)
/usr/local/go/path/pkg/mod/github.com/libp2p/[email protected]/validation.go:328 +0x8e fp=0xc08148ff60 sp=0xc08148feb0 pc=0xc1c0ce
github.com/libp2p/go-libp2p-pubsub.(*validation).validate.func1(0xc0005ddfb0, 0xc080ca72a0, 0xc07d9c9740, 0x26, 0xc085969680, 0xc080ce0d70)
/usr/local/go/path/pkg/mod/github.com/libp2p/[email protected]/validation.go:298 +0x79 fp=0xc08148ffb0 sp=0xc08148ff60 pc=0xc20e59
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc08148ffb8 sp=0xc08148ffb0 pc=0x588861
created by github.com/libp2p/go-libp2p-pubsub.(*validation).validate
/usr/local/go/path/pkg/mod/github.com/libp2p/[email protected]/validation.go:297 +0x4a5

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7ff264758238, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:220 +0x55
internal/poll.(*pollDesc).wait(0xc00082ca18, 0x72, 0x0, 0x0, 0x2b4223f)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00082ca00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc00082ca00, 0x20301d, 0xc015983380, 0xc077e2c540)
/usr/local/go/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc077e2ac00, 0x50, 0x50, 0x29a37a0)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc077e2ac00, 0x2a0ac60, 0x7ff29d1f17d0, 0xc0005a0400, 0x30)
/usr/local/go/src/net/tcpsock.go:261 +0x65
github.com/multiformats/go-multiaddr/net.(*maListener).Accept(0xc077e2ac80, 0xc000181b30, 0xc0005f1510, 0x5265d8, 0x30)
/usr/local/go/path/pkg/mod/github.com/multiformats/[email protected]/net/net.go:243 +0x37
github.com/multiformats/go-multiaddr/net.(*netListenerAdapter).Accept(0xc077e2c4f0, 0x2a0ac60, 0xc077e28d50, 0x2860f00, 0x512e390)
/usr/local/go/path/pkg/mod/github.com/multiformats/[email protected]/net/net.go:222 +0x33
net/http.(*Server).Serve(0xc07698c460, 0x2ec0220, 0xc077e2c4f0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2937 +0x266
main.serveRPC(0x2ef6560, 0xc0003a4700, 0xc078463cc0, 0x2ee6a20, 0xc07853c060, 0xc000182900, 0x0, 0x0)
/home/gogo/lotus/cmd/lotus/rpc.go:86 +0x5f4
main.glob..func2(0xc000564700, 0x0, 0x0)
/home/gogo/lotus/cmd/lotus/daemon.go:288 +0x1316
github.com/urfave/cli/v2.(*App).RunAsSubcommand(0xc0000ec600, 0xc0005642c0, 0x0, 0x0)
/usr/local/go/path/pkg/mod/github.com/urfave/cli/[email protected]/app.go:432 +0x9bf
github.com/urfave/cli/v2.(*Command).startApp(0x5152500, 0xc0005642c0, 0x2b409f1, 0x7ffd01798774)
/usr/local/go/path/pkg/mod/github.com/urfave/cli/[email protected]/command.go:279 +0x6bb
github.com/urfave/cli/v2.(*Command).Run(0x5152500, 0xc0005642c0, 0x0, 0x0)
/usr/local/go/path/pkg/mod/github.com/urfave/cli/[email protected]/command.go:94 +0x9cd
github.com/urfave/cli/v2.(*App).RunContext(0xc0000ec480, 0x2eca2a0, 0xc00019c000, 0xc0001a8020, 0x2, 0x2, 0x0, 0x0)
/usr/local/go/path/pkg/mod/github.com/urfave/cli/[email protected]/app.go:306 +0x81f
github.com/urfave/cli/v2.(*App).Run(...)
/usr/local/go/path/pkg/mod/github.com/urfave/cli/[email protected]/app.go:215
github.com/filecoin-project/lotus/cli.RunApp(0xc0000ec480)
/home/gogo/lotus/cli/helper.go:34 +0x7c
main.main()
/home/gogo/lotus/cmd/lotus/main.go:73 +0x631

goroutine 19 [select]:
github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc000197ad0)
/usr/local/go/path/pkg/mod/github.com/ipfs/[email protected]/writer/writer.go:71 +0x128
created by github.com/ipfs/go-log/writer.NewMirrorWriter
/usr/local/go/path/pkg/mod/github.com/ipfs/[email protected]/writer/writer.go:36 +0xb9

goroutine 20 [runnable]:
go.opencensus.io/stats/view.encodeWithKeys(0x0, 0x0, 0x0, 0x0, 0xc078501dd8, 0x5289e5, 0xc078501e88)
/usr/local/go/path/pkg/mod/[email protected]/stats/view/collector.go:61 +0x376
go.opencensus.io/stats/view.(*viewInternal).addSample(0xc07846fbe0, 0x0, 0x3ff0000000000000, 0x0, 0xbfd13476a2ed4d8e, 0xd4868bfb, 0x52bdd60)
/usr/local/go/path/pkg/mod/[email protected]/stats/view/view.go:160 +0x5e
go.opencensus.io/stats/view.(*recordReq).handleCommand(0xc082836e00, 0xc0001de400)
/usr/local/go/path/pkg/mod/[email protected]/stats/view/worker_commands.go:166 +0x225
go.opencensus.io/stats/view.(*worker).start(0xc0001de400)
/usr/local/go/path/pkg/mod/[email protected]/stats/view/worker.go:278 +0x1bb
created by go.opencensus.io/stats/view.init.0
/usr/local/go/path/pkg/mod/[email protected]/stats/view/worker.go:34 +0x68

goroutine 15 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:147 +0x9d
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:150 +0x45

goroutine 67 [chan receive]:
github.com/filecoin-project/lotus/cli.ReqContext.func1(0xc0001c8000, 0xc000580080)
/home/gogo/lotus/cli/cmd.go:271 +0x34
created by github.com/filecoin-project/lotus/cli.ReqContext
/home/gogo/lotus/cli/cmd.go:270 +0x108

[skyqhxiao]

fixed by this parameter

If you are building Lotus 0.7.1 and have an older Intel or AMD processor, set:

export CGO_CFLAGS="-D__BLST_PORTABLE__"