Error running milestone solidification thread on 1.8.6

[yillkid]

Bug description

lastSnapshottedMilestoneIndex / latestSolidSubtangleMilestoneIndex stuck after upgrade IRI from 1.85. to 1.8.6.

{
    "appName": "IRI",
    "appVersion": "1.8.6-COACH",
    "coordinatorAddress": "EQSAUZXULTTYZCLNJNTXQTQHOMOFZERHTCGTXOLTVAHKSA9OGAZDEKECURBRIXIJWNPFCQIOVFVVXJVD9",
    "dbSizeInBytes": 270028399542,
    "duration": 0,
    "features": [
        "dnsRefresher",
        "zeroMessageQueue",
        "RemotePOW",
        "WereAddressesSpentFrom"
    ],
    "jreAvailableProcessors": 4,
    "jreFreeMemory": 404790688,
    "jreMaxMemory": 4294967296,
    "jreTotalMemory": 1324351488,
    "jreVersion": "11.0.7",
    "lastSnapshottedMilestoneIndex": 1050000,
    "latestMilestone": "999999999999999999999999999999999999999999999999999999999999999999999999999999999",
    "latestMilestoneIndex": 0,
    "latestSolidSubtangleMilestone": "999999999999999999999999999999999999999999999999999999999999999999999999999999999",
    "latestSolidSubtangleMilestoneIndex": 1050000,
    "milestoneStartIndex": 1050381,
    "neighbors": 4,
    "packetsQueueSize": 0,
    "time": 1592193158456,
    "tips": 749,
    "transactionsToRequest": 0
}

IRI config:

[IRI]
HEADLESS = true

# Uncommend this line and set user and password
# in the format: `user:password` to password protect
# the IRI API. Change requires restart of iri.
# If enabled, the API will have to be called using
# basic auth. For example, with curl:
# curl http://user:password@localhost:14265 ...
#REMOTE_AUTH = iota:password

# set max requests value
#MAX_REQUESTS_LIST = 9999

# set max find transactions value
#MAX_FIND_TRANSACTIONS = 100000

ZMQ_ENABLED = true
ZMQ_PORT = 5556

; Local Snapshots Settings
LOCAL_SNAPSHOTS_ENABLED = false
LOCAL_SNAPSHOTS_DEPTH = 200
LOCAL_SNAPSHOTS_PRUNING_ENABLED = false
LOCAL_SNAPSHOTS_PRUNING_DELAY = 40000
LOCAL_SNAPSHOTS_INTERVAL_SYNCED = 10
LOCAL_SNAPSHOTS_INTERVAL_UNSYNCED = 100
TIP_SOLIDIFIER_ENABLED = false

; *Network rewrite options*
# Defines the interval at which to try to reconnect/disconnect wanted neighbors.
#RECONNECT_ATTEMPT_INTERVAL_SECONDS = 60

# Controls auto-tethering, this was previously controlled via TESTNET true,
# default is false (also in testnet mode).
#AUTO_TETHERING_ENABLED = false

# Rename of MAX_PEERS, defines the max number of connected neighbors.
MAX_NEIGHBORS = 20

# These two values are configured elsewhere in the iri-playbook's installtion:
# Debian/Ubuntu: /etc/default/iri. Centos: /etc/sysconfig/iri
#NEIGHBORING_SOCKET_ADDRESS
#NEIGHBORING_SOCKET_PORT

Version

• Version: 1.8.6

IOTA network

Which node are you connected to and which IOTA network is it in?

• Node URL: http://node10.uyuma.org:14265
• Network: Mainnet

Hardware specification

• Operating system: Ubuntu 18.04.3 LTS
• RAM: 10 GB
• Number of cores: 4
• Device type:

Steps To reproduce the bug

Explain how the maintainer can reproduce the bug.

1. Follow the guideline Node owners who configured IRI without local snapshots to upgrade IRI
2. Restart iri
3. journalctl -u iri

Expected behaviour

lastSnapshottedMilestoneIndex / latestSolidSubtangleMilestoneIndex will increasing

Actual behaviour

Got Error running milestone solidification thread error and milestone index stuck.

Errors

Paste any errors that you see.

Jun 15 03:38:52 tangle iri[12761]: 06/15 03:38:52.262 [main] ERROR MilestoneSolidifierImpl:121 - Error starting milestone solidification thread
Jun 15 03:38:52 tangle iri[12761]: com.iota.iri.service.ledger.LedgerException: unexpected error while restoring the ledger state
Jun 15 03:38:52 tangle iri[12761]:         at com.iota.iri.service.ledger.impl.LedgerServiceImpl.restoreLedgerState(LedgerServiceImpl.java:86) ~[iri-1.8.6-COACH.jar:1.8.6-COACH]
Jun 15 03:38:52 tangle iri[12761]:         at com.iota.iri.service.milestone.impl.MilestoneSolidifierImpl.start(MilestoneSolidifierImpl.java:111) ~[iri-1.8.6-COACH.jar:1.8.6-COACH]
Jun 15 03:38:52 tangle iri[12761]:         at com.iota.iri.Iota.init(Iota.java:200) ~[iri-1.8.6-COACH.jar:1.8.6-COACH]
Jun 15 03:38:52 tangle iri[12761]:         at com.iota.iri.IRI$IRILauncher.main(IRI.java:135) ~[iri-1.8.6-COACH.jar:1.8.6-COACH]
Jun 15 03:38:52 tangle iri[12761]:         at com.iota.iri.IRI.main(IRI.java:64) ~[iri-1.8.6-COACH.jar:1.8.6-COACH]
Jun 15 03:38:52 tangle iri[12761]: Caused by: com.iota.iri.service.snapshot.SnapshotException: failed to replay the state of the ledger
Jun 15 03:38:52 tangle iri[12761]:         at com.iota.iri.service.snapshot.impl.SnapshotServiceImpl.replayMilestones(SnapshotServiceImpl.java:133) ~[iri-1.8.6-COACH.jar:1.8.6-COACH]
Jun 15 03:38:52 tangle iri[12761]:         at com.iota.iri.service.ledger.impl.LedgerServiceImpl.restoreLedgerState(LedgerServiceImpl.java:83) ~[iri-1.8.6-COACH.jar:1.8.6-COACH]
Jun 15 03:38:52 tangle iri[12761]:         ... 4 common frames omitted
Jun 15 03:38:52 tangle iri[12761]: Caused by: com.iota.iri.service.snapshot.SnapshotException: milestone # 1050001 is missing
Jun 15 03:38:52 tangle iri[12761]:         at com.iota.iri.service.snapshot.impl.SnapshotServiceImpl.replayMilestones(SnapshotServiceImpl.java:109) ~[iri-1.8.6-COACH.jar:1.8.6-COACH]
Jun 15 03:38:52 tangle iri[12761]:         ... 5 common frames omitted
Jun 15 03:38:52 tangle iri[12761]: 06/15 03:38:52.267 [main] INFO  TransactionSolidifier:388 - Starting [Transaction Solidifier] (starts in 0ms / runs every 100ms) ...
Jun 15 03:38:52 tangle iri[12761]: 06/15 03:38:52.517 [main] INFO  TransactionSolidifier:403 - [Transaction Solidifier] Started (execution #1) ...

[GalRogozinski]

I just started a node from scratch (global snapshot) and didn't get an error.
@JakeSCahill when you have time can you try to follow the guide and see if something is wrong?