ADM Blockchain Recovering Procedure

[al-onyxprotocol]

In ADAMANT blockchain node software, forks may happen if multiple delegate nodes produce blocks at the same time. When this occurs, the network needs to decide which block to keep.

Currently, the consensus rule is:

• Pick the oldest block (assuming the timestamp is trustworthy), or
• If timestamps are equal, choose the block with the smallest ID.

Such short forks happen regularly in blockchains and do not create larger or dangerous forks.

Deleting the Last Block

“Deleting last block” occurs when a fork is detected (either during recovery or upon receiving a new block).

Example case, the fork is caused by reason 5:

The node received a block with the same height and previous block ID, but a different block ID.

Example Log

[WRN] 2025-08-06 00:00:09 | Deleting last block - {"version":0,"totalAmount":12119967437,"totalFee":150000000,"reward":15000000,"payloadHash":"b04ed7f935bec1f74b7a179ff07557a36ca5ea38d6966381f1a5794ca9df7125","timestamp":250066805,"numberOfTransactions":3,"payloadLength":351,"previousBlock":"17914802164676259429","generatorPublicKey":"547919533beb2612d76e3a7d5f38e61145a2b0ecb96d971277f34fb8efd3bcb3","transactions":[{"type":0,"timestamp":250066800,"amount":2893807848,"senderPublicKey":"817c961b57678c2c55bc55358bab9551b7aa65e572a812f50f11c1976aaaebc6","senderId":"U14982290023542615154","asset":{},"recipientId":"U8153746339485828029","signature":"8b9c7f4ac1759f94a5c9cf2a466763bb719cd08bd582c93b8608d395462fdb976fe5194b9e2177d5baee3b7e31489539f3762dc62840f67eaf29b6d57a7f4e05","id":"18272966159919349432","fee":50000000,"relays":3,"receivedAt":"2025-08-06T00:00:03.315Z","blockId":"15500937233239097229","height":48093334,"block_timestamp":250066805},{"type":0,"timestamp":250066800,"amount":3359198799,"senderPublicKey":"817c961b57678c2c55bc55358bab9551b7aa65e572a812f50f11c1976aaaebc6","senderId":"U14982290023542615154","asset":{},"recipientId":"U12130826829283598554","signature":"448b237827ea65288026fb099e74b7b32d6f32ef5986f7bd17dd37ab226c851caa7fcd38f2c566be24f4c458ca4d9318a13557a86a437e993600985f0c2aba03","id":"6994430507695336221","fee":50000000,"relays":3,"receivedAt":"2025-08-06T00:00:03.118Z","blockId":"15500937233239097229","height":48093334,"block_timestamp":250066805},{"type":0,"timestamp":250066800,"amount":5866960790,"senderPublicKey":"817c961b57678c2c55bc55358bab9551b7aa65e572a812f50f11c1976aaaebc6","senderId":"U14982290023542615154","asset":{},"recipientId":"U16963107530576131637","signature":"bd83e38db1ab672be95d669b89b8cf9a1e3d81ad34aa150672abc8ca7d77506d57aac50146403235105b53d610329bbd7d4ad52f3f78f25991b3ba740ef7c305","id":"12712691363231971651","fee":50000000,"relays":3,"receivedAt":"2025-08-06T00:00:03.260Z","blockId":"15500937233239097229","height":48093334,"block_timestamp":250066805}],"blockSignature":"64be4069540ab86f195d3bfb477ac8b2514dc99c33b881d98cdab103101b95e21c3f0cb1d6df5a128977a13992956c5390d7e69b18eb99055250c55d9b734802","height":48093334,"id":"15500937233239097229","relays":3}

Summary

• Forks are resolved automatically by picking the oldest block or the block with the smallest ID
• “Deleting last block” is a normal recovery action when conflicting blocks with the same height are detected
• This ensures the blockchain stays consistent without producing deeper forks

[al-onyxprotocol]

[Bug] Internal Discarded block, a node unable to recover

[adamantmm]

“Deleting last block” ≠ data loss. This action only rolls back the very last block in order to replace it with the correct fork branch. Transactions inside that block are not lost; they are simply re-applied if included in the winning block.

[adamantmm]

@yoxira Do you have something to add?

[metalisk]

Investigation update for adamant#145.

This incident is a combination of several different problems. It should not be treated as one normal short-fork recovery case.

Sequence and findings

1. The old Discarded block warning was partly a logging problem. Release v0.10.0 included the changes from adamant#120: blocks received before startup readiness and already-known/old blocks are no longer reported as alarming discarded blocks. A later readiness guard was added to onReceiveBlock, and commit 8ea50401 (August 15, 2025) fixed a missing sequence callback that could otherwise stop the main queue.

2. NamecheapUS-[ROG]-5 is still running very old node code. It is on v0.7.1-era code from November 2022, not v0.10.0 or current dev. Its current height and top delegate vote weights matched public mainnet nodes during this investigation, so the historical vote-weight mismatch from the issue is not reproducible now. The node is advancing, but its logs contain a very large number of repeated old/stale block deliveries and main-queue warnings. The v0.10.0 logging/readiness changes are therefore not present on that server.

3. The supplied ClientWs.emit fatal error has a separate, deterministic cause. On Clown, wsClient.enabled is false. The ClientWs constructor used return false, but JavaScript ignores primitive return values from a constructor invoked with new. This left a truthy, partially initialized object without describes and logger. During applyUnconfirmed, code called clientWs.emit(): the first access failed, and the catch block then failed again on this.logger.debug, escaping into the dapps domain and terminating the node during synchronization.

   The exact pasted stack is no longer present in the retained PM2 log files, but the deployed configuration and source reproduce its code path exactly. This bug was still present in current dev before the patch prepared for this issue.

4. Clown also shows prolonged stale-block pressure, not only the WebSocket crash. The retained log contains about 1.24 million discarded-block messages, 7,901 synchronization cycles, and 6,360 main-queue warnings. The same blocks were delivered repeatedly while broadhash consensus was often only 3-4%. The final retained shutdown on March 20, 2026 was an explicit PM2 SIGINT with cleanup, rather than an OOM kill.

5. adm-zabbix-node stopped for an operational memory reason, amplified by the queue behavior. The host has about 8 GB RAM, no swap, and several memory-heavy Node.js applications under the same PM2 service.

   • On March 23, 2026, the kernel OOM killer directly killed the ADM process at roughly 2.3 GB anonymous RSS. PM2 restarted it.
   • The restarted node was about 44,900 blocks behind. While it synchronized, live block notifications continued to be appended to the main sequence. The queue reached 15,584 entries at sync completion and then had to drain thousands of stale/future block callbacks.
   • On March 26, 2026, the OOM killer killed another Node.js pool process at roughly 2.5 GB anonymous RSS. Restarting the shared PM2 systemd service stopped ADM as well. The saved PM2 process list restored the pool applications but did not restore ADM, leaving it stopped.

Status in dev

• Already fixed before this investigation: false high-severity reporting for several harmless old/not-ready block cases; the missing onReceiveBlock sequence callback that could deadlock the main queue.
• Not fixed before the current patch: disabled ClientWs creating a truthy incomplete instance; incoming live blocks being appended to the main sequence while the loader is already synchronizing or a round is ticking.
• Not a node-code fix: the host-wide OOM condition and PM2 restore configuration on adm-zabbix-node. That host needs memory headroom (or swap as an emergency buffer), independent PM2/service failure domains where appropriate, and a saved process list that actually includes ADM.

Patch prepared

The patch for this issue:

• makes disabled ClientWs a valid no-op instance, so transaction processing cannot crash through emit();
• rejects live block notifications before adding them to the main sequence when the node is loading, synchronizing, or ticking a round;
• keeps the readiness check inside the queued callback as a second check, because state can change after enqueue;
• adds unit coverage for both cases.

This does not change block validation, serialization, consensus ordering, replay behavior, or activation rules. Synchronization continues through the loader; the change prevents redundant live notifications from consuming the serialized state-mutation queue while that loader owns block application.

Validation completed for the patch:

• focused unit tests: 6 passing;
• full unit suite: 1,050 passing, with one unrelated pre-existing failure in transaction sorting (Unknown transaction type 5), reproducible in isolation;
• targeted ESLint: 0 errors (existing JSDoc warnings remain);
• full API run reached a healthy synchronized testnet node but the Mocha process stalled in the existing blocks cache suite and was stopped after no progress; the node was then shut down gracefully and completed cleanup.

The practical recovery advice remains: first identify whether the node is merely receiving old blocks, has a blocked main sequence, crashed during unconfirmed transaction re-application, or was killed by the operating system. Deleting the last block is appropriate only for a confirmed short-fork recovery; it does not address these queue, crash, OOM, or process-manager causes.

[adamant-al]

@metalisk Thank you for solid explanation.