Node stopped synchronizing after usage of the new docker image

[MirekR]

Hi @maebeam, I'm back with server stopped synchronisation.

After yesterday updated of the docker image to "19c446547510f1e0b83d56611e732e3fa6a0b32d" server stopped syncing, I can see last post 12h ago. It has also started ignoring super_admins, at the moment I can see only new post but not server stats ( I was able to for some time yesterday).

Docker compose file:

version: "3.7"
services:
  backend:
    container_name: backend
    image: docker.io/bitclout/backend:19c446547510f1e0b83d56611e732e3fa6a0b32d
    command: run
    volumes:
    - db:/db
    - ./:/bitclout/run  
    ports:
    - 17001:17001
    - 17000:17000
    env_file:
    - dev.env
    expose:
    - "17001"
    - "17000"
  frontend:
    container_name: frontend 
    image: docker.io/bitclout/frontend:23d22a586e70b2f6700f01ab4feabe98e53ea991
    ports:
    - 8080:8080
    volumes:
    - ./:/app
    env_file:
    - dev.env
    expose:
    - "8080"
  nginx: 
    container_name: nginx
    image: nginx:latest
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
    volumes:
      - ./nginx.dev:/etc/nginx/nginx.conf
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    depends_on: 
      - backend
      - frontend
    ports:
      - 80:80
      - 443:443
  certbot:
    image: certbot/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
volumes:
  db:

Full logs attached.
full.log

[tijno]

@MirekR anything in dmesg like running out of memory, too many open files or disc space?

Main reason i have seen nodes crash or stop syncing is for those 2

[tijno]

Ah looking at your logs you did not clear the data in /db volume. As per the changelog on backend you need to clear and resync.

[maebeam]

Please reopen this if wiping the /db volume does not resolve.

[marnimelrose]

I did clear the db and I am stuck, should we re-open?

[maebeam]

Sure

[MirekR]

We shouldn't need to wipe /db volume everytime change comes in, it kills the global feed as well.

[tijno]

@MikeKR We haven't had to - before NFTs mine has not had to resync for 2 months.

Also the main issue with resync is not so much the blocks - they happen relatively quickly.

Its the TxIndex which is getting replaced by new DB soon that should make it all much faster and easier.

[MirekR]

Performance of re-sync is one question, loosing nodes global feed is another and from the end user perspective more serious issue.

[tijno]

You can keep the global feed in 2 ways:

1 - dont delete the whole db - eg make sure you keep the one in /db/badgerdb/globalstate or

2 - use the config option to load global state from a central node

# The IP:PORT or DOMAIN:PORT corresponding to a node that can be used to
# set/get global state. When this is not provided, global state is set/fetched
# from a local DB. Global state is used to manage things like user data, e.g.
# emails, that should not be duplicated across multiple nodes.
GLOBAL_STATE_REMOTE_NODE=

[MirekR]

Its the TxIndex which is getting replaced by new DB soon that should make it all much faster and easier.

Do I read this correctly as "we'll need to re-sync again soon and loose global again"?

[tijno]

@MirekR Im sure the upgrade can be done in such a way that you keep global state.

And note my reply above about ways to not loose global state.

[maebeam]

We're in the process of the backing store to postgres which will make these updates much less painful and more efficient. Global state migration tools will be provided.

[addanus]

@maebeam may I ask where we are with this?

And... has anyone considered a postgres docker image for a faster sync - at least at some set block height?

I was having this reorg issue as recently as last weekend (9/4/21) - deso-protocol/core#98