Many Don´t have rc hash error - Not prticipating in challenges

[ApfelBirneKreis]

Hello devs,

I have an insane ammount of Don´t have RC hash errors. Therefore im not participating in ally challenges...

Logs:
Just an example:

2021-09-05T17:31:35.072 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 67d5fca16b44781ee73d94f6f6e9708a16234ccf3f9a2fc9c63a37b5f2f1d318, caching EOS
2021-09-05T17:31:35.073 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 99720986fbed5175ede3ad5f32df15752259baf865ff23fa147de15f7bf5b436
2021-09-05T17:31:35.118 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 67d5fca16b44781ee73d94f6f6e9708a16234ccf3f9a2fc9c63a37b5f2f1d318, caching EOS
2021-09-05T17:31:35.119 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 99720986fbed5175ede3ad5f32df15752259baf865ff23fa147de15f7bf5b436
2021-09-05T17:31:35.126 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 67d5fca16b44781ee73d94f6f6e9708a16234ccf3f9a2fc9c63a37b5f2f1d318, caching EOS
2021-09-05T17:31:35.127 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 99720986fbed5175ede3ad5f32df15752259baf865ff23fa147de15f7bf5b436
2021-09-05T17:31:35.135 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 67d5fca16b44781ee73d94f6f6e9708a16234ccf3f9a2fc9c63a37b5f2f1d318, caching EOS
2021-09-05T17:31:35.137 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 99720986fbed5175ede3ad5f32df15752259baf865ff23fa147de15f7bf5b436
2021-09-05T17:31:35.333 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 67d5fca16b44781ee73d94f6f6e9708a16234ccf3f9a2fc9c63a37b5f2f1d318, caching EOS
2021-09-05T17:31:35.334 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 99720986fbed5175ede3ad5f32df15752259baf865ff23fa147de15f7bf5b436
2021-09-05T17:31:35.350 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 67d5fca16b44781ee73d94f6f6e9708a16234ccf3f9a2fc9c63a37b5f2f1d318, caching EOS
2021-09-05T17:31:35.350 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 99720986fbed5175ede3ad5f32df15752259baf865ff23fa147de15f7bf5b436
2021-09-05T17:31:35.576 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 67d5fca16b44781ee73d94f6f6e9708a16234ccf3f9a2fc9c63a37b5f2f1d318, caching EOS
2021-09-05T17:31:35.577 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 99720986fbed5175ede3ad5f32df15752259baf865ff23fa147de15f7bf5b436

My config:

ALERTS_URL: https://download.chia.net/notify/mainnet_alert.txt
CHIA_ALERTS_PUBKEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
chia_ssl_ca:
  crt: config/ssl/ca/chia_ca.crt
  key: config/ssl/ca/chia_ca.key
daemon_port: 55400
daemon_ssl:
  private_crt: config/ssl/daemon/private_daemon.crt
  private_key: config/ssl/daemon/private_daemon.key
farmer:
  full_node_peer:
    host: 127.0.0.1
    port: 8444
  harvester_peer:
    host: 127.0.0.1
    port: 8448
  logging: &id001
    log_filename: log/debug.log
    log_level: INFO
    log_maxfilesrotation: 3
    log_stdout: false
    log_syslog: false
    log_syslog_host: 127.0.0.1
    log_syslog_port: 514
  network_overrides: &id002
    config:
      mainnet:
        address_prefix: xch
        default_full_node_port: 8444
      testnet0:
        address_prefix: txch
      testnet1:
        address_prefix: txch
      testnet2:
        address_prefix: txch
      testnet3:
        address_prefix: txch
      testnet4:
        address_prefix: txch
      testnet7:
        address_prefix: txch
        default_full_node_port: 58444
    constants:
      mainnet:
        GENESIS_CHALLENGE: ccd5bb71183532bff220ba46c268991a3ff07eb358e8255a65c30a2dce0e5fbb
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: 3d8765d3a597ec1d99663f6c9816d915b9f68613ac94009884c4addaefcce6af
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: d23da14695a188ae5708dd152263c4db883eb27edeb936178d4d988b8f3ce5fc
        NETWORK_TYPE: 0
      testnet0:
        GENESIS_CHALLENGE: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: 3d8765d3a597ec1d99663f6c9816d915b9f68613ac94009884c4addaefcce6af
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: d23da14695a188ae5708dd152263c4db883eb27edeb936178d4d988b8f3ce5fc
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
      testnet2:
        DIFFICULTY_CONSTANT_FACTOR: 10052721566054
        GENESIS_CHALLENGE: ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: 3d8765d3a597ec1d99663f6c9816d915b9f68613ac94009884c4addaefcce6af
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: d23da14695a188ae5708dd152263c4db883eb27edeb936178d4d988b8f3ce5fc
        INITIAL_FREEZE_PERIOD: 100
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
      testnet3:
        DIFFICULTY_CONSTANT_FACTOR: 10052721566054
        GENESIS_CHALLENGE: ca7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015af
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: 3d8765d3a597ec1d99663f6c9816d915b9f68613ac94009884c4addaefcce6af
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: d23da14695a188ae5708dd152263c4db883eb27edeb936178d4d988b8f3ce5fc
        MEMPOOL_BLOCK_BUFFER: 10
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
      testnet4:
        DIFFICULTY_CONSTANT_FACTOR: 10052721566054
        DIFFICULTY_STARTING: 30
        EPOCH_BLOCKS: 768
        GENESIS_CHALLENGE: dd7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015af
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: 3d8765d3a597ec1d99663f6c9816d915b9f68613ac94009884c4addaefcce6af
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: d23da14695a188ae5708dd152263c4db883eb27edeb936178d4d988b8f3ce5fc
        INITIAL_FREEZE_END_TIMESTAMP: 1618776900
        MEMPOOL_BLOCK_BUFFER: 10
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
      testnet5:
        DIFFICULTY_CONSTANT_FACTOR: 10052721566054
        DIFFICULTY_STARTING: 30
        EPOCH_BLOCKS: 768
        GENESIS_CHALLENGE: ee7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015af
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: 3d8765d3a597ec1d99663f6c9816d915b9f68613ac94009884c4addaefcce6af
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: d23da14695a188ae5708dd152263c4db883eb27edeb936178d4d988b8f3ce5fc
        INITIAL_FREEZE_END_TIMESTAMP: 1618869600
        MEMPOOL_BLOCK_BUFFER: 10
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
      testnet7:
        DIFFICULTY_CONSTANT_FACTOR: 10052721566054
        DIFFICULTY_STARTING: 30
        EPOCH_BLOCKS: 768
        GENESIS_CHALLENGE: 117816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015af
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: 3d8765d3a597ec1d99663f6c9816d915b9f68613ac94009884c4addaefcce6af
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: d23da14695a188ae5708dd152263c4db883eb27edeb936178d4d988b8f3ce5fc
        INITIAL_FREEZE_END_TIMESTAMP: 1619661600
        MEMPOOL_BLOCK_BUFFER: 10
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
  pool_public_keys: !!set
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: null
  pool_share_threshold: 1000
  port: 8447
  rpc_port: 8559
  selected_network: mainnet
  ssl:
    private_crt: config/ssl/farmer/private_farmer.crt
    private_key: config/ssl/farmer/private_farmer.key
    public_crt: config/ssl/farmer/public_farmer.crt
    public_key: config/ssl/farmer/public_farmer.key
  start_rpc_server: true
  xch_target_address: xchxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
full_node:
  database_path: db/blockchain_v1_CHALLENGE.sqlite
  dns_servers:
  - dns-introducer.chia.net
  enable_profiler: false
  enable_upnp: true
  exempt_peer_networks: []
  farmer_peer:
    host: 127.0.0.1
    port: 8447
  introducer_peer:
    host: introducer.chia.net
    port: 8444
  logging: *id001
  max_inbound_farmer: 20
  max_inbound_timelord: 10
  max_inbound_wallet: 10
  network_overrides: *id002
  peer_connect_interval: 30
  peer_db_path: db/peer_table_node.sqlite
  port: 8444
  recent_peer_threshold: 600
  rpc_port: 8555
  sanitize_weight_proof_only: false
  selected_network: mainnet
  send_uncompact_interval: 0
  short_sync_blocks_behind_threshold: 20
  simulator_database_path: sim_db/simulator_blockchain_v1_CHALLENGE.sqlite
  simulator_peer_db_path: sim_db/peer_table_node.sqlite
  ssl:
    private_crt: config/ssl/full_node/private_full_node.crt
    private_key: config/ssl/full_node/private_full_node.key
    public_crt: config/ssl/full_node/public_full_node.crt
    public_key: config/ssl/full_node/public_full_node.key
  start_rpc_server: true
  sync_blocks_behind_threshold: 300
  target_outbound_peer_count: 20
  target_peer_count: 8
  target_uncompact_proofs: 100
  timelord_peer:
    host: 127.0.0.1
    port: 8446
  wallet_peer:
    host: 127.0.0.1
    port: 8449
  weight_proof_timeout: 360
harvester:
  chia_ssl_ca:
    crt: config/ssl/ca/chia_ca.crt
    key: config/ssl/ca/chia_ca.key
  farmer_peer:
    host: 127.0.0.1
    port: 8447
  logging: *id001
  network_overrides: *id002
  num_threads: 30
  parallel_read: true
  plot_directories:
  - /plots1
  - /plots2
  - /plots3
  - /plots4
  - /plots5
  - /plots6
  - /plots7
  plot_loading_frequency_seconds: 300
  port: 8448
  private_ssl_ca:
    crt: config/ssl/ca/private_ca.crt
    key: config/ssl/ca/private_ca.key
  rpc_port: 8560
  selected_network: mainnet
  ssl:
    private_crt: config/ssl/harvester/private_harvester.crt
    private_key: config/ssl/harvester/private_harvester.key
  start_rpc_server: true
inbound_rate_limit_percent: 100
introducer:
  host: 127.0.0.1
  logging: *id001
  max_peers_to_send: 20
  network_overrides: *id002
  port: 8445
  recent_peer_threshold: 6000
  selected_network: mainnet
  ssl:
    public_crt: config/ssl/full_node/public_full_node.crt
    public_key: config/ssl/full_node/public_full_node.key
logging: *id001
min_mainnet_k_size: 32
network_overrides: *id002
outbound_rate_limit_percent: 30
ping_interval: 120
pool:
  logging: *id001
  network_overrides: *id002
  pool_list:
  - authentication_public_key: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
    launcher_id: '0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
    owner_public_key: '0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
    p2_singleton_puzzle_hash: '0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
    payout_instructions: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    pool_url: https://eu1.pool.space
    target_puzzle_hash: '0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
  selected_network: mainnet
  xch_target_address: xchxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
private_ssl_ca:
  crt: config/ssl/ca/private_ca.crt
  key: config/ssl/ca/private_ca.key
selected_network: mainnet
self_hostname: 127.0.0.1
timelord:
  fast_algorithm: false
  full_node_peer:
    host: 127.0.0.1
    port: 8444
  logging: *id001
  max_connection_time: 60
  network_overrides: *id002
  port: 8446
  sanitizer_mode: false
  selected_network: mainnet
  ssl:
    private_crt: config/ssl/timelord/private_timelord.crt
    private_key: config/ssl/timelord/private_timelord.key
    public_crt: config/ssl/timelord/public_timelord.crt
    public_key: config/ssl/timelord/public_timelord.key
  vdf_clients:
    ip:
    - 127.0.0.1
    - 127.0.0.1
    - 127.0.0.1
    ips_estimate:
    - 150000
  vdf_server:
    host: 127.0.0.1
    port: 8000
timelord_launcher:
  logging: *id001
  port: 8000
  process_count: 3
ui:
  daemon_host: 127.0.0.1
  daemon_port: 55400
  daemon_ssl:
    private_crt: config/ssl/daemon/private_daemon.crt
    private_key: config/ssl/daemon/private_daemon.key
  logging: *id001
  network_overrides: *id002
  port: 8222
  rpc_port: 8555
  selected_network: mainnet
  ssh_filename: config/ssh_host_key
wallet:
  database_path: wallet/db/blockchain_wallet_v1_CHALLENGE_KEY.sqlite
  enable_profiler: false
  full_node_peer:
    host: 127.0.0.1
    port: 8444
  initial_num_public_keys: 100
  initial_num_public_keys_new_wallet: 5
  introducer_peer:
    host: introducer.chia.net
    port: 8444
  logging: *id001
  network_overrides: *id002
  num_sync_batches: 50
  peer_connect_interval: 60
  port: 8449
  recent_peer_threshold: 6000
  rpc_port: 9256
  selected_network: mainnet
  short_sync_blocks_behind_threshold: 20
  ssl:
    private_crt: config/ssl/wallet/private_wallet.crt
    private_key: config/ssl/wallet/private_wallet.key
    public_crt: config/ssl/wallet/public_wallet.crt
    public_key: config/ssl/wallet/public_wallet.key
  start_height_buffer: 100
  starting_height: 0
  target_peer_count: 15
  testing: false
  trusted_peers:
    trusted_node_1: config/ssl/full_node/public_full_node.crt
  wallet_peers_path: wallet/db/wallet_peers.sqlite

Pls help its getting worse by every day........

[loppefaaret]

if you look at the log entries, it spans from 2021-09-05T17:31:35.072 - 2021-09-05T17:31:35.577. just over ½ a second - the peers you are connected to are gossiping the next signage point data, before your node has progressed to it.
being opened on 8444 has you better connected to the network, but it also means that you are more likely to be connected to faster nodes, that can verify blockchain data faster than your node can, and this behavior appears
being ½ a second behind is not really going to have a negative effect on a well working farming node, as long as it can proccess the round in 9~ average, and never go 25~ seconds behind, it should be fine

[djdookie]

I have the same problem for some time now with a RaspberryPi4 on all versions 1.2.3 to 1.2.5.
This rc caching leads to a lot of lost partials because of them being too late for the pool.
Pool response: {'error_code': 2, 'error_message': 'Received partial in 46.02203941345215. Make sure your proof of space lookups are fast, and network connectivity is good.Response must happen in less than 25 seconds. NAS or network farming can be an issue'}

I don't have any other connectivity issues and I didn't recognize any CPU or RAM problems on the Raspi.
My client is currently connected to 12 good full nodes, so I have no idea where those new problems are coming from now.

This is pretty unfortunate, because this leads to losing money and an ineffective farming.
As pooling were new, I didn't have issues like this.

Any ideas?
Could the size of the blockchain be a problem? My microSD card is over 90% full now, so I'll try to relocate the chia folder soon.

[djdookie]

I solved this by moving the .chia folder to an attached HDD and symbolic linking it.
As I assumed, my Raspi had problems handling the meanwhile big blockchain and wallet files an it's quite full micro SD card.
No more late partials since then for hours now. Using current latest branch (because some memory leak is also fixed there, but not in the last release yet).

[loppefaaret]

the latest branch is the release branch, so being on latest would have you on the latest release - thank you for following up on the issue, and providing the solution

[djdookie]

I meant the current main branch, sorry. ;)
Meanwhile the fix for the memory leak is included in the "latest" release.

[BrandtH22]

@ApfelBirneKreis , since this ticket was opened a number of years ago without further comments it seems this issue has been resolved.
We will be closing this ticket but if we have closed this ticket in error do not hesitate to reach out to us again with any followup questions or comments.

The best place to reach our support team is on Discord (https://discord.gg/chia) or by reopening this ticket.