@dmtrjsg commented on Fri May 13 2022

Context

Incorrect balances vs anticipated for the WG Debt

Scope

• Conduct 2 tests as per this comment
• Isolate the scenario and provide a view on the actual mechanics to be fixed if incorrect
• Fix described issue on the QN side: Pioneer is correctly filtering for isActive, but the query node is not updating debts for NewMissedRewardLevelReached events.

Additional info

Martin: I tried:

query workersDebt {
  workers (limit:1000000, where:{groupId_eq:"contentWorkingGroup"}) {
    isActive
    missingRewardAmount
  }
}

and got:

{
  "data": {
    "workers": [
      {
        "isActive": false,
        "missingRewardAmount": "144300"
      },
      {
        "isActive": false,
        "missingRewardAmount": "113787"
      },
      {
        "isActive": false,
        "missingRewardAmount": "48670"
      },
      {
        "isActive": false,
        "missingRewardAmount": "48670"
      },
      {
        "isActive": true,
        "missingRewardAmount": "548257"
      },
      {
        "isActive": true,
        "missingRewardAmount": "86580"
      },
      {
        "isActive": true,
        "missingRewardAmount": "432900"
      },
      {
        "isActive": true,
        "missingRewardAmount": null
      }
    ]
  }
}

Adding these numbers up -> 1423164
Just the isActive -> 1067737 - which matches pioneer.

@dmtrjsg commented on Tue May 17 2022

@bwhm is it the case where they get synced up in arrears at a certain point in time regularly?

@dmtrjsg commented on Wed Jun 08 2022

@bwhm pls advise :)

@bwhm commented on Wed Jun 08 2022

For the distribution group as of block #1,099,400, it looks like this:

The budget, compared to the chain state is correct, but the last round of rewards for the group, at block #1,086,750, we can see there was no debt accumulated, nor any old debts to be repaid.

My (educated) guess is that it's caused by one of these two issues:

1. Every time a reward can not be paid, and the event is ${wg}.NewMissedRewardLevelReached, with a workerId and the "new" amount(owed) the amount is ADDED to the debt. If correct, that means that the ${wg}.RewardPaid where the RewardPaymentType is NOT RegularReward, but MissedReward (meaning the amount paid to the worker is all/some part of the accumulated debt repaid) is ignored.
2. Same as above, except the QN actually does reduce the debt correctly for a MissedReward. However, if a worker gets fired or quits, and the workerId is no longer in state, while being owed a debt, the QN will "count" that debt forever. The runtime however, will give the WG a "jubilee" in that case, as their owed rewards is no longer kept in state.

How to debug:
Take a group WITH debt (such as distributors), get their reward period, and check the state for each worker in the group group at that block. Sum up all the debts for each of them, and compare. If same number -> 1.

If not same number -> probably 2. To confirm: Do the same, but sum up the number of debts for all workerId that are NOT part of the group in the next rewardperiod.

@bwhm commented on Wed Jun 08 2022

I tried:

query workersDebt {
  workers (limit:1000000, where:{groupId_eq:"contentWorkingGroup"}) {
    isActive
    missingRewardAmount
  }
}

and got:

{
  "data": {
    "workers": [
      {
        "isActive": false,
        "missingRewardAmount": "144300"
      },
      {
        "isActive": false,
        "missingRewardAmount": "113787"
      },
      {
        "isActive": false,
        "missingRewardAmount": "48670"
      },
      {
        "isActive": false,
        "missingRewardAmount": "48670"
      },
      {
        "isActive": true,
        "missingRewardAmount": "548257"
      },
      {
        "isActive": true,
        "missingRewardAmount": "86580"
      },
      {
        "isActive": true,
        "missingRewardAmount": "432900"
      },
      {
        "isActive": true,
        "missingRewardAmount": null
      }
    ]
  }
}

Adding these numbers up -> 1423164
Just the isActive -> 1067737 - which matches pioneer.

Conclusion:
Pioneer is correctly filtering for isActive, but the query node is not updating debts for NewMissedRewardLevelReached events.

@polikosi commented on Mon Jul 04 2022

Verified on staging.
Distribution group:

{"data":{"workers":[{"missingRewardAmount":"8692550","__typename":"Worker"},{"missingRewardAmount":"1695648","__typename":"Worker"}]}}

Add 8,692,550 and 1,695,648 = 10,388,198

@singulart commented on Tue Jul 05 2022

As @bwhm found, this is a QN issue