SJIP: Likelihood is not considered during judging

[WangSecurity]

Description

Clarify that likelihood shouldn't be considered when identifying the validity and the sevetity of the report.

Judging Guidelines PR

https://github.com/sherlock-protocol/sherlock-v2-docs/pull/55/files

Rationale

It's often the case that Watsons/Judges are basing their judgement on the likelihood of the issue happening. However, on Sherlock, issues should be judged based on the impact and constraints. Hence, the goal of this SJIP is to clarify the Watsons/Judges shouldn't base their judgements on the likelihood.

Relevant Issue Discussions

No specific discussion.

[midori-fuse]

Please define "likelihood" and "constraints".

Brute forcing a private key/CREATE2 collision has a non-zero likelihood of success, and the only constraint is that I have to own a computer.

[IllIllI000]

perhaps the logic used during the maker contest could be generalized and codified. I believe the discussion was deleted, so Sherlock would have to retrieve it

[midori-fuse]

Generalizing this, an "attack path/vuln path" is a scenario where the impact may occur. An issue's "constraint" is a set of conditions that must be met for the attack path/vuln path to be possible. Each of the conditions have a probability to occur, then judging based on constraints is equivalent to judging based on the likelihood of these constraints happening.

Then how is it possible to judge based on constraints but not likelihood? What is the difference between these two?

[WangSecurity]

It's a very good question, tbh, and indeed, these two terms are very similar. The main point here is to prevent situations where the judgment is made because the issue has a "low likelihood". As you know, Sherlock's definitions for high and med severity do not consider the likelihood. An example with private key/CREATE2 collision is also very good, but here is the common sense that comes in that was involved a couple of months ago, and validating this such a finding (slightly) conflicts with common sense, considering all the constraints (at least for now).

Another important moment to consider is that this change in guidelines doesn't change how the issues are judged, this is only the clarification. Returning to the earlier point, the high and med definitions do not include likelihood and previous judgements (before this SJIP) were not considered based on likelihood. This change is just to prevent argumentations like "the issue is med severity and low likelihood so it should be invalid". The purpose here is not to change how the issues are judged.

[midori-fuse]

The proposed changes as they are will still only result in arguments like "the issue is med severity and low likelihood so it should be valid". On one side you have a low likelihood that should be a medium because "likelihood is not used during judging", on the other you get an astronomically low likelihood that should not be a medium but still is because "likelihood is not used during judging".

You cannot use any wording that resembles "likelihood is completely ignored". Sherlock's stance has not been ignoring likelihood, but rather "impact weights much more heavily than likelihood", where issues are downgraded less often based on likelihood than with other platforms (e.g. only astronomically low likelihood is grounds for invalidation).

You can make clear of that general philosophy, but again you cannot completely ignore likelihood, as that's exactly what separates purely theoretical scenarios from realistic and actual issues.

[WangSecurity]

Could you please elaborate on "Sherlock's stance has not been ignoring likelihood, but rather "impact weights much more heavily than likelihood"? Why do you have such an understanding? Or is it based on the fact that constraints are very similar to likelihood?

[0xHuntoor]

This issue was invalidated on the base of likely hood as far as i remember, can you clarify the difference between judging reason there and this SJIP that says likelihood is not a factor?

[WangSecurity]

Unfortunately, I wasn't Sherlock Judge on this contest, but after looking at the report, my decision would be that it's invalid as the difference here is only one wei, which is dust, and it can fairly be a design decision to use ">" instead of ">=".

[0xHuntoor]

Ye it doesn't matter how it ended. Its just that iam still confused of likelihood/constraints differ.

And that issue was a good example at least for me.

Having that 1 wei diff reverted a valid trading txn which is time sensitive.

Maybe it was invalidated from the aspect that it a design decision and doesn't cause loss of funds i don't know

Do you think that that one wei is a constraint or the issue was just invalidated with common sense?(Which i actually agree with)

Can you elaborate for me please?

[WangSecurity]

For me, it is "invalidated with common sense". Additionally, I don't see that one wei difference as an issue worth medium/high severity (though it could be based on the impact).

Here's another issue about one wei difference, which I judged as invalid, but this one is about liquidation and not slippage, however, the issue is about using ">" rather than ">=".

[Flashloan44]

If we remove "likelihood" , many findings will be validated with unrealistic attack path or conditions.

Considering that Sherlock prioritize findings with impact of losing funds, it should always base the judgment in likelihood on whether it could happen in real life or not.

[WangSecurity]

You can refer to my comment here:

this change in guidelines doesn't change how the issues are judged, this is only the clarification. Returning to the earlier point, the high and med definitions do not include likelihood and previous judgements (before this SJIP) were not considered based on likelihood. This change is just to prevent argumentations like "the issue is med severity and low likelihood so it should be invalid". The purpose here is not to change how the issues are judged

[bluenights004]

So in this statement, need to clarify on what it really says,

This change is just to prevent argumentations like "the issue is med severity and low likelihood so it should be invalid"

The goal of this SJIP is to remove the "low likelihood" here in judge decision making? So if we remove low likelihood, this is considered valid? Am I right? because you only considered impact?

However if we go back to Sherlock docs, , it says "requires certain external conditions" but how can you measure the reality of external conditions if we don't factor the likelihood?

How to identify a medium issue:
Causes a loss of funds but requires certain external conditions or specific states, or a loss is highly constrained. The loss must be relevant to the affected party.

[WangSecurity]

because you only considered impact

Nope, it's not only impact, it's impact + constraints. And this is what is referred to in the medium severity definition as "certain external conditions". It's not likelihood but rather what is required for the attack/vulnerability to happen. As you can see in the High-severity definition, it says "Direct loss of funds without (extensive) limitations of external conditions", so if the requirements are not extensive or the attack/vulnerability can happen, then it's high. And on the contrary, if there are certain requirements or limitations for the issue to occur, then it's medium. Does it make sense?

[bluenights004]

How about in this sample situation, the lead judge make the issue invalid because the certain requirements is impossible or very low to occur. In this case, the effect of this SJIP will make this issue from invalid to valid?

[WangSecurity]

About the "the certain requirements is impossible", in this case, the issue should be invalid because it's impossible to happen.

About low likelihood, even before this SJIP, we were tracking the escalations and LJ's decisions so they don't base them on the low likelihood. Yes, there was one case recently in the Ethos contest, but I've clarified it after the LJ explaining why the issue was invalid without low likelihood.

Hence, yes, since we don't consider low likelihood, some issues can be valid while they would be invalid considering low likelihood. However, this depends on the context, and I cannot say for sure without a concrete example. If you want a concrete example, you can look into this comment.