Support equivalent words in license detection #4190

Handle similar words in license detection by allowing multiple
"legalese words" to have the same token id.

Regenerate the tokens ids accordingly.

Convert Index.tokens_by_tid to a computed property, available on demand.
Convert tokens_by_tid to a dictionary from a list.
Ensure that all code relying on the tokens_by_tid is updated as needed.
All locations were used only for testing and debugging.

Deprecate all rules that are duplicated under this new regime, where
tokens like "license" and "licence" are not treated as identical.

Update test suite to test the detection of all deprecated licenses and
rules as a sanity check. A rule with "relevance" set to 0 is not tested
if deprecated, as some rules are deprecated because they are false
positive and should no longer be detected. Also improved the validation
and loading of rules relevance, including the case for zero relevance.

Update ambiguous or conflicting rules as needed.
In particular ensure that all rules in the style of "MIT or GPL"
without a GPL version are now reported consistently as:
"mit or gpl-1.0-plus"

Reference: #4190
Signed-off-by: Philippe Ombredanne <[email protected]>

Author: pombredanne

src/licensedcode/data/rules/agpl-1.0-plus_3.RULE

@@ -1,6 +1,7 @@
 ---
 license_expression: agpl-1.0-plus
 is_license_reference: yes
+is_deprecated: yes
 relevance: 100
 ---
 

src/licensedcode/data/rules/agpl-1.0-plus_4.RULE

@@ -1,6 +1,7 @@
 ---
 license_expression: agpl-1.0-plus
 is_license_reference: yes
+is_deprecated: yes
 relevance: 100
 ---
 

src/licensedcode/data/rules/agpl-1.0-plus_43.RULE

@@ -1,6 +1,7 @@
 ---
 license_expression: agpl-1.0-plus
 is_license_tag: yes
+is_deprecated: yes
 relevance: 100
 ---
 

src/licensedcode/data/rules/agpl-1.0-plus_44.RULE

@@ -1,6 +1,7 @@
 ---
 license_expression: agpl-1.0-plus
 is_license_tag: yes
+is_deprecated: yes
 relevance: 100
 ---
 

src/licensedcode/data/rules/agpl-1.0_24.RULE

@@ -1,6 +1,7 @@
 ---
 license_expression: agpl-1.0
 is_license_tag: yes
+is_deprecated: yes
 relevance: 100
 ---
 

src/licensedcode/data/rules/agpl-1.0_25.RULE

@@ -1,6 +1,7 @@
 ---
 license_expression: agpl-1.0
 is_license_tag: yes
+is_deprecated: yes
 relevance: 100
 ---
 

src/licensedcode/data/rules/agpl-1.0_7.RULE

@@ -1,6 +1,7 @@
 ---
 license_expression: agpl-1.0
 is_license_reference: yes
+is_deprecated: yes
 relevance: 95
 ---
 

src/licensedcode/data/rules/agpl-1.0_9.RULE

@@ -1,6 +1,7 @@
 ---
 license_expression: agpl-1.0
 is_license_reference: yes
+is_deprecated: yes
 relevance: 100
 ---
 

src/licensedcode/data/rules/agpl-2.0_25.RULE

@@ -1,6 +1,7 @@
 ---
 license_expression: agpl-2.0
 is_license_tag: yes
+is_deprecated: yes
 relevance: 100
 ---
 

src/licensedcode/data/rules/agpl-2.0_26.RULE

@@ -1,6 +1,7 @@
 ---
 license_expression: agpl-2.0
 is_license_tag: yes
+is_deprecated: yes
 relevance: 100
 ---