Percent-encoding of qualifier values

[mprpic]

@pombredanne @johnmhoran @jkowalleck I finally caught up with all of the discussions in #398 and wanted to just double-confirm that my understanding is right. If I'm reading the rules for a qualifier value:

A value MAY be composed of any character and all characters MUST be encoded as described in the "Character encoding" section.

and then combine it with the guidance in the "Character encoding section":

A canonical purl is an ASCII string composed of these characters:

• alphanumeric characters A to Z, a to z, 0 to 9,
• the purl separators :/@?=&# (colon ':', slash '/', at sign '@', question mark '?', equal sign '=', ampersand '&' and pound sign '#'), and
• these punctuation marks %.-_~ (percent sign '%', period '.', dash '-', underscore '_' and tilde '~').

All other characters MUST be encoded as UTF-8 and then percent-encoded. In addition, each component specifies its permitted characters and its percent-encoding rules.

and the final modifier:

When applying percent-encoding or decoding to a string, use the rules of RFC 3986 section 2 (https://datatracker.ietf.org/doc/html/rfc3986#section-2).

Each component defines when and how to apply percent-encoding and decoding to its content.

When percent-encoding is required, all characters MUST be encoded except for the colon ':'.

Taking all this together, if I have a qualifier that has a key of repository_url and value of https://example.repo.io/commons, the key remains as is, and the value should be percent-encoded (except for the colon character) to:

https:%2F%2Fexample.repo.io%2Fcommons

where a / (slash) character is represented by %2F and the colon is left unencoded. The final purl would then be:

pkg:[...]?repository_url=https:%2F%2Fexample.repo.io%2Fcommons

Is that a correct interpretation of the spec?

[mprpic]

@johnmhoran @jkowalleck Judging by your 👍, I take it that that is the correct encoding. Does that then mean that most of the purls in the test data are incorrect:

master!purl-spec > rg 'repository_url=.*/.*' test-suite-data.json 
113:    "canonical_purl": "pkg:maven/org.apache.xmlgraphics/[email protected]?classifier=sources&repository_url=repo.spring.io/release",
125:    "canonical_purl": "pkg:maven/org.apache.xmlgraphics/[email protected]?extension=pom&repository_url=repo.spring.io/release",
484:    "purl": "pkg:huggingface/microsoft/deberta-v3-base@559062ad13d311b87b2c455e67dcd5f1c8f65111?repository_url=https://hub-ci.huggingface.co",
485:    "canonical_purl": "pkg:huggingface/microsoft/deberta-v3-base@559062ad13d311b87b2c455e67dcd5f1c8f65111?repository_url=https://hub-ci.huggingface.co",
508:    "purl": "pkg:mlflow/CreditFraud@3?repository_url=https://adb-5245952564735461.0.azuredatabricks.net/api/2.0/mlflow",
509:    "canonical_purl": "pkg:mlflow/creditfraud@3?repository_url=https://adb-5245952564735461.0.azuredatabricks.net/api/2.0/mlflow",
520:    "purl": "pkg:mlflow/CreditFraud@3?repository_url=https://westus2.api.azureml.ms/mlflow/v1.0/subscriptions/a50f2011-fab8-4164-af23-c62881ef8c95/resourceGroups/TestResourceGroup/providers/Microsoft.MachineLearningServices/workspaces/TestWorkspace",
521:    "canonical_purl": "pkg:mlflow/CreditFraud@3?repository_url=https://westus2.api.azureml.ms/mlflow/v1.0/subscriptions/a50f2011-fab8-4164-af23-c62881ef8c95/resourceGroups/TestResourceGroup/providers/Microsoft.MachineLearningServices/workspaces/TestWorkspace",
532:    "purl": "pkg:mlflow/trafficsigns@10?model_uuid=36233173b22f4c89b451f1228d700d49&run_id=410a3121-2709-4f88-98dd-dba0ef056b0a&repository_url=https://adb-5245952564735461.0.azuredatabricks.net/api/2.0/mlflow",
533:    "canonical_purl": "pkg:mlflow/trafficsigns@10?model_uuid=36233173b22f4c89b451f1228d700d49&repository_url=https://adb-5245952564735461.0.azuredatabricks.net/api/2.0/mlflow&run_id=410a3121-2709-4f88-98dd-dba0ef056b0a",

^ all of these are "is_invalid": false.

[johnmhoran]

@ppkarwasz "the remaining reserved characters NOT used as separators MUST NOT be encoded." is new and not my understanding of what is intended or expressed currently, but I will happily include language for that when and if it receives approval. Meanwhile I'll push my PR next week, post that fact in this discussion, and wait for agreement (i.e., whatever you and others in the community decide). (edited for clarity)

[ppkarwasz]

"the remaining reserved characters NOT used as separators MUST NOT be encoded." is new and not my understanding of what is intended or expressed currently

Sorry, I use one "NOT" to much. I edited my comment.

[mprpic]

Can I ask one more time what the benefit of percent-encoding is within qualifier values?

[johnmhoran]

I assume you are directing your question to @pombredanne ....

[mprpic]

A question for anyone who considers this to be correct/intended, really: pkg:[...]&repository_url=repo.spring.io%2Frelease.

[johnmhoran]

@pombredanne @mprpic @jkowalleck @ppkarwasz @matt-phylum Please see my proposed updated to the "Character encoding" section and related provisions. #461

[johnmhoran]

Re @matt-phylum 's suggestion (above) that we change the RFC 3986 section 2 reference to just section 2.1, such a focus solely on the percent-encoding process itself could reduce ambiguity and make implementation more straightforward. After all, while the PURL spec is related to RFC 3986, it differs in significant respects from 3986, including in its percent-encoding rules (e.g., what and when). That seems like an important distinction that's worth emphasizing.