Hash encodings should be converted to UTF-8

[joelweinberger]

I was taking a brief look at the CSP 3 script/style hash definitions, and it seems to get the sources for computing a digest on directly from the HTML spec. For example, https://html.spec.whatwg.org/multipage/scripting.html#prepare-a-script step 11 calls https://w3c.github.io/webappsec-csp/#match-element-to-source-list.

However, in the HTML spec, these sources (such as textContext) are, I believe UTF-16. In practice, this contradicts UA behavior. Chrome (and I believe Firefox as well) explicitly converts source to UTF-8 before computing the digest. See https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/frame/csp/ContentSecurityPolicy.cpp?sq=package:chromium&rcl=1471966011&l=466. While an argument could be made that the UAs should change, this has been their behavior for several years now, and changing the UTF-8 conversion to UTF-16 could potentially break a number of sites (although that's measurable, I suppose). The simplest solution, it seems to me, is to add an explicit UTF-8 conversion in the CSP spec.

[annevk]

They are not UTF-16. They are USC-2. What happens with unpaired surrogates? If those get the replacement character, you'd give different scripts the same hash.

[joelweinberger]

I'll take your word that it's UCS-2, not UTF-16 :-) I'm confused because the source sent in prepare a script is the child text content, which I thought was the concatenation of DOMStrings, which I thought were UTF-16... but this was off the cuff spelunking.

Unfortunately, I don't understand the unpaired surrogates question. What are "unpaired surrogates"?

In any case, I don't actually care too much about the particular outcome, but I think the CSP spec should clarify what the input string should be (or it should be explicitly converted).

[mikewest]

/cc @fmarier

[andypaicu]

I know close to nothing about encodings, what should be put in the spec to make everyone happy?

[annevk]

I suspect you want to invoke the hash function upon the https://encoding.spec.whatwg.org/#utf-8-encode upon the https://infra.spec.whatwg.org/#javascript-string-convert of the input string. Does that help?

[annevk]

(And you want to add tests.)

[andypaicu]

This needs WPT tests to be closed.