Fetching submission context from cloudflare fails

[F-Node-Karlsruhe]

When fetching your presentation_submission context ( https://identity.foundation/presentation-exchange/submission/v1 ) from our server, the request fails, as it gets stuck in the cloudflare bot filter.

Could you consider not hosting the context on cloudflare, but on a more machine accessible location?

[bumblefudge]

I'm pretty sure all DIF specs are published via GitPages, and I'm not sure how we would publish @Context files otherwise... could we get more debugging info? maybe a curl output or some kind of log?
@OR13 - would you recommend pre-loading/document-loader best practice in this case, or some kind of caching proxy?

[F-Node-Karlsruhe]

It fails on preload into cache. Workaround would be to hard code the context into cache, but i think a context should still be resolvable in any case.

curl -v https://identity.foundation/presentation-exchange/submission/v1
* Expire in 0 ms for 6 (transfer 0x5593b8220c10)
* Expire in 1 ms for 1 (transfer 0x5593b8220c10)
* Expire in 0 ms for 1 (transfer 0x5593b8220c10)
... x100
* Expire in 3 ms for 1 (transfer 0x5593b8220c10)
* Expire in 3 ms for 1 (transfer 0x5593b8220c10)
* Expire in 4 ms for 1 (transfer 0x5593b8220c10)
*   Trying 188.114.97.3...
* TCP_NODELAY set
* Expire in 149994 ms for 3 (transfer 0x5593b8220c10)
* Expire in 200 ms for 4 (transfer 0x5593b8220c10)
* Connected to identity.foundation (188.114.97.3) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Jun  6 00:00:00 2022 GMT
*  expire date: Jun  5 23:59:59 2023 GMT
*  subjectAltName: host "identity.foundation" matched cert's "identity.foundation"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5593b8220c10)
> GET /presentation-exchange/submission/v1 HTTP/2
> Host: identity.foundation
> User-Agent: curl/7.64.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 403 
< date: Thu, 16 Mar 2023 17:50:43 GMT
< content-type: text/html; charset=UTF-8
< cross-origin-embedder-policy: require-corp
< cross-origin-opener-policy: same-origin
< cross-origin-resource-policy: same-origin
< permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
< referrer-policy: same-origin
< x-frame-options: SAMEORIGIN
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< server-timing: cf-q-config;dur=7.9999990703072e-06
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=DvrQ5viNYkpk4JkJ34VE0UJRHQaXfy3ei7dTK%2FkW%2BK%2FFxnl9nXvj7TBxErw6h4NTs0uAfJ0vF4pgE82M4DOGHqC19QClhdxaxydyLE0rtQ0Tl7pGikUh2ZDy2nW4T2rHQ%2BvGBSVM"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 7a8ed710df548ff4-FRA
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
< 
<!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">
    

</head>
<body class="no-js">
    <div class="main-wrapper" role="main">
    <div class="main-content">
        <h1 class="zone-name-title h1">
            <img class="heading-favicon" src="/favicon.ico" alt="Icon for identity.foundation"
                 onerror="this.onerror=null;this.parentNode.removeChild(this)">
            identity.foundation
        </h1>
        <h2 class="h2" id="challenge-running">
            Checking if the site connection is secure
        </h2>
        <noscript>
            <div id="challenge-error-title">
                <div class="h2">
                    <span class="icon-wrapper">
                        <div class="heading-icon warning-icon"></div>
                    </span>
                    <span id="challenge-error-text">
                        Enable JavaScript and cookies to continue
                    </span>
                </div>
            </div>
        </noscript>
        <div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/managed/nojs/transparent.gif?ray=7a8ed710df548ff4')"></div>
        <div id="challenge-body-text" class="core-msg spacer">
            identity.foundation needs to review the security of your connection before proceeding.
        </div>
        <form id="challenge-form" action="/presentation-exchange/submission/v1?__cf_chl_f_tk=LouclF6vdo7g4k0MZH835oiC_kFjIdV_1woJiiQi8MQ-1678989043-0-gaNycGzNCPs" method="POST" enctype="application/x-www-form-urlencoded">
            <input type="hidden" name="md" value="GyKw0c7phPrzTI8UQAHW06wiJY5as6DuYjatWGOSHGE-1678989043-0-AZL-luTdYTnPL9s1A_YfD3TgitLYT_J9yvtJVzAuCRt_-cjVUlAVPTaLwB7De82BigmgDcybF5CKhfmd81r1RYYZegfDNdyK3g1kPr8yeI060WYwDIPCYSUuOOP2MT_aZ_aJxLOB51COI4V7QHK7UemcLIGaW6HC7XgTVafYxIzeLjGDMtwmj0TQG63BBez18u5ZJv89X39NNN-v5HJNiHTdPqg-fVQ2TshOylV-iOh9YRivw_4Um9Az9zchn-flqbUKacSLDdj9cZ2vjrEbjq-cCRKw7HpkigTuk9mrC9VBGknqUDf8Gph4AHz5pIyj2T6H3j57EAL48bf4gWCVu_l5JJzuEbur_H1GHex9jNeJau98o2H0T_eQHm2DTJKGcbCaSi1xo3kMhGy7RWBDXoz8FkJs0ifyw-D_1TRLrx-C6UNlQRtYXa2Bi-KqBHa5UcC1wBmmNR6CDhEY7Q6CbdECv0h8R6ZW4iB8eFCCPuKeKX0abZrT-OHQ6ZByaUruvOEcAQgK6JRdjIdD6ybH3B-nNSz4pvUBOBrVjeCK2f1SNjftWbOdNyfXtE17XLM_qZI8809en8ZhdIu-KKfnS9w8v_Ty4IF7NpG2yBxpMBnB4x9nnvuiKR6NyitVbohcLQrvBETG_w_vVf7LsxBZoIsvcNU1Jz9Q7kuck_-WsiJwm70gK5pERIskhYrPYNbX3duO_JDWU4SU6p9OS5JIRlpqFNBUuF2cmdol__WOAtkuTmUoDYSxdjATC43x96C11q-qH2Gw96wpudnur3KlDclcZoa03bb2Wo-du6S6Fba0nxMR0s5i381T4s88RjdwTl_tgN5jPmp3SIQypva0YeXHTuUXpkpfp0kEmwZGwVdUuKtFRB4xEqrDMVH1ZVTGhtZ1XJJfAej62DibNmX03YLuqMTwbgrIOmW2cWrly5_CC2uvzF9yftOaC7mtjTYyOQgouykz_rw0EpKzjFqITJlgJRtFahhgs55fc1HsxNGc5lzNcvIk7ckdGpFrQwnX8YsJYqrID6mJIHd9H0I4Ctku07VKmjVAdErzmURg2qnlbZz35pFCjo7oHWPrrLssczF4I7ydH71Xo1rnrRoPWgHFIByr3urbo7Kp-hfqqJKDXisW6ZkCeGlbcMi5bghBz1FKlodNEbSxaaWzE-A8JXw4WYobYKheYhUQeCBv62fVj3jaVguEKwOlClEL5XcgsZqBWNXV9m5TirEfk0cyuULDXE0ffqMkdaUEzep6HRZWp5PydlO7MtTjejF92CxNoKe9YZCsBe8rc1NdaNZMo3YuayEdOU2Kt4XQNHdy61BlzWibAwMuf0usuT9MTZqIin9jDQIt9KYi8NjzfUKb_dVRnrdu5TOgAem-Gre3xV-i5x-gkDH8SIKrc1tkLiMENh847hELy6CbT-MhpcOy_0UgCDwnE_Hztirua0fp9HnE30bLCkhnnnj4hGRvu5SXkHlBMG_nA_JA9klaOlqgbhAKtVEr01ClWJ7byJIFSok71fhzjJTZGSBcPzFNoPBWwwHX-yJsmRralyeya-kG7X9Ke9rTBaR2oSnCFPHEXvdTqLQabkEPMGG4eYRs2N9TJ4pCT6XNArpsyEB_RTjaiOM_RxnW0_zoH-fdd6gCiAQOi0sj3BMVM5QYp5bkA5C5gmuk3b-cDov6t5z45TckIhs-TWmiJFncpPBgYbsywPfF3KWhvbgjmW3ADUxjs6-EDl8z4-aYLeDSQnQzC5e8EByyNwUBLkOerlMnnuzrRKXj8FMtou5Xu04F9l1HO6DXbmQjqgQBKlXNrnfaxxNb9i5a4yvczWIrBQP6aki66NsYFtvzjjZw3FiH0-XSKrqyYnZN2CHBNRj-3w-Oa1gy-LtIBoysHI_EzoINa31Pq5k5fzF2CsihwkXdtKuc4fBhhotZiBG5lavXvJrMnX78bgB-TtPUaEFlBP5lfQEMdqEi0bevsZEaRw1s25rnHDH-FjROf0FSPB4BzjpYJHs9qneqzmsojYSX4JOpeMO12M0fShtu-iJKsJbasbJHiDcRQgt2ddYIT6gVwcL1lwmKvRJaq7eq_vwXsd3EaKcMP1noa6MKHfCbw4tQIVtePO1WTzjXEbGyn1NeUHmLFqaAJpI0E9AShWhTHIxfJYiPK0-s6mQR9YLdJAaVwyUFly9QANaE3aXAlMyhmihxWQxDK1RpJiM9iDdvQR2LtRJJVJbpg-jCnVdhSTi9V114OlBbkw">
        </form>
    </div>
</div>
<script>
    (function(){
        window._cf_chl_opt={
            cvId: '2',
            cZone: 'identity.foundation',
            cType: 'managed',
            cNounce: '85370',
            cRay: '7a8ed710df548ff4',
            cHash: 'dd8978b4c1f6d8d',
            cUPMDTk: "\/presentation-exchange\/submission\/v1?__cf_chl_tk=LouclF6vdo7g4k0MZH835oiC_kFjIdV_1woJiiQi8MQ-1678989043-0-gaNycGzNCPs",
            cFPWv: 'g',
            cTTimeMs: '1000',
            cMTimeMs: '0',
            cTplV: 4,
            cTplB: 'cf',
            cK: "",
            cRq: {
                ru: 'aHR0cHM6Ly9pZGVudGl0eS5mb3VuZGF0aW9uL3ByZXNlbnRhdGlvbi1leGNoYW5nZS9zdWJtaXNzaW9uL3Yx',
                ra: 'Y3VybC83LjY0LjA=',
                rm: 'R0VU',
                d: 'y3uEKpvTpu3YAry5Q0wwmSohzEXxASL+DjhXILlGHEev90hIiU6rJS53yYJqAZXKtdaiXVD6r6n+5Q9S8QYvBrLRIdqYDFHNY60+OYzPGT4VKjHtG9nuqi9CDQclv2SbwQ+B0oU/MVYwWlFXtQvn/YKF2At+EMvh1avqm7g7JXE9dyQcoVyM14yPhEIwBJghNGefjkaR4xwwG4Tgaroqiu+8RWGkYtzll9X2nB7cnE5mursFdTHpsF9WcAgpeW77zfgPt7n1i+Ah787M82X99Twk9TiMAZjHN0v37slyi+NKfmAxqFiDkLZGfOgbXdSz/g0LaUtCkcbvf/jHEm/2aOKhws5mU+Or+mbLYb4ShbP+L4b6AKkX4UdENIlQeaUc1UKn9Y2pHksEWFTG/0SsRubQVifoJUK501Be/iX8Hl8stASU3zIhG74qqKnMr88cVMN8dokyB2VqEJYLUj3oca1PrHILkfXDb8oxOB6C7poLjcAfYHt1MErwTIrXO6yhnI6R7d2+AgMVv8PfS3MIg++Y/fvuDyYz3PZ4get3s53Mig+jhLxDnzHGiCdvebv0Zj0LrXUjkA6zePMxJYfSIaKUhmxWbbW9rpbnFyLkCBF0XoYKdzQ5v3N/NndQTk50YBx2UVY691675m51w8wJBs39WAHPj2zihWAh6Ep1Jp15ZZdBHGX45NGoo95pyMqi',
                t: 'MTY3ODk4OTA0My4zMzMwMDA=',
                m: 'q72tQe9HY/UL1Q5LNtUM+fpfhPN55SYEP2x9cLzbPUk=',
                i1: 'VC63gBdgLxO7KgE8i7vKgA==',
                i2: 'VjU2WrXOgsZDAdjH6vC6pA==',
                zh: '35403PcnLzgZRNse1zFB+qwPXRKHsgxEaEV1/fJuqTM=',
                uh: 'yms5jSnGBs3VDH1kGpihZ1WDyKZou9hLAVWd3YzjCVM=',
                hh: 'BQTE2OW2SnDXtyQZ67fpqwnOyfugvJ4MkN+gLC93agw=',
            }
        };
        var trkjs = document.createElement('img');
        trkjs.setAttribute('src', '/cdn-cgi/images/trace/managed/js/transparent.gif?ray=7a8ed710df548ff4');
        trkjs.setAttribute('alt', '');
        trkjs.setAttribute('style', 'display: none');
        document.body.appendChild(trkjs);
        var cpo = document.createElement('script');
        cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/managed/v1?ray=7a8ed710df548ff4';
        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
        if (window.history && window.history.replaceState) {
            var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
            history.replaceState(null, null, "\/presentation-exchange\/submission\/v1?__cf_chl_rt_tk=LouclF6vdo7g4k0MZH835oiC_kFjIdV_1woJiiQi8MQ-1678989043-0-gaNycGzNCPs" + window._cf_chl_opt.cOgUHash);
            cpo.onload = function() {
                history.replaceState(null, null, ogU);
            };
        }
        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
</script>


    <div class="footer" role="contentinfo">
        <div class="footer-inner">
            <div class="clearfix diagnostic-wrapper">
                <div class="ray-id">Ray ID: <code>7a8ed710df548ff4</code></div>
            </div>
            <div class="text-center" id="footer-text">Performance &amp; security by <a rel="noopener noreferrer" href="https://www.cloudflare.com?utm_source=challenge&utm_campaign=m" target="_blank">Cloudflare</a></div>
        </div>
    </div>
</body>
</html>
* Connection #0 to host identity.foundation left intact

[OR13]

On my machine:

❯ curl -v https://identity.foundation/presentation-exchange/submission/v1
*   Trying 2606:4700:3031::ac43:8ca4:443...
* Connected to identity.foundation (2606:4700:3031::ac43:8ca4) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Jun  6 00:00:00 2022 GMT
*  expire date: Jun  5 23:59:59 2023 GMT
*  subjectAltName: host "identity.foundation" matched cert's "identity.foundation"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x14c80ec00)
> GET /presentation-exchange/submission/v1 HTTP/2
> Host: identity.foundation
> user-agent: curl/7.79.1
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 301 
< date: Thu, 16 Mar 2023 18:44:55 GMT
< content-type: text/html
< location: https://identity.foundation/presentation-exchange/submission/v1/
< access-control-allow-origin: *
< expires: Thu, 16 Mar 2023 18:54:55 GMT
< cache-control: max-age=600
< x-proxy-cache: MISS
< x-github-request-id: 3EFC:4AF0:122A965:18974DB:641363A7
< accept-ranges: bytes
< via: 1.1 varnish
< age: 0
< x-served-by: cache-dfw-kdfw8210032-DFW
< x-cache: MISS
< x-cache-hits: 0
< x-timer: S1678992296.933003,VS0,VE41
< vary: Accept-Encoding
< x-fastly-request-id: 6360387387b637f4165dd0dc4ff369ee00d8ef50
< cf-cache-status: DYNAMIC
< server-timing: cf-q-config;dur=5.9999999848515e-06
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=IX4fSKGC5d0dtnnF1bKlGlpk39AjHspaeb9HKw0uk832YvUedvVRna7t5Tq%2FtYx1ZSdpAE63%2BEZx3mwoW27Ub%2F%2Fr2WKT%2F4H9jMwoKPjEwZjSyYM0XcPgY%2BYDWYSM%2BXz47MLnBwy2NuUdqtxms66m9yJg"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 7a8f26797bfc2e73-DFW
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
<script defer src="https://static.cloudflareinsights.com/beacon.min.js/vaafb692b2aea4879b33c060e79fe94621666317369993" integrity="sha512-0ahDYl866UMhKuYcW078ScMalXqtFJggm7TmlUtp0UlD4eQk0Ixfnm5ykXKvGJNFjLMoortdseTfsRT8oCfgGA==" data-cf-beacon='{"rayId":"7a8f26797bfc2e73","version":"2023.2.0","r":1,"token":"78fb5e9ad69b466097a9153ffa9afbe7","si":100}' crossorigin="anonymous"></script>
</body>
</html>
* Connection #0 to host identity.foundation left intact

[OR13]

At any rate, I am not DIF tech support, I can't help resolve GitHub Pages hosting, or DIF Cloudflare account related issues.

[F-Node-Karlsruhe]

True that. This is why i'm a big fan of ld context on IPFS ;)

digitalbazaar/jsonld.js#512

[bumblefudge]

Haha, of course not, Orie, was just wondering if you had guidance on the resolvability requirements and/or preloading of such things. I noticed the context in question had been uploaded by one of your commits.

curling from here in Germany gets me a response that looks just like Orie's (i.e., 301), not like OP's, although in a browser the same URL returns:

Maybe as a quick fix I could just upload this particular document to public IPFS for you? :D

[bumblefudge]

Maybe DIF should tack on an IPNS verificationMethod to its did:web... 💃

[F-Node-Karlsruhe]

Maybe as a quick fix I could just upload this particular document to public IPFS for you? :D

Nice Idea :D i already did and replaced the URL with ipfs://QmRSE16DGVRzNdzEeoA14nsG9iLZzHUz9sbYPqaZncCEJ1

Works/verifies like a charm now :)

[F-Node-Karlsruhe]

Maybe DIF should tack on an IPNS verificationMethod to its did:web... woman_dancing

As mentioned above, we are about to switch towards referencing contexts via IPFS mostly as this mitigates any issues with versioning, availability and integrity

https://github.com/european-epc-competence-center/vc-verifier/blob/main/api/src/services/documentLoader/index.ts#L53

[F-Node-Karlsruhe]

I made a small PR to the (i guess) most used JSON-LD library by digitalbazaar, for those interested in the topic :)

digitalbazaar/jsonld.js#513

[kimdhamilton]

This is a DIF-wide issue; need to determine where to track.

• Question: if anyone can give guidance on how to fix this site-wide, DIF staff can address
• Probably need to update content type at domain level
• Near-term: may add warning to spec that hosted contexts and schemas shouldn't be relied upon