Any reason valid links to pdf files might raise false alarms

[markcmiller86]

I am getting false positives both of which have to do with .pdf files, https://github.com/betterscientificsoftware/bssw.io/actions/runs/7734371239/job/21088244489?pr=1633

Any reason to suspect the checker has trouble with .pdf files?

[vsoch]

A pdf file is not a text machine readable file, so you should not ask the checker to parse it (or add to ignore).

[markcmiller86]

Hmm...did you follow the link to the failed tests? I am not using it to check links in pdf files. It is failing on links to PDF files which I can browse to fine.

[vsoch]

My apologies - I did not! It looks like it has nothing to do with the PDF files, those servers have bad certificates:

HTTPSConnectionPool(host='iitj.ac.in', port=443): Max retries exceeded with url: /uploaded_docs/cc/HPC_training/mcmuserguide.pdf (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

You can reproduce with two lines of python:

import requests
requests.get('https://iitj.ac.in/uploaded_docs/cc/HPC_training/mcmuserguide.pdf')

You can ask the webmasters to update their certs, and if you don't have that control, you'll have to add them to the skiplist.

[markcmiller86]

You can ask the webmasters to update their certs, and if you don't have that control, you'll have to add them to the skiplist.

Any chance you'd be willing to add a feature to ignore bad certs? (maybe even make it the default). Its a common scenario and asking people to populate skip lists for such a common scenario seems onerous. And, its confusing why my browser is able to follow links fine that the urlchecker action deems "broken".

[vsoch]

The browser, and actually depending on the browser, does a lot of wonky things to "just make the page load." If you use command line / core tools that enforce best practices to check certificates, you tend to see the truth. And actually, we go to some lengths to try to emulate a web driver, but it's not perfect.

We can definitely consider that feature. You'll still have the timeout issue on the second PDF, however.

[vsoch]

okay, I have a branch for you to test, will post shortly. Note that this is an action for urlchecker-python, so you can run the tool manually on your directory to check.

[vsoch]

Here you go! Please test this out locally, and let me know if the new option works.

urlstechie/urlchecker-python#89

urlchecker check  --branch master --no-check-certs --no-print --verbose --file-types .md --exclude-patterns http://localhost:4000,[https://preview.bssw.io,https://github.com/](https://preview.bssw.io,https//github.com/)<your-github-handle> --retry-count 3 --timeout 10 --files .github/workflows/check-urls.yml,.github/workflows/README.md,Articles/Blog/2020-01-usrse.md,Articles/Blog/2020-11-PSIP4HDF5.md,Articles/Blog/2021-09-CollegevilleReportDay1.md,Articles/Blog/2021-12-sc21-swe-cse-bof.md,Articles/Blog/ConnectingSoftwareDevelopers.md,Articles/Blog/Covid19WorkstationCleanliness.md,Articles/Blog/HowToEnablePerformancePortability.md,Articles/Blog/HowToWriteGoodDocumentation.md,Articles/Blog/URSSI.md,CuratedContent/GoodEnoughPracticesInScientificComputing.md,CuratedContent/LanguageReferenceOnLine.md,CuratedContent/TeamOfTeamsUNPUB.md,CuratedContent/kitchen-sink-TEST.md,Site/BSSwFellowshipProgram/People/2020-F-Eisty.md .

[markcmiller86]

@vsoch thanks so much!

Lemme give this a try.

[vsoch]

Thank you!!

Heads up I'm breaking for dinner, but will be back later.

[markcmiller86]

Am running into ssl version issues...

(myenv) sh-3.2$ urlchecker check ../bssw.io/CuratedContent/LanguageReferenceOnLine.md 
/Users/miller86/ideas-ecp/urlchecker-python/myenv/lib/python3.8/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020

[vsoch]

You need to add the --no-check-certs flag as I showed above. I'm not going to take off checking by default, it's a "use at your own risk" feature.

[markcmiller86]

I am using that flag though the command and error I pasted didn't include it. Its an issue with macOS ssl, python and virtual env.

[vsoch]

I don't have a Mac that I use for programming, but I'd follow that GitHub link and see if you can track down the issue. This is unrelated to urlchecker and the PR - it seems like it's an issue with the Python/ssl versions on your system.

[markcmiller86]

Ok, well it might help if I was in the correct branch of the clone. I've done that now. And, I built a docker ubuntu container..., but strangley, I am getting cert errors...

# urlchecker check  --branch master --no-check-certs --no-print --verbose --file-types .md /tmp
           original path: /tmp
              final path: /tmp
               subfolder: None
                  branch: master
                 cleanup: False
                  serial: False
              file types: ['.md']
                   files: []
               print all: False
                 verbose: True
           urls excluded: []
   url patterns excluded: []
  file patterns excluded: []
              force pass: False
             retry count: 2
                    save: None
                 timeout: 5
https://yaml.org/spec/1.2.2/
https://docs.python.org/dev/reference/
https://en.wikipedia.org/wiki/POSIX
HTTPSConnectionPool(host='iitj.ac.in', port=443): Max retries exceeded with url: /uploaded_docs/cc/HPC_training/mcmuserguide.pdf (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))
https://iitj.ac.in/uploaded_docs/cc/HPC_training/mcmuserguide.pdf
HTTPSConnectionPool(host='iitj.ac.in', port=443): Max retries exceeded with url: /uploaded_docs/cc/HPC_training/mcmuserguide.pdf (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))
https://iitj.ac.in/uploaded_docs/cc/HPC_training/mcmuserguide.pdf
https://docs.python.org/3/reference/
https://docs.microsoft.com/en-us/cpp/standard-library/cpp-standard-library-reference?view=msvc-170
https://www.open-mpi.org/doc/v4.0/
https://wg5-fortran.org/N1151-N1200/N1191.pdf
https://chapel-lang.org/docs/language/spec/index.html