How to specify default protocol

[dotnetCarpenter]

It's a common pattern to not include protocol in your links and let the browser choose the protocol based on the current URL. That way, all links will have the https protocol in production, where you use HTTP redirects to enforce it but still work in development, where you don't necessarily have the infrastructure to setup certificates.

E.I: Given a link like //github.com/lycheeverse/

If the current URL is https://github.com/ then the above will become https://github.com/lycheeverse/. Using port 443.

If the current URL is http://github.com/ then the above will become http://github.com/lycheeverse/. Using port 80.

Current URL	naked link	becomes	port
https://github.com	//github.com/lycheeverse/	https://github.com/lycheeverse/	443
http://github.com	//github.com/lycheeverse/	http://github.com/lycheeverse/	80
https://github.com	/book/pt-br/	https://github.com/book/pt-br/	443
http://localhost:1312	/book/pt-br/	http://localhost:1312/book/pt-br/	1312
http://localhost:1312	//localhost:1313/book/pt-br/	http://localhost:1312/book/pt-br/	1312
And so on...

However, this scheme does not appear to work with lychee. And I could not find any documentation how to configure lychee to behave like a browser.

I need lychee to analyse the naked links.

Any help in how to configure lychee to this end or pointers to where in lychee this could be implemented, is greatly appreciated!

[mre]

Yes, you're right, and I know about that behavior, which is commonly used in browsers as far as I'm aware.
At the moment, lychee doesn't support that.

In order to do so, we'd have to "track" the origin of a link across the entire chain. We could introduce a parent URL to a request here:

lychee/lychee-lib/src/types/request.rs

Lines 9 to 27 in d6bbf85

	pub struct Request {
	/// A valid Uniform Resource Identifier of a given endpoint, which can be
	/// checked with lychee
	pub uri: Uri,
	/// The resource which contained the given URI
	pub source: InputSource,
	/// Specifies how the URI was rendered inside a document
	/// (for example `img`, `a`, `pre`, or `code`).
	/// In case of plaintext input the field is `None`.
	pub element: Option<String>,
	/// Specifies the attribute (e.g. `href`) that contained the URI
	pub attribute: Option<String>,
	/// Basic auth credentials
	pub credentials: Option<BasicAuthCredentials>,
	}

I guess it would be totally achievable, however, it might be quite some work to follow through. Namely, we support Markdown and Plaintext formats, and we also support local HTML files, for which there wouldn't be any kind of "parent" or source URL.

So the field would be more like

parent: Option<Url>

or even

parent: Option<Request>

And when we build the request URL, we would have to check that source and take it into account.

If you like, feel free to experiment with the code and see if it would be possible to add the functionality without too much hassle.

[dotnetCarpenter]

It should definitely be called base and not parent 😉 Thanks for the pointers. I will look into this as soon as I have time.

[dotnetCarpenter]

heh! MDN says:

This feature is well established and works across many devices and browser versions. It’s been available across browsers since July 2015.

More like 1997! ... Sorry 1994

[dotnetCarpenter]

I think the first version should go with the 1995 spec definition of base.
https://www.w3.org/MarkUp/html-spec/html-spec_5.html#SEC5.2.2

Should probably both be configurable and be able to parse <base href="[absolute URL]"> in a local HTML file.

[nobkd]

Just linking an old related comment #1481 (comment)

[nobkd]

I think my problem is similar to this, so I'm commenting here.

How can I check links to webpages, when I'm checking files on the filesystem, and the links to webpages are naked.

example:

<!-- index.html -->
<a href="//example.com">mylink</a> <!-- fails, is naked url -->
<a href="//developer.mozilla.org/en-US/docs/Web/CSS/@starting-style">@starting-style</a> <!-- excluded, as mail -->
<a href="file.html">myfile</a> <!-- works, is local file -->

$ lychee . -v        
   [WARN ] Error creating request: InvalidPathToUri("//example.com")
[EXCLUDED] mailto://developer.mozilla.org/en-US/docs/Web/CSS/@starting-style
     [200] file:///<redacted>/file.html

🔍 2 Total (in 0s) ✅ 1 OK 🚫 0 Errors 👻 1 Excluded

$ lychee --version 
lychee 0.18.1

[mre]

First off, the InvalidPathToUri comes from me breaking relative links in #1489.
I have a fix for that in #1624 and the output will look slightly differently:

   [ERROR] file:///private/tmp/testfile/file.html | Cannot find file
[EXCLUDED] mailto://developer.mozilla.org/en-US/docs/Web/CSS/@starting-style
   [ERROR] file:///example.com | Cannot find file

Issues found in 1 input. Find details below.

[./index.html]:
   [ERROR] file:///example.com | Cannot find file
   [ERROR] file:///private/tmp/testfile/file.html | Cannot find file

🔍 3 Total (in 0s) ✅ 0 OK 🚫 2 Errors 👻 1 Excluded

Of course, file:///example.com is still incorrect; it should be http://example.com or https://example.com. (Probably http, as that's what curl uses, but I guess the default schema for HTTP URLs should be configurable as well, but that's beside the point)

But, as you said, how would we know that the link refers to an HTTP URL in that case? Technically, //example.com could be a valid path, even though it would be a pretty strange one.

I don't see a way to automate schema detection right now, but maybe I'm overlooking a possibility.

[dotnetCarpenter]

Current browser behaviour is to assume https, port 443. Unless a current URL or base says differently. If that doesn’t work then it’s invalid.

[mre]

We support local files, though, where file paths are much more likely. We could assume that two slashes at the beginning indicate a URL, but that's just an educated guess, not a strict rule. That's the problem.