Login page and Shibboleth auth refactored to work with new InCommon services

[landreev]

What this PR does / why we need it:

See the issue. In short, the shibboleth framework had to be reconfigured since InCommon has discontinued the metadata feed around which our current implementation was built. The new implementation is based on the recommended replacement parts - the MDQ protocol for the federation metadata and the WayFinder service for the login page authentication redirect.

Which issue(s) this PR closes:

• Closes Refactor the Shibboleth parts of the Login Page to accommodate the InCommon XML feed being discontinued #11404

Special notes for your reviewer:

I preserved the old, DiscoFeed-based implementation, in case an instance has a need to keep using it. I'm guessing a use case for this will be an installation that is not part of InCommon, that got their shib. auth. to work with a static XML metadata file. They cannot use, and have no need for switching to WayFinder, so they will be able to keep their old setup intact by setting one feature flag.

In order to make the WayFinder redirect, Dataverse needs to supply the entityID of itself as a Service Provider. In most cases this will be the siteUrl + "/sp", but it's not guaranteed. One way of knowing this entityID would be to add a required configuration setting for it. Which I chose not to do in favor of calling shibd to retrieve it. One other option would be to just read it from the config file /etc/shibboleth/shibboleth2.xml, since shibd is almost certainly running on the same server. But I figured there may be some complicated load balancer situation where that is not the case. So I went to some trouble of making Dataverse call /Shibboleth.sso/Metadata once, on startup, when the authentication provider is initiated, where it is then cached for shibservice to use. Feedback welcome about this implementation.

Suggestions on how to test this:

A Dataverse instance with Shibboleth configured and properly registered with InCommon is required to test the functionality. I'm going to proceed with obtaining the registration for dataverse-internal, which hopefully will be done by the time the PR goes into QA. [this is still pending as of 05-27]

I will try to add more info here, on specific parts that need to be tested and confirmed.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

Preview at https://dataverse-guide--11502.org.readthedocs.build/en/11502/installation/shibboleth.html#identity-federation

[coveralls]

coverage: 23.424% (-0.03%) from 23.456%
when pulling a507380 on 11404-shib-login-mdq
into f223fb2 on develop.

[pdurbin]

I made some tweaks to the docs in 4ecb090 and I'm leaving a little non-critical feedback in this review. I hope I'm understanding the InCommon vs non-InCommon divide properly! Overall, this is great!

[pdurbin]

http://localhost is hard coded here and in the shibservicebean (within the string http://localhost/Shibboleth.sso/DiscoFeed). Should http://localhost be factored into a common location? Should https://localhost (https instead of http) be an option?

[landreev]

It actually needs to be http:, not https: - since that will result in a failure if called from Java, since the ssl certificate is not going to be valid for localhost. And this in turn was the main reason why I chose not to make this the default behavior, since calling http://localhost/Shibboleth.sso/DiscoFeed would not work on an instance that followed our recommendations for the Apache configuration. I.e., things would break for most Shibboleth-using instances without a configuration change - and, seeing how this localhost gimmick is extremely unlikely to be needed by any instance other than ours, I absolutely wanted to avoid that.

[pdurbin]

Suggested change

	* It is kept in the code for now, under the assumption that somebody
	* may still have reasons to keep using the DiscoFeed-based model.
	* It is kept in the code for now because the majority of Dataverse
	* installations using Shib are likely not using InCommon.

In the release notes we say that if you're not using InCommon you should set dataverse.feature.shibboleth-use-discofeed=true. I'm pretty sure Harvard Dataverse and UNC Dataverse are the only ones using InCommon. All the other shib installations (of which there are many, I think) will operate in this non-InCommon mode (DiscoFeed-mode), as far as I understand.

[landreev]

Thank you for pointing out/clarifying this. This is totally on me, I asked which case was more common (it was an in-person conversation by the coffee machine, I think) - but I must have misunderstood your answer as the exact opposite of the above. ... It does make far more sense of course, that our use case is in fact fringe. I changed the feature flag etc. accordingly (to an optional dataverse.feature.shibboleth-use-wayfinder=true). Will update the release note and docs momentarily.

[landreev]

I'm going to mention this during standup, but note that I removed the 6.7 milestone from this last week.
At the same time, there was actually some progress - while at DCM, I pinged the Harvard iam group and finally got them to get to complete the InCommon registration.
But this of course is only 50% of the process - as we also need InCommon to grant this sp the status of a "Research & Scholarship" entity, to match the prod. configuration 1:1. In the process of getting that part done now. But unsure of how long that's going to take, hence not pushing for the inclusion in 6.7.

[landreev]

As of yesterday, the word I got was that we would get the answer on the R&S application "within 2 business days", i.e. by tomorrow.

[landreev]

Oh wow, I haven't heard back from our contacts, but

% curl "https://mdq.incommon.org/entities/https%3A%2F%2Fdataverse-internal.iq.harvard.edu%2Fsp" | grep -i research

        <saml:AttributeValue>http://refeds.org/category/research-and-scholarship</saml:AttributeValue>

This looks very promising. I will update the "how to test" section in preparation for (maybe) finally being able to test and merge this thing.

[landreev]

Working on fixing one bug I had reported on demo: redirects not working consistently. I.e., the user gets logged in successfully, but they not redirected to the proper destination page necessarily.

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:11404-shib-login-mdq

ghcr.io/gdcc/configbaker:11404-shib-login-mdq

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.