aws-lc-rs dependency is a breaking change

[rukai]

Describe the bug

A recent change to aws-smithy-http-client has resulted in the rustls feature aws_lc_rs feature being enabled.

For projects using the ring backend this is a breaking change as enabling multiple rustls backends results in the error no process-level CryptoProvider available -- call CryptoProvider::install_default() before this point.

Its been a week now, so I think the boat has sailed on that one and I accept it.

But it is very difficult to actually fix this breaking change on our end.
I can see that in #1263 it was suggested to manually create a client and manually disable set all the features needed by aws-* crates.
Is it possible for AWS to provide documentation or examples that demonstrate this?
It is important to keep aws-lc-rs out of our build to avoid introducing new build time dependencies.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

It should be easy to depend on ring instead of aws-lc-rs

Possible Solution

No response

Additional Information/Context

No response

Version

latest

Environment details (OS name and version, etc.)

any

Logs

No response

[aajtodd]

This was discussed a little bit here.

It looks like you can try installing a crypto provider as the default instead of relying on crate features to define the default. That would be the approach I'd be looking to as it seems the most explicit and less prone to these feature enablement/unification problems. There appears to be more discussion here in this rustls issue.

To disable the default HTTPS client features and enable only the old HTTPS stack would look something like:

aws-config = { version = "1", default-features = false, default = ["rt-tokio", "credentials-process", "sso"] }

# repeat for each service SDK in use
aws-sdk-s3 = { version = "1", default-features = false, features = ["behavior-version-latest", "sigv4a", "rt-tokio", "rustls"] }

The problem with this is feature unification, any library you use that enables the aws-lc feature in rustls will expose this. Also overriding these default features puts the burden on you to pay attention to if you should update them.

It is important to keep aws-lc-rs out of our build to avoid introducing new build time dependencies.

For many platforms aws-lc-rs doesn't introduce new build time dependencies other than a working C compiler. Did your build break (if so what platform) or is this just a general concern?

---

In hindsight I think we should not have bundled a default HTTPS client with the SDK and provided aws-smithy-http-client as a convenience for creating a client with different common configurations. We launched the SDK with a bundled default though so that ship has sailed. The change to aws-lc was an intentional choice to use something AWS has experience operating at scale and can customize to meet service team needs. Additionally given the future of ring is a bit up in the air, defaulting to something AWS maintains (and is the default for rustls now anyway) was also a consideration.

[rukai]

Thanks for the detailed response, it was really helpful and made me reevaluate my assumptions.

When rustls originally swapped to aws-lc-rs as the default, aws-lc-rs did not build on the ubuntu github actions images due to the older gcc on ubuntu 20.04. So our project had to use ring at that time. This is however, no longer an issue as ubuntu 20.04 is deprecated, so we've moved on to ubuntu 22.04. (we need to build on the oldest available ubuntu to remain compatible with the most glibc versions)

Another thing spooking me was that aws-lc-rs pulls a bunch of weird dependencies into the Cargo.lock, after inspection it looks like they don't actually get built despite being listed in the Cargo.lock, I've made a PR to address this.

For many platforms aws-lc-rs doesn't introduce new build time dependencies other than a working C compiler. Did your build break (if so what platform) or is this just a general concern?

aws-lc-rs has a cmake/make dependency which ring does not have.
I know this is true at least for ubuntu 22.04 where I spun up a docker image to test this just now.
Without the cmake package installed the build failed, once I installed the cmake package the build succeeded.

To disable the default HTTPS client features and enable only the old HTTPS stack would look something like:

I tried that but I couldnt get it to work, not sure whats going wrong.

It looks like you can try installing a crypto provider as the default instead of relying on crate features to define the default.

You are correct that if we are to use ring as the default, we should really install it as a default crypto provider. The AWS SDK is not the first project to break us in this way.

In conclusion I think the best approach for us at the moment is to just accept the new cmake dependency and go all in on aws-lc-rs. I really don't want multiple crypto implementations floating around the project, for both build time and security reasons.

Leaving this open in case you have any ideas to shed the cmake dependency, since you didnt expect there to be one.

[aajtodd]

aws-lc-rs has a cmake/make dependency which ring does not have.
I know this is true at least for ubuntu 22.04 where I spun up a docker image to test this just now.
Without the cmake package installed the build failed, once I installed the cmake package the build succeeded.

This surprises me. The build requirements for vanilla x64 linux list only a working C compiler. Were you cross compiling? If not you might submit a ticket to aws-lc-rs as they've done a lot of work to remove the need for anything other than a working C compiler on most architectures for linux.

[rukai]

Ok, I will absolutely file that issue, thanks for your time.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[rukai]

While trying to reproduce the issue I realized that its actually openssl being pulled in for one of our integration tests that has the make requirement.
aws-lc-rs does not have any external requirements.
My apologies, that was very careless of me.