Easy to use webpki-roots

[joshtriplett]

Describe the bug

aws-sdk-rust includes an example of how to configure hyper-rustls to use webpki-roots and TLS 1.3, in examples/tls/src/lib.rs.

However, this example has two separate versions of hyper-rustls involved (0.23.0 and 0.22.1), and while hyper-rustls 0.23.0 is configured to use webpki-roots, something in aws-sdk-rust is still attempting to initialize hyper-rustls 0.22.1, which is failing with this error:

thread 'main' panicked at 'no CA certificates found', /home/josh/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-rustls-0.22.1/src/connector.rs:45:13

(The easiest way to test this is to build a static binary using the x86_64-unknown-linux-musl target and run it in an otherwise empty chroot.)

Expected Behavior

I'd like to use aws-sdk-rust with only webpki-roots, in a standalone static binary that has no system certificate store.

Current Behavior

thread 'main' panicked at 'no CA certificates found', /home/josh/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-rustls-0.22.1/src/connector.rs:45:13

Reproduction Steps

Here's a cut down version of the example (skipping the TLS 1.3 bits):

Cargo.toml

[package]
name = "awstest"
version = "0.1.0"
edition = "2021"

[dependencies]
anyhow = "1.0.58"
async-std = { version = "1.12.0", features = ["tokio1", "attributes"] }
aws-config = { version = "0.15.0", default-features = false, features = ["rustls"] }
aws-sdk-sts = { version = "0.15.0", default-features = false }
aws-smithy-client = { version = "0.45.0", default-features = false }
hyper-rustls = { version = "0.23.0", default-features = false, features = ["http1", "http2", "tls12", "webpki-roots", "webpki-tokio"] }

(aws-config needs rustls enabled due to an unrelated bug. And the hyper-rustls dependency is coming in via aws-smithy-client: despite default-features = false, there's an indirect dependency on aws-smithy-client's default features via aws-types, among other things.)

src/main.rs

#[async_std::main]
async fn main() -> anyhow::Result<()> {
    let https_connector = hyper_rustls::HttpsConnectorBuilder::new()
        .with_webpki_roots()
        .https_only()
        .enable_http1()
        .enable_http2()
        .build();
    let provider_config = aws_config::provider_config::ProviderConfig::default()
        .with_tcp_connector(https_connector.clone())
        .load_default_region()
        .await;
    let aws_sdk_config = aws_config::from_env()
        .configure(provider_config)
        .load()
        .await;

    let sts = aws_sdk_sts::Client::from_conf_conn(
        aws_sdk_sts::Config::from(&aws_sdk_config),
        aws_smithy_client::hyper_ext::Adapter::builder().build(https_connector),
    );

    let identity = sts.get_caller_identity().send().await?;
    dbg!(identity);
    Ok(())
}

Some debug prints suggest that the issue happens when trying to create aws_sdk_config. The example in "possible solution" gives a workaround.

Possible Solution

Alternate example that does work:

src/main.rs

#[async_std::main]
async fn main() -> anyhow::Result<()> {
    let https_connector = hyper_rustls::HttpsConnectorBuilder::new()
        .with_webpki_roots()
        .https_only()
        .enable_http1()
        .enable_http2()
        .build();
    let provider_config = aws_config::provider_config::ProviderConfig::default()
        .with_tcp_connector(https_connector.clone())
        .load_default_region()
        .await;
    let adapter = aws_smithy_client::hyper_ext::Adapter::builder().build(https_connector);
    let dyn_connector = aws_smithy_client::erase::DynConnector::new(adapter.clone());
    let aws_sdk_config = aws_config::from_env()
        .http_connector(aws_smithy_client::http_connector::HttpConnector::Prebuilt(Some(dyn_connector)))
        .configure(provider_config)
        .load()
        .await;

    let sts = aws_sdk_sts::Client::from_conf_conn(
        aws_sdk_sts::Config::from(&aws_sdk_config),
        adapter,
    );

    let identity = sts.get_caller_identity().send().await?;
    dbg!(identity);
    Ok(())
}

Additional Information/Context

No response

Version

├── aws-config v0.15.0
│   ├── aws-http v0.15.0
│   │   ├── aws-smithy-http v0.45.0
│   │   │   ├── aws-smithy-types v0.45.0
│   │   ├── aws-smithy-types v0.45.0 (*)
│   │   ├── aws-types v0.15.0
│   │   │   ├── aws-smithy-async v0.45.0
│   │   │   ├── aws-smithy-client v0.45.0
│   │   │   │   ├── aws-smithy-async v0.45.0 (*)
│   │   │   │   ├── aws-smithy-http v0.45.0 (*)
│   │   │   │   ├── aws-smithy-http-tower v0.45.0
│   │   │   │   │   ├── aws-smithy-http v0.45.0 (*)
│   │   │   │   ├── aws-smithy-types v0.45.0 (*)
│   │   │   ├── aws-smithy-http v0.45.0 (*)
│   │   │   ├── aws-smithy-types v0.45.0 (*)
│   ├── aws-sdk-sso v0.15.0
│   │   ├── aws-endpoint v0.15.0
│   │   │   ├── aws-smithy-http v0.45.0 (*)
│   │   │   ├── aws-types v0.15.0 (*)
│   │   ├── aws-http v0.15.0 (*)
│   │   ├── aws-sig-auth v0.15.0
│   │   │   ├── aws-sigv4 v0.15.0
│   │   │   │   ├── aws-smithy-http v0.45.0 (*)
│   │   │   ├── aws-smithy-http v0.45.0 (*)
│   │   │   ├── aws-types v0.15.0 (*)
│   │   ├── aws-smithy-async v0.45.0 (*)
│   │   ├── aws-smithy-client v0.45.0 (*)
│   │   ├── aws-smithy-http v0.45.0 (*)
│   │   ├── aws-smithy-http-tower v0.45.0 (*)
│   │   ├── aws-smithy-json v0.45.0
│   │   │   └── aws-smithy-types v0.45.0 (*)
│   │   ├── aws-smithy-types v0.45.0 (*)
│   │   ├── aws-types v0.15.0 (*)
│   ├── aws-sdk-sts v0.15.0
│   │   ├── aws-endpoint v0.15.0 (*)
│   │   ├── aws-http v0.15.0 (*)
│   │   ├── aws-sig-auth v0.15.0 (*)
│   │   ├── aws-smithy-async v0.45.0 (*)
│   │   ├── aws-smithy-client v0.45.0 (*)
│   │   ├── aws-smithy-http v0.45.0 (*)
│   │   ├── aws-smithy-http-tower v0.45.0 (*)
│   │   ├── aws-smithy-query v0.45.0
│   │   │   ├── aws-smithy-types v0.45.0 (*)
│   │   ├── aws-smithy-types v0.45.0 (*)
│   │   ├── aws-smithy-xml v0.45.0
│   │   ├── aws-types v0.15.0 (*)
│   ├── aws-smithy-async v0.45.0 (*)
│   ├── aws-smithy-client v0.45.0 (*)
│   ├── aws-smithy-http v0.45.0 (*)
│   ├── aws-smithy-http-tower v0.45.0 (*)
│   ├── aws-smithy-json v0.45.0 (*)
│   ├── aws-smithy-types v0.45.0 (*)
│   ├── aws-types v0.15.0 (*)
├── aws-sdk-sts v0.15.0 (*)
├── aws-smithy-client v0.45.0 (*)

Environment details (OS name and version, etc.)

Debian, latest sid

Logs

No response

[Velfi]

We are working on reducing the number of branching feature paths. We plan to accomplish this by unifying the HTTP connection providers and possibly providing them through a separate crate.

Ultimately, we want to avoid the situation where people (including us) have to understand complex branching rules in order to understand the ways that HTTP/HTTPS client creation and configuration happens.

[joshtriplett]

Ultimately, we want to avoid the situation where people (including us) have to understand complex branching rules in order to understand the ways that HTTP/HTTPS client creation and configuration happens.

I can definitely appreciate that. My primary goal with this was simplicity as well; I don't just want to reduce dependencies (which I care about as well), but also, omitting native certificate support entirely seems like the easiest way to be confident that no possible code path will attempt to read them.

Nonetheless, I'd be happy with any solution path that makes that possible, including a refactor to do HTTP initialization in a different way. I'd just like to make sure that it's not substantially harder to run with webpki-roots than to run with native roots.

Thank you for aws-sdk-rust!

[rcoh]

In either case, I think there's a bug here in aws-config where something isn't building a connector properly. Thanks for the reproducer, we'll track this down

[joshtriplett]

Checking back in on this: it still seems like there's no way to configure the built-in usage of hyper-rustls to use webpki-roots, and the only solution is overriding it completely and making sure the built-in version never gets invoked. I'd love to have a simpler and more robust solution than that.

[jmklix]

Changing this to feature request to make webpki-roots easier to use.