Skip to content

Conversation

@3axap4eHko
Copy link

What

Adds support for custom CA certificates in OAuth login flows to fix authentication failures behind corporate
proxies with SSL/TLS inspection.

Why

The Codex CLI fails to authenticate when running behind corporate proxies that perform SSL/TLS inspection with
custom CA certificates. The OAuth token exchange fails because reqwest::Client::new() only trusts system
default certificate roots, causing it to reject connections intercepted by corporate proxies presenting
certificates signed by custom CAs.

This affects all Codex CLI users in enterprise environments with:

  • SSL/TLS inspection proxies (Zscaler, Palo Alto Networks, etc.)
  • Custom internal CA certificates
  • Air-gapped environments with internal certificate authorities

Fixes #6849

How

Implementation:

  • Added build_login_http_client() function that creates an HTTP client with custom CA certificate support
  • Reads CA certificate path from environment variables (priority order):
    1. CODEX_CA_CERTIFICATE (primary)
    2. SSL_CERT_FILE (fallback, standard across many tools)
  • Updated all three authentication flows to use the custom client:
    • exchange_code_for_tokens() (OAuth code exchange)
    • obtain_api_key() (API key exchange)
    • run_device_code_login() (device code flow)
  • Follows the same pattern as existing otel_provider.rs implementation

Testing:

  • Added comprehensive test coverage with 3 new tests:
    • Certificate loading via CODEX_CA_CERTIFICATE env var
    • Certificate loading via SSL_CERT_FILE fallback
    • Invalid certificate rejection with proper error handling
  • All existing tests continue to pass
  • Added serial_test dependency to safely test env var manipulation

Usage:

export CODEX_CA_CERTIFICATE=/path/to/corporate-ca.pem
codex login

or using the standard env var:

export SSL_CERT_FILE=/path/to/corporate-ca.pem
codex login

Testing:

  • ✅ All tests pass (cargo test)
  • ✅ No clippy warnings (cargo clippy --tests -- -D warnings)
  • ✅ Code formatted (cargo fmt --check)
  • ✅ Tested with both environment variables
  • ✅ Invalid certificate properly rejected with clear error message

@github-actions
Copy link
Contributor

github-actions bot commented Nov 19, 2025

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@3axap4eHko
Copy link
Author

I have read the CLA Document and I hereby sign the CLA

github-actions bot added a commit that referenced this pull request Nov 19, 2025
@etraut-openai
Copy link
Collaborator

@codex review

Copy link
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@3axap4eHko
Copy link
Author

@codex review

1 similar comment
@etraut-openai
Copy link
Collaborator

@codex review

@chatgpt-codex-connector
Copy link
Contributor

Codex Review: Didn't find any major issues. Breezy!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@etraut-openai
Copy link
Collaborator

@3axap4eHko, thanks for the contribution! It looks like there are some minor code format issues. Please fix those when you have time.

@etraut-openai etraut-openai added needs-response Additional information is requested and removed needs-response Additional information is requested labels Nov 20, 2025
@etraut-openai
Copy link
Collaborator

@3axap4eHko, the PR is looking good. We'll need to document the new environment variables. Ideally, this should be part of the same PR. Do you want to take a shot at adding the documentation?

Copy link
Collaborator

@joshka-oai joshka-oai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Brief review notes / questions to help work out whether to do a deeper review.

Copy link
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@joshka-oai
Copy link
Collaborator

I'm assuming that your environment sets some sort of CA root authority that your general tools (browser etc.) properly check, so you don't tend to get errors in non-cli tooling. Is this something that we could generally load from the environment in the same way without having to manually configure this with environment variables? I.e. perhaps configuring the tooling to load from the OS Cert store where this is setup. There's probably a crate that sets this up automatically if that's the case. I'm not sure whether this is relevant to the use cases here, so please forgive the naive question.

@3axap4eHko
Copy link
Author

I'm assuming that your environment sets some sort of CA root authority that your general tools (browser etc.) properly check, so you don't tend to get errors in non-cli tooling. Is this something that we could generally load from the environment in the same way without having to manually configure this with environment variables? I.e. perhaps configuring the tooling to load from the OS Cert store where this is setup. There's probably a crate that sets this up automatically if that's the case. I'm not sure whether this is relevant to the use cases here, so please forgive the naive question.

Totally fair question. Out of the box, reqwest already trusts the platform store (via either system OpenSSL/Schannel/Security‑Framework on the blocking TLS stack or rustls-native-certs when using the rustls backend). That’s the reason browsers and “normal” CLI calls don’t need extra configuration. The new code only kicks in when you explicitly point CODEX_CA_CERTIFICATE or SSL_CERT_FILE at a custom bundle, because corporate proxies often publish their own root chain that isn’t in the OS store.

So, for users whose proxy installs its cert into the regular Windows/macOS/Linux trust store, nothing special is required—the default client still works. The env knob is just for the cases where IT hands you a standalone PEM bundle instead of touching the OS store. In that scenario you still need manual configuration somewhere, either by importing the PEM into the OS trust store (which is painful to automate across platforms) or by telling the app exactly which file to add. The helper we added is that second option.

If we ever run into a platform where the OS store is available but reqwest/rustls can’t discover it, the fix would likely be as simple as enabling the rustls-native-certs crate (or the native-tls backend) for that target. But so far there hasn’t been a report that the default trust roots are being ignored; every bug we’ve seen came from proxy-issued certs that the OS store rightfully doesn’t trust.

@etraut-openai
Copy link
Collaborator

@3axap4eHko, I wanted to give you an update on this PR. Since it involves security-sensitive code, we have pulled in some security experts outside of the codex team to provide a more thorough code review. This is a holiday week in the U.S. (Thanksgiving), so many OpenAI employees are out of the office. That means we probably won't make any further progress on this until next week.

@joshka-oai
Copy link
Collaborator

After chatting about this with our internal security, we'd prefer to have the settings for this added to the config.toml file rather than as an environment variable. If you're concerned that you need single call setups (where you don't want to set this at the entire machine level) codex has a --config flag that can generally be used to pass config key/values like this. We don't need to add a specific command line flag for this. Let's call this setting login_ca_certificate.

Let's use rustls-pki-types to open / parse the pem files rather thank hand coded.

Happy to continue the implementation on this with codex, or let you complete it. The things I'd generally want to make sure are covered here:

@etraut-openai etraut-openai added the needs-response Additional information is requested label Nov 27, 2025
@3axap4eHko
Copy link
Author

After chatting about this with our internal security, we'd prefer to have the settings for this added to the config.toml file rather than as an environment variable. If you're concerned that you need single call setups (where you don't want to set this at the entire machine level) codex has a --config flag that can generally be used to pass config key/values like this. We don't need to add a specific command line flag for this. Let's call this setting login_ca_certificate.

Let's use rustls-pki-types to open / parse the pem files rather thank hand coded.

Happy to continue the implementation on this with codex, or let you complete it. The things I'd generally want to make sure are covered here:

@joshka-oai One thing I want to push back on in the "after chatting with internal security" part: environment-based CA configuration is a very common pattern on Unix. SSL_CERT_FILE is the standard OpenSSL convention and is used by tools like curl, git, Python requests, etc. It’s not inherently less secure than putting the same path in config.toml; both are just configuration channels, but it allows to avoid micromanage every cli tool.

If the Codex policy is "all TLS-related overrides must live in config.toml and we intentionally ignore SSL_CERT_FILE", that’s a valid product/design decision. But it’s different from saying that using SSL_CERT_FILE is a security concern. I’d prefer we frame this explicitly as "Codex chooses config-over-env for CA overrides" (and document that) rather than implying the standard env-based mechanism is problematic.

@joshka-oai
Copy link
Collaborator

I'm in general agreement with you, but I'm not sure if there's things I don't know about this yet.
I'll follow up with our security folk and will get back to you on this shortly.

@joshka-oai
Copy link
Collaborator

joshka-oai commented Dec 1, 2025

I clarified with our Security team. We're fine with using environment variables on this. I misread a recommendation in a message.

The main part of getting this over the line is testing this correctly in a way that doesn't cause other usages of this code path to be impacted. Leaving the ball with you for now to take a look, but feel free to ask more questions or kick it back to us if you're not sure of the right direction on this.

3axap4eHko and others added 3 commits December 11, 2025 15:15
Wraps env::set_var and env::remove_var calls in unsafe blocks as required
by newer Rust versions. Also updates test certificate to valid RSA format
and adds better error messages to test assertions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@3axap4eHko
Copy link
Author

@joshka-oai Oh thanks for helping, I didn't expect that. I was a bit off for holidays

@joshka-oai
Copy link
Collaborator

Rebased on main, and got codex to implement the necessary parts of this that

@joshka-oai Oh thanks for helping, I didn't expect that. I was a bit off for holidays

No problem - take a quick look to see if this change still solves your problem in real testing if you have a few moments.
I think it's likely mostly equivalent to your change, but it's always possible something was missed in the rewrite (e.g. rustls-pki-types does something different in handling the pem load).

@3axap4eHko 3axap4eHko force-pushed the ISSUE-6849 branch 2 times, most recently from b5feeb4 to 16e15ce Compare December 12, 2025 18:05
@3axap4eHko
Copy link
Author

@codex review

@chatgpt-codex-connector
Copy link
Contributor

Codex Review: Didn't find any major issues. Nice work!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@joshka-oai
Copy link
Collaborator

Quick clarification on what happened here, since this is a workflow not everyone runs into:

As a maintainer, I have push access to contributor branches. I used that access to rebase your branch on current main and then push a set of fixes (PEM parsing via rustls‑pki‑types, env‑safe tests, clearer errors, docs). That’s not something we do often, but it’s sometimes useful for unblocking review feedback. The important part: it rewrites the published history on your branch, which can be confusing if you’re not expecting it.

image

After that, the branch was force‑pushed again and those maintainer changes were overwritten. So the PR currently doesn’t include the rewrite.

To get us back to the version that includes our fixes, can you please reset your local branch back to the commit where that work landed, test it, and start any new changes from there:

git fetch your-remote-name (e.g. origin)
git checkout ISSUE-6849
git reset --hard 2eac899ac7c43157c648c9cbb2b9825940daeab7
# run your usual tests
git push --force

My main goal here is to ensure that you've tested the changes I made to this in your environment with a real TLS cert that is setup for this.

Once you’ve done that, any additional changes should be made on top of that commit (2eac899) so we don’t lose the rewrite again.

If anything in the force‑push after that point was intentional and should be preserved, let me know and I’ll help fold it back in.

As a side note, I generally avoid merging from main except right before merging as it makes it difficult to follow the history of the commits. I rebased this PR to avoid that problem and start us off from a good point. We squash merge all our PRs so having the merge commits in there is not a big issue, but don't worry too much about getting the absolute latest code from main all the time.

Enterprise TLS inspection proxies use custom roots, so OAuth token exchanges
fail when we only trust system CAs. This change switches PEM parsing to
rustls-pki-types (multi-cert bundles included) and surfaces clearer, user-
facing errors that explain how to fix invalid or empty CA files via
CODEX_CA_CERTIFICATE/SSL_CERT_FILE.

To avoid cross-test races with process-wide env vars, CA path selection now
uses a small EnvSource abstraction in unit tests, and environment-dependent
behavior is verified via an assert_cmd-driven login_ca_probe helper binary.
This keeps normal tests isolated while still validating env precedence and
error messaging.

Also updates login dev-deps (assert_cmd/pretty_assertions), removes serial_test,
and re-exports build_login_http_client for the probe helper.
@joshka-oai
Copy link
Collaborator

joshka-oai commented Dec 13, 2025

Sorry for the churn here — this one ended up more confusing than it should have, and that’s on me.

Very important workflow note for this PR (please read once):

🚫 Please do NOT merge main into this branch.
🚫 Please do NOT force-push over the branch history.

I’m intentionally keeping this branch rebased as a maintainer so we can reason about the changes cleanly. When main moves, I’ll handle rebasing on my side. Merging main or force-pushing replaces that work and makes review harder, which is how we ended up with two different “expected” versions of the PR.

I’ve force-pushed the branch back to a canonical state that includes:

  • the rustls-pki-types PEM parsing,
  • test hardening,
  • and the doc updates.

From here on, please treat the current tip of the branch as the source of truth.


What I need from you (low effort, no Git surgery)

If you have a few minutes, please sanity-check the current branch in your real environment (your actual CA / proxy setup) and confirm:

  • does this version still solve the original problem for you?

That confirmation is the main thing blocking this from landing.


If you do need to make changes locally

To make sure you’re working from the correct base and don’t accidentally overwrite things, the safest flow is:

git fetch origin
git checkout ISSUE-6849
git reset --hard origin/ISSUE-6849

Then:

  • make commits on top of that
  • push normally (no force-push)
  • and don’t merge main

If something from your earlier commits was intentional and got lost in the rewrite, just say so — I’m happy to help fold it back in.

I think the main missing commit in your work on this is 181c08b

Thanks for sticking with this, and sorry again for the workflow back-and-forth. Once we confirm it works for you, we should be in good shape to wrap this up.

@joshka-oai
Copy link
Collaborator

I added back in the docs commit, which got lost in the shuffle.

@3axap4eHko
Copy link
Author

@joshka-oai is it safe to pick up the current branch state, and work based on it?

@joshka
Copy link

joshka commented Dec 13, 2025

Yes. Go for it.

@joshka-oai
Copy link
Collaborator

joshka-oai commented Dec 15, 2025

Can you please add some extra justification for the new changes? E.g. references to standards / documentation of the specific firewalls that are adding the new parts of this added in 299047b and 77e87f0

If I'm reading between lines here, you tested this in your environment and found that you needed extra config to align. It would be helpful to make sure we capture the rationale here rather than just the changes.

I see that rustls intentionally decided not to support pemfiles with "BEGIN TRUSTED CERTIFICATE" in them over in rustls/pemfile#52

I also see that rustls has pem file certs that have CRLs (but can't see any obvious parsing for this). I'd prefer all the parsing code for pem files to be in the library where possible. Doing anything with funky parsing of certs requires every developer that modifies that code to dig into the details of how the parsing should work, what specs it meets, etc. Pushing those decisions off to a library instead of manually implementing it allows the wisdom of developers more knowledgable about such things to be applied.

Anything outside of the absolutely normal parsing of these files needs to either be well justified or use libraries in their normal form.

@3axap4eHko
Copy link
Author

@joannas-oai Of course, here’s the rationale behind the two parsing tweaks and a path to make them less bespoke:

Why we normalized “TRUSTED CERTIFICATE”

  • Enterprise root bundles commonly ship with “BEGIN TRUSTED CERTIFICATE” (OpenSSL’s “trusted cert” form, see man
    x509 and OpenSSL docs on “Trusted Certificates”). Corporate proxy vendors (e.g., Zscaler, Palo Alto) often export
    their roots that way. Rustls’ pki-types rejects that label (see Certificates in PEM files that starts "BEGIN TRUSTED CERTIFICATE" are ignored rustls/pemfile#52), so users hit “no certificates
    found”. Normalizing the label lets those bundles load while discarding the extra OpenSSL trust bits, which rustls
    wouldn’t consume anyway.

Why we stripped CRL blocks

  • Some enterprise exports bundle CRLs alongside roots (standard PEM label “X509 CRL”, RFC 5280). Rustls/reqwest does
    not consume CRLs today; leaving them in causes the certificate-only iterator to error. We strip them to accept
    mixed bundles while still loading the certs. We are not attempting CRL enforcement; just avoiding a parse failure.

If we want to reduce ad-hoc parsing

  • We can switch to rustls-pemfile::read_all and pick out Item::X509Certificate, ignoring everything else (including
    CRLs). That uses library parsing instead of manual replacements.
  • “TRUSTED CERTIFICATE” would still need a policy decision: either keep the normalization (dropping auxiliary trust
    info) or reject them and document a conversion step (openssl x509 -in trusted.pem -out cert.pem).
  • CRL handling could be clarified in docs: we currently ignore CRLs because rustls does; adding real CRL checking
    would require a custom verifier and is out-of-scope.

Let me know if you’d prefer:
A) keep normalization + CRL stripping but add the above justification to code comments/docs, or
B) move to rustls-pemfile::read_all and only keep a minimal, explicit normalization for trusted labels, or
C) drop the trusted/CRL tolerance entirely and document required conversions.

@joshka-oai
Copy link
Collaborator

Ok, it sounds like the TRUSTED prefix seems reasonable to think about. My concern is that several upstream things have decided to intentionally not support that. How about we split the PR at this on this part to a second PR?

Same deal with CRL - is it possible to make the use of rustls-pki-types tolerant of the CRL without having to remove it upfront?

In generaly, we shouldn't want to switch back to rustls-pemfile or generally do much ad-hoc parsing / modification of the certs.

  • pemfile because the upstream developers are actively working on the replacement rustls-pki-types
  • manual parsing / modification because doing this opens us up to making mistakes on insecure edge cases and increases the maintenance burden to avoid doing so

@3axap4eHko
Copy link
Author

@codex review

Copy link
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@3axap4eHko
Copy link
Author

@joshka-oai is there something here that needs to be addressed?

@joshka-oai
Copy link
Collaborator

Hey there, I haven't forgotten about this. Thanks for hanging in there and putting up with the delay on this. It's just below my threshold for where I can allocate time to get this over the line, and the majority of the team has been taking a break over the holiday period. I'll talk with a few people early next week to get some extra eyes on this.

Something to note is the changes coming in reqwest 0.13, which moves to using rustls by default and has some extra things around merging tls certs. I haven't looked into that change yet to see if there's overlap on this, but if there is, then we likely should upgrade reqwest first and then rebase this on top of those newer approaches.
https://seanmonstar.com/blog/reqwest-v013-rustls-default/

@3axap4eHko
Copy link
Author

Hey there, I haven't forgotten about this. Thanks for hanging in there and putting up with the delay on this. It's just below my threshold for where I can allocate time to get this over the line, and the majority of the team has been taking a break over the holiday period. I'll talk with a few people early next week to get some extra eyes on this.

Something to note is the changes coming in reqwest 0.13, which moves to using rustls by default and has some extra things around merging tls certs. I haven't looked into that change yet to see if there's overlap on this, but if there is, then we likely should upgrade reqwest first and then rebase this on top of those newer approaches. https://seanmonstar.com/blog/reqwest-v013-rustls-default/

I'll review changes in reqwest

@3axap4eHko
Copy link
Author

@joshka-oai I confirm, upgrade will allow to get rid of some crates and reduce amount of code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-response Additional information is requested

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OAuth Login Fails Behind Corporate Proxy with Custom CA Certificates

5 participants