Buck2 not respecting config file

[achint-trc]

Context
A monorepo setup with Buck2 build. I have a .buckconfig holding all the config details including [buck2_re_client] for remote builds.

Issue
In my .buckconfg the remote build section looks like:

[buck2_re_client]
action_cache_address = <my_rbe.com>
engine_address = <my_rbe.com>
cas_address = <my_rbe.com>
instance_name = default

Since I want to do remote builds via CI as well, I was planning to create another file like .buckconfig.ci which shall hold the certs

[buck2_re_client]
tls_ca_certs = <path/to/ca/cert/in/ci>
tls_client_cert = <path/to/client/cert/in/ci>

and build like :

buck2 build --prefer-remote --config-file .buckconfig.ci <targets>

Expected
The Buck2 client merges the configs (as it does when users have a .buckconfig and .buckconfig.local on their dev setup locally) and does the build.

Actual
It fails with Unable to verify certificate or transport error: invalid peer certificate. If I copy paste the key-value pairs mentioned in .buckconfig.ci to .buckconfig itself and run, it works smoothly.

What changed?
We were earlier using http_headers to pass JWT token for auth with our RBE. In .buckconfig we kept the token as an environment variable (which would be exposed via CI) and for developers, they could simply override via .buckconfig.local

Constraints
I can't put these certs in .buckconfig as we want developers to use the JWT token method only.

Note
I also tried using --config as mentioned in https://buck2.build/docs/concepts/buckconfig/#precedence-of-buck2-configuration-specifications but didn't help.

[IanChilds]

.buckconfig.local is explicitly specified in buck2 as a possible location for buckconfigs, but .buckconfig.ci is not, so I don't think you'd expect that to work unfortunately.

I also tried using --config as mentioned in https://buck2.build/docs/concepts/buckconfig/#precedence-of-buck2-configuration-specifications but didn't help.

What do you mean by that? I would expect that to work, you can specify buckconfigs on the command line. What were you actually doing that didn't work there?

[achint-trc]

I agree that .buckconfig.local is an explicit acceptable location, but we do have --config-file as well and I shared above that I am running the build command as :

buck2 build --prefer-remote --config-file .buckconfig.ci ....

and the reason I say it doesn't respect the input from config-file is because it fails with the certificate related errors whereas the same error goes away if I directly specify the cert in .buckconfig

[IanChilds]

Oh yeah, sorry. What does buck2 log show | grep ".buckconfig.ci" give you? Perhaps it is not finding that file correctly?

[achint-trc]

It looks like

{"command_line_args":["<buck2 binary path>","build","--prefer-remote","--config-file",".buckconfig.ci","//examples:example"],"expanded_command_line_args":["<buck2 binary path>","build","--prefer-remote","--config-file",".buckconfig.ci","//examples:example"],"working_dir":"<path/to/monorepo>","trace_id":"8e78578a-dc68-4e27-ba77-79409963b7a8","start_time":[1773062257,283276010]}

{"Event":{"timestamp":[1773062257,956945234],"trace_id":"8e78578a-dc68-4e27-ba77-79409963b7a8","span_id":1,"parent_id":0,"data":{"SpanStart":{"data":{"Command":{"metadata":{"materializer":"deferred","os":"linux","arch":"x86_64","io_provider":"fs","daemon_uuid":"e19101df-2744-4fb0-9e05-73bd15e573c2","vpnless":"false","hostname":"<hostname>","os_version":"<os_version>","client":"terminal-fallback","http_versions":"1,2"},"cli_args":["<buck2 binary path>","build","--prefer-remote","--config-file",".buckconfig.ci","//examples:example"],"tags":["dice-detect-cycles:Disabled","hash-all-commands:true","sqlite-materializer-state:true","paranoid:false","remote-dep-files:false","disable-eager-write-dispatch-v2:false","use-eden-thrift-read:false","memory_tracker-enabled:false","action-freezing-enabled:false","has-cgroup:false"],"data":{"Build":{}}}}}}}}

{"Event":{"timestamp":[1773062257,958855415],"trace_id":"8e78578a-dc68-4e27-ba77-79409963b7a8","span_id":0,"parent_id":4,"data":{"Instant":{"data":{"BuckconfigInputValues":{"components":[{"data":{"GlobalExternalConfigFile":{"values":[],"origin_path":"/etc/buckconfig"}}},{"data":{"GlobalExternalConfigFile":{"values":[],"origin_path":"<my_nfs_dir>/.buckconfig.local"}}},{"data":{"ConfigFile":{"data":{"GlobalExternalConfig":{"values":[{"section":"buck2_re_client","key":"tls_ca_certs","value":"<path/to/monorepo>/build/certs/root-ca.pem","cell":null,"is_cli":true},{"section":"buck2_re_client","key":"tls_client_cert","value":"<path/to/monorepo>/cert.pem","cell":null,"is_cli":true}],"origin_path":"<path/to/monorepo>/.buckconfig.ci"}}}}}]}}}}}}

<my_nfs_dir>/.buckconfig.local doesn't exist. I have trimmed some names but that should be irrelevant hopefully.

Also, on doing a buck2 audit config --config-file .buckconfig.ci, it shows the config correctly i.e. both of .buckconfig and .buckconfig.ci merged.

Note: I have retried in a fresh clone and also after a buck2 killall

[achint-trc]

Also, I see that a test was introduced for --config-file in buck2 RE tests, d0d5e2f
I believe this test is working as expected but still I haven't got a clue what's going wrong at my end.

@KapJI if you have any idea?

[achint-trc]

I tried printing the external-configs after the failure

$ buck2 log external-configs
Showing external configs from: <buck2-binary> build --prefer-remote --config-file buckconfig.ci <targets>

buck2_re_client.tls_ca_certs = <tls_ca_certs_path>                 <path_to_buckconfig.ci>
buck2_re_client.tls_client_cert = <tls_client_cert_path>          <path_to_buckconfig.ci>