elasticsearch/opensearch fails to authenticate with 403

[mhristof]

im trying to setup a simple stdin -> opensearch pipeline and im having some trouble authenticating to opensearch.

Im running vector 0.29.0 (x86_64-unknown-linux-gnu 33b3868 2023-04-11 22:19:54.512366263) and my config is

[sources.in]
type = "stdin"

[sinks.opensearch]
inputs = ["in"]
type = "elasticsearch"
auth.strategy = "aws"
endpoints = ["https://redacted:443"]
aws.region = "eu-west-2"

the verbose output is something like this

2023-04-19T17:54:07.962263Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-04-19T17:54:07.962305Z  INFO vector::app: Log level is enabled. level="vector=debug,codec=debug,vrl=debug,file_source=debug,tower_limit=trace,rdkafka=debug,buffers=debug,lapin=debug,kube=debug"
2023-04-19T17:54:07.963038Z  INFO vector::app: Loading configs. paths=["os.toml"]
2023-04-19T17:54:07.963400Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2023-04-19T17:54:07.964324Z DEBUG vector::topology::builder: Building new source. component=parse_json
2023-04-19T17:54:07.964677Z DEBUG vector::topology::builder: Building new sink. component=opensearch
2023-04-19T17:54:07.964892Z  INFO vector::sources::file_descriptors: Capturing stdin.
2023-04-19T17:54:08.011376Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-04-19T17:54:08.019012Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-04-19T17:54:08.028330Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=https://redacted:443/_cluster/state/version method=GET version=HTTP/1.1 headers={"x-amz-date": "20230419T175408Z", "authorization": Sensitive, "x-amz-security-token": "redacted1", "user-agent": "Vector/0.29.0 (x86_64-unknown-linux-gnu 33b3868 2023-04-11 22:19:54.512366263)", "accept-encoding": "identity"} body=[empty]
2023-04-19T17:54:08.064250Z DEBUG http: vector::internal_events::http_client: HTTP response. status=403 Forbidden version=HTTP/1.1 headers={"date": "Wed, 19 Apr 2023 17:54:08 GMT", "content-type": "application/json", "content-length": "1971", "connection": "keep-alive", "x-amzn-requestid": "275016dc-e965-42a8-8fdb-a15b59dbfa4a", "access-control-allow-origin": "*"} body=[1971 bytes]
2023-04-19T17:54:08.065084Z DEBUG vector::sinks::elasticsearch::common: Assumed ElasticsearchApi based on config setting suppress_type_name. assumed_version=8 config.suppress_type_name=false
2023-04-19T17:54:08.065127Z  WARN vector::sinks::elasticsearch::common: Failed to determine Elasticsearch version from `/_cluster/state/version`. Please fix the reported error or set an API version explicitly via `api_version`. assumed_version=8 error=Unexpected response from Elasticsearch endpoint `/_cluster/state/version`. Missing `version`. Consider setting `api_version` option.
2023-04-19T17:54:08.076573Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-04-19T17:54:08.082719Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-04-19T17:54:08.089383Z  INFO vector::topology::running: Running healthchecks.
2023-04-19T17:54:08.089413Z DEBUG vector::topology::running: Connecting changed/added component(s).
2023-04-19T17:54:08.089431Z DEBUG vector::topology::running: Configuring outputs for source. component=parse_json
2023-04-19T17:54:08.089447Z DEBUG vector::topology::running: Configuring output for component. component=parse_json output_id=None
2023-04-19T17:54:08.089504Z DEBUG vector::topology::running: Connecting inputs for sink. component=opensearch
2023-04-19T17:54:08.089526Z DEBUG vector::topology::running: Adding component input to fanout. component=opensearch fanout_id=parse_json
2023-04-19T17:54:08.089556Z DEBUG vector::topology::running: Spawning new source. key=parse_json
2023-04-19T17:54:08.089597Z DEBUG vector::topology::running: Registered new allocation group. component_kind="source" component_type="stdin" component_id="parse_json" group_id="2"
2023-04-19T17:54:08.089643Z DEBUG vector::topology::running: Registered new allocation group. component_kind="sink" component_type="elasticsearch" component_id="opensearch" group_id="3"
2023-04-19T17:54:08.089728Z  INFO vector: Vector has started. debug="false" version="0.29.0" arch="x86_64" revision="33b3868 2023-04-11 22:19:54.512366263"
2023-04-19T17:54:08.089748Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-04-19T17:54:08.089850Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=https://redacted:443/_cluster/health method=GET version=HTTP/1.1 headers={"x-amz-date": "20230419T175408Z", "authorization": Sensitive, "x-amz-security-token": "redacted2", "user-agent": "Vector/0.29.0 (x86_64-unknown-linux-gnu 33b3868 2023-04-11 22:19:54.512366263)", "accept-encoding": "identity"} body=[empty]
2023-04-19T17:54:08.089936Z DEBUG source{component_kind="source" component_id=parse_json component_type=stdin component_name=parse_json}: vector::topology::builder: Source starting.
2023-04-19T17:54:08.089997Z DEBUG sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}: vector::topology::builder: Sink starting.
2023-04-19T17:54:08.089927Z DEBUG source{component_kind="source" component_id=parse_json component_type=stdin component_name=parse_json}: vector::topology::builder: Source pump supervisor starting.
2023-04-19T17:54:08.090081Z DEBUG source{component_kind="source" component_id=parse_json component_type=stdin component_name=parse_json}: vector::topology::builder: Source pump starting.
2023-04-19T17:54:08.090506Z DEBUG sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}: vector::utilization: utilization=0.04690143198966257
2023-04-19T17:54:08.106926Z DEBUG http: vector::internal_events::http_client: HTTP response. status=403 Forbidden version=HTTP/1.1 headers={"date": "Wed, 19 Apr 2023 17:54:08 GMT", "content-type": "application/json", "content-length": "1964", "connection": "keep-alive", "x-amzn-requestid": "32d11167-3559-42c2-acb3-4614a3d6b58c", "access-control-allow-origin": "*"} body=[1964 bytes]
2023-04-19T17:54:08.107001Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Unexpected status: 403 Forbidden component_kind="sink" component_type="elasticsearch" component_id=opensearch component_name=opensearch
2023-04-19T17:54:13.091129Z DEBUG sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}: vector::utilization: utilization=0.0046901633564615285

Im pretty confident the IAM/opensearch part is setup correctly because if i run from the same ec2 instance a logstash process, it can authenticate and send stuff to my opensearch, for example

docker run -it --rm \
	--name logstash \
	public.ecr.aws/opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2 \
	-e 'input { stdin { } } output { stdout {}
   opensearch {
     hosts => ["https://redacted:443"]
     index => "logstash-%{+YYYY.MM.dd}"
     auth_type => {
     	type => "aws_iam"
     	region => "eu-west-2"
     }
     ssl => true
     ecs_compatibility => disabled
     ssl_certificate_verification => true
   }
 }'

Am i missing something obvious in my config ?

[bruceg]

If I'm reading the logs right, it appears that the API version check and health check failed, but no actual traffic was sent to the ES/OS server. If that is right, could you try sending some traffic and see if that also fails? We have seen situations where the access permissions can be set to allow writing new data but the access checks are forbidden.

[mhristof]

If i send an event, im getting the same http: vector::internal_events::http_client: HTTP response. status=403 Forbidden version=HTTP/1.1 error.

The role attached to the ec2 instance im trying to write data from has all_access which is * in the opensearch world.

[jszwedko]

You could try setting the environment variable: export VECTOR_LOG=aws_config=debug. That will output detailed logs about the loading of AWS credentials. Maybe something will jump out there.

[mhristof]

nothing interesting from that extra debug

2023-04-27T12:38:53.799795Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-04-27T12:38:53.799800Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2023-04-27T12:38:53.799816Z DEBUG provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profile context=No profiles were defined
2023-04-27T12:38:53.801018Z DEBUG provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=WebIdentityToken context=$AWS_WEB_IDENTITY_TOKEN_FILE was not set
2023-04-27T12:38:53.801422Z DEBUG provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=EcsContainer context=ECS provider not configured
2023-04-27T12:38:53.801443Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loading credentials from IMDS
2023-04-27T12:38:53.803087Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loaded profile profile=profile
2023-04-27T12:38:53.803743Z DEBUG provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=Ec2InstanceMetadata
test-message
2023-04-27T12:38:59.546697Z DEBUG provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache

[jszwedko]

It looks like it is successfully loading credentials from the IMDSv2 metadata service. My next guess would be that the credentials don't have the right permissions. You could try checking CloudTrail if those requests are recorded there.