Vector TLS setup

[kevinremoue]

Hi guys,

I have few questions on TLS setup for vector sink (on vector agent) to vector source (on a vector aggregator) communication.
I'm on Kubernetes. Here are my versions:

vector       	default    	4       	2022-09-15 16:46:06.698968764 +0200 CEST	deployed	vector-0.16.0      	0.24.0-distroless-libc
vector-server	default    	3       	2022-09-15 16:33:51.280147386 +0200 CEST	deployed	vector-0.16.0      	0.24.0-distroless-libc

I did the following config for testing. (generating two self signed)
On my vector agent:

    to_vector:
      type: vector
      inputs:
        - prom
      address: vector-server.default.svc.cluster.local:6000
      version: "2"
      healthcheck: false
      tls:
        verify_hostname: false
        verify_certificate: true
        crt_file: /etc/tls-vector/client.crt
        key_file: /etc/tls-vector/client.pem
        key_pass: test

On my vector aggregator:

  sources:
    vector_in:
      address: 0.0.0.0:6000
      type: vector
      version: "2"
      tls:
        enabled: true
        verify_certificate: true
        crt_file: /etc/tls-vector/server.crt
        key_file: /etc/tls-vector/server.pem
        key_pass: test
        ca_file: /etc/tls-vector/client.crt

But I got the following errors on agent:

022-09-15T14:54:46.275749Z DEBUG hyper::client::connect::dns: resolving host="vector-server.default.svc.cluster.local"
2022-09-15T14:54:46.282132Z DEBUG sink{component_kind="sink" component_id=to_vector component_type=vector component_name=to_vector}:request{request_id=1}: hyper::client::connect::http: connecting to 10.43.81.73:6000
2022-09-15T14:54:46.282848Z DEBUG sink{component_kind="sink" component_id=to_vector component_type=vector component_name=to_vector}:request{request_id=1}: hyper::client::connect::http: connected to 10.43.81.73:6000
2022-09-15T14:54:46.282966Z DEBUG sink{component_kind="sink" component_id=to_vector component_type=vector component_name=to_vector}:request{request_id=1}: h2::client: binding client connection
2022-09-15T14:54:46.283150Z DEBUG sink{component_kind="sink" component_id=to_vector component_type=vector component_name=to_vector}:request{request_id=1}: h2::client: client connection bound
2022-09-15T14:54:46.283228Z DEBUG sink{component_kind="sink" component_id=to_vector component_type=vector component_name=to_vector}:request{request_id=1}: h2::codec::framed_write: send frame=Settings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }
2022-09-15T14:54:46.283366Z DEBUG sink{component_kind="sink" component_id=to_vector component_type=vector component_name=to_vector}:request{request_id=1}:Connection{peer=Client}: h2::codec::framed_write: send frame=WindowUpdate { stream_id: StreamId(0), size_increment: 5177345 }
2022-09-15T14:54:46.283583Z DEBUG sink{component_kind="sink" component_id=to_vector component_type=vector component_name=to_vector}:request{request_id=1}: hyper::client::pool: pooling idle connection for ("http", vector-server.default.svc.cluster.local:6000)
2022-09-15T14:54:46.284773Z DEBUG sink{component_kind="sink" component_id=to_vector component_type=vector component_name=to_vector}:request{request_id=1}:Connection{peer=Client}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
2022-09-15T14:54:46.284878Z DEBUG sink{component_kind="sink" component_id=to_vector component_type=vector component_name=to_vector}:request{request_id=1}:Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(1) }
2022-09-15T14:54:46.285013Z DEBUG hyper::proto::h2::client: connection error: connection reset
2022-09-15T14:54:46.285056Z DEBUG hyper::proto::h2::client: client response error: broken pipe
2022-09-15T14:54:46.285125Z DEBUG hyper::client::client: client connection error: connection error: broken pipe
2022-09-15T14:54:46.285273Z  WARN sink{component_kind="sink" component_id=to_vector component_type=vector component_name=to_vector}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=Request failed: status: Unknown, message: "connection error: broken pipe", details: [], metadata: MetadataMap { headers: {} }

On aggregator:

2022-09-15T14:53:54.233395Z DEBUG hyper::server::server::new_svc: connection error: connection error: TLS handshake failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:332:
2022-09-15T14:53:56.239503Z DEBUG h2::codec::framed_write: send frame=Settings { flags: (0x0), initial_window_size: 1048576, max_frame_size: 16384, max_header_list_size: 16777216 }

Another issue, even if I configure verify_certificate: false on the agent, it isn´t working. I don´t understand how it is supposed to be setup.
Thanks!

[kevinremoue]

I fixed my issue.
tls.enabled: true must be set for the sink configuration too. This is missing in the documentation.
https://vector.dev/docs/reference/configuration/sinks/vector/#tls
https://vector.dev/docs/reference/configuration/sources/vector/#tls.enabled

[jszwedko]

Ah, good catch @kevinremoue . Indeed either tls.enabled must be set to true or https:// is used in the sink config like:

address: https://vector-server.default.svc.cluster.local:6000

I'll see about re-adding that option to the docs.