Vector reports fields in GCP service account credentials.json file are unknown

[danhul]

I configured vector to ship logs to gcp_stackdriver(Google operations)

  sinks:
    google_stackdriver_sink:
      type: "gcp_stackdriver_logs"
      credentials_path: "/path/to/credentials.json"
      project_id: "<project-id>"
      inputs:
        - logs.apache
      log_id: "<log_id>"
      resource:
        type: global
      batch:
        max_size: 1048576
        timeout_secs: 1

I downloaded the credentials.json file from Google Cloud but as shown below Vector claims the fields in the credentials.json file are unknown.

2023-05-11T11:14:29.844667Z ERROR vector::cli: Configuration error. error=unknown field type at line 12 column 3

The first standard field in credentials.json is "type": "service_account". If I put the line "project_id" before "type", vector reports "project_id" is unknown field.

[jszwedko]

Hi @danhul ! Could you provide the credentials file with the credentials themselves redacted?

[danhul]

Hi @jszwedko,

Here is the content of the credentials.json file

{
    "type": "service_account",
    "project_id": "project-id",
    "private_key_id": "private-key-id",
    "private_key": "-----BEGIN PRIVATE KEY-----\n ... \n-----END PRIVATE KEY-----\n",
    "client_email": "client@... .iam.gserviceaccount.com",
    "client_id": "client-id",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/....iam.gserviceaccount.com"
}

[danhul]

Hi @jszwedko did you get the time to look at the credentials file I sent last week?

[jszwedko]

Hi @danhul ! I was able to try this just now, but I don't get a configuration error with 0.29.1 at least. This is the config I'm using:

sources:
  in:
    type: stdin
sinks:
  google_stackdriver_sink:
    type: "gcp_stackdriver_logs"
    credentials_path: "credentials.json"
    project_id: "foo"
    inputs:
      - stdin
    log_id: "bar"
    resource:
      type: global
    batch:
      max_size: 1048576
      timeout_secs: 1

and the credentials file you provided. Vector starts up OK -- it just fails to send events -- which seems different than the deserialization issue you were seeing. What version of Vector are you trying?

[danhul]

Hi @jszwedko I was running 0.20. With 0.29.1 Vector starts running but it shows random text in its logs, I don't get it what it means.

{"appname":"ahmadajmi","facility":"auth","hostname":"we.de","message":"Great Scott! We're never gonna reach 88 mph with the flux capacitor in its current state!","msgid":"ID374","procid":9558,"severity":"debug","timestamp":"2023-05-28T05:37:56.082Z","version":1}
{"appname":"ahmadajmi","facility":"local3","hostname":"we.org","message":"You're not gonna believe what just happened","msgid":"ID709","procid":7755,"severity":"info","timestamp":"2023-05-28T05:37:57.082Z","version":1}
{"appname":"meln1ks","facility":"syslog","hostname":"up.net","message":"Great Scott! We're never gonna reach 88 mph with the flux capacitor in its current state!","msgid":"ID521","procid":1844,"severity":"warning","timestamp":"2023-05-28T05:37:58.082Z","version":1}
{"appname":"Karimmove","facility":"user","hostname":"random.us","message":"#hugops to everyone who has to deal with this","msgid":"ID723","procid":131,"severity":"debug","timestamp":"2023-05-28T05:37:59.081Z","version":1}
{"appname":"meln1ks","facility":"alert","hostname":"make.us","message":"Maybe we just shouldn't use computers","msgid":"ID17","procid":5179,"severity":"info","timestamp":"2023-05-28T05:38:00.081Z","version":1}
{"appname":"Karimmove","facility":"daemon","hostname":"for.de","message":"#hugops to everyone who has to deal with this","msgid":"ID279","procid":8241,"severity":"debug","timestamp":"2023-05-28T05:38:01.082Z","version":1}
{"appname":"devankoshal","facility":"alert","hostname":"we.net","message":"A bug was encountered but not in Vector, which doesn't have bugs","msgid":"ID871","procid":4570,"severity":"notice","timestamp":"2023-05-28T05:38:02.082Z","version":1}
{"appname":"devankoshal","facility":"cron","hostname":"up.us","message":"Maybe we just shouldn't use computers","msgid":"ID575","procid":1548,"severity":"notice","timestamp":"2023-05-28T05:38:03.081Z","version":2}
{"appname":"devankoshal","facility":"news","hostname":"random.net","message":"Pretty pretty pretty good","msgid":"ID578","procid":2160,"severity":"warning","timestamp":"2023-05-28T05:38:04.082Z","version":1}
{"appname":"Karimmove","facility":"kern","hostname":"we.net","message":"#hugops to everyone who has to deal with this","msgid":"ID595","procid":6569,"severity":"notice","timestamp":"2023-05-28T05:38:05.082Z","version":1}
{"appname":"ahmadajmi","facility":"syslog","hostname":"up.org","message":"There's a breach in the warp core, captain","msgid":"ID649","procid":6538,"severity":"warning","timestamp":"2023-05-28T05:38:06.082Z","version":2}
{"appname":"devankoshal","facility":"ftp","hostname":"we.de","message":"#hugops to everyone who has to deal with this","msgid":"ID192","procid":3582,"severity":"info","timestamp":"2023-05-28T05:38:07.082Z","version":1}
{"appname":"devankoshal","facility":"clockd","hostname":"some.de","message":"Pretty pretty pretty good","msgid":"ID261","procid":257,"severity":"notice","timestamp":"2023-05-28T05:38:08.082Z","version":2}
{"appname":"jesseddy","facility":"local6","hostname":"we.org","message":"A bug was encountered but not in Vector, which doesn't have bugs","msgid":"ID706","procid":655,"severity":"info","timestamp":"2023-05-28T05:38:09.082Z","version":2}
{"appname":"devankoshal","facility":"news","hostname":"for.us","message":"There's a breach in the warp core, captain","msgid":"ID179","procid":2683,"severity":"crit","timestamp":"2023-05-28T05:38:10.081Z","version":2}
{"appname":"ahmadajmi","facility":"local4","hostname":"up.de","message":"You're not gonna believe what just happened","msgid":"ID100","procid":9492,"severity":"info","timestamp":"2023-05-28T05:38:11.082Z","version":2}
{"appname":"jesseddy","facility":"kern","hostname":"make.org","message":"Take a breath, let it go, walk away","msgid":"ID772","procid":7930,"severity":"info","timestamp":"2023-05-28T05:38:12.082Z","version":2}
{"appname":"ahmadajmi","facility":"audit","hostname":"we.net","message":"You're not gonna believe what just happened","msgid":"ID438","procid":6649,"severity":"err","timestamp":"2023-05-28T05:38:13.082Z","version":1}
{"appname":"devankoshal","facility":"auth","hostname":"some.net","message":"Take a breath, let it go, walk away","msgid":"ID701","procid":6350,"severity":"err","timestamp":"2023-05-28T05:38:14.082Z","version":2}

[jszwedko]

Hi @danhul ! Those look like dummy logs from a demo_logs source. Do you happen to have one of those in your configuration?

[danhul]

Thanks for your answer @jszwedko
We have been running vector on a docker container that we build based on our config files and the image from timberio/vector. When I last tested version 0.29.1, I tried the image from timberio directly and that was why I got the random log messages.

Now I updated our Dockerfile to use newer versions of vector image from "timberio/vector" but vector failed to start and the error in the logs is shown below (This error starts showing on all vector version starting from 0.25.0)

For more information try '--help'
error: "/etc/vector" isn't a valid value for '--watch-config'
  [possible values: true, false]

I checked the release note for Vector ver. 0.25.0 and I didn't get any change related to the "--watch-config" option. Do you think of a change in Vector that is related to this argument?

The dockerfile that we have been using to generate custom image until now is shown below. (Note: The "config" directory that is copied to /etc/vector on line nr. 4 contains output and processing configuration files).

FROM timberio/vector:0.25.0-alpine
RUN apk add --no-cache curl
RUN rm /etc/vector/*
COPY config /etc/vector
ENV VECTOR_CONFIG_DIR /etc/vector/*
ENV VECTOR_WATCH_CONFIG /etc/vector

[spencergilbert]

I suspect a change in the underlying CLI library caused the change - VECTOR_WATCH_CONFIG should be true|false, but I suspect previous libraries probably accepted "content" as true.

[danhul]

Hi @spencergilbert I tested it by changing the VECTOR_WATCH_CONFIG to true and now it goes back to the original problem that Vector claims the fields of "credentials.json" file as unknown. This time, as shown below, it starts from the field auth_provider_x509_cert_url . But this is a valid field in the credential file of a Google service account.

2023-05-31T14:52:53.508046Z ERROR vector::cli: Configuration error. error=unknown field auth_provider_x509_cert_url

[jszwedko]

Is this using Vector 0.30.0? I think the GCP SDKs have been upgraded since v0.25.0.

[danhul]

@jszwedko I get the "Unknown field ..." error in both vector version v0.25.0 and v0.30.0.

[jszwedko]

Hmm, that's weird. I'm not able to reproduce this with the JSON you provided above. Is the JSON you are trying now maybe different?

[danhul]

Yeha that is weird. I am using the same sink configuration and credentials.json file that I sent you.

[danhul]

Hi @jszwedko, is the configuration file you used different from the one I shared previously? If so, is it possible to get your config?

[jszwedko]

Hey! It was the config I shared here: #17382 (reply in thread)