feat(vrl): add functions for internal vector metrics access in VRL #23430

esensar · 2025-07-25T11:46:16Z

Summary

Adds 3 new functions: get_vector_metric, find_vector_metrics and aggregate_vector_metrics, enabling access to current Vector metrics (with some delay, to reduce locking).

Vector configuration

api:
  enabled: true
  address: "0.0.0.0:8686"

metrics_storage_refresh_period: 1

sources:
  demo_logs:
    type: demo_logs
    format: "json"

transforms:
  remap_vrl_metrics:
    type: remap
    inputs: ["demo_logs"]
    source: |
      .utilization = find_vector_metrics("utilization", tags: { "component_id": "console" })
      .utilization_sum = aggregate_vector_metrics("sum", "utilization")
      .utilization_avg = aggregate_vector_metrics("avg", "utilization")
      .utilization_min = aggregate_vector_metrics("min", "utilization")
      .utilization_max = aggregate_vector_metrics("max", "utilization")

sinks:
  console:
    inputs: ["remap_vrl_metrics"]
    type: console
    encoding:
      codec: json

How did you test this PR?

Ran Vector with the above configuration.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Closes: Make prometheus values available in VRL #23284

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - cargo fmt --all
  - cargo clippy --workspace --all-targets -- -D warnings
  - cargo nextest run --workspace (alternatively, you can run cargo test --all)
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

Sponsored by Quad9

Adds 2 new functions: `get_vector_metric` and `find_vector_metrics`, enabling access to current Vector metrics (with some delay, to reduce locking). Closes: vectordotdev#23284

src/topology/running.rs

esensar · 2025-07-25T11:52:30Z

@pront this is still not done (I need to implement filtering by tags and maybe aggregation too), but I wanted to check if the way this global metrics cache is plugged into the topology looks alright to you.

esensar · 2025-08-06T07:36:27Z

@pront
Just checking if you had time to look at this one.

pront · 2025-08-06T13:23:08Z

@pront Just checking if you had time to look at this one.

Hi @esensar, no we tend to not review draft PRs. In the future if you want us to take look please mark it as "ready for review" and ping @vectordotdev/vector

esensar · 2025-08-06T13:30:03Z

@pront Just checking if you had time to look at this one.

Hi @esensar, no we tend to not review draft PRs. In the future if you want us to take look please mark it as "ready for review" and ping @vectordotdev/vector

Oh, sorry about that. I left it as a draft because it is not really complete. It is missing the additional function arguments, for filtering by tags and similar, but I was wondering if the way I hooked up the metrics storage with the topology was alright. I can add these missing arguments and finalize the functions if you would prefer that to be done first.

pront · 2025-08-06T13:55:44Z

@pront Just checking if you had time to look at this one.

Hi @esensar, no we tend to not review draft PRs. In the future if you want us to take look please mark it as "ready for review" and ping @vectordotdev/vector

Oh, sorry about that. I left it as a draft because it is not really complete. It is missing the additional function arguments, for filtering by tags and similar, but I was wondering if the way I hooked up the metrics storage with the topology was alright. I can add these missing arguments and finalize the functions if you would prefer that to be done first.

No problem. Just to manage expectations, I don't think we have enough time to put in this release. But will prioritize it for the next one.

esensar · 2025-08-06T14:03:49Z

Oh, sorry about that. I left it as a draft because it is not really complete. It is missing the additional function arguments, for filtering by tags and similar, but I was wondering if the way I hooked up the metrics storage with the topology was alright. I can add these missing arguments and finalize the functions if you would prefer that to be done first.

No problem. Just to manage expectations, I don't think we have enough time to put in this release. But will prioritize it for the next one.

Yeah, understood, that is fine. I am pretty sure I won't have it ready for the release anyways.

thomasqueirozb

I merged master and resolved the conflicts but it seems like tests are failing (and have been even before my changes). I'll circle back to this PR once I see new commits pushed

lib/vector-vrl-metrics/src/aggregate_vector_metrics.rs

esensar · 2025-12-23T16:40:10Z

I have fixed the one about using all() from vrl stdlib. Other tests passed for me.

thomasqueirozb · 2025-12-23T17:14:13Z

Other tests passed for me.

cd lib/vector-vrl-metrics && cargo nextest run is failing for me. Letting CI run to see if it's replicable.

thomasqueirozb · 2025-12-23T17:35:41Z

Looks like tests are failing in the CI also, not sure if something is different with your local environment that is making these tests pass.

lib/vector-vrl-metrics/src/common.rs

src/topology/running.rs

thomasqueirozb

Good to go! I'll enable auto merge and you can resolve all comments if you think no changes are necessary

src/config/transform.rs

thomasqueirozb · 2025-12-23T21:49:07Z

It actually looks like playground is failing. I think that maybe we need to omit this new crate from vector_vrl_functions::all on wasm

esensar · 2025-12-24T10:45:42Z

It actually looks like playground is failing. I think that maybe we need to omit this new crate from vector_vrl_functions::all on wasm

Yeah, I did that. It should be working now (it worked locally 😄 -- wasm-pack build --target web --out-dir public/pkg in lib/vector-vrl/web-playground).

thomasqueirozb · 2026-01-05T18:24:18Z

Hey @esensar, apologies for the delay - I was off for the last two weeks. Everything looks fine. Deny is failing due to an unrelated reason which I'll fix in another PR. This is good to go and I'll enable auto merge once deny is fixed, the only thing you'll need to do is to resolve all comments

feat(vrl): add functions for internal vector metrics access in VRL

74c3064

Adds 2 new functions: `get_vector_metric` and `find_vector_metrics`, enabling access to current Vector metrics (with some delay, to reduce locking). Closes: vectordotdev#23284

esensar added 2 commits July 25, 2025 13:48

Add changelog entry

8bf2f25

Merge branch 'master' into feature/vrl-metrics-access

cd451fd

esensar commented Jul 25, 2025

View reviewed changes

src/topology/running.rs Show resolved Hide resolved

esensar added 9 commits October 27, 2025 13:58

Merge branch 'master' into feature/vrl-metrics-access

f0033e1

Remove old TODO

4814283

Add filtering by tags

f4202b5

Add basic get/find vector metrics tests

20a3743

Add tests note

949f422

Add wildcard support

dadff47

Update example for get/find vector metrics

ff99044

Add aggregate_vector_metrics function

2802ceb

Update changelog

47999e8

esensar marked this pull request as ready for review October 29, 2025 10:46

esensar requested a review from a team as a code owner October 29, 2025 10:46

esensar added 2 commits October 29, 2025 12:10

Update metrics_storage_refresh_period docs for aggregation

795f9bb

Update configuration docs

746ca8e

esensar requested a review from a team as a code owner October 29, 2025 11:29

github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Oct 29, 2025

thomasqueirozb reviewed Dec 23, 2025

View reviewed changes

lib/vector-vrl-metrics/src/aggregate_vector_metrics.rs Outdated Show resolved Hide resolved

thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Dec 23, 2025

Remove needless full qualification for Result

0fd602b

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Dec 23, 2025

Fix disallowed_methods error

d987fec

thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Dec 23, 2025

esensar added 2 commits December 23, 2025 20:05

Fix function names in examples

3a84523

Fix example tests for vector-vrl-metrics

135ad46

esensar commented Dec 23, 2025

View reviewed changes

lib/vector-vrl-metrics/src/common.rs Outdated Show resolved Hide resolved

Remove ; from metric_into_vrl macro to fix all failing tests

5aec1c6

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Dec 23, 2025

thomasqueirozb reviewed Dec 23, 2025

View reviewed changes

src/topology/running.rs Show resolved Hide resolved

thomasqueirozb approved these changes Dec 23, 2025

View reviewed changes

src/config/transform.rs Outdated Show resolved Hide resolved

esensar added 3 commits December 24, 2025 10:05

Add conversion for LogNamespace into bool

901abce

Make metrics_storage_refresh_period required to use metrics in VRL

baf77b7

Disable vector-vrl-metrics in vrl web-playground

1190997

esensar requested a review from thomasqueirozb January 5, 2026 09:40

Merge branch 'master' into feature/vrl-metrics-access

b181b49

thomasqueirozb enabled auto-merge January 7, 2026 07:43

thomasqueirozb added this pull request to the merge queue Jan 7, 2026

Merged via the queue into vectordotdev:master with commit acd4a73 Jan 7, 2026
50 checks passed

github-actions bot locked and limited conversation to collaborators Jan 7, 2026

esensar deleted the feature/vrl-metrics-access branch January 7, 2026 09:08

feat(vrl): add functions for internal vector metrics access in VRL #23430

feat(vrl): add functions for internal vector metrics access in VRL #23430

Uh oh!

Conversation

esensar commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

Uh oh!

esensar commented Jul 25, 2025

Uh oh!

esensar commented Aug 6, 2025

Uh oh!

pront commented Aug 6, 2025

Uh oh!

esensar commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pront commented Aug 6, 2025

Uh oh!

esensar commented Aug 6, 2025

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

esensar commented Dec 23, 2025

Uh oh!

thomasqueirozb commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thomasqueirozb commented Dec 23, 2025

Uh oh!

Uh oh!

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thomasqueirozb commented Dec 23, 2025

Uh oh!

esensar commented Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thomasqueirozb commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

esensar commented Jul 25, 2025 •

edited

Loading

esensar commented Aug 6, 2025 •

edited

Loading

thomasqueirozb commented Dec 23, 2025 •

edited

Loading

esensar commented Dec 24, 2025 •

edited

Loading

thomasqueirozb commented Jan 5, 2026 •

edited

Loading