Is there a way for me to preserve aggregated data via transforms between crashes?

[Kannaj]

Example config

api:
  enabled: true

sources:
  agent_logs:
    type: vector
    address: "0.0.0.0:8700"

  vector_logs:
    type: internal_logs


transforms:
  long_term_agg:
    type: aggregate
    inputs:
      - agent_logs
    interval_ms: 1800000


sinks:
  vm_short:
    type: "prometheus_remote_write"
    inputs:
      - "agent_logs"
    endpoint: ${SHORT_SINKURL}
    auth:
      strategy: basic
      user: test_user
      password: test_user_123

  vm_long:
    type: "prometheus_remote_write"
    inputs:
      - "long_term_agg"
    endpoint: ${LONG_SINKURL}
    auth:
      strategy: basic
      user: test_user
      password: test_user_123

  console:
    type: console
    inputs:
      - vector_logs
    encoding:
      codec: text

Here im aggregating the data for 30 mins before writing to victoria metrics. What if vector crashes during this period - is there a way for me to recover the state before the crash?

[jszwedko]

Unfortunately not at the moment. Persisting more state per-component is something we've considered, but haven't yet implemented. There is some precedent for state storage with the file source which writes checkpoints to disk.