[High Data Loss] Async NATS crash leads to thread never restarting

[octave-ati]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hello,

I have run into this issue when the NATS subject I listen to (when using * or >) is not UTF-8 compatible :

thread 'vector-worker' panicked at /cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-nats-0.42.0/src/connection.rs:180:63: called Result::unwrap()on anErrvalue: Utf8Error { valid_up_to: 38, error_len: Some(1) } note: run withRUST_BACKTRACE=1 environment variable to display a backtrace 2025-12-17T13:00:23.962120Z INFO source{component_kind="source" component_id=balance component_type=nats}: vector::sources::nats::source: NATS source drained and shut down gracefully.

The problem is that the NATS Source thread never restarts after the shut down, which led to huge data loss on our side (about 1 day of data).

Steps to reproduce :

• Create a NATS source that listens to 'balance.*'
• nats pub $'balance.\xFF' '{}'

Would it be possible to fix this by handling this error more gracefully and making sure that the thread restarts?

Configuration

[sources.alva_warrants_balance]
    type = "nats"
    subject = "balances.*"
    url = "${NATS_BUSINESS_URL}"
    connection_name = "internal_data_vector"
    queue = "dataplatform_internaldata_vector_queue"
    [sources.alva_warrants_balance.auth]
        strategy = "user_password"
        [sources.alva_warrants_balance.auth.user_password]
            user = "${NATS_BUSINESS_USER}"
            password = "${NATS_BUSINESS_PASSWORD}"

Version

0.49.0

Debug Output

thread 'vector-worker' panicked at /cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-nats-0.42.0/src/connection.rs:180:63:
called `Result::unwrap()` on an `Err` value: Utf8Error { valid_up_to: 38, error_len: Some(1) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2025-12-17T13:00:23.962120Z  INFO source{component_kind="source" component_id=balance component_type=nats}: vector::sources::nats::source: NATS source drained and shut down gracefully.

Example Data

No response

Additional Context

No response

References

No response

[thomasqueirozb]

Hi @octave-ati, have you run this with RUST_BACKTRACE=1? This seems like an issue with the async-nats library and it's hard to know where in Vector code the issue lies.

Either way I'd suggest raising an issue in the async-nats repo, as the .unwrap() is still present in the latest version.

[octave-ati]

Hi @thomasqueirozb,

For me the problem isn't with async-nats, the error is to be expected if you look at the codebase, the problem is how Vector handles the error, it should either error out or retry the thread that panicks, not just do a silent failure

Here is the full stacktrace :

vector-1             | called `Result::unwrap()` on an `Err` value: Utf8Error { valid_up_to: 10, error_len: Some(1) }
vector-1             | stack backtrace:
vector-1             |    0:     0xaaaab05c6cec - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h6393d514ac6e9cd2
vector-1             |    1:     0xaaaaafa1b50c - core::fmt::write::h3d5451b97dda5b0e
vector-1             |    2:     0xaaaab05c669c - std::io::Write::write_fmt::hdd3503b3866de1d7
vector-1             |    3:     0xaaaab05c6b30 - std::sys::backtrace::BacktraceLock::print::h9d5d0fd803c544d7
vector-1             |    4:     0xaaaab05c62e0 - std::panicking::rust_panic_with_hook::h6e7e6d3e8b95d45e
vector-1             |    5:     0xaaaab0602910 - std::panicking::begin_panic_handler::{{closure}}::hac60eebb8a81f899
vector-1             |    6:     0xaaaab0602888 - std::sys::backtrace::__rust_end_short_backtrace::h62db1ed7db49b861
vector-1             |    7:     0xaaaab0606f1c - __rustc[4794b31dd7191200]::rust_begin_unwind
vector-1             |    8:     0xaaaaaf15bb38 - core::panicking::panic_fmt::h53676893f0005007
vector-1             |    9:     0xaaaaaf15bf80 - core::result::unwrap_failed::hb63fd6e35169bf07
vector-1             |   10:     0xaaaaaf38ac3c - async_nats::connection::Connection::poll_read_op::hf7595fd6c18ae667
vector-1             |   11:     0xaaaaaf39b9a8 - <async_nats::ConnectionHandler::process::{{closure}}::ProcessFut as core::future::future::Future>::poll::h45cb990608c253fc
vector-1             |   12:     0xaaaab136124c - async_nats::connect_with_options::{{closure}}::{{closure}}::hc7c33cd582deda46
vector-1             |   13:     0xaaaab13640c4 - tokio::runtime::task::raw::poll::h5472f5ffceed43d7
vector-1             |   14:     0xaaaab06675ec - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h9a8c506b3304719a
vector-1             |   15:     0xaaaab06657d0 - tokio::runtime::scheduler::multi_thread::worker::run::hd5da029507cc4087
vector-1             |   16:     0xaaaab066e32c - tokio::runtime::task::raw::poll::hfd4c03c3875ee5b0
vector-1             |   17:     0xaaaab06418ac - std::sys::backtrace::__rust_begin_short_backtrace::h41cf12444ece1658
vector-1             |   18:     0xaaaab06499ec - core::ops::function::FnOnce::call_once{{vtable.shim}}::h2c11a928f837c00d
vector-1             |   19:     0xaaaab060744c - std::sys::pal::unix::thread::Thread::new::thread_start::h9909ea7a9d24fe08
vector-1             |   20:     0xffff7f65eea0 - <unknown>
vector-1             |   21:     0xffff7f6c7b1c - <unknown>
vector-1             |   22:                0x0 - <unknown>```

[thomasqueirozb]

For me the problem isn't with async-nats, the error is to be expected if you look at the codebase

That isn't correct, the method can return an error but doesn't properly handle invalid UTF-8 data and panics at the library level. async_nats::ConnectOptions::connect calls async_nats::connect_with_options which creates a ConnectionHandler which contains a Connection. The connection's poll_read_op method is called here, which then calls try_read_op, which does not properly handle invalid UTF-8 data (source).

Given that information this is an issue with the async_nats library. Historically we don't handle panics in worker threads for sources/sinks/transforms. Handling that properly would be a big project but if that's what you think Vector should do we can discuss further in a separate issue. Again, this should be brought up and fixed on the async_nats repo