Parsing Best Practices

[nicholaswilde]

Firstly, thanks 🙏🏽 for this project!

I'm not that knowledgeable about logs, but this project is helping me understand it a bit better.

I have rsyslog installed on my Proxmox node and I'm successfully capturing the logs with Fluent Bit. Currently, my LogWard (LW) logs show the Service column as unknown and I'm wondering what's the best practice to categorize these logs.

My high level goal 🥅 is to capture logs for each of my LXC containers so that I can at least parse them by each container. A secondary goal is to parse them by service.

This is my first introduction to rsyslog, Vector, and Fluent Bit and so I'm not sure the best route is to show at least the hostnames in LW from where the logs are coming from. I haven't done a deep dive into those services, but I am wondering where to start.

The types of integrations/services that can send data is quite extensive and thanks for documenting it. I like the rsyslog approach from a high level. I just need to figure out how to filter it. I also like the ability to use logger for bash scripts, or one-off logs using curl. There's a lot of flexibility 🦾 here.

Anyway, thanks 🙏🏽 again for the project!

[Polliog]

Hey! Thanks so much for the kind words and for sharing your setup, this is exactly the kind of feedback that helps shape LogWard's direction!

Great to hear you're successfully getting logs from Proxmox via rsyslog → Fluent Bit → LogWard. Let me address your questions:

Service Categorization

The Service field in LogWard is populated from the service field in your log entries. Since you're seeing "unknown", it means this field isn't being set by Fluent Bit when forwarding the logs.

Quick fix for Fluent Bit:
You can add the service name in your Fluent Bit configuration using the record_modifier filter:

[FILTER]
    Name record_modifier
    Match *
    Record service rsyslog

[FILTER]
    Name record_modifier
    Match syslog.*
    Record service ${HOSTNAME}

This will at least categorize them. For more granular control per LXC container, you could:

1. Tag-based approach: Use Fluent Bit's routing to tag logs by container, then set the service field accordingly
2. Parse hostname: Extract the container name from the syslog hostname field and use it as the service name

LXC Container Filtering

For your primary goal (parsing by container), I'd recommend:

Option 1 - Via rsyslog templates:
Configure rsyslog to include the hostname/container name in a structured way:

# In your rsyslog config
template(name="LogWardJSON" type="list") {
    constant(value="{")
    constant(value="\"message\":\"")      property(name="msg" format="json")
    constant(value="\",\"hostname\":\"")  property(name="hostname")
    constant(value="\",\"container\":\"") property(name="hostname") 
    constant(value="\"}")
}

Option 2 - Via Fluent Bit parsing:
Use Fluent Bit's parser to extract container info from the syslog message and add it as metadata.

Where to Start

Given you're new to this stack, I'd suggest:

1. Start with simple hostname tagging in Fluent Bit (the record_modifier example above)
2. Use LogWard's search with hostname:your-container-name to filter
3. Once comfortable, create dedicated services per container using more advanced Fluent Bit routing

Would you be open to sharing a sample of your current Fluent Bit config? I can give you more specific suggestions. Also, this discussion is making me realize we should add a dedicated guide for Proxmox/LXC setups, it's a common use case!

Quick Reference

For your secondary goal (service-level parsing), you could:

• Use different rsyslog facilities per service type
• Configure each LXC container's applications to log with distinct syslog tags
• Use Fluent Bit's lua scripting for advanced log enrichment

Let me know if you want to dive deeper into any of these approaches

[nicholaswilde]

I appreciate your lengthy responses and I think it gives me a good place to start. What you're suggestion is kind of what I was thinking and so it's nice to get confirmation.

Would you be open to sharing a sample of your current Fluent Bit config? I can give you more specific suggestions.

It's the stock FB config from your docs, but here's a link. I ended up having to compile my own FB Docker image because I'm running it on a RPi5.

Let me know if you want to dive deeper into any of these approaches

I don't think I need any more at the moment. I'll start with hostnames and then see if I want to go any further.

❓ One question that's not related, can I delete logs from LW? My logs are quite lengthy at the moment and I was wondering what LW's method is for dealing with large logs (time base deletion, db size, log entry qty, etc.).

[Polliog]

Cool, thank tou for the link. I will see if i can pre-compile the images for the next releases.
In this moment there is no UI for deteting logs.

LogWard uses TimescaleDB's retention policies for automatic log management. (logs retention and compression)
In this moment i can say that 10milion logs are around 1gb,

But I will add for sure a manual deletion in my roadmap. (I think that i will share it here in a new discussion)

[nicholaswilde]

If I were to use this for my rsyslog, what would the fluent-bit FILTER, INPUT, and OUTPUT look like to pipe to LW?

# In your rsyslog config
template(name="LogWardJSON" type="list") {
    constant(value="{")
    constant(value="\"message\":\"")      property(name="msg" format="json")
    constant(value="\",\"hostname\":\"")  property(name="hostname")
    constant(value="\",\"container\":\"") property(name="hostname") 
    constant(value="\"}")
}

[SERVICE]
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name          tcp
    Listen        0.0.0.0
    Port          514
    Format        json
    Tag           rsyslog.json

[OUTPUT]
    Name   stdout
    Match  *
    Format json_lines

[OUTPUT]
    Name                http
    Match               syslog.*
    Host                ${LOGWARD_API_HOST}
    Port                8080
    URI                 /api/v1/ingest/single
    Format              json_lines
    Header              X-API-Key ${LOGWARD_API_KEY}
    Header              Content-Type application/json
    Json_date_key       time
    Json_date_format    iso8601
    Retry_Limit         3
    tls                 Off

[Polliog]

For rsyslog → Fluent Bit → LogWard, you don't need the JSON template in rsyslog. Just forward raw syslog and let Fluent Bit handle the parsing.

Simplified rsyslog config:

# /etc/rsyslog.d/50-logward.conf
*.* @@fluent-bit-server-ip:514

That's it! Use @@ for TCP (recommended) or @ for UDP.

Fluent Bit Configuration:

[INPUT]
    Name        syslog
    Parser      syslog-rfc3164
    Listen      0.0.0.0
    Port        514
    Mode        tcp
    Tag         rsyslog

[FILTER]
    Name                modify
    Match               rsyslog
    Copy                ident service
    Add                 level info

# Map syslog severity to LogWard levels
[FILTER]
    Name                lua
    Match               rsyslog
    script              /fluent-bit/etc/map_syslog_level.lua
    call                map_syslog_level

[FILTER]
    Name                record_modifier
    Match               rsyslog
    Remove_key          pri
    Remove_key          ident
    Remove_key          pid

[OUTPUT]
    Name                http
    Match               rsyslog
    Host                ${LOGWARD_API_HOST}
    Port                8080
    URI                 /api/v1/ingest/single
    Format              json_lines
    Header              X-API-Key ${LOGWARD_API_KEY}
    Header              Content-Type application/json
    Json_date_key       time
    Json_date_format    iso8601
    Retry_Limit         3
    tls                 Off

Lua script (map_syslog_level.lua) for severity mapping:

function map_syslog_level(tag, timestamp, record)
    local pri = tonumber(record["pri"])
    if not pri then
        return 0, 0, 0
    end
    
    -- Extract severity from PRI (severity = PRI mod 8)
    local severity = pri % 8
    
    -- Map syslog severity to LogWard levels
    local level_map = {
        [0] = "critical", -- Emergency
        [1] = "critical", -- Alert
        [2] = "critical", -- Critical
        [3] = "error",    -- Error
        [4] = "warn",     -- Warning
        [5] = "info",     -- Notice
        [6] = "info",     -- Informational
        [7] = "debug"     -- Debug
    }
    
    record["level"] = level_map[severity] or "info"
    return 1, timestamp, record
end

Key differences from your config:

• No JSON template needed in rsyslog (Fluent Bit parses raw syslog)
• Use syslog input plugin instead of tcp
• The parser handles RFC 3164 format automatically
• Lua script maps syslog severity levels to LogWard's schema

This approach is cleaner and handles standard syslog formats without custom templates. Full docs at your LogWard instance: /docs/syslog