Need to add some custom-fields to the event data returned for Metrics

[biskitboy]

Hey guys! I need the ability to add some static tags to the SNMP Metric data prior to being sent to Splunk. In our topologies, we will often have various testbeds that will share IP/hostname etc. Or, we'll have testbeds that just have IP address/DNS overlap. We use Metric Filters to help isolate these topologies in Splunk Enterprise when we use Metric data.

For event based data we can usually use Lookup Tables or prepend meta data on ingestion so we know the testbed source. Our current SNMP tool that we built adds these tags to the metric data so we can sort them in SPLUNK properly.

For example, here is what filters look like today:

Here is what the filter looks like for sc4snmp:

I was hoping I could just append data to the inventory and have it auto-append this to the metric data, but that doesn't work. Here's an example of something I hope could be implemented. Maybe even having "Groups" of hosts with their own metric data would be better than what I have shown below.

poller:
  inventory: |
     address,port,version,community,secret,securityEngine,walk_interval,profiles,SmartProfiles,delete,testbed,other
     10.10.50.42,,2c,read,,,60,generic_switch;cusp_stats,,my_testbed,my_other_data

Maybe there is a way for me to do this in this tool already, if so, I'd love to know how.

Thanks!

[omrozowicz-splunk]

Hi @biskitboy,
Sorry for a late response, but for some reason I don't get notification when a new conversation starts in Discussions panel - what means I've seen this post today for the first time 🙈
Currently there's no way to append additional field to the metrics, but the idea you have makes a lot of sense, that must be a common usecase to have network devices working in sites/groups. We'll discuss internally if we can do it and when, and let you know.
Thanks!

[biskitboy]

Awesome! Thanks for checking this out!

[omrozowicz-splunk]

Hi, I need to gather some more info in order to plan and estimate the task.
I'm not a fan of a solution of appending new fields to the inventory, I'd prefer to configure it from different place. I was thinking about two solutions:

1. You mentioned you have overlapped IP addresses - that means you use many instances of SC4SNMP in different networks (every instance in different testbed? Because if so, we could just configure global additional_metrics that would be added to every metric sent from the SC4SNMP instance, like:

additional_metrics:
     testbed: my_testbed

2. Or we can append metrics per subnet/ip address. From my basic networking knowledge I know that usually we manage whole sites more than a single device. Maybe let's have a config that defines metrics per subnet rather just per host. That would look like:

additional_metrics:
     192.22.1.0/24:
          - testbed: my_testbed

So that would add my_testbed testbed to every metric coming from host of 192.22.1.0/24 subnet.

Does it make sense and fit your needs? Or maybe do you have some other idea? Please let me know 😃

[biskitboy]

I prefer item #2. We try to leverage shared pollers when possible so we don't have to spin up additional services. For example, one of our shared systems polls our storage/compute systems and adds a production and/or infrastructure tag. This same system also has hooks into a few test topologies that don't have overlapping addresses.

Thank you for diving into this for us!

[omrozowicz-splunk]

Hello, I'm here with the update. We're not planning exactly to do the feature we discussed here, but decided to go with something similar that may solve your problem as well. We're working on allowing to configure groups of hosts instead of hosts one by one, which means you'll be able to define the site/group, like:

sites:
    testbed_one:
        - 1.2.3.4
        - 2.2.2.2
        - 3.3.3.3

And then, refer to the whole group in the inventory as:

poller:
  inventory: |
     address,port,version,community,secret,securityEngine,walk_interval,profiles,SmartProfiles,delete
     testbed_one,,2c,read,,,60,generic_switch;cusp_stats,

In polled metrics you'd see some appended field, like group: testbed_one so you can filter.

[biskitboy]

This seems like it should work! I can still change community strings and other items on a per IP basis correct? Seems like it.

…

-Steve

On Aug 24, 2022, at 4:19 AM, Olga Malita ***@***.***> wrote: Hello, I'm here with the update. We're not planning exactly to do the feature we discussed here, but decided to go with something similar that may solve your problem as well. We're working on allowing to configure groups of hosts instead of hosts one by one, which means you'll be able to define the site/group, like: sites: testbed_one: - 1.2.3.4 - 2.2.2.2 - 3.3.3.3 And then, refer to the whole group in the inventory as: poller: inventory: | address,port,version,community,secret,securityEngine,walk_interval,profiles,SmartProfiles,delete testbed_one,,2c,read,,,60,generic_switch;cusp_stats, In polled metrics you'd see some appended field, like group: testbed_one so you can filter. — Reply to this email directly, view it on GitHub <#482 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGCVJTORT4Z66QB4HWLJI23V2XLKPANCNFSM5YVGLOTA>. You are receiving this because you were mentioned.

[omrozowicz-splunk]

Yes, there will be two options, per host and per group. But I think you raised a very important point, I thought I'll just make users be able to define a list of hosts that will always share the same community string, secret, walk interval, and so on... But now I'm thinking, that not always all of the group members will have it the same. So maybe, it would be better to allow configuration to override this values when needed, like:

groups: |
  group1:
    - address: 1.1.1.1
      port: 162
    - address: 2.2.2.2
      port: 777
      community: other_string

[biskitboy]

Looks good to me!

[omrozowicz-splunk]

Hey, version 1.8.0 with the group feature we discussed here was released 🚀
Here is the documentation of how to configure it: https://splunk.github.io/splunk-connect-for-snmp/main/configuration/configuring-groups/

I recommend you to first upgrade to 1.8.0, and then run another helm3 upgrade with the configuration change, so that you'll see appended indexes from the very beginning. When you do change and upgrade at once, the change might be visible in Splunk during the next walk process.

[biskitboy]

Thank you! I'm hopeful to try this out this week. I'll let you know how it goes.

[biskitboy]

Things are looking pretty good over here. I have minor issue that I'm hoping is just an error on my part. I was hoping to have the name "testbed" show up in the metrics. But it looks like I'm getting the term "group" instead. Is it possible to change the name group to anything else?

I tried changing the term "groups" to "testbed", but that didn't work.

For Example:

scheduler:
  logLevel: "INFO"
  groups: |
       fake_testbed:
           - address: 10.10.50.42
             port: 161
poller:
  inventory: |
    address,port,version,community,secret,securityEngine,walk_interval,profiles,SmartProfiles,delete
    fake_testbed,,2c,read,,,60,generic_switch;cusp_stats,

Screenshot Example

Thanks!

[omrozowicz-splunk]

Hey, I must admit that I didn't know that the testbed word is so crucial here 😅 I decided to name it group as it is a bit more general that the testbed word, at least from my perspective. But I believe that Splunk has the capability to rename the field so you can see whatever field name you want. I can do some investigation about that when I'll have a minute.

[biskitboy]

Thanks! We can rename it after the initial search, but I don't think I can change it at ingestion time like I can non-metric data. I know the metric indexes have more limitations than the traditional indexes. I'll try to check more on this since I've never tried to manipulate a metric field like this before.

The main reason I was hoping it was configurable was because of the user community that uses this. They already know when the see 'testbed' what it indicates. Group is something new I'd need to share.

Whatever happens, it works today! Thank you very much for getting this implemented.