Steps to Change field types for custom apps logs integration

[samuel809]

Taking into consideration that:

1. In fleet you can add a custom integration and dissect custom applications logs using processors in the integration. or that
2. in SecurityOnion (im at 2.4.80) has its customs pipeline configurations settings in the SOC Administration Elasticsearch->Config->Pipelines->Custom001..010.

Ingesting is working fine (for option 1, option 2 not tested yet) all fiedls as keywords but if i wanted to change field types, let's say two fields one to integer and the second to float.

Where should i introduce this in the process?

should it be modifying the mappings on the integration component table?
or in option 2 above?
or by adding a new local mapping in and modifying the /opt/so/saltstack/default/salt/elasticsearch/defaults.yaml?

thanks,

[reyesj2]

Most integrations by default have a component template like

logs-1password.signin_attempts@custom

That @custom is normally empty or doesn't exist. You can create it or modify it and add your mappings. (index has to rollover for new mappings to take place)

Alternatively, you can create your own component template with mappings and then add it to any index template it should be applied to under the 'composed_of' section

[samuel809]

Thanks @reyesj2; I'm testing both ways.

in the first suggestion, I have edited the @custom component table.

what way would be the recommended to perform the index roll over?

for the Alternatively, Will this new component table be assigned to so-logs index?

[reyesj2]

It won't be @custom. In my example above I showed logs-1password.signin_attempts@custom. By default from 1password the integration would come with several packages named like logs-1password.signin_attempts@package

The <integration>@custom runs after everything else for the integration. So that is where you can make changes to overwrite the defaults from the integration.

Index rollover can be done using _rollover
https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover
something like the kibana Dev tools makes that pretty simple to do

POST logs-zeek-so/_rollover

[samuel809]

Very insightful information, thanks again. @reyesj2
Allow me to recap.

1. integration@pakage runs first. from what i am understanding - the mapping to change field types should be done here.
2. integration@custom runs after. if we need post treatment, this is the component table to edit.

will test index rollover now POST index/_rollover

regarding the creation of a custom component table, if so, should it be linked to the so-logs index? or are any other considerations | recommendations? let's say for same fields linked to different datasets.

[samuel809]

My best results whereby adding the custom component table (dyor) to the so-logs index.