Skip to content

Conversation

consulthys
Copy link

Here is my attempt at tackling issue #90

@jsvd
Copy link
Member

jsvd commented Jun 22, 2017

this has been implemented in #84

@jsvd jsvd closed this Jun 22, 2017
@jsvd jsvd reopened this Jun 22, 2017
@jsvd
Copy link
Member

jsvd commented Jun 22, 2017

I may have closed this too soon. it seems this feature is essentially a rename + delete:

mutate {
    copy => { "syslog" => "source" }
    remove_field => ["syslog"]
}

Do we still need this?

@consulthys
Copy link
Author

The original need was expressed in #90 so if you think that covers it, I'm fine with it.

@jsvd
Copy link
Member

jsvd commented Jun 22, 2017

ah I missed a few use cases:

  • moving a nested structured to the root of the event (the copy operation doesn't seem to support moving to root)
  • resetting an event's data with a portion of its nested structure (new event + moving data to root, or moving data to root + deleting everything else)

we can't close this then as these two features don't exist yet

@consulthys
Copy link
Author

Weird, I thought my PR was complete... at least it supported the use cases I needed back then

@jsvd
Copy link
Member

jsvd commented Jun 22, 2017

sorry, I didn't explain myself properly. I meant to say that currently the features requested in #90 aren't fulfilled in master because, although the plugin now supports copy and remove, it doesn't support copying to the root, nor clearing all event data minus a certain key.

These needs are covered in this PR, I'm just thinking if it should be refocused on providing these two features instead of implementing copy again PLUS the two features.

@consulthys
Copy link
Author

Indeed. You guys should have merged this earlier ;-)

@LorenKeagle
Copy link

Is there a reason this is still sitting here after more than a year? Is there another feature that performs the use case discussed? I have json data from filebeat in a "json" field, and I need to move all fields up to the root event.

The reason is that ever since filebeat 6.3+, our json logs lose fields such as "host" and "input" because those are (undocumented) reserved by filebeat for some reason (why they can't just put all their beat fields in @metadata is beyond me). So we can't use the json input codec in filebeat, but we need logstash now to translate the "json" field back to root.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants