pack: add 'epoch_ms' format

[jeongki-kim]

This PR adds epoch_ms format which represents UNIX epoch in milliseconds.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

• [N/A] Documentation required for this feature

Example configuration snippet

[OUTPUT]
    Name  stdout
    Match *
    Format json
    json_date_key timestamp
    json_date_format epoch_millis
    # ...

Debug log output

Fluent Bit v1.9.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/09/28 11:54:06] [ info] Configuration:
[2021/09/28 11:54:06] [ info]  flush time     | 0.100000 seconds
[2021/09/28 11:54:06] [ info]  grace          | 5 seconds
[2021/09/28 11:54:06] [ info]  daemon         | 0
[2021/09/28 11:54:06] [ info] ___________
[2021/09/28 11:54:06] [ info]  inputs:
[2021/09/28 11:54:06] [ info]      tail
[2021/09/28 11:54:06] [ info] ___________
[2021/09/28 11:54:06] [ info]  filters:
[2021/09/28 11:54:06] [ info] ___________
[2021/09/28 11:54:06] [ info]  outputs:
[2021/09/28 11:54:06] [ info]      stdout.0
[2021/09/28 11:54:06] [ info] ___________
[2021/09/28 11:54:06] [ info]  collectors:
[2021/09/28 11:54:06] [ info] [engine] started (pid=30920)
[2021/09/28 11:54:06] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2021/09/28 11:54:06] [debug] [storage] [cio stream] new stream registered: tail.0
[2021/09/28 11:54:06] [ info] [storage] version=1.1.1, initializing...
[2021/09/28 11:54:06] [ info] [storage] in-memory
[2021/09/28 11:54:06] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/09/28 11:54:06] [ info] [cmetrics] version=0.2.1
[2021/09/28 11:54:06] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2021/09/28 11:54:06] [debug] [input:tail:tail.0] inotify watch fd=25
[2021/09/28 11:54:06] [debug] [input:tail:tail.0] scanning path test.log
[2021/09/28 11:54:06] [debug] [input:tail:tail.0] inode=11797143 with offset=1665152 appended as test.log
[2021/09/28 11:54:06] [debug] [input:tail:tail.0] scan_glob add(): test.log, inode 11797143
[2021/09/28 11:54:06] [debug] [input:tail:tail.0] 1 new files found on path 'test.log'
[2021/09/28 11:54:06] [debug] [stdout:stdout.0] created event channels: read=27 write=28
[2021/09/28 11:54:06] [debug] [router] match rule tail.0:stdout.0
[2021/09/28 11:54:06] [ info] [sp] stream processor started
[2021/09/28 11:54:06] [debug] [input:tail:tail.0] inode=11797143 file=test.log promote to TAIL_EVENT
[2021/09/28 11:54:06] [ info] [input:tail:tail.0] inotify_fs_add(): inode=11797143 watch_fd=1 name=test.log
[2021/09/28 11:54:09] [debug] [input:tail:tail.0] inode=11797143 events: IN_MODIFY 
[2021/09/28 11:54:09] [debug] [task] created task=0x7f70d8013410 id=0 OK
[{"logTime":1632797649178,"log":"testlog"}]
[2021/09/28 11:54:09] [debug] [out coro] cb_destroy coro_id=0
[2021/09/28 11:54:09] [debug] [task] destroy task=0x7f70d8013410 (task_id=0)
[2021/09/28 11:54:09] [debug] [input:tail:tail.0] inode=11797143 events: IN_MODIFY 
[{"logTime":1632797649778,"log":"testlog"}][2021/09/28 11:54:09] [debug] [task] created task=0x7f70d8013c10 id=0 OK

[2021/09/28 11:54:09] [debug] [out coro] cb_destroy coro_id=1
[2021/09/28 11:54:09] [debug] [task] destroy task=0x7f70d8013c10 (task_id=0)
[2021/09/28 11:54:10] [debug] [input:tail:tail.0] inode=11797143 events: IN_MODIFY 
[{"logTime":1632797650393,"log":"testlog"}]
[2021/09/28 11:54:10] [debug] [task] created task=0x7f70d8013e40 id=0 OK
[2021/09/28 11:54:10] [debug] [out coro] cb_destroy coro_id=2
[2021/09/28 11:54:10] [debug] [task] destroy task=0x7f70d8013e40 (task_id=0)
^C[2021/09/28 11:54:22] [engine] caught signal (SIGINT)
[2021/09/28 11:54:22] [ info] [input] pausing tail.0
[2021/09/28 11:54:22] [ warn] [engine] service will stop in 5 seconds
[2021/09/28 11:54:27] [ info] [engine] service stopped
[2021/09/28 11:54:27] [debug] [input:tail:tail.0] inode=11797143 removing file name test.log
[2021/09/28 11:54:27] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=11797143 watch_fd=1

Valgrind output

$ valgrind --leak-check=full --error-limit=no bin/fluent-bit -c test.conf
...
^C[2021/09/28 11:49:14] [engine] caught signal (SIGINT)
[2021/09/28 11:49:14] [ info] [input] pausing tail.0
[2021/09/28 11:49:14] [ warn] [engine] service will stop in 5 seconds
==28047== 
==28047== HEAP SUMMARY:
==28047==     in use at exit: 95,888 bytes in 3,216 blocks
==28047==   total heap usage: 65,968 allocs, 62,752 frees, 262,848,238 bytes allocated
==28047== 
==28047== LEAK SUMMARY:
==28047==    definitely lost: 0 bytes in 0 blocks
==28047==    indirectly lost: 0 bytes in 0 blocks
==28047==      possibly lost: 0 bytes in 0 blocks
==28047==    still reachable: 95,888 bytes in 3,216 blocks
==28047==         suppressed: 0 bytes in 0 blocks
==28047== Reachable blocks (those to which a pointer was found) are not shown.
==28047== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==28047== 
==28047== For counts of detected and suppressed errors, rerun with: -v
==28047== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[edsiper]

thanks for this contribution.

Can you make the default name epoch_ms and epoch_millis as another option for compat/understable mode ?

[jeongki-kim]

Thank you for your suggestion. I think that's a better idea.

[PettitWesley]

@jeongki-kim If rebase this with the latest (another format was added), I think we can merge it. Thanks for this contribution and sorry for the time it has taken to review.

[marcosdiez]

Hi @PettitWesley and @edsiper

Since @jeongki-kim already fixed his PR to work with master, I just wrote two small documentation PRs: fluent/fluent-bit-docs#715 and jeongki-kim#1

The latter is trivial and can be easily cherry picked in case you guys want to go ahead and merge this one directly.

[PettitWesley]

@jeongki-kim Thank You 😃

[PettitWesley]

@edsiper This is ready. Also, Did the requirement on the number of maintainer approvals change? I thought after approving this I could merge it myself... anyway please merge it now..

[jeongki-kim]

@marcosdiez Thank you for your help!

I think this PR is ready for review now.

[patrick-stephens]

@edsiper This is ready. Also, Did the requirement on the number of maintainer approvals change? I thought after approving this I could merge it myself... anyway please merge it now..

No changes I'm aware of @PettitWesley, it looked like we had a push after this comment as well so can't check now as needed an approval to run the tests again.

[PettitWesley]

@edsiper please merge

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[marcosdiez]

this should be merged! (message sent to remove stale PR)

[patrick-stephens]

@PettitWesley the requirement is for codeowner approval, not just any maintainer (although that helps). You're not down as a codeowner for the files affected so I think this is why your approval does not allow merge: https://github.com/fluent/fluent-bit/blob/master/CODEOWNERS
@edsiper should be able to approve and merge - or any of the other codeowners auto-requested for review.

[jeongki-kim]

Will this PR eventually be merged anytime soon?

[tarokkk]

Do we waiting for something here?

[ahma]

@edsiper Could you please take a look into this?
Thanks for your support.

[leonardo-albertovich]

Could you create the function flb_time_to_millisec in flb_time.c and move that conversion there so it can be reused by others in the future? Also, please use uint64_t like flb_time_to_nanosec.

[ahma]

@jeongki-kim Could you make these changes, please?

[jeongki-kim]

Thank you for your review. Had I found this earlier, I'd have followed your suggestion.

[jeongki-kim]

Addressed by #6114