Commit 2df84a9
authored
Use JSON instead of Yajl to improve performance of store/load files (#4759)
**Which issue(s) this PR fixes**:
Fixes #
**What this PR does / why we need it**:
When I read 10 GB file using in_tail plugin, it calls Yajl methods many
times when it stores to buffer file.
Recently, Ruby's JSON has huge improvement and it is much faster.
Ref.
https://byroot.github.io/ruby/json/2024/12/15/optimizing-ruby-json-part-1.html
* Before
* It spent 88.533409329 sec to handle 10 GB file
* After
* It spent 72.324643557 sec to handle 10 GB file
* config
```
<source>
@type tail
path /home/watson/prj/sandbox/fluentd/log/access*.log
pos_file /home/watson/prj/sandbox/fluentd/log/access.log.pos
tag log
read_from_head true
follow_inodes true
# rotate_wait 0
refresh_interval 5
# open_on_every_update true
<parse>
@type none
</parse>
</source>
<match **>
@type file
path /home/watson/prj/sandbox/fluentd/log/log
</match>
```
**Docs Changes**:
**Release Note**:
Signed-off-by: Shizuo Fujita <[email protected]>1 parent 5080f81 commit 2df84a9
2 files changed
+10
-6
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
| 49 | + | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
| 93 | + | |
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
| |||
114 | 114 | | |
115 | 115 | | |
116 | 116 | | |
117 | | - | |
| 117 | + | |
118 | 118 | | |
119 | 119 | | |
120 | 120 | | |
| |||
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
133 | | - | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
134 | 138 | | |
135 | 139 | | |
136 | 140 | | |
| |||
0 commit comments