Skip to content

Commit 2e896ec

Browse files
authored
memory_chunk: clear internal memory use in buffer string (#4845)
**Which issue(s) this PR fixes**: Fixes #1657 **What this PR does / why we need it**: Seems that a full GC have to run in order to collect the wasted String object in `@chunk`. However, full GC is not performed frequently. This patch purges internal memory in wasted String object immediately. Here is memory usage retrieved by `ObjectSpace.memsize_of_all`. ![](https://github.com/user-attachments/assets/a90755ac-15c3-4240-97b3-87e4e4d88617) > [!NOTE] > Ruby has various optimization, such as reusing freed heap areas by pooling them. > For this reason, it may not be reflected in the process's memory usage immediately even if it clear the object. ## Reproduce * Configuration ``` <source> @type tcp tag testing format json port 10130 </source> <match **> @type forward require_ack_response true <server> host 127.0.0.1 port 10131 </server> ## Tune the buffer parameters <buffer tag> flush_mode interval flush_interval 1s </buffer> </match> ``` * test data ```ruby require "socket" require "json" i = 0 Thread.new do s = TCPSocket.open("localhost", 10130) data = {"message": "a" * 512 * 1024} loop do puts "[#{i}]" data["number"] = i s.puts data.to_json i += 1 sleep 0.01 end s.close end sleep 20 ``` **Docs Changes**: Not needed. **Release Note**: buf_memory: improve memory performance --------- Signed-off-by: Shizuo Fujita <[email protected]>
1 parent c182b6a commit 2e896ec

File tree

2 files changed

+4
-12
lines changed

2 files changed

+4
-12
lines changed

lib/fluent/plugin/buffer/memory_chunk.rb

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -68,13 +68,13 @@ def empty?
6868

6969
def purge
7070
super
71-
@chunk = ''.force_encoding("ASCII-8BIT")
71+
@chunk.clear
7272
@chunk_bytes = @size = @adding_bytes = @adding_size = 0
7373
true
7474
end
7575

7676
def read(**kwargs)
77-
@chunk
77+
@chunk.dup
7878
end
7979

8080
def open(**kwargs, &block)

test/plugin/test_output_as_buffered_compress.rb

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -122,11 +122,7 @@ def waiting(seconds)
122122
assert_equal :gzip, @i.buffer.compress
123123

124124
@i.register(:write) do |c|
125-
compressed_data = c.instance_variable_get(:@chunk)
126-
if compressed_data.is_a?(File)
127-
compressed_data.seek(0, IO::SEEK_SET)
128-
compressed_data = compressed_data.read
129-
end
125+
compressed_data = c.read(compressed: :gzip)
130126
c.write_to(io)
131127
end
132128

@@ -160,11 +156,7 @@ def waiting(seconds)
160156

161157
@i.register(:format) { |tag, time, record| "#{record}\n" }
162158
@i.register(:write) { |c|
163-
compressed_data = c.instance_variable_get(:@chunk)
164-
if compressed_data.is_a?(File)
165-
compressed_data.seek(0, IO::SEEK_SET)
166-
compressed_data = compressed_data.read
167-
end
159+
compressed_data = c.read(compressed: :gzip)
168160
c.write_to(io)
169161
}
170162

0 commit comments

Comments
 (0)