Skip to content

Track temp dirs and remove when staled.#30

Draft
mashhurs wants to merge 2 commits intologstash-plugins:mainfrom
mashhurs:track-temp-dirs-and-remove
Draft

Track temp dirs and remove when staled.#30
mashhurs wants to merge 2 commits intologstash-plugins:mainfrom
mashhurs:track-temp-dirs-and-remove

Conversation

@mashhurs
Copy link
Contributor

@mashhurs mashhurs commented Jun 9, 2023

Description

This PR mainly focuses on two changes:

  1. Tracks every created temp files make sure they are removed.
    Introduced temp_files container (FileRepository locks for thread safety) to keep and track all rotated files. When file is removed from track list?
  • when recently created file becomes stale, means no event written to the file

When file and tempdir are physically deleted?

  • when uploaded to S3 - both temp dir and file are deleted.
  • when becomes stale: no event written to the file during 15 mins - both temp dir and file are deleted.
  • !NEW: deleting file (file.delete!) didn't succeed, means third party process (especially on Windows, security scanners) is keeping open the IO and Logstash removes the file. Physically, for this case, file will be deleted but temp dir stays. - both temp dir is deleted.

asdas

+def remove_staled_files
+  with_lock do |factory|
+    factory.temp_files = factory.temp_files.delete_if do |temp_file|
+     is_staled = temp_file.size == 0 && (Time.now - temp_file.ctime > @stale_time)
+      is_temp_dir_empty = false
+     begin
+        # checking Dir emptiness and remove file
+        # reading file and checking size doesn't make sense as it will not possible after temp_file.size == 0 filter
+        temp_file.delete! if is_staled
+        is_temp_dir_empty = Dir.empty?(::File.join(temp_file.temp_path, temp_file.prefix)) unless is_staled
+        temp_file.delete! if is_temp_dir_empty
+      rescue => e
+        @logger.error("An error occurred while sweeping temp dir.", :exception => e.class, :message => e.message, :path => temp_file.path, :backtrace => e.backtrace)
+      end
+      is_staled || is_temp_dir_empty
+    end
+    # mark as deleted once we finish tracking all temp files created
+    @is_deleted = factory.temp_files.length == 0
+  end
+end
  1. Moves the periodic stale sweeper to higher level, same level as periodic file rotator to make KISS. Also, confusing codes are cleaned such as properly namings (ex: periodic file rotator, periodic file sweeper), correcting misleading comments/log-descriptions.

@mashhurs mashhurs self-assigned this Sep 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant