radar-output-restructure 1.1.6.1

Changes since version 1.1.6:

• Hotfix: solves incompatibility between Azure SDK 12.9.0 and Jackson 2.12.0 by downgrading Jackson.

radar-output-restructure 1.1.6

Changes since version 1.1.5:

• Updated dependencies
• Run in Docker as non-root user
• Remove separate HDFS build
• Fixed Dokka build
• Make S3 the default target storage

radar-output-restructure 1.1.5

Changes since version 1.1.4.1:

• Fixed deduplication when no duplicates were found (fixes #84)
• Updated dependencies
• Use Java 11 for LTS support

radar-output-restructure 1.1.4.1

Changes since release 1.1.4:

• Fixes an issue where the data checking part does not remove its temporary directories. This may cause storage inodes to fill up.

radar-output-restructure version 1.1.4

Changes since version 1.1.3:

• Reduce log size when offsets are missing.

radar-output-restructure version 1.1.3

Changes since version 1.1.2:

• Updated Minio, including updated syntax
• Read object tags from S3 and object metadata in Azure to determine end offset of file.
• Wrapped JedisPool in a RedisHolder that does converts thrown JedisException to IOException.
• Wrapped RadarKafkaRestructure and SourceDataCleaner into a Job to manage starting and catching exceptions. This fixes the occurrence that an OutOfMemoryException was not properly handled.
• Fixed off-by-one error in OffsetIntervals
• Reduced deduplication memory consumption, especially if the format: deduplication: distinctFields property is set.

radar-output-restructure version 1.1.2

Changes since version 1.1.1:

• Tested and updated Azure storage
• Added Azure configuration example
• Optimized imports

radar-output-restructure version 1.1.1

Changes since version 1.1.0:

• Added logging messages on why a file cannot be cleaned
• Only extract files if they have reached a certain age, controlled by the worker.minimumFileAge configuration property.

radar-output-restructure version 1.1.0

Changes since version 1.0.1

• Fixed S3 directory listing and file name parsing. Added by #73 and #74
• Added a cleaner process that does an extraction validation of old files and remove them if valid. Added by #71
• Increased coverage of integration tests

radar-output-restructure version 1.0.1

Changes from Release 1.0.0

• Allows endOffset of filenames to be optional
  Closes #67
• Create target storage factory only once. Fixes #66.
• Adds configurable batch size