RELEASE v2.9.0

Features

• feat: allow to configure value schema using Avro by @fhussonnois in #393

Improvements & Bugfixes

• fix: Add application/gzip to recognize gzipped files by @loquisgon in #398
• fix(api): fix connect schema mapping for map and array by @fhussonnois in #397

Docs

Sub-Tasks

• build(deps-dev): bump google-cloud-nio from 0.124.15 to 0.126.0 by @dependabot in #354
• build(deps): bump azure-storage-blob from 12.20.0 to 12.20.1 by @dependabot in #351
• build(deps): bump antlr4.version from 4.10.1 to 4.11.1 by @dependabot in #350
• build(deps): bump maven-compiler-plugin from 3.8.1 to 3.10.1 by @dependabot in #342
• build(deps): bump Saxon-HE from 11.3 to 11.4 by @dependabot in #359
• build(deps): bump jackson-databind from 2.13.4.2 to 2.14.1 by @dependabot in #358
• build(deps): bump httpclient from 4.5.13 to 4.5.14 by @dependabot in #360
• build(deps): bump actions/stale from 5 to 7 by @dependabot in #356
• build(deps): bump junit.version from 5.9.0 to 5.9.1 by @dependabot in #357
• build(deps): bump commons-compress from 1.21 to 1.22 by @dependabot in #364
• build(deps): bump junit.version from 5.9.1 to 5.9.2 by @dependabot in #367
• build(deps): bump json from 20220320 to 20230227 by @dependabot in #390
• build(deps): bump libraries-bom from 25.4.0 to 26.9.0 by @dependabot in #387
• build(deps): bump testcontainer.version from 1.17.5 to 1.17.6 by @dependabot in #365
• build(deps): bump jackson-core from 2.14.0 to 2.14.2 by @dependabot in #395
• build(deps-dev): bump json-path from 5.2.0 to 5.3.0 by @dependabot in #394
• build(deps): bump aws-java-sdk.version from 1.12.324 to 1.12.421 by @dependabot in #396
• build(deps-dev): bump google-cloud-nio from 0.126.7 to 0.126.8 by @dependabot in #402
• build(deps): bump Saxon-HE from 11.4 to 12.0 by @dependabot in #399
• build(deps): bump maven.surefire.plugin.version from 3.0.0-M7 to 3.0.0-M9 by @dependabot in #401
• build(deps): bump maven-compiler-plugin from 3.10.1 to 3.11.0 by @dependabot in #400

New Contributors

• @loquisgon made their first contribution in #398

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.9.0

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.9.0.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.9.0/streamthoughts-kafka-connect-file-pulse-2.9.0.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.9.0.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.9.0/streamthoughts-kafka-connect-file-pulse-2.9.0.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.9.0.zip.asc streamthoughts-kafka-connect-file-pulse-2.9.0.zip

Full Changelog: v2.8.0...v2.9.0

Release v2.8.0

Features

Improvements & Bugfixes

• 8c5fe6c fix(plugins): fix FilePulseSourceTask can fail to stop gracefully
• fix(api): fix filter-chain behavior to handle errors (#325) by @fhussonnois in #326

Docs

Sub-Tasks

• b5c7866 build(deps): bump confluentinc/cp-kafka-connect-base to 7.3.0
• 30a652a build(deps): bump kafka-connect-avro-converter to 7.3.0
• aef0e28 build(deps): bump kafka version to 3.3.1
• build(deps): bump avro from 1.11.0 to 1.11.1 by @dependabot in #312
• build(deps): bump azure-storage-blob from 12.14.2 to 12.18.0 by @dependabot in #307
• build(deps): bump junit.version from 5.8.2 to 5.9.0 by @dependabot in #315
• build(deps): bump azure-identity from 1.2.5 to 1.6.0 by @dependabot in #330
• build(deps-dev): bump google-cloud-nio from 0.122.11 to 0.124.15 by @dependabot in #329
• build(deps): bump aws-java-sdk.version from 1.12.210 to 1.12.275 by @dependabot in #314
• build(deps): bump jackson-databind from 2.13.3 to 2.13.4.2 by @dependabot in #338
• build(deps): bump azure-storage-blob from 12.18.0 to 12.20.0 by @dependabot in #336
• build(deps): bump maven-failsafe-plugin from 3.0.0-M3 to 3.0.0-M7 by @dependabot in #335
• build(deps): bump jackson-core from 2.13.3 to 2.14.0 by @dependabot in #343
• build(deps-dev): bump json-path from 5.1.0 to 5.2.0 by @dependabot in #340
• build(deps): bump testcontainer.version from 1.17.2 to 1.17.5 by @dependabot in #347
• build(deps): bump aws-java-sdk.version from 1.12.275 to 1.12.324 by @dependabot in #339

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.8.0

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.8.0.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.8.0/streamthoughts-kafka-connect-file-pulse-2.8.0.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.8.0.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.8.0/streamthoughts-kafka-connect-file-pulse-2.8.0.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.8.0.zip.asc streamthoughts-kafka-connect-file-pulse-2.8.0.zip

Full Changelog: v2.7.0...v2.8.0

Release v2.7.0

Features

042cca4 feat(expression): add new built-in function from_bytes
0623e61 feat(expression): add new built-in function extract_struct_field
4bb15d3 feat(expression): add new built-in function parse_url
8484877 feat(filters): add new built-in CSVFilter (#249)
1bb3c9b feat(filesystem): add support for AmazonS3MoveCleanupPolicy (#238)
a6abb9d feat(filesystem): allow to configure S3 client RetryPolicy with max.retries and back.off delay (#247)

Improvements & Bugfixes

b37628f fix(filters): fix wrong arg type for DelimitedRowFilterConfig
5b5b501 fix(filters): enhance DateFilter to support parsing simple date (#277)
c4998d5 fix(api): fix wrong argurments passed to TypedStruct
e62dc04 fix(api): fix TypedValue should default to string on non standard type-value (#wq108)
689c903 fix(filters): fix DelimitedRowFilter to not discarded trailing empty strings (#257)
2d37f89 refactor(filters): refactor and normalize config names for DelimitedRowFilterConfig
82c3597 fix(api): fix Arrayschema to support merging given empty array
3bef574 Fixed Add Support for AWS AssumeRole #251
ab74f57 fix(api): remove dot from connector-name when storing file states (#231)
7a9a333 last maven version ( > 3.8.1 ) require https repository endpoint

Docs

06a1832 docs: update installation guide
d8b176b docs: update AWS config properties
11450ae docs: fix FileSystem Listing section (#246)
934056c docs: fix getting-started
919b430 Fix documentation for AppendError

Sub-Tasks

6786156 build(deps): bump maven-surefire-plugin from 3.0.0-M3 to 3.0.0-M7
ba06a87 build(deps): bump maven-source-plugin from 3.2.0 to 3.2.1
ec2e1e8 build(deps): bump libraries-bom from 19.2.1 to 25.4.0
fd3d2ce build(deps): bump maven-gpg-plugin from 1.5 to 3.0.1
c416183 build(deps): bump findsecbugs-plugin from 1.10.1 to 1.12.0
4b65855 build(deps-dev): bump json-path from 3.3.0 to 5.1.0
5908d62 build(deps): bump spotbugs from 4.5.3 to 4.7.0
ad72fd0 build(deps): bump jackson-core from 2.13.2 to 2.13.
d3a084a build(deps): bump antlr4.version from 4.7.1 to 4.10.1
d85ce86 test: add junit5 and fix integration-test
b50e247 build: add spotbugs maven plugin
529a44f build(deps-dev): bump testcontainers from 1.11.1 to 1.17.2
41fca73 build(deps): bump json from 20210307 to 20220320
276e0ba build(deps): bump github/codeql-action from 1 to 2
38855e8 build(deps): bump actions/setup-node from 1 to 3
3dc560e build(deps): bump jackson-annotations from 2.13.2 to 2.13.3
4069ffd build(deps): bump maven-jar-plugin from 3.0.2 to 3.2.2
fd07ee3 build(deps): bump avro from 1.10.1 to 1.11.0
90613b5 build(deps): bump actions/checkout from 2 to 3
4a8da8d build(deps): bump org.apache.kafka.version from 2.8.0 to 3.2.0
1e5d48f build(deps): bump kafka-connect-maven-plugin from 0.11.3 to 0.12.0
9453b5d build(deps): bump docker/login-action from 1 to 2
350f49e build(deps): bump actions/setup-java from 2 to 3
04155ac build(deps): bump docker/build-push-action from 2 to 3
265e03e build(deps): bump jackson-databind from 2.12.6.1 to 2.13.3
27f0bfd build(deps): bump Saxon-HE from 9.9.1-5 to 11.3
45952f9 ci: update git actions workflows
364d6b7 build: bump version for next iteration
c836ded build: fix jackson version
adfec26 gh: add issue template
329536d build(deps): bump jackson-databind from 2.12.2 to 2.12.6.1

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.7.0

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.7.0.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.7.0/streamthoughts-kafka-connect-file-pulse-2.7.0.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.7.0.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.7.0/streamthoughts-kafka-connect-file-pulse-2.7.0.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.7.0.zip.asc streamthoughts-kafka-connect-file-pulse-2.7.0.zip

New Contributors

• @ogawa-takeshi made their first contribution in #252
• @bertrandcedric made their first contribution in #256

Full Changelog: v2.6.0...v2.7.0

Release v2.6.0

Features

10259c5 feat(plugin): add built-in SizeFileListFilter

Improvements & Bugfixes

32d55e4 fix(api): fix typed value parsing may result in a loss of precision
d17298f fix(plugin): add generation-id prop to task-config to force reconfiguraton

Docs

Sub-Tasks

a8ddbf1 script: update debug.sh

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.6.0

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.6.0.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.6.0/streamthoughts-kafka-connect-file-pulse-2.6.0.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.6.0.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.6.0/streamthoughts-kafka-connect-file-pulse-2.6.0.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.6.0.zip.asc streamthoughts-kafka-connect-file-pulse-2.6.0.zip

Release v2.5.2

Features

Improvements & Bugfixes

e8cbbf2 (origin/master, origin/HEAD) fix(api): fix filter-pipeline to always copy record context between filters (#235)
f639209 fix(filesystem): update LocalFileStorage to fallback to a simple move implements as copy+delete (#234)

Docs

Sub-Tasks

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.5.2

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.2.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.2/streamthoughts-kafka-connect-file-pulse-2.5.2.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.2.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.2/streamthoughts-kafka-connect-file-pulse-2.5.2.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.5.2.zip.asc streamthoughts-kafka-connect-file-pulse-2.5.2.zip

Release v2.5.1

Features

8b1491b feat(plugin): allow to keep leading underscores on field names

Improvements & Bugfixes

d60e574 fix(api): connector should throw an error when merging struct with string
6f6decc fix(plugin): files should be cleanup when an error occurred while converting data to connect record
b308e70 fix(api): fix connect.value.schema is not used when sink topic is computed from record
6aa8123 fix(dataformat): ignore empty XML elements that contain only whitespaces
79deb60 fix(docker): fix docker-compose version

Docs

Sub-Tasks

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.5.1

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.1.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.1/streamthoughts-kafka-connect-file-pulse-2.5.1.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.1.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.1/streamthoughts-kafka-connect-file-pulse-2.5.1.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.5.1.zip.asc streamthoughts-kafka-connect-file-pulse-2.5.1.zip

Release v2.5.0

Features

b8dd10b feat(plugin): allow to configure cache.max.size.capacity for InMemoryFileObjectStateBackingStore
4f892a8 feat: add new config props to define file ordering
f964de0 feat(dataformat): add new XML config to force content field
2a78283 feat: externalize configs for XML filter and reader
b1b8de9 feat: add new config xml.text.node.value.field.name

Improvements & Bugfixes

9163b74 fix(dataformat): fix invalig config and NPE in XMLNodeToStructConverter
a3fe274 fix(api): fix wrong schema for double in TypedStruct (#230)
2884fe0 fix(filters): ignore blank values during DelimitedRowFilter type conversion (#228)
d7a4738 fix(filesystem-s3): fix AmazonS3FileSystemListing get stuck when bucket contains more than max response items (#226)

Docs

5f517f9 docs: fix config props for XML filter and reader
37adaa9 docs: fix typo and config error on AzureBlobStorageFileSystemListing (#229)
a4e28d5 docs: fix yaml
c5cb5a3 docs: fix typos and reformat

Sub-Tasks

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.5.0

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.0.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.0/streamthoughts-kafka-connect-file-pulse-2.5.0.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.0.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.0/streamthoughts-kafka-connect-file-pulse-2.5.0.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.5.0.zip.asc streamthoughts-kafka-connect-file-pulse-2.5.0.zip

Release v2.4.3

Features

fbe8b6f feat(scel-api): add is_empty expression for string/array fields

Improvements & Bugfixes

2232579 fix(plugins): fix task resources should be closed correctly (#224)
8d266f1 fix(amazons3-fs): fix NPE hen configuring credential provider (#219)

Docs

Sub-Tasks

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.4.3

For verifying distirbution zip file

$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.4.3.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.4.3/streamthoughts-kafka-connect-file-pulse-2.4.3.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.4.3.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.4.3/streamthoughts-kafka-connect-file-pulse-2.4.3.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.4.3.zip.asc streamthoughts-kafka-connect-file-pulse-2.4.3.zip

Release v2.4.2

Features

Improvements & Bugfixes

948687d fix(api): fix stackoverflow while merging incompatible schemas (#208)
d7106c5 build(deps): include slf4j-log4j12 only for test (#213)
b89a913 fix(dataformat): ignore XML document node type for converting data (#207)

Docs

0fd88e8 fix filename typo

Sub-Tasks

01e4b4c Bump to actions/setup-java@v2, use Zulu Java 11
4d36de6 test(docker): add akhq to docker-compose-debug.yml
302889e fix(build): fix makefile

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.4.2

For verifying distirbution zip file

$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.4.2.zip.asc streamthoughts-kafka-connect-file-pulse-2.4.2.zip

Release v2.4.1

Features

Improvements & Bugfixes

c6c9d3b fix(api): schema value for collection should be inferred from all items
680685e bump azure blob version to 12.14.2
3883456 refactor(filesystems): improve logs and add check for file is readable
917ef7b fix(plugin): ensure that completed files are cleaned before starting task(#194)
05a0ba8 fix(plugins): fix invalid object-file should be reported as invalid (#193)
85368d8 builder(deps): bump kafka-connect base-image version (#192)
333e1ee fix(filesystems): fix GcsStorage should support blob URI with underscore (#190)
1818f00 fix(expression): fix MapAdaptable/Reflective accessors should support dotted-notation (#189)

Docs

9c6ea7e docs(site): add details about configuring connector with SSL/SASL
ae05829 docs(site): add release 2.4 blog post

Sub-Tasks

0967479 build(docker): refactor and cleanup docker build
a9fd0b4 build(scripts): remove build-deploy-site.sh
ff5f8c6 build(github-action): add workflow git docker
413f8f6 build(github): migrate github-pages build to github actions
68aadda build(deps): update kafka and confluent version
372e27e -build(maven): update java version for ossh

Docker Images

$ docker pull streamthoughts/kafka-connect-file-pulse:2.4.1

For verifying distirbution zip file

$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.4.1.zip.asc streamthoughts-kafka-connect-file-pulse-2.4.1.zip