Skip to content

GH-10372: Fix the ignoring logic for tmp files #10373

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 3, 2025
+13 −16
Merged

GH-10372: Fix the ignoring logic for tmp files #10373

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 10 additions & 13 deletions ...k/integration/file/remote/synchronizer/AbstractInboundFileSynchronizingMessageSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URI;
import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.regex.Pattern;

import org.jspecify.annotations.Nullable;
Expand All @@ -33,7 +33,7 @@
import org.springframework.integration.file.DirectoryScanner;
import org.springframework.integration.file.FileHeaders;
import org.springframework.integration.file.FileReadingMessageSource;
import org.springframework.integration.file.filters.CompositeFileListFilter;
import org.springframework.integration.file.filters.ChainFileListFilter;
import org.springframework.integration.file.filters.FileListFilter;
import org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter;
import org.springframework.integration.file.filters.RegexPatternFileListFilter;
Expand All @@ -51,9 +51,7 @@
* from an 'inbound' adapter).
* <p>
* The base class supports configuration of whether the remote file system and
* local file system's directories should be created on start (what 'creating a
* directory' means to the specific adapter is of course implementation
* specific).
* local file system's directories should be created on start.
* <p>
* This class is to be used as a pair with an implementation of
* {@link AbstractInboundFileSynchronizer}. The synchronizer must
Expand Down Expand Up @@ -94,7 +92,8 @@ public abstract class AbstractInboundFileSynchronizingMessageSource<F>
@SuppressWarnings("NullAway.Init")
private File localDirectory;

private @Nullable FileListFilter<File> localFileListFilter;
@SuppressWarnings("NullAway.Init")
private FileListFilter<File> localFileListFilter;

/**
* Whether the {@link DirectoryScanner} was explicitly set.
Expand Down Expand Up @@ -196,9 +195,7 @@ protected void onInit() {
this.fileSource.setDirectory(this.localDirectory);
initFiltersAndScanner();
BeanFactory beanFactory = getBeanFactory();
if (beanFactory != null) {
this.fileSource.setBeanFactory(beanFactory);
}
this.fileSource.setBeanFactory(beanFactory);
this.fileSource.afterPropertiesSet();
this.synchronizer.afterPropertiesSet();
}
Expand Down Expand Up @@ -263,7 +260,7 @@ public boolean isRunning() {
* @param maxFetchSize the maximum files to fetch.
*/
@Override
public @Nullable final AbstractIntegrationMessageBuilder<File> doReceive(int maxFetchSize) {
public final @Nullable AbstractIntegrationMessageBuilder<File> doReceive(int maxFetchSize) {
AbstractIntegrationMessageBuilder<File> messageBuilder = this.fileSource.doReceive();
if (messageBuilder == null) {
this.synchronizer.synchronizeToLocalDirectory(this.localDirectory, maxFetchSize);
Expand All @@ -284,9 +281,9 @@ public boolean isRunning() {
}

private FileListFilter<File> buildFilter() {
Pattern completePattern = Pattern.compile("^.*(?<!" + this.synchronizer.getTemporaryFileSuffix() + ")$");
return new CompositeFileListFilter<>(
Arrays.asList(this.localFileListFilter, new RegexPatternFileListFilter(completePattern)));
var ignoreTemporaryFilesPattern = Pattern.compile("^.*(?<!" + this.synchronizer.getTemporaryFileSuffix() + ")$");
return new ChainFileListFilter<>(
List.of(new RegexPatternFileListFilter(ignoreTemporaryFilesPattern), this.localFileListFilter));
}

/**
Expand Down
2 changes: 1 addition & 1 deletion src/reference/antora/modules/ROOT/pages/ftp/inbound.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ metadata store on every update (if the store implements `Flushable`).

IMPORTANT: Further, if you use a distributed `MetadataStore` (such as xref:redis.adoc#redis-metadata-store[Redis]), you can have multiple instances of the same adapter or application and be sure that each file is processed only once.

The actual local filter is a `CompositeFileListFilter` that contains the supplied filter and a pattern filter that prevents processing files that are in the process of being downloaded (based on the `temporary-file-suffix`).
The actual local filter is a `ChainFileListFilter` that contains a pattern filter that prevents processing files that are in the process of being downloaded (based on the `temporary-file-suffix`) and the supplied filter.
Files are downloaded with this suffix (the default is `.writing`), and the file is renamed to its final name when the transfer is complete, making it 'visible' to the filter.

The `remote-file-separator` attribute lets you configure a file separator character to use if the default '/' is not applicable for your particular environment.
Expand Down
4 changes: 2 additions & 2 deletions src/reference/antora/modules/ROOT/pages/sftp/inbound.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ You can handle any other use-cases by using `CompositeFileListFilter` (or `Chain

The above discussion refers to filtering the files before retrieving them.
Once the files have been retrieved, an additional filter is applied to the files on the file system.
By default, this is an`AcceptOnceFileListFilter`, which, as discussed in this section, retains state in memory and does not consider the file's modified time.
By default, this is an `AcceptOnceFileListFilter`, which, as discussed in this section, retains state in memory and does not consider the file's modified time.
Unless your application removes files after processing, the adapter re-processes the files on disk by default after an application restarts.

Also, if you configure the `filter` to use a `SftpPersistentAcceptOnceFileListFilter` and the remote file timestamp changes (causing it to be re-fetched), the default local filter does not allow this new file to be processed.
Expand All @@ -86,7 +86,7 @@ metadata store on every update (if the store implements `Flushable`).

IMPORTANT: Further, if you use a distributed `MetadataStore` (such as xref:redis.adoc#redis-metadata-store[Redis Metadata Store]), you can have multiple instances of the same adapter or application and be sure that one and only one instance processes a file.

The actual local filter is a `CompositeFileListFilter` that contains the supplied filter and a pattern filter that prevents processing files that are in the process of being downloaded (based on the `temporary-file-suffix`).
The actual local filter is a `ChainFileListFilter` that contains a pattern filter that prevents processing files that are in the process of being downloaded (based on the `temporary-file-suffix`) and the supplied filter.
Files are downloaded with this suffix (the default is `.writing`), and the files are renamed to their final names when the transfer is complete, making them 'visible' to the filter.

See the https://github.com/spring-projects/spring-integration/tree/main/spring-integration-core/src/main/resources/org/springframework/integration/config[schema] for more detail on these attributes.
Expand Down