Skip to content

[BUG] OpenSearch source has expiring PIT #6352

@dlvenable

Description

@dlvenable

Describe the bug

I'm seeing issues in the opensearch source that indicate that the PIT is expiring while reading from an index.

2025-12-11T07:59:49.545 [pool-16-thread-1] ERROR org.opensearch.dataprepper.plugins.source.opensearch.worker.PitWorker - Unknown exception while processing index 'my_index':
org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [illegal_argument_exception] Point In Time id: [UJZuozGIhKWjjjWIHGayhg] doesn't exist
 at org.opensearch.client.transport.aws.AwsSdk2Transport.extractAndWrapCause(AwsSdk2Transport.java:720)
 at org.opensearch.client.transport.aws.AwsSdk2Transport.performRequest(AwsSdk2Transport.java:222)
 at org.opensearch.client.opensearch.OpenSearchClient.search(OpenSearchClient.java:1386)
 at org.opensearch.dataprepper.plugins.source.opensearch.worker.client.OpenSearchAccessor.searchWithSearchAfter(OpenSearchAccessor.java:270)
 at org.opensearch.dataprepper.plugins.source.opensearch.worker.client.OpenSearchAccessor.searchWithPit(OpenSearchAccessor.java:132)
 at org.opensearch.dataprepper.plugins.source.opensearch.worker.PitWorker.processIndex(PitWorker.java:189)
 at org.opensearch.dataprepper.plugins.source.opensearch.worker.PitWorker.lambda$run$0(PitWorker.java:115)
 at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
 at org.opensearch.dataprepper.plugins.source.opensearch.worker.PitWorker.run(PitWorker.java:115)
 at org.opensearch.dataprepper.plugins.source.opensearch.OpenSearchService.lambda$start$0(OpenSearchService.java:108)
 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 at java.base/java.lang.Thread.run(Thread.java:840)

To Reproduce

I'm not sure how to reproduce this.

Expected behavior

I believe there should be two things:

  1. There seems to be a bug with how we extend the PIT timeout.
  2. Pipeline authors should be able to configure the PIT timeouts to prevent this.

Screenshots

N/A

Environment (please complete the following information):

Data Prepper 2.13.0

Additional context

N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status

    Unplanned

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions