Process source Opensearch index just once and stop - is there a way for such behaviour ? #5660
Unanswered
tzaleski-vertexinc
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
I've set up a pipeline to copy indices from Opensearch on prem to AWS Opensearch Serverless. The sink uses time series collection which means that one can't index by custom document id.
With just one index set to be included from the source when the pipeline is started PitWorker processes the index, the data is copied but after a while PitWorker starts processing the same index and the data is being copied again (and this happens again and again...). The outcome is that the "new" index has n times more documents than the source index. Is there a way to stop processing indices after the first time?
I've tried setting on the source
or
but the source index has been processed twice.
Thx in advance,
Beta Was this translation helpful? Give feedback.
All reactions