Reconcile increasing documents before annotating

[nsorros]

I am using the annotation tool to annotate some data scraped from a wiki. As the annotation progresses, I am also making changes to the scraping script which changes the documents somewhat. What is the best way to reconcile differences while keeping the annotations that have been made?

An easy scenario is that some additional documents were added in the latest change so I want to include them without adding duplicates. Do I need to deduplicate before uploading?

Also in cases where the context has changed, is there some way to update those documents in the annotation tool again without having to keep the old and new version?

[nsorros]

An easy scenario is that some additional documents were added in the latest change so I want to include them without adding duplicates. Do I need to deduplicate before uploading?

From a quick check it seems that if the txt documents are exactly the same then deduplication happens automatically.

Also in cases where the context has changed, is there some way to update those documents in the annotation tool again without having to keep the old and new version?

I would assume the best way to do that is to upload a csv with a document id which is the same as the one that exists but what seems to happen in the case is that it creates a new document.

[bogdankostic]

Hi @nsorros! Yes, duplicate documents won't be added if they already exist in the annotation tool. About updating Documents where the context has changed: As far as I know, this is not possible, and there's a good reason for this. If you already produced labels for a Document and want to change the context of that Document afterwards, the produced labels won't be valid anymore.

[nsorros]

That makes sense. There are cases where the labels may still be valid but i understand its better to assume it has been corrupted and they need to be produced again.