The avtable_import_set command can create duplicate sets

[JoshEvans33]

Hi,

It was recently brought to our attention that the avtable_import_set command will create duplicate entities within a data set in Terra if the same code is ran twice. It appears that the code doesn't check if the entities already exist and adds them again.

If I create a small data frame of "sample1", "sample2", and "sample3". Then use avtable_import_set to add those entities to a data set in my Terra workspace. I will get 6 entities if I run that command again instead of system checking if there was any change to the data or ignoring duplicates.

[mtmorgan]

Sorry to be slow @JoshEvans33 -- I'm not sure that this is a problem or unexpected behavior with the AnVIL package per se? For instance I can easily use the Terra user interface to edit the members of an entity set to contain duplicate entries. I'm also concerned that this could be quite expensive on a large set -- the entire set would have to be downloaded for verification. I can see how checking for duplicates would be a valuable feature...

Internally, the AnVIL package uses this entry point. I'll see if I can get some guidance from the Terra technical support team.

[mtmorgan]

Terra support tells me the issue comes from https://support.terra.bio/hc/en-us/community/posts/5077709219355-Bug-report-duplicate-entities-appearing-in-sets (@smgogarten); I'll investigate a bit more...

[mtmorgan]

Some follow-up. Duplicate entities within a set can be accomplished through the UI (either using the 'edit' button, or actually re-importing the same tsv file, provided the elements are of type 'list') so this is an issue with the Terra environment. I understand that there is an open ticket, and will wait to make any modifications to the AnVIL package until whatever upstream steps have been taken.

[JoshEvans33]

Hi, sorry for the delayed response. I'm actually a member of Terra Support and we were able to determine that there is indeed a bug in the Terra UI that this package interfaces with to cause this behavior. Our engineers are aware of the issue and we'll look into resolving it on our end. Thanks for taking a look at this!