Write_Dataframe when I know some elements (rows) will NOT exist in target cube?

[FHoile]

I receive a medium-sized transactional dataset from a source system outside TM1 and bring it into a dataframe. Sometimes there are records (elements) in that dataset that have NOT been created in TM1 yet and that's fine. The code should just ignore those ones if they aren't in TM1 yet.

The dimension in TM1 with the sometimes-missing elements is MASSIVE and VERY slow to query!

Hence I am looking for a way to send a tm1.cubes.cells.write_dataframe command to TM1 when I know I will have rows in the dataframe that won't have matching elements in the cube. I'm hoping TM1Py can just send the rows that it does have an ignore the others without throwing a critical error. (Similar to TM1Ignore_BadTuples but in reverse).

I know I could query TM1 first into a dataframe and then union-filter my source dataframe to the TM1 dataframe and send the filtered result. That's what I do currently to run without errors. However that TM1 dataframe query makes the script run 20 times slower than it would otherwise run, so I'm asking the community if there's any solution I can use where I don't have to check for element-existence in TM1 first.

EDIT: Confirmed that the existing records ARE being sent at least. For now I'll just nest in a try-catch-pass. As a feature, would be nice if could have a parameter to hide just this one error messages and suppress the "TM1ProcessError...tm1py....log" error file in instances where I expect each time that some rows will fail. That way I could use without the try-catch in case other errors were generated. If there's a better way than try-catch please let me know. Thanks!

[MariusWirtz-cubewise]

Not sure it is ideal to simply ignore the minor write errors.
However if it does fail on missing elements TM1py it should raise a special TM1pyWritePartialFailureException exception if remember correctly.

Perhaps you could speed up the existence check if you make use of MDX.
You could perhaps do a set MDX execution based on df['Product'].unique() set of elements to verify which ones do exist and which ones dont'.

[FHoile]

Thanks Marius.

The MDX pre-validation is what I have done historically. It works fine but it makes the script literally 20 times slower needing to process that MDX pre-check.

The list of source items I need to check is about 100,000 rows, all of them against a distinct TM1 element. Of those rows, maybe 15 won’t be in the target TM1 dimension, and those 15 are fine to get rejected. The target dimension has around 4,000,000 unique elements, hence is very slow to query for pre-checks.

So for now, for speed reasons, I’m keeping the MDX pre-check code commented out, and I’m just going to continue mashing the rows I have into the cube and ignoring the errors.