Getting zero-shot inference working

[AlexanderZeilmann]

I am currently trying to get a zero-shot segmentation (= not using the classes in label_dict.json) working, i.e. something similar like the following segment anything approach but for 3D

from segment_anything import SamPredictor, sam_model_registry
sam = sam_model_registry["<model_type>"](checkpoint="<path/to/checkpoint>")
predictor = SamPredictor(sam)
predictor.set_image(<your_image>)
masks, _, _ = predictor.predict(<input_prompts>)

Is there a tutorial/documentation how to do this?
Here is what I tried so far, and the various points I'm stuck:

from scripts.infer import InferClass
# Default config file.
# 1. Question: Is there documentation for the config file?
ic = InferClass()

# Create a test image
image = torch.rand(1, 3, 100,100,100) * 0.5
image[:, : , 25:75, 25:75, 25:75] = image[:, :, 25:75, 25:75, 25:75] * 2

res = ic.infer(image, [[50, 50, 50], [90, 90, 90]] , [1, 0])

Which tells me that my image is not working. 2. Question: how can I use a tensor as image?
3. Questions: Am I using the arguments point and point_label correctly? (I want to mark class 1 at (50,50,50) and class 0 at (90,90,90))

Okay, let's try an nii.gz example image:

from scripts.infer import InferClass
# Default config file.
ic = InferClass()
res = ic.infer('Example.nii.gz' , [[100, 100, 100], [100, 160, 160]], [1, 0])

This seems to work, but the segmentation looks rather bad. 4. Question: Is this expected or am I missing something?