API

At this moment, all features for single image have been accessible via API. I am not planning to support batch process. However, if you really need, you can submit an issue or pull request.

GET /sam/heartbeat

You can use this API to check if this extension is working.

Example:

import requests
url = "http://localhost:7861/sam/heartbeat"
response = requests.get(url)
reply = response.json()
print(reply["msg"])

If this extension is working, you should get Success!

POST /sam/sam-predict

You can use this API to get masks from SAM.

Parameters:

• sam_model_name: str = "sam_vit_h_4b8939.pth" [Optional] SAM model name. You should manually download models before using them. Please do not change model name. See here for how to pick the model you want. Optional parameter with default assignment.
• input_image: str [Required] base64 image in string format.
• sam_positive_points: List[List[float]] = [] [Optional] Positive point prompts in N * 2 python list.
• sam_negative_points: List[List[float]] = [] [Optional] Negative point prompts in N * 2 python list.
• dino_enabled: bool = False [Optional] Whether to use GroundingDINO to generate bounding boxes from text to guide SAM to generate masks.
• dino_model_name: Optional[str] = "GroundingDINO_SwinT_OGC (694MB)" [Optional] Choose one of "GroundingDINO_SwinT_OGC (694MB)" and "GroundingDINO_SwinB (938MB)" as your desired GroundingDINO model.
• dino_text_prompt: Optional[str] = None [Optional] Text prompt for GroundingDINO to generate bounding boxes. Separate different categories with .
• dino_box_threshold: Optional[float] = 0.3 [Optional] Threshold for selecting bounding boxes. Do not use a very high value, otherwise you may get no box.
• dino_preview_checkbox: bool = False [Optional] Whether to preview checkbox. You can enable preview to select boxes you want if you have accessed API dino-predict
• dino_preview_boxes_selection: Optional[List[int]] = None [Optional] Choose the boxes you want. Index start from 0.

One of point prompts and GroundingDINO text prompts must be provided to generate masks.

Returns:

• msg Message of the execution information. May contain some common error messages.
• blended_images List of 3 base64 images with masks and bounding boxes.
• masks List of 3 base64 masks generated from SAM
• masked_images List of 3 base64 masked images. Unmasked region will become transparent.

Example:

import base64
import requests
from PIL import Image
from io import BytesIO

def filename_to_base64(filename):
    with open(filename, "rb") as fh:
        return base64.b64encode(fh.read())

img_filename = "<something>.png"
url = "http://localhost:7861/sam/sam-predict"
payload = {
    "input_image": filename_to_base64(img_filename).decode(),
    "dino_enabled": True,
    "dino_text_prompt": "the girl with blue hair",
    "dino_preview_checkbox": False,
}
response = requests.post(url, json=payload)
reply = response.json()
print(reply["msg"])

grid = Image.new('RGBA', (3 * 512, 3 * 512))
def paste(img, row):
    for idx, img in enumerate(img):
        img_pil = Image.open(BytesIO(base64.b64decode(img))).resize((512, 512))
        grid.paste(img_pil, (idx * 512, row * 512))
paste(reply["blended_images"], 0)
paste(reply["masks"], 1)
paste(reply["masked_images"], 2)
grid.show()

The output should be very similar to the demo

POST /sam/dino-predict

You can get GroundingDINO bounding boxes with this API.

Parameters:

• input_image: str [Required] base64 image in string format.
• dino_model_name: str = "GroundingDINO_SwinT_OGC (694MB)" [Optional] Choose one of "GroundingDINO_SwinT_OGC (694MB)" and "GroundingDINO_SwinB (938MB)" as your desired GroundingDINO model.
• text_prompt: str [Required] Text prompt for GroundingDINO to generate bounding boxes. Separate different categories with .
• box_threshold: float = 0.3 [Optional] Threshold for selecting bounding boxes. Do not use a very high value, otherwise you may get no box.

Returns:

• msg Message of the execution information. May contain some common error messages.
• image_with_box base64 image string with bounding boxes. Each bounding boxes are associated with an index on the left-top corner of the box.

Example:

url = "http://localhost:7861/sam/dino-predict"
payload = {
    "dino_model_name": "GroundingDINO_SwinT_OGC (694MB)",
    "input_image": filename_to_base64(img_filename).decode(),
    "text_prompt": "the girl with red hair",
}
response = requests.post(url, json=payload)
reply = response.json()
print(reply["msg"])

grid = Image.new('RGBA', (512, 512))
paste([reply["image_with_box"]], 0)
grid.show()

POST /sam/dilate-mask

You can use this API to expand the mask created by SAM.

Parameters:

• input_image: str [Required] base64 image in string format.
• mask: str [Required] base64 mask image in string format.
• dilate_amount: int = 10 [Optional] Mask expansion amount from 0 to 100.

Returns:

• blended_image base64 image with masks and bounding boxes.
• masks base64 mask generated from SAM
• masked_images base64 masked image. Unmasked region will become transparent.

Example:

url = "http://localhost:7861/sam/dilate-mask"
payload = {
    "input_image": filename_to_base64(img_filename).decode(),
    "mask": reply["mask"],
}
response = requests.post(url, json=payload)
reply = response.json()

grid = Image.new('RGBA', (3 * 512, 512))
paste([reply["blended_image"], reply["mask"], reply["masked_image"]], 0)
grid.show()