Skip to content

Segmentation

The Segmenter task supports YOLO-seg heads (v8-seg, v11-seg, v26-seg). It does everything the detector does and, on top of that, decodes the mask prototypes into per-instance binary masks.

Building the segmenter

from ort_vision_sdk import Segmenter

seg = Segmenter(
    "yolov8n-seg.onnx",
    head="yolo-seg",            # decoder family (default)
    labels="coco",              # default — 80-class COCO preset
    input_size=(640, 640),      # default
    conf_threshold=0.25,
    iou_threshold=0.45,
    max_detections=300,
    mask_threshold=0.5,         # soft → binary mask cutoff
)

import { Segmenter } from "@mauriciobenjamin700/ort-vision-sdk-web";

const seg = await Segmenter.create("/models/yolov8n-seg.onnx", {
  head: "yolo-seg",             // default
  labels: "coco",               // default
  inputSize: [640, 640],        // default
  confThreshold: 0.25,
  iouThreshold: 0.45,
  maskThreshold: 0.5,
});

Predicting

result = seg.predict("street.jpg")[0]

# Same Boxes view as the detector …
print(result.boxes.xyxy, result.boxes.cls, result.boxes.conf)

# … plus per-instance binary masks
for inst in result:
    print(inst.name, inst.conf, inst.box.xyxy)
    print(inst.mask.shape)            # (h, w) uint8 ∈ {0, 255}, cropped to the bbox
    print(inst.segmented_image.shape) # (h, w, 3) RGB with background zeroed

On Web:

const result = (await seg.predict("/images/street.jpg"))[0];
for (const inst of result) {
  console.log(inst.className, inst.confidence, inst.bbox.asXyxy());
  console.log(inst.mask.width, inst.mask.height);  // cropped binary mask
  // inst.segmentedImage: RGBImage with the background zeroed out
}

The Masks view

Beyond boxes, the segmentation envelope exposes the bulk masks view (masks.data, masks.xyxy), mirroring Ultralytics' Masks interface.

Per instance, the mask is cropped to the bounding box:

  • Python: inst.mask is an (h, w) uint8 ndarray with values in {0, 255}, and inst.segmented_image is the RGB crop with the background zeroed.
  • Web: inst.mask is a Mask object (data/width/height, row-major layout), and inst.segmentedImage is an RGBImage.

Python and Web produce the same mask

Both SDKs follow the same algorithm: combine the prototypes, apply sigmoid, resample to the bounding box with half-pixel bilinear, and binarize at the same cutoff. Shared fixtures under fixtures/parity/ check that on both sides, comparing the bitmaps pixel for pixel — so running the same model on the backend and in the browser gives you the same masks.

Masks produced up to 0.6.0 differ at the border

Up to 0.6.0 the Python side resampled the mask through uint8, which put the input to the >= 0.5 test on a grid of 1/255 steps and shifted border pixels for no reason. If you have masks stored from an earlier version, expect a few border pixels to differ — the new ones are the correct ones (they agree 100% with a float64 reference; the old ones, 99.7%).

When finding nothing is an error

Segmenter takes the same raise_on_empty as Detector, with the same default (False) and the same message — see When finding nothing is an error.

seg = Segmenter("yolov8n-seg.onnx", conf_threshold=0.6, raise_on_empty=True)
seg.predict("img.jpg")   # nothing >= 0.6 -> NoDetectionsError

See also