Fused pipelines (detection → classification)

You have two models. A detector finds the objects; a classifier says which sub-type each object is. The natural flow is to chain them:

detections = detector.predict("flock.jpg")[0]
for d in detections:
    sub = classifier.predict(d.cropped_image)[0]  # 😐 one more session, one more round trip

That works — and it is expensive. It means two sessions, two model loads, and a round trip through Python (or JavaScript) for every crop: slice, resize, restack a batch, call the second runtime. On a phone or in a browser tab, that round trip is frequently the dominant cost.

The ort_vision_sdk.compose module removes both. It rewrites the two .onnx files into a single graph: the detector, a bridge that crops and resizes the boxes, and the classifier. One file, one session, one load — and the crops never leave the runtime.

flowchart LR
    A[letterboxed<br/>image] --> B[detector]
    B --> C[NMS]
    C --> D[RoiAlign<br/>crop + resize]
    A --> D
    D --> E[normalize]
    E --> F[classifier]
    F --> G[probs]
    C --> H[boxes / scores / classes]

Fusing is a build step, running is not

Fusing needs the onnx library (the [compose] extra), because it rewrites protobufs. What comes out is a plain .onnx: running it needs nothing beyond the onnxruntime the SDK already depends on — including in the browser, where the web SDK loads the very same file.

Installing the extra

pip install "ort-vision-sdk[compose]"

Fusing the two models

from ort_vision_sdk.compose import fuse_detect_classify

fuse_detect_classify(
    "yolov8n.onnx",       # detector: anchor-free YOLO head (v8..v26)
    "resnet18.onnx",      # classifier: one NCHW input, one (batch, classes) output
    "pipeline.onnx",      # where to write the fused model
    max_detections=20,    # how many boxes the pipeline reports per image
    conf_threshold=0.25,  # score threshold, baked into the graph's NMS
    iou_threshold=0.45,   # IoU threshold, same
)

That's it. pipeline.onnx is a self-contained model.

The fusion validates itself

Before returning, fuse_detect_classify runs the fused graph once in ONNX Runtime. That is what catches the most common failure — a classifier whose graph only accepts the batch size it was exported with (a Reshape with a hardcoded 1 inside it). You find out at fusion time, not in production. Pass validate=False to skip it.

Running it

PythonWeb (browser)

from ort_vision_sdk import DetectClassify

pipeline = DetectClassify("pipeline.onnx")
result = pipeline.predict("flock.jpg")[0]

for d in result:
    print(d.name, d.conf, d.box.xyxy)                    # what the detector found
    print(d.classification.name, d.classification.conf)  # what the classifier said

import { DetectClassify } from "@mauriciobenjamin700/ort-vision-sdk-web";

const pipeline = await DetectClassify.create("/models/pipeline.onnx");
const result = (await pipeline.predict("/images/flock.jpg"))[0];

for (const d of result) {
  console.log(d.name, d.conf, d.box.xyxy);
  console.log(d.classification?.name, d.classification?.conf);
}

You restated no configuration when loading. The letterbox resolution, the crop size, whether the output still needs a softmax, the class names of both stages — all of it was decided at fusion time and written into the file's own metadata. It is the same idea as The model decides, applied to a whole pipeline.

Two label spaces, kept apart

A detector that finds sheep feeding a classifier that answers famacha_3 shares no class ids with it. So the envelope carries two maps, and the classifier's answer lives in its own field:

result.names             # {0: 'sheep', 1: 'goat'}                  — detection stage
result.classifier_names  # {0: 'famacha_1', 1: 'famacha_2', ...}    — classification stage

d = result[0]
d.cls, d.name              # detector class
d.classification.cls       # classifier class — different space, different id

Do not compare d.cls with d.classification.cls

They answer different questions: what kind of object this is and which sub-category the object belongs to. Collapsing them into one field would lose one of the two answers.

Choosing where the crops come from

This is the decision that changes the result the most.

detector_input (default)original

Crops from the 640×640 letterboxed tensor itself. The graph has a single input and is the simplest to operate.

fuse_detect_classify(det, clf, "pipeline.onnx", crop_source="detector_input")

The cost: a small object is classified from its downscaled copy. A 40×40 px box inside a 640 letterbox becomes a 224×224 crop upsampled from 40×40 pixels of real detail.

Adds a second input at native resolution. The bridge undoes the letterbox inside the graph and crops from the original image.

fuse_detect_classify(det, clf, "pipeline.onnx", crop_source="original")

That is two tensors to feed — but the SDK builds both for you, and it is still one session and one load. Use this when the classifier depends on fine detail (texture, mucous-membrane colour, a small lesion).

The boxes are identical in both modes

The graph always reports boxes in the detector's letterboxed space, and the runtime undoes that transform the same way in both cases. Switching crop_source changes crop quality, never coordinates.

How many boxes the pipeline reports

By default the pipeline has a fixed number of rows: max_detections. Surplus rows are zero-padded, and the num_detections output says how many are real — the runtime already ignores the rest for you.

fuse_detect_classify(det, clf, "pipeline.onnx", max_detections=20)  # 20 rows, always

That is what keeps every shape in the graph static, and static shapes are what TensorRT, NNAPI and WebGPU need in order to compile the model. It is also what removes the zero-detection case: the classifier always gets K crops, never an empty batch that some execution providers refuse to run.

The price is running the classifier K times even when there are 2 objects. If that outweighs static compilation, use the dynamic mode:

fuse_detect_classify(det, clf, "pipeline.onnx", max_detections=None)

What the dynamic mode requires

The classifier must have been exported with a dynamic batch axis, and must tolerate a batch of zero rows (which is what happens when nothing clears the threshold). The fusion's validation run exercises exactly that case, so you find out immediately if your model cannot take it.

Classifier normalization

The crop leaves the graph in [0, 1]. What has to happen next depends entirely on how your classifier was trained — and the two most common families disagree:

Family	Expects	Preset
torchvision / timm	a tensor normalized with the ImageNet mean and deviation	`"imagenet"`
Ultralytics (`YOLO(...).export()`)	raw `[0, 1]`, no normalization at all	`"ultralytics"`

So the default is normalization="auto": the fusion reads the classifier's metadata and picks for you.

from ort_vision_sdk.compose import fuse_detect_classify

# Nothing to decide: the file says what it is.
fuse_detect_classify(det, "yolo11s-cls.onnx", "pipeline.onnx")

Every Ultralytics export stamps author: Ultralytics and task: classify into the .onnx itself. That pair is deterministic, and the SDK already read this block to carry over class names — so the information needed to get it right was always inside the file you hand over.

The failure mode this removes

Up to 0.8.0 the default was ImageNet for everybody. Fusing an Ultralytics classifier with that default inserts a Sub/Div the model never saw in training — and nothing complains: no exception, no warning, and validate=True passes (it checks shape, not semantics). The graph runs, returns a probs row of the right shape, and classifies worse. Without measuring agreement against the unfused route, there is no way to notice.

Choosing by hand

fuse_detect_classify(det, clf, "pipeline.onnx", normalization="imagenet")
fuse_detect_classify(det, clf, "pipeline.onnx", normalization="ultralytics")
fuse_detect_classify(det, clf, "pipeline.onnx", normalization="none")

"none" is the same arithmetic as "ultralytics" — identity — under a name that says "this model wants raw [0, 1]" rather than naming a vendor.

Your own values

A model whose normalization no preset describes takes the numbers directly:

fuse_detect_classify(
    det, clf, "pipeline.onnx",
    mean=(0.5, 0.5, 0.5),
    std=(0.5, 0.5, 0.5),
    input_scale=1.0,     # 255.0 if your model expects 0..255
)

mean and std are independent

Passing only one of the two leaves the other at whatever the preset auto would pick — passing a mean does not quietly reset the deviation to 1.

Passing mean/std together with normalization raises ValueError: they are two answers to one question, and guessing which one you meant is how a graph ends up normalized one way with metadata claiming another.

And if the classifier is an Ultralytics export and you ask for a normalization that is not the identity, you get a UserWarning — the graph is still built, but you know about it.

What the file records

The choice goes into the fused model's metadata, under ovs.classifier_normalization:

python -c "import onnx; print({k.key: k.value for k in onnx.load('pipeline.onnx').metadata_props}['ovs.classifier_normalization'])"
# ultralytics

Once fused, the normalization is a handful of Sub/Div nodes in the middle of a graph — invisible. "Which preprocessing does this pipeline assume?" is the first question when a fused model classifies worse than the cascade it replaced, and now the file answers it.

Steps that would be no-ops (zero mean, unit deviation, unit scale) are not emitted as nodes — a classifier that wants the raw [0, 1] crop pays for no arithmetic at all.

Three stages: segmenting inside the crop

Sometimes the crop is not enough: you want to classify the object only, without the background a box inevitably drags along. That is what a third stage is for.

from ort_vision_sdk.compose import fuse_detect_segment_classify

fuse_detect_segment_classify(
    "yolov8n.onnx",
    "unet.onnx",
    "resnet18.onnx",
    "pipeline.onnx",
    max_detections=10,
)

The resulting graph does all of this in one file: detect → crop → segment each crop → multiply the crop by the mask → classify.

flowchart LR
    A["images"] --> B["detector"]
    B --> C["NMS + RoiAlign"]
    C --> D["crops"]
    D --> E["segmenter"]
    E --> F["binary mask"]
    D --> G["Mul"]
    F --> G
    G --> H["classifier"]
    F --> I["masks"]
    H --> J["probs"]

This is not for a YOLO-seg

A YOLO-seg export already detects and segments in one model. Putting a separate detector in front of it duplicates the work. Three stages are for when the segmenter is a plain image-to-mask network (U-Net and relatives) with no detector of its own, which does better on a tight crop than on a full frame.

Reading the masks

from ort_vision_sdk import DetectClassify

pipeline = DetectClassify("pipeline.onnx")
result = pipeline.predict("photo.jpg")[0]

for d in result.detections:
    print(d.name, d.conf, d.mask.shape)   # (box_height, box_width)

d.mask arrives already shaped to the box, binary (0/255) uint8 — the same contract Segmenter produces. Code that already handles a segmenter's mask handles this one unchanged.

Inside the graph, the mask lives in the crop's space

The .onnx's masks output is shaped (K, 1, crop_h, crop_w), because that is where the segmenter computed it. Mapping it back to the original image is one resize per instance at a per-box geometry — the task does that for you (nearest-neighbour, the values being binary), and anyone driving the graph by hand does it from the matching boxes row.

What you can tune

Parameter	For
`mask_threshold`	Probability above which a pixel is foreground (default `0.5`).
`mask_channel`	Which segmenter channel is the foreground. `0` for a single-channel head.
`mask_activation`	`"sigmoid"` (one channel) or `"softmax"` (competing channels). `None` decides from the graph.
`apply_mask`	`False` reports masks without gating the classifier — use it when the classifier was trained on un-masked crops.
`segmenter_mean` / `segmenter_std`	The segmenter's normalization, independent of the classifier's.

The last two stages share crop_size

The mask is computed in the crop's space and multiplies that crop directly. If the segmenter and the classifier declare different resolutions, the fusion refuses and names both, rather than resampling behind your back and handing you a mask that no longer lines up with what it describes.

Checking that the mask reached the classifier

The test the SDK uses for this doubles as a recipe: fix a uniformly coloured crop and compare the probabilities with apply_mask=True and False. Masking half of a pure-red crop takes the channel mean from 1.0 to 0.5 — a number that can only appear if the mask actually multiplied.

When finding nothing is an error

A pipeline that finds nothing returns an empty envelope — the classifier never even gets a row to answer about. If the next step depends on something being there, raise_on_empty turns that into an exception, exactly as on Detector:

pipeline = DetectClassify("pipeline.onnx", raise_on_empty=True)
pipeline.predict("empty-field.jpg")   # -> NoDetectionsError

The threshold named in the message is the effective one: the higher of what was frozen into the graph's NMS at fusion time and any stricter conf_threshold passed on the call.

When fusing pays off

Fusing is not always faster. Intuition says it is — one artifact, one session, no round trip through the host — but the real cost depends on where the time actually is, and in a detection pipeline it is almost never in the bridge.

These numbers come from a real pipeline (eye-mucosa detection at 640 → classification at 224, 12-core i9-13900F, RTX 4070 Ti SUPER, onnxruntime 1.24.3), 200 timed passes after 20 warm-up passes, median of 20 repeats:

Device	Topology	Sessions	Latency	vs cascade
CPU	cascade `.onnx`, 6+6 threads	2	33.06 ms	1.00×
CPU	fused	1	50.65 ms	0.65×
CPU	cascade `.pt` (torch)	2	69.45 ms	0.48×
GPU	cascade `.onnx`, 6+6 threads	2	5.97 ms	1.00×
GPU	fused	1	5.74 ms	1.04×
GPU	cascade `.pt` (torch)	2	11.24 ms	0.53×

On CPU the fused graph was 53% slower than two well-configured sessions. On GPU it tied.

Why the ceiling is low

The cascade's per-stage breakdown on CPU explains all of it:

Stage	ms	% of total
detector	25.79	78.0%
crop (`cv2` on the host)	1.49	4.5%
classifier	5.00	15.1%

Fusing can only remove the host-side crop and one hand-off between sessions — a 4.5% ceiling. And the ~68 bridge nodes (NMS, TopK, Pad, clamp, RoiAlign) cost +17.6 ms on CPU, well over the cv2.resize they replace. On GPU those nodes are cheap and the tie reappears.

Capability has a price

A minimal hand-written bridge (ArgMax + RoiAlign, 22 nodes, no NMS, no padding, a single object) measured 0.99× on that same machine. The gap between 0.99× and 0.65× is the price of thresholded NMS, TopK, padding to a static shape, and clamping — fair for what it buys, but large on CPU.

The trap that inverts the conclusion

A misconfigured cascade makes fusing look 5× faster

The first measurement in that study gave 148 ms for the cascade against 30 ms fused. The number was false: the detector alone measured 110 ms inside the cascade, and the fused graph contains exactly the same detector.

The cause is ONNX Runtime giving each InferenceSession its own intra-op pool. In a cascade the two stages alternate, and the pool that just finished sits in spin-wait stealing cores from the one that is running:

Configuration	Total (CPU)
12 + 12 threads	140.6 ms
12 + 12, `intra_op.allow_spinning = 0`	49.4 ms
6 + 6 threads	33.7 ms

The fused graph has one pool and is immune to this by construction — so comparing against a misconfigured cascade hands fusing a win it did not earn. The honest baseline splits the thread budget between the sessions. (torch does not suffer from this: torch.set_num_threads configures a single process-wide pool.)

Reproducing the cascade numerically

For the fused graph to reproduce a cascade that crops with cv2.INTER_LINEAR:

sampling_ratio=1 — a plain bilinear resample. The default 0 adapts to the box size and anti-aliases: better in general, but it samples differently from a cv2.resize.
coordinate_transformation_mode="half_pixel" on RoiAlign, which is what the bridge already uses: it reproduces cv2.INTER_LINEAR to a mean absolute error of 4·10⁻⁵ on an image in [0, 1], against 185× more for output_half_pixel.

With both, prediction agreement between fused and cascade came out at 0.982 (3 disagreements in 169 samples, all with confidence between 0.47 and 0.60 — right on the decision boundary), median probability delta 3.4·10⁻⁵, and an identical selected box (0.000 px).

Recommendation

Worth it for portability and operations: one artifact, one session, static shapes (TensorRT/NNAPI/WebGPU), the same file running in the web SDK.
Not worth it for CPU latency, and a tie on GPU, when the alternative is a cascade with its thread budget split.
Before measuring, look at the per-stage breakdown with timings: if the detector is ~80% of the cost, the ceiling on what fusing can win is whatever is left.

Limits worth knowing

The head must be anchor-free YOLO — output (1, 4 + nc, N), the same family Detector accepts. Heads with an explicit objectness channel (v5/v6/v7) or with built-in NMS (v10 end2end) are refused with a clear message rather than silently reading the wrong channels.
Thresholds are frozen into the graph. conf_threshold and iou_threshold go inside the NMS node. At runtime you can filter further (predict(img, conf_threshold=0.6)), never looser — to lower the threshold, fuse again.
The fused NMS scores every class. The Python decoder collapses each anchor to its argmax before suppressing; ONNX's NonMaxSuppression scores each class independently. An anchor that clears the threshold for two classes yields two rows here and one row there.
Opsets are reconciled upwards. If the two models were exported at different versions, the older one is converted — and the floor is opset 16, required by the bridge's RoiAlign.
boxes is the box that was classified. A box running off the frame is clamped to the image bounds before it becomes a crop, and it is that clamped version that comes out in boxes — drawing the rectangle and looking at the crop show the same region. Up to 0.8.0 the two diverged: boxes reported the whole box while RoiAlign received the clamped one. Padded rows stay zero.

Recap

fuse_detect_classify turns a detector plus a classifier into one .onnx.
Fusing needs the [compose] extra; running needs nothing extra.
DetectClassify (Python and Web) loads the file and configures itself from the metadata written at fusion time.
crop_source="original" costs one extra input and gives the classifier back its native resolution.
A fixed max_detections keeps shapes static; None trades that for less work per run.

Next steps

Detection — the head the detection stage must have.
Classification — the normalization the second stage expects.
The model decides — why configuration lives in the file.
Python API and Web API.