TensorRT: A Real-Time Semantic Segmentation

Why Semantic Segmentation on the Edge

Semantic segmentation goes one step beyond image classification: a classification network answers "what is in this image", while a segmentation network answers "for every pixel, what class does it belong to". This dense, per-pixel output is a prerequisite for many edge scenarios:

Pushing this computation to the cloud means continuously uploading raw video: a single 1080p camera easily produces 4–8 Mbps, and multi-camera deployments quickly make both bandwidth cost and latency uncontrollable — worse, many field sites simply have no reliable internet. Once inference is consolidated onto an edge box like the AIBOX, the only thing going upstream is structured results (masks, region ratios, alarm events) — orders of magnitude less data, and local perception keeps working when the network is down.

TensorRT's role here is making this idea practical. Semantic segmentation is a dense, per-pixel task with an order of magnitude more compute than classification; running it on an edge GPU through vanilla PyTorch/ONNX Runtime rarely reaches real-time frame rates. TensorRT lifts the achievable FPS on the same Jetson hardware several-fold to tens-fold through graph optimization and precision quantization, turning "real-time streamed segmentation on a 60–160 TOPS box" into a deliverable engineering goal.

NVIDIA Series AIBOX

The AIBOX-OrinNano and AIBOX-OrinNX are both equipped with original NVIDIA Jetson Orin core modules. They come standard with industrial-grade all-metal casings and aluminum alloy structures for heat dissipation. The top cover features a strip grille design on the side for efficient cooling, ensuring computational performance and stability even under high-temperature operating conditions, meeting various industrial application requirements.

TensorRT: The Inference Accelerator on the Edge

The NVIDIA series AIBOX supports the deep learning framework TensorRT. TensorRT is an SDK and runtime ecosystem for high-performance deep learning inference, providing low latency and high throughput for production deployments. The ecosystem includes TensorRT, TensorRT-LLM, TensorRT Model Optimizer, and TensorRT Cloud.

NVIDIA's official list of TensorRT advantages:

1. Inference speed increased by up to 36x
2. Optimized inference performance
3. Accelerated various workloads
4. Deploy, run, and scale using Triton

In engineering terms, TensorRT does four things:

Semantic Segmentation and SegNet

Semantic segmentation is based on image recognition, but classification is performed at the pixel level rather than on the entire image. This is achieved by convolutionalizing a pre-trained image recognition backbone, converting the model into a Fully Convolutional Network (FCN) capable of per-pixel annotation. Semantic segmentation is particularly useful for environmental perception, as it allows for dense per-pixel classification of many different potential objects (including foreground and background) in each scene.

SegNet Model

The novelty of SegNet lies in the way the decoder upsamples its lower-resolution input feature maps. Specifically, the decoder uses the pooling indices computed in the corresponding encoder's max-pooling steps to perform non-linear upsampling. The upsampled feature maps are sparse, so trainable convolution kernels are subsequently used to generate dense feature maps. This "pooling-index reuse" design avoids the cost of learned upsampling parameters and lands on a memory-vs-accuracy trade-off well suited to edge deployment — one reason SegNet/FCN-style architectures have long been the default segmentation entry point in the Jetson ecosystem.

Pre-trained Segmentation Models in jetson-inference

The segNet module ships with a set of FCN-ResNet18 pre-trained models covering street scenes, humans, indoor scenes, and outdoor trails:

Hands-On Deployment

System Preparation

Set the power mode to maximum first:

$ sudo nvpmodel -m 0        # MAX-N mode (Orin Nano defaults to 15W)
$ sudo jetson_clocks        # Lock CPU/GPU/EMC clocks to maximum

All official performance figures are measured under MAX-N. At the default power mode the GPU clock is capped and frame rates drop noticeably. Also keep jtop or tegrastats handy for live monitoring:

$ tegrastats               # CPU/GPU utilization, clocks, temperature, power draw

Download the Source Code

$ git clone --recursive --depth=1 https://github.com/dusty-nv/jetson-inference
$ cd jetson-inference

Install Dependencies and Build

$ sudo apt-get update
$ sudo apt-get install git cmake libpython3-dev python3-numpy
$ mkdir build && cd build
$ cmake ../
$ make -j$(nproc)
$ sudo make install
$ sudo ldconfig

Reference: https://github.com/dusty-nv/jetson-inference/blob/master/docs/building-repo-2.md

Download the Models

$ cd jetson-inference/tools
$ ./download-models.sh

The script presents an interactive menu for selecting models. Segmentation-only, non-interactive download is also available:

$ ./download-models.sh --non-interactive

Models land in jetson-inference/data/networks/, each consisting of .onnx weights plus class labels and a color map.

Run the Example

Segment a single image:

$ cd jetson-inference/build/aarch64/bin
$ ./segnet.py --network=fcn-resnet18-cityscapes input.jpg output.jpg

View the result once processing completes:

$ eog output.jpg

The output is the original image overlaid with the segmentation mask, one color per class. Use the --overlay and --alpha options to switch between raw mask, blended, or plain overlay.

Feeding Live Video Streams

Single images are just a smoke test — real deployments are stream-based. jetson-inference abstracts inputs and outputs behind unified videoSource/videoOutput interfaces:

# USB camera (/dev/video0) live segmentation, rendered to the HDMI display
$ ./segnet.py --network=fcn-resnet18-cityscapes /dev/video0

# CSI camera (IMX219/IMX477, etc.)
$ ./segnet.py --network=fcn-resnet18-cityscapes csi://0

# RTSP camera in, segmented result streamed out over RTP
$ ./segnet.py --network=fcn-resnet18-cityscapes \
    rtsp://admin:password@192.168.1.100:554/stream1 \
    rtp://192.168.1.50:1234

# Video file in, MP4 out (offline batch processing)
$ ./segnet.py --network=fcn-resnet18-cityscapes input.mp4 output.mp4

Adding --stats prints per-frame timing broken down by stage (capture / pre-process / network / post-process / render) — first-hand data for locating pipeline bottlenecks:

$ ./segnet.py --network=fcn-resnet18-cityscapes --stats /dev/video0