Welcome to Firefly
English
Firefly Docsss
Last Updated: 2026-11-18 14:24:01

Large Language Model Deployment

Introduction

Firefly-R3572A, based on the RK3572 NPU (RKNPU3), supports on-board private deployment of large language models (LLMs), covering Qwen, GLM, MiniCPM, Gemma and other series, suitable for edge-side offline conversation, multimodal understanding and other scenarios.

Important

LLM deployment on RK3572 uses RKLLM3 from the RKNN3 SDK, which is different from and incompatible with the rknn-llm (RKLLM-Toolkit / llm_demo) used on RK3588/RK3576. The RKNPU2-era rknn-llm, rkllm_api_demo and librknnrt.so do not apply to RK3572.

RKLLM3 Overview

RKLLM3 is the LLM deployment solution in the RKNN3 SDK. Its core is rkllm3-server, an LLM server based on RKNPU3 and llama.cpp, providing an OpenAI API-compatible chat interface.

  • Model inference: runs LLM inference on-board through the RKNPU3 driver
  • Chat interface: OpenAI API-compatible REST interface for easy integration
  • Multimodal support: supports LLM + Vision (image) and Audio (speech) multimodal models
  • Multi-card cascading: supports multi-device cascaded inference (--stage-count)

rkllm3-server is located in the SDK external/rknn3-runtime/rkllm3-server/, providing executables for Linux aarch64 and Android arm64-v8a as well as librkllm3-server.so.

Supported Models

The LLMs and multimodal models supported by RK3572 include (refer to the SDK external/rknn3-runtime/README_CN.md):

CategoryRepresentative models
LLMQwen2.5 / Qwen3 / Qwen3.5, GLM-Edge, Youtu-LLM, MiniCPM5, FunctionGemma, LFM2.5
VLMQwen2.5-VL / Qwen3-VL / Qwen3.5-VL, MiniCPM-V-4, InternVL3, SmolVLM2
OmniQwen2.5-Omni / Qwen3-Omni, Gemma4
ASR / TTSQwen3-ASR, Whisper, SenseVoice, Qwen3_TTS, VITS
Embedding / RerankerQwen3-Embedding, Qwen3-Reranker

For more models and pre-converted RKNN models, refer to rknn3-model-zoo and the SDK network drive (pre-converted models are under RKNN3_SDK/rknn3_models/v1.1.0).

Getting the Model

For model conversion, refer to the RKNN3-Toolkit documentation and the LLM examples in rknn3-model-zoo. Alternatively, download pre-converted RKNN models from the SDK network drive (under RKNN3_SDK/rknn3_models/v1.1.0).

A complete LLM model usually consists of the following files, which should be copied to the board together:

  • *.rknn: the converted model body
  • *.gguf (--vocab): the vocabulary file
  • *.embed.bin (--embed): the embedding file (not required in tie-word mode)
  • A weight file *.weight (required by some models)

Deployment and Running

Copy rkllm3-server (or librkllm3-server.so), the model and related files to the board, then start the server. The following is the standard-mode example from the SDK README (modify the paths accordingly):

# Located in SDK external/rknn3-runtime/rkllm3-server/bin/linux-aarch64/
# LLM model (standard mode, requires an embed file)
./rkllm3-server -m qwen2.5-3b.rknn \
    --vocab qwen2.5-3b.tokenizer.gguf \
    --embed qwen2.5-3b.embed.bin \
    --host 0.0.0.0 --port 8080 -c 768 --n_predict 512

Example in tie-word mode (no embed file required):

./rkllm3-server -m Qwen3-VL-2B-llm-tie-word.rknn \
    --weight Qwen3-VL-2B-llm-tie-word.weight \
    --vocab Qwen3-VL-2B-llm_quant.tokenizer.gguf \
    --tie-word-embedding 1 --host 0.0.0.0 --port 8080 -c 4096 --n_predict 512

Once started, the server listens on --host / --port (default 127.0.0.1

) and provides chat through the OpenAI API-compatible REST interface.

Note

The commands above are parameter examples; the actual model file names, parameters and file combinations depend on the rkllm3-server README and the corresponding model directory in rknn3-model-zoo in your SDK. Run ./rkllm3-server --help for the full list of parameters (such as the --core-mask core mask and sampling parameters --temp/--top-k/--top-p).

NPU Driver Requirements

Firefly-R3572A uses the RKNPU3 driver (device node /dev/rknpu3, sysfs directory /sys/class/misc/rknpu3), with the current firmware shipping driver v1.0.3. Before running RKLLM3, confirm the driver is loaded:

root@firefly:/# dmesg | grep -i RKNPU3
[    4.692690] RKNPU3 26b00000.npu: RKNPU3 Driver v1.0.3 (20260613) loaded successfully
root@firefly:/# ls /dev/rknpu3
/dev/rknpu3

Note

cat /sys/kernel/debug/rknpu/version belongs to the legacy rknpu driver; the RK3572 uses the RKNPU3 driver and has no such debugfs path. Use the dmesg method above instead.

FAQs

Q1: The model fails to load after starting the server?

A1: Make sure the model was converted with RKNN3-Toolkit (matching the RKNN3 Runtime version), that the --vocab/--embed/--weight files are complete and correctly pathed, and that /dev/rknpu3 exists with the RKNPU3 driver loaded.

Q2: Can I use RK3588's rknn-llm / llm_demo models?

A2: No. RK3588/RK3576 use RKNPU2 + rknn-llm, which is a different stack from RK3572's RKNPU3 + RKLLM3; the toolchains and model formats are incompatible. Use RKNN3-Toolkit to convert models or download pre-converted RKNN3 models.

Q3: Insufficient PC memory when converting a large model?

A3: Larger models require more memory to convert. Increase the swapfile or use a PC with more memory.

On this page