Welcome to Firefly
Switch language
Switch language
Firefly Docsss
Last Updated: 2026-08-14 16:48:17

API Reference

llamapi-server provides OpenAI-compatible chat, Embeddings, and model-query APIs, together with LlamaPi extensions for on-device model instance management and hardware platform discovery.

Overview

ItemDescription
Base URLhttp://127.0.0.1:9265
API prefixOpenAI-compatible and management APIs use /v1; health does not
Request formatPOST uses a JSON body; GET does not require a body
Response formatJSON for regular APIs, SSE for streaming chat, plain text for health
Request body limit64 MiB
AuthenticationNone required
CORSCross-origin requests are allowed

The current llamapi-server has no API authentication and permits CORS. Add a firewall, reverse proxy, or other access control before exposing it to an untrusted network.

Quick Example

Load a model:

curl -s http://127.0.0.1:9265/v1/models/load \
  -H 'Content-Type: application/json' \
  -d '{
    "model_id": "Qwen3",
    "model_path": "/var/lib/llamapi/models/rkllm/rk3588/qwen3-4b"
  }'

Call the chat API:

curl -s http://127.0.0.1:9265/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3",
    "messages": [
      { "role": "user", "content": "Hello" }
    ]
  }'

Conventions

ItemDescription
Model IDmodel or model_id must identify a loaded model
StreamingChat with "stream": true returns SSE and ends with data: [DONE]
Multimodal inputThe protocol accepts text, image_url, and input_audio
Error responseRegular application errors use OpenAI-style { "error": ... } JSON
SSE errorA mid-stream failure uses event: error

Discover models and platforms through /v1/models and /v1/platforms. Do not hard-code runtime model IDs, platform names, chip types, or model paths.

Value Conventions

Closed Values

FieldValuesDescription
messages[].rolesystem, user, assistant, toolChat message roles
messages[].content[].typetext, image_url, input_audioSupported content parts
Data URL image formatpng, jpeg, jpg, webpDoes not apply to local paths
input_audio.formatwav, mp3A model may support only a subset
encoding_formatfloat, base64Embedding output format
model_kindchat, embeddingModel capability
error.typeinvalid_request_error, rate_limit_exceeded, server_errorError category

Known but Extensible Response Values

FieldKnown valuesClient guidance
choices[].finish_reasonstop, length, tool_callsAccept future string values
objectchat.completion, chat.completion.chunk, list, embedding, modelProcess according to endpoint and structure
error.codeSee ErrorsTreat unknown values as generic errors

Open Values

Do not treat these fields as fixed enumerations:

FieldDescription
model, model_idDefined when a model is loaded
idResponse ID generated by the llamapi-server
platformDiscovered from the platform API
owned_byCurrently formatted as llamapi/{platform}; clients should not rely on it
model_pathllamapi-server filesystem path
detected_chips[].chip_typeChip type detected on the current host
messageHuman-readable text; do not parse it for program logic
tool_call_id, tool_calls[].idDefined by model output or request context
tools[].typefunction is recommended, but the llamapi-server parses a string
tools[].function.nameDefined by the client

Endpoints

MethodPathTypeDescription
POST/v1/chat/completionsOpenAI-compatibleChat completion with JSON or SSE
POST/v1/embeddingsOpenAI-compatibleText vectors for single or batched input
GET/v1/modelsOpenAI-compatibleList loaded models
GET/v1/models/{model_id}OpenAI-compatibleGet one loaded model
POST/v1/models/loadLlamaPi extensionLoad a model dynamically
POST/v1/models/resizeLlamaPi extensionResize a model group
POST/v1/models/unloadLlamaPi extensionUnload a model
GET/v1/platformsLlamaPi extensionList platforms and detected chips
GET/healthHealthCheck whether HTTP is running

Errors

Application errors use an OpenAI-style structure:

{
  "error": {
    "message": "Model 'demo' not found",
    "type": "invalid_request_error",
    "param": "model",
    "code": "model_not_found"
  }
}
HTTP statustypecodeCondition
400invalid_request_errorunsupported_platformThe model platform is unsupported
400invalid_request_errorplatform_detection_failedThe platform cannot be detected from the model directory
400invalid_request_errorwrong_model_typeChat uses an Embedding model, or Embeddings uses a chat model
400invalid_request_errorunsupported_content_part_typeAn unsupported content part is present
400invalid_request_errorinvalid_content_partImage, audio, or another content part is malformed
400invalid_request_errorcontext_length_exceededInput exceeds the model context limit
400invalid_request_errorinvalid_instance_countInstance count is 0
404invalid_request_errormodel_not_foundThe model is not loaded
409invalid_request_errormodel_already_existsThe model ID already exists
429rate_limit_exceededqueue_fullAll instances and queue slots are occupied
500server_errornullEngine, configuration, or internal error

A negative instance count fails during JSON parsing.

Chat Completions

POST /v1/chat/completions

The model must have model_kind=chat. stream defaults to false.

Request Fields

FieldTypeRequiredDescription
modelstringYesLoaded model ID
messagesarrayYesChat messages
streambooleanNoReturn SSE when true
temperaturenumberNoOverride temperature
top_pnumberNoOverride top-p
top_kintegerNoOverride top-k
repeat_penaltynumberNoOverride repetition penalty
frequency_penaltynumberNoOverride frequency penalty
presence_penaltynumberNoOverride presence penalty
max_tokensintegerNoMaximum generated tokens
max_completion_tokensintegerNoEquivalent to max_tokens; takes priority when both are present
stopstring arrayNoStop sequences
toolsarrayNoOpenAI-style function tools
tool_choicestringNoParsed, but not currently used by the llamapi-server
enable_thinkingbooleanNoOverride model thinking mode

Message Fields

messages[] supports:

FieldTypeRequiredDescription
rolestringYessystem, user, assistant, or tool
contentstring or arrayNoPlain text or content parts
tool_call_idstringNoTool call ID for a tool message
tool_callsarrayNoTool calls in an assistant message

Multimodal Content

When content is an array, supported parts are:

Text:

{
  "type": "text",
  "text": "Describe this image."
}

Image:

{
  "type": "image_url",
  "image_url": {
    "url": "/path/to/image.jpg"
  }
}

Image URLs can be:

  • A local path on the llamapi-server.
  • A base64 data URL for png, jpeg, jpg, or webp.

Remote http:// and https:// image URLs are not supported.

Audio:

{
  "type": "input_audio",
  "input_audio": {
    "format": "wav",
    "data": "<base64>"
  }
}

The protocol accepts wav and mp3; a specific model may support only a subset.

video, file, and unknown types return unsupported_content_part_type.

Tool Definitions

tools[] uses the OpenAI function-tool shape:

FieldTypeRequired
typestringYes; function recommended
function.namestringYes
function.descriptionstringYes
function.parametersJSONYes

Non-Streaming Request

curl -s http://127.0.0.1:9265/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3",
    "messages": [
      { "role": "user", "content": "Introduce LlamaPi in one sentence." }
    ],
    "max_tokens": 128
  }'

Non-Streaming Response

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1710000000,
  "model": "Qwen3",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "...",
        "tool_calls": []
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 14,
    "total_tokens": 27
  }
}

Streaming Request

curl -N http://127.0.0.1:9265/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3",
    "messages": [
      { "role": "user", "content": "Hello" }
    ],
    "stream": true
  }'

Streaming Response

Each SSE event contains a chat.completion.chunk JSON object. After the finish chunk, the llamapi-server sends a separate usage chunk and then [DONE]:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1710000000,"model":"Qwen3","choices":[{"index":0,"delta":{"role":"assistant","content":"Hel"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1710000000,"model":"Qwen3","choices":[{"index":0,"delta":{"content":"lo"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1710000000,"model":"Qwen3","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1710000000,"model":"Qwen3","choices":[],"usage":{"prompt_tokens":13,"completion_tokens":14,"total_tokens":27}}

data: [DONE]

A mid-stream failure sends:

event: error
data: {"error":{...}}

The stream then ends without [DONE].

Embeddings

POST /v1/embeddings

The model must have model_kind=embedding.

Embeddings Request Fields

FieldTypeRequiredDescription
modelstringYesLoaded Embedding model ID
inputstring or string arrayYesSingle or batched text
encoding_formatstringNofloat or base64; default float
dimensionsintegerNoParsed, but the llamapi-server does not truncate vectors
userstringNoParsed, but not currently used

base64 encodes raw little-endian f32 bytes.

Load the Embedding Model

Use the llamapi-cli:

llamapi load bge-m3
llamapi ps

Or call the management API:

curl -s http://127.0.0.1:9265/v1/models/load \
  -H 'Content-Type: application/json' \
  -d '{
    "model_id": "bge-m3",
    "model_path": "/var/lib/llamapi/models/rknn2/rk3588/bge-m3"
  }'

Embeddings Request Example

curl -s http://127.0.0.1:9265/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "bge-m3",
    "input": ["hello", "LlamaPi"],
    "encoding_format": "float"
  }'

Embeddings Response Example

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [0.1, 0.2],
      "index": 0
    }
  ],
  "model": "bge-m3",
  "usage": {
    "prompt_tokens": 2,
    "total_tokens": 2
  }
}

Query Models

List Models

GET /v1/models
curl -s http://127.0.0.1:9265/v1/models

Response:

{
  "object": "list",
  "data": [
    {
      "id": "Qwen3",
      "object": "model",
      "created": 0,
      "owned_by": "llamapi/rkllm",
      "platform": "rkllm",
      "instance_count": 1,
      "model_path": "/var/lib/llamapi/models/rkllm/rk3588/qwen3-4b",
      "model_kind": "chat"
    }
  ]
}

Get One Model

GET /v1/models/{model_id}
curl -s http://127.0.0.1:9265/v1/models/Qwen3

The response fields match an item in /v1/models data[]. A missing model returns 404 model_not_found.

Load a Model

POST /v1/models/load

Load Model Request Fields

FieldTypeRequiredDescription
model_idstringYesModel ID exposed to clients
model_pathstringYesModel directory on the llamapi-server filesystem
instance_countintegerNoTarget count; default 1, minimum 1
request_queue_sizeintegerNoModel queue capacity; uses the llamapi-server setting when omitted
default_paramsobjectNoModel generation defaults

default_params supports:

  • temperature
  • top_p
  • top_k
  • repeat_penalty
  • frequency_penalty
  • presence_penalty
  • max_tokens
  • max_context_len
  • stop
  • enable_thinking

Load Model Request Example

curl -s http://127.0.0.1:9265/v1/models/load \
  -H 'Content-Type: application/json' \
  -d '{
    "model_id": "Qwen3",
    "model_path": "/var/lib/llamapi/models/rkllm/rk3588/qwen3-4b",
    "instance_count": 2,
    "default_params": {
      "temperature": 0.7,
      "max_tokens": 512
    }
  }'

Load Model Response Example

{
  "success": true,
  "message": "model 'Qwen3' loaded",
  "requested_instance_count": 2,
  "actual_instance_count": 2
}

When at least one instance loads, the API returns 200 OK:

  • Complete success uses model '<id>' loaded.
  • Partial success uses model '<id>' partially loaded.
  • requested_instance_count is the target.
  • actual_instance_count is the actual count.

Coprocessor Instance Limitation

Requesting multiple model instances on a coprocessor through /v1/models/load or /v1/models/resize may cause loading failure, chip communication failure, and abnormal service state. Keep instance_count at 1 for coprocessor models. See Coprocessor Communication Failure After Loading Multiple Model Instances for recovery.

Resize a Model

POST /v1/models/resize
FieldTypeRequiredDescription
model_idstringYesLoaded model ID
instance_countintegerYesTarget count, at least 1
curl -s http://127.0.0.1:9265/v1/models/resize \
  -H 'Content-Type: application/json' \
  -d '{
    "model_id": "Qwen3",
    "instance_count": 1
  }'

Response:

{
  "success": true,
  "message": "model 'Qwen3' resized",
  "requested_instance_count": 1,
  "actual_instance_count": 1
}

A partial expansion can still return 200 OK with model '<id>' partially resized.

Unload a Model

POST /v1/models/unload
FieldTypeRequiredDescription
model_idstringYesLoaded model ID
curl -s http://127.0.0.1:9265/v1/models/unload \
  -H 'Content-Type: application/json' \
  -d '{ "model_id": "Qwen3" }'

Response:

{
  "success": true,
  "message": "model 'Qwen3' unloaded"
}

Query Platforms

GET /v1/platforms

Returns registered chat and Embedding platforms and chips detected on the current host.

curl -s http://127.0.0.1:9265/v1/platforms

Example response:

{
  "platforms": [
    {
      "id": "rknn3",
      "display_name": "RKNN3",
      "available": true,
      "detected_chips": [
        {
          "chip_type": "RK1828",
          "count": 2
        }
      ]
    }
  ]
}

detected_chips groups devices by chip type. It is an empty array when no chip is detected.

Health

GET /health
curl -s http://127.0.0.1:9265/health

Response:

ok

Health only confirms that the HTTP service is running. Confirm model availability with /v1/models and an inference request.

Service Configuration and Operations

Previous Page

On this page

API ReferenceOverviewQuick ExampleConventionsValue ConventionsClosed ValuesKnown but Extensible Response ValuesOpen ValuesEndpointsErrorsChat CompletionsRequest FieldsMessage FieldsMultimodal ContentTool DefinitionsNon-Streaming RequestNon-Streaming ResponseStreaming RequestStreaming ResponseEmbeddingsEmbeddings Request FieldsLoad the Embedding ModelEmbeddings Request ExampleEmbeddings Response ExampleQuery ModelsList ModelsGet One ModelLoad a ModelLoad Model Request FieldsLoad Model Request ExampleLoad Model Response ExampleCoprocessor Instance LimitationResize a ModelUnload a ModelQuery PlatformsHealth