Speech SDK

Note: This is a temporary version and will be consolidated into the AI SDK in a future release.

Speech Model SDK (sm-sdk) is a comprehensive AI speech service platform that brings together multiple speech processing technologies in a single unified solution: Voice Activity Detection (VAD), Automatic Speech Recognition (ASR), Machine Translation, Text-to-Speech (TTS), Keyword Spotting (KWS), and Speaker Diarization.

Platform Support

✨ Key Features

🔍 Voice Activity Detection (VAD)

• Segment splitting: Detects speech and silence boundaries within a continuous audio stream.
• Stream-friendly: Chunk-based processing designed for real-time use cases.

🎤 Automatic Speech Recognition (ASR)

• Real-time streaming: Low-latency speech-to-text transcription.
• Multi-language support: Chinese, English, Japanese, Korean, Cantonese, and more.

🌐 Machine Translation

• Real-time streaming: Low-latency machine translation.
• Multi-language support: Chinese, English, Japanese, Korean, Spanish, and more.

🔊 Text-to-Speech (TTS)

• High-quality synthesis: Natural, fluent speech output.
• Multi-language support: Chinese and English.
• Adjustable parameters: Configurable speech rate, pitch, and more.

🎯 Keyword Spotting (KWS)

• Real-time wake-word detection: Custom keyword detection with minimal latency.
• Low power consumption: Optimized algorithm suited for always-on, long-running deployments.
• Configurable threshold: Adjustable detection sensitivity.

👥 Speaker Diarization

• Multi-speaker support: Tags each segment of a transcription with a speaker identity label.
• ASR pipeline integration: Chains directly with ASR output to produce speaker-labeled transcripts.

Model List

The CPU Core column shows which core type on K3 each model targets by default.

🚀 Quick Start

sm-sdk supports two usage modes:

• From source: Recommended for development, debugging, custom pipeline work, and direct embedding.
• As a system service: Recommended for production deployments on Bianbu systems.

Install Dependencies

1. Install python

   sudo apt update
   sudo apt install python

2. Install llama-server

   HY-MT-1.5 and Qwen3-ASR require a local llama-server. Skip this step if you are only using the ONNX model path.

   Bianbu:

   sudo apt update
   sudo apt install llama.cpp-tools-spacemit

   Ubuntu:

   sudo apt update
   sudo apt install -y build-essential cmake git libcurl4-openssl-dev
   
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   
   cmake -B build -S . -DGGML_CURL=ON -DCMAKE_INSTALL_PREFIX=$HOME/.local
   cmake --build build -j$(nproc)
   cmake --install build
   
   export PATH="$HOME/.local/bin:$PATH"

   To persist this across sessions, add the following line to your shell profile:

   echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
   source ~/.bashrc

From Source

The source-based mode currently targets the following environments:

• Bianbu / RISC-V
• Ubuntu / x86-64

1. Create a virtual environment

   cd path/to/sm-sdk
   curl -LsSf https://astral.sh/uv/install.sh | sh
   source ~/.bashrc
   
   uv venv --python 3.13
   source .venv/bin/activate

2. Install dependencies

   Ubuntu:

   uv pip install .

   Bianbu:

   export UV_EXTRA_INDEX_URL=https://git.spacemit.com/api/v4/projects/33/packages/pypi/simple
   export UV_INDEX_URL=https://mirrors.aliyun.com/pypi/simple
   uv pip install .
   UV_EXTRA_INDEX_URL= uv pip install sherpa-onnx==1.13.1 --index-url https://git.spacemit.com/api/v4/projects/81/packages/pypi/simple --no-cache-dir

3. Prepare models

   On Ubuntu, default ONNX models are downloaded automatically during component initialization according to the active configuration.

   Note: Qwen3-ASR is not yet supported on Ubuntu.

   On Bianbu, download the pre-quantized model package (produced with xslim). Unaccelerated open-source models can also be downloaded automatically via configuration.

   cd path/to/sm-sdk
   wget https://archive.spacemit.com/spacemit-ai/yumeet/yumeet_models.tar.gz
   tar -xf yumeet_models.tar.gz

4. Start the service

   python main.py --start-server

As a System Service

The system service mode currently targets Bianbu environments only.

1. Install the service

   Required model files (~3 GB) are downloaded and prepared automatically on first install:

   sudo apt update
   sudo apt install sm-sdk

2. Manage the service

   The service starts automatically after apt install sm-sdk. Use systemctl for manual lifecycle management:

   sudo systemctl start sm-sdk.service
   sudo systemctl stop sm-sdk.service
   sudo systemctl restart sm-sdk.service
   sudo systemctl status sm-sdk.service

3. View logs

   journalctl -u sm-sdk.service

Additional Notes

Some model backends launch independent processes on dedicated ports. The main service remains the recommended entry point for all external traffic:

Service Configuration

Both usage modes are driven by configuration files that control model selection, pipeline feature toggles, and service parameters.

The configuration directory is resolved in the following priority order:

# Environment variable takes highest priority
$SM_SDK_CONFIG_DIR

# Default directory when running from source
./config/

# Common directory for system service installations
/etc/sm-sdk/

# Fallback directory
/var/lib/sm-sdk/config/

The model directory is resolved in the following priority order:

$SM_SDK_MODEL_DIR
./models/
/var/lib/sm-sdk/models/

Configuration files and their roles:

• asr.yaml: ASR model type, model name, thread count, inference provider, and recognition parameters.
• audio_stream.yaml: Base sample rate for audio input; reused by other components.
• diarization.yaml: Speaker diarization model and similarity threshold parameters.
• keywords.txt: Keyword list for keyword spotting.
• kws.yaml: KWS model, detection threshold, and repeat-detection interval.
• pipeline.yaml: Pipeline-level feature flags — controls whether translation, TTS, punctuation restoration, KWS, and diarization are active.
• server.yaml: FastAPI service listen address, port, and worker count.
• terms.json: Post-processing correction rules for proper nouns and common ASR misrecognitions.
• translation.yaml: Translation model configuration, including llama-server path, port, context length, and thread count.
• tts.yaml: TTS model configuration, including language, thread count, and provider.
• vad.yaml: VAD model and segmentation threshold configuration.

After changing any configuration file, restart the service for the changes to take effect:

# From source
python main.py --start-server

# As a system service
sudo systemctl restart sm-sdk.service

API Reference

WebSocket

The real-time streaming endpoint is:

ws://<host>:<port>/ws/stream

Protocol overview:

1. Once the connection is established, the server sends a ready status message.
2. The client sends raw audio as a continuous byte stream in 16 kHz, mono, PCM16 format.
3. The server emits transcription results (transcript) or keyword detection events (kws) as they become available.
4. The client may send control messages at any time to enable or disable STT, KWS, and translation.
5. When the client sends {"type": "finish"}, the server flushes any remaining output, sends a termination message, and closes the session.

Message formats:

Server ready message

{
    "type": "status",
    "status": "ready",
    "message": "Ready to receive audio"
}

Transcription result message

{
    "type": "transcript",
    "segment": {
        "text": "Hello, world",
        "speaker_id": 0,
        "speaker_label": "Speaker",
        "start_time": 0.0,
        "end_time": 1.2,
        "language": "zh",
        "translation": "Hello, world",
        "is_final": true,
        "is_pseudo": false
    },
    "is_final": true,
    "timestamp": 1710000000.0
}

Keyword detection message

{
    "type": "kws",
    "segment": {
        "keyword": "小迭小迭",
        "confidence": 1.0,
        "start_time": 2.1,
        "end_time": 2.5
    },
    "timestamp": 1710000000.0
}

Control message

{
    "type": "control",
    "enable_stt": true,
    "enable_kws": false,
    "enable_translation": true
}

Finish message

{
    "type": "finish"
}

Note: The server operates in singleton session mode. If the session is already held by another WebSocket or REST request, new connections will receive an error response and be rejected.

HTTP

Once the main service is running, interactive API documentation (Swagger UI) is available at:

http://localhost:8060/docs

Available HTTP endpoints:

curl -X POST http://localhost:8060/v1/components/load

curl -X POST http://localhost:8060/v1/components/release

curl -X POST http://localhost:8060/v1/audio/transcriptions \
    -F "file=@audio.wav" \
    -F "model=asr" \
    -F "response_format=json"

curl -X POST http://localhost:8060/v1/audio/translate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "Hello, welcome to Speech SDK."
    }'

curl -X POST http://localhost:8060/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, welcome to Speech SDK.",
        "response_format": "wav",
        "speed": 1.0,
        "speaker_id": 0
    }' \
    --output output.wav

curl -X POST http://localhost:8060/v1/audio/pipeline \
    -F "file=@audio.wav" \
    -F "model=asr" \
    -F "response_format=json" \
    -F "speed=1.0" \
    -F "speaker_id=0"

Programmatic Usage

import numpy as np

from src.base.config import load_session_config
from src.core.pipeline import SpeechPipeline


config = load_session_config()
pipeline = SpeechPipeline(config)
pipeline.initialize()


def on_segment(segment):
    print(segment.model_dump())


pipeline.add_callback(on_segment)

pcm16 = open("audio.pcm", "rb").read()
chunk = np.frombuffer(pcm16, dtype=np.int16).astype(np.float32) / 32768.0

pipeline._on_audio_chunk(chunk)
pipeline.flush()
pipeline.reset()

import soundfile as sf

from src.base.config import load_session_config
from src.core.speech_components import SpeechComponentManager


audio, sample_rate = sf.read("audio.wav", dtype="float32")
if audio.ndim > 1:
    audio = audio.mean(axis=1)

config = load_session_config()
components = SpeechComponentManager()
components.configure(config)
components.initialize_all()

asr_result = components.asr.recognize(audio)
print(asr_result.text)

if components.translation is not None:
    translated = components.translation.translate(
            asr_result.text,
            source_lang=asr_result.language,
    )
    print(translated)

Source Code Download

On Bianbu, the sm-sdk source package can be retrieved directly via apt.

Types: deb deb-src
URIs: https://archive.spacemit.com/bianbu4/
Suites: resolute resolute-security resolute-updates resolute-backports resolute-porting resolute-customization
Components: main universe restricted multiverse

sudo apt update

mkdir -p ~/Workspace/sm-sdk-src
cd ~/Workspace/sm-sdk-src
apt source sm-sdk

📁 Project Structure

sm-sdk/
├── main.py                    # Entry point
├── src/                       # Core implementation
│   ├── server.py              # FastAPI service entry point
│   ├── api/                   # HTTP / WebSocket interfaces
│   ├── base/                  # Configuration and base data structures
│   ├── core/                  # Speech pipeline and component management
│   ├── models/                # Model engine implementations
│   └── utils/                 # Download, audio, messaging, and other utilities
├── config/                    # Runtime configuration files
├── docs/                      # Internal project documentation
├── models/                    # Local model directory
├── debian/                    # Debian / Bianbu packaging files
└── benchmark/                 # Performance benchmarks and results

Module responsibilities:

• src/api/: HTTP and WebSocket interface layer exposed to external clients.
• src/core/: Pipeline orchestration, pseudo-streaming output, and component lifecycle management.
• src/models/: Per-engine initialization, inference, and reset logic.
• config/: The single source of truth for runtime behavior — feature flags and model selection.
• examples/: End-user code samples and a starting point for custom integrations.