Deployment Routines
AI algorithm deployment
EC-A1684JD4 FD has a comprehensive local private deployment capability of AI algorithms, whether it is a novel large language model with high computing requirements, or a classic YOLOv5 object detection model, EC-A1684JD4 FD can be competent.
This chapter mainly introduces the SOPHON-DEMO project. For large model deployment, refer to the LLM deployment chapter.
SOPHON-DEMO project introduction
SOPHON-DEMO contains a series of porting routines for mainstream AI algorithms, as well as detailed deployment documentation to make it easy for users to run.
Project repository link: SOPHON-DEMO。
Examples list
The examples provided by SOPHON-DEMO are divided into three modules from easy to difficult: tutorial, sample and application.
tutorialmodule stores some examples of basic interfaces.samplemodule stores some serial examples of classic algorithms on SOPHONSDK.applicationmodule stores some typical applications in typical scenarios.
| tutorial | code |
|---|---|
| resize | C++/Python |
| crop | C++/Python |
| crop_and_resize_padding | C++/Python |
| ocv_jpubasic | C++/Python |
| ocv_vidbasic | C++/Python |
| blend | C++/Python |
| stitch | C++/Python |
| contents | category | code | BModel |
|---|---|---|---|
| LPRNet | License Plate Recognition | C++/Python | FP32/FP16/INT8 |
| ResNet | Image classification | C++/Python | FP32/FP16/INT8 |
| RetinaFace | Face detection | C++/Python | FP32 |
| segformer | Semantic segmentation | C++/Python | FP32/FP16 |
| SAM | Semantic segmentation | Python | FP32/FP16 |
| yolact | Instance segmentation | C++/Python | FP32/FP16/INT8 |
| YOLOv8_seg | Instance segmentation | C++/Python | FP32/FP16/INT8 |
| PP-OCR | OCR | C++/Python | FP32/FP16 |
| OpenPose | Keypoint detection | C++/Python | FP32/FP16/INT8 |
| C3D | Video recognition | C++/Python | FP32/FP16/INT8 |
| DeepSORT | Object tracking | C++/Python | FP32/FP16/INT8 |
| ByteTrack | Object tracking | C++/Python | FP32/FP16/INT8 |
| CenterNet | Object Detection + pose estimation | C++/Python | FP32/FP16/INT8 |
| YOLOv5 | Object Detection | C++/Python | FP32/FP16/INT8 |
| YOLOv34 | Object Detection | C++/Python | FP32/INT8 |
| YOLOX | Object Detection | C++/Python | FP32/INT8 |
| SSD | Object Detection | C++/Python | FP32/INT8 |
| YOLOv7 | Object Detection | C++/Python | FP32/FP16/INT8 |
| YOLOv8_det | Object Detection | C++/Python | FP32/FP16/INT8 |
| YOLOv5_opt | Object Detection | C++/Python | FP32/FP16/INT8 |
| ppYOLOv3 | Object Detection | C++/Python | FP32/FP16/INT8 |
| ppYoloe | Object Detection | C++/Python | FP32/FP16 |
| WeNet | Speech Recognition | C++/Python | FP32/FP16 |
| BERT | Language | C++/Python | FP32/FP16 |
| ChatGLM2 | Language | C++/Python | FP16/INT8/INT4 |
| Llama2 | Language | C++ | FP16/INT8/INT4 |
| ChatGLM3 | Language | Python | FP16/INT8/INT4 |
| Qwen | Language | Python | FP16/INT8/INT4 |
| Qwen1_5 | Language | Python | FP16/INT8/INT4 |
| StableDiffusionV1.5 | Image Generation | Python | FP32/FP16 |
| GroundingDINO | MultiModal Object Detection | Python | FP16 |
| application | scenarios | code |
|---|---|---|
| VLPR | Multi-streams Vehicle License Plate Recognition | C++/Python |
| YOLOv5_multi | Multi-streams Object Detection | C++ |
| YOLOv5_multi_QT | Multi-streams Object Detection + QT_HDMI display | C++ |
LLM deployment
LLM-TPU project introduction
LLM-TPU contains migration deployment routines for various open source generative AI models, mainly LLM, also Stable Diffusion (AI painting).
Project repository link: LLM-TPU.
Examples list
| Model | INT4 | INT8 | FP16/BF16 | Huggingface Link |
|---|---|---|---|---|
| Baichuan2-7B | ✔ | LINK | ||
| ChatGLM3-6B | ✔ | ✔ | ✔ | LINK |
| CodeFuse-7B | ✔ | ✔ | LINK | |
| DeepSeek-6.7B | ✔ | ✔ | LINK | |
| Falcon-40B | ✔ | ✔ | LINK | |
| Phi-3-mini-4k | ✔ | ✔ | ✔ | LINK |
| Qwen-7B | ✔ | ✔ | ✔ | LINK |
| Qwen-14B | ✔ | ✔ | ✔ | LINK |
| Qwen-72B | ✔ | LINK | ||
| Qwen1.5-0.5B | ✔ | ✔ | ✔ | LINK |
| Qwen1.5-1.8B | ✔ | ✔ | ✔ | LINK |
| Llama2-7B | ✔ | ✔ | ✔ | LINK |
| Llama2-13B | ✔ | ✔ | ✔ | LINK |
| LWM-Text-Chat | ✔ | ✔ | ✔ | LINK |
| Mistral-7B-Instruct | ✔ | ✔ | LINK | |
| Stable Diffusion | ✔ | LINK | ||
| Stable Diffusion XL | ✔ | LINK | ||
| WizardCoder-15B | ✔ | LINK | ||
| Yi-6B-chat | ✔ | ✔ | LINK | |
| Yi-34B-chat | ✔ | ✔ | LINK |
RAG private knowledge base
Retrieval-Augmented Generation (RAG) is a technology that utilizes information from private or proprietary data sources to enhance text generation. It combines the generative capabilities of large language models (LLMs) with the ability to retrieve relevant information from external knowledge bases, thereby producing more accurate and contextually relevant responses or text content.
By leveraging RAG techniques, key issues inherent in large models such as outdated knowledge, limitations in contextual understanding, and uncertainty of information sources can be addressed.
This section provides three RAG deployment cases that can be deployed on EC-A1684JD4 FD, namely FireflyChat, ChatDoc-TPU and LangChain-Chatchat-TPU.
FireflyChat project introduction
FireflyChat is a graphical application platform for LLM developed by the Firefly team. It requires only simple installation with no need for compilation, allowing for quick experience of the enhancement that RAG brings to LLM.
For details, see FireflyChat.
ChatDoc-TPU project introduction
ChatDoc-TPU is a fully localized inference document conversation tool whose main goal is to simplify interactions with documents and extract valuable information by using natural language.
Project repository link: ChatDoc-TPU
LangChain-Chatchat-TPU project introduction
LangChain-Chatchat-TPU is a fully localized inference knowledge base enhancement scheme based on Langchain-Chatchat.
Project repository link: LangChain-Chatchat-TPU
TTS and ASR
VITS-TPU project introduction
VITS is an end-to-end text-to-speech (TTS) model. The VITS-TPU project implements TPU algorithm porting to VITS.
Project repository link: VITS-TPU
Whisper-UI-TPU project introduction
Whisper is an open source speech recognition (ASR) model developed by OpenAI that enables real-time, multilingual speech recognition.
The Whisper-UI-TPU implements TPU algorithm porting to Whisper, and also provides an easy-to-use WebUI.
Project repository link: Whisper-UI-TPU
Image content retrieval
CLIP-TPU project introduction
CLIP, which stands for Constrastive Language-Image Pre-training, is a text-image pre-training model implemented by OpenAI using contrast learning. CLIP has a very impressive performance in zero-shot text-image retrieval, zero-shot image classification, Vincennes chart task guidance, open-domain detection segmentation and other tasks.
The CLIP-TPU project implements the TPU algorithm transplantation of CLIP, and users can use this project to verify the actual performance of CLIP on EC-A1684JD4 FD.
Project repository link: CLIP-TPU
ImageSearch-tpu project introduction
ImageSearch-tpu project realizes the function of searching a large number of images based on CLIP, and provides a visual WebUI for users to use.
Project repository link: ImageSearch-tpu