AI Tools Radar › Categories › LLM inference & serving
LLM inference engines, serving & gateways
Inference engines, model serving, quantization, LLM gateways and proxies, model routing and API endpoints. 36 items in today's radar (data fetched 21 Oct 2026, 14:13 (Berlin time)). Categories are assigned by fixed keyword rules on GitHub topics, repo names, descriptions and Show HN titles; an item can be in several categories. How categories work.
All 393Agents & automation 196Coding assistants & dev tools 168Chat apps & assistants 28LLM inference & serving 36RAG & search 28Training & fine-tuning 11Models & research 36Image, video & audio generation 52Voice & speech 21Computer vision 8MCP, skills & plugins 146Evaluation & observability 22Data & datasets 30Local / on-device AI 45Security & sandboxing 33Learning & resources 48Other 16
Trending on GitHub today (6)
| Repository | Stars today | Total stars | Language | License | Created |
|---|---|---|---|---|---|
| 1. Comfy-Org/ComfyUI The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world. | 1,206 | 141,213 | Python | GPL-3.0 | 2023-01-17 |
| 2. tashfeenahmed/freellmapi 7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only. | 684 | 38,646 | TypeScript | MIT | 2026-04-21 |
| 3. lyogavin/airllm AirLLM 2.8T LLM inference with single 4GB GPU | 386 | 40,040 | Jupyter Notebook | Apache-2.0 | 2023-06-12 |
| 4. vllm-project/vllm A high-throughput and memory-efficient inference and serving engine for LLMs | 128 | 94,666 | Python | Apache-2.0 | 2023-02-09 |
| 5. trymirai/uzu A high-performance inference engine for AI models | 15 | 2,465 | Rust | MIT | 2025-06-23 |
| 6. facebookresearch/sam2 The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model. | 10 | 20,032 | Jupyter Notebook | Apache-2.0 | 2024-07-29 |
Trending on GitHub this week (11)
| Repository | Stars this week | Total stars | Language | License | Created |
|---|---|---|---|---|---|
| 1. debpalash/VoiceStudio VoiceStudio is the open-source, fully-local ElevenLabs alternative — voice cloning, voice design, video dubbing, dictation, transcription & audiobook creation in 646 languages. | 17,057 | 79,695 | Python | AGPL-3.0 | 2026-04-09 |
| 2. tashfeenahmed/freellmapi 7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only. | 3,710 | 38,646 | TypeScript | MIT | 2026-04-21 |
| 3. Comfy-Org/ComfyUI The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world. | 3,364 | 141,213 | Python | GPL-3.0 | 2023-01-17 |
| 4. lyogavin/airllm AirLLM 2.8T LLM inference with single 4GB GPU | 1,694 | 40,040 | Jupyter Notebook | Apache-2.0 | 2023-06-12 |
| 5. lightseekorg/tokenspeed TokenSpeed is a speed-of-light LLM inference engine. | 503 | 2,737 | Python | MIT | 2026-05-06 |
| 6. gitleaks/gitleaks Find secrets with Gitleaks 🔑 | 278 | 30,185 | Go | MIT | 2018-01-27 |
| 7. trymirai/uzu A high-performance inference engine for AI models | 175 | 2,465 | Rust | MIT | 2025-06-23 |
| 8. UFund-Me/Qbot [🔥updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. 📃 online docs: https://ufund-me.github.io/Qbot ✨ :news: qbot-mini: https://github.com/Charmve/iQuant | 88 | 18,712 | Jupyter Notebook | MIT | 2022-11-23 |
| 9. facebookresearch/segment-anything The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model. | 68 | 55,039 | Jupyter Notebook | Apache-2.0 | 2023-03-23 |
| 10. facebookresearch/sam2 The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model. | 39 | 20,032 | Jupyter Notebook | Apache-2.0 | 2024-07-29 |
| 11. llm-d/llm-d-router llm-d Router: The intelligent entry point for inference requests | 25 | 428 | Go | Apache-2.0 | 2025-05-05 |
New repos of the last 30 days (21)
| Repository | Age | Total stars | Language | License | Created |
|---|---|---|---|---|---|
| 1. Ebony-Vinyl/dsh-our-free-model 在 dsh 里装上这个插件即可，无需登录、注册或填 API Key，就能使用包括 DeepSeek V4.1 Flash、Kimi K3 在内的前沿模型——完全免费，不限量。 All you do is install this plugin in dsh: no login, no sign-up, no API key — the frontier models are just there, DeepSeek V4.1 Flash and Kimi K3 among them. Completely free, with no usage cap. | 26 d | 32,653 | JavaScript | MIT | 2026-09-24 |
| 2. Niko1221/Strata Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input. | 26 d | 29,588 | C++ | MIT | 2026-09-24 |
| 3. deepseek-ai/DeepInfer DeepInfer: High-performance, large-scale inference framework for DeepSeek models | 5 d | 16,691 | Python | Apache-2.0 | 2026-10-16 |
| 4. ARahim3/DigUp Search your computer by what's in your files: photos, PDFs, documents, recordings, videos and code. On-device on Mac, Windows and Linux, built on Google DeepMind's EmbeddingGemma 2. | 12 d | 5,295 | Swift | MIT | 2026-10-08 |
| 5. onehumanbeing/esp32-iphone-agent Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vision model. No Mac, no developer account, no jailbreak. | 9 d | 3,457 | C | MIT | 2026-10-11 |
| 6. mw00/project-maya Project Maya - run GLM-5.3-Flash (321B MoE) and DeepSeek-V4.1-Flash-1017 (748B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools. | 14 d | 1,350 | C++ | MIT | 2026-10-06 |
| 7. StayLameBro/backburner Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable | 19 d | 1,092 | Python | MIT | 2026-10-01 |
| 8. StartLuxLabs/StartLux-Decision Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX. | 22 d | 1,073 | Python | Apache-2.0 | 2026-09-29 |
| 9. datalab-to/indra Fast CPU OCR: line detection, CTC recognition, reading order, markdown | 8 d | 873 | Python | Apache-2.0 | 2026-10-13 |
| 10. Zfinix/eleven Agent-first reverse engineering for macOS, Windows and Linux: any ACP agent reads, renames and patches binaries, Android apps and GPU kernels next to you. | 7 d | 816 | TypeScript | Apache-2.0 | 2026-10-13 |
| 11. cobanov/PS5LM llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8 | 19 d | 621 | C++ | GPL-3.0 | 2026-10-01 |
| 12. eagle-hq/EAGLE-4 Official Inference Code of EAGLE-4 | 0 d | 593 | Python | Other | 2026-10-20 |
| 13. hanzohanzhe/Grandet Find the cheapest API for any model and connect it from your AI agent: Grandet Skill and an anonymous MCP server, backed by listed prices from 1,800+ official, inference and relay providers on grandet.ai. | 16 d | 558 | JavaScript | Other | 2026-10-05 |
| 14. kanaworksai-hue/audio-studio-mcp Local AI BGM and sound effects for Claude Code and any MCP client. Stable Audio 3 on Apple Silicon (MLX) and Windows NVIDIA GPUs (PyTorch). No cloud, no API key. | 3 d | 397 | Python | MIT | 2026-10-18 |
| 15. VecSzn/PolyStrata Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost. | 14 d | 324 | C++ | MIT | 2026-10-06 |
| 16. Chengjun023/agent-smith Token 超支？快给史密斯打电话！ Better Call Smith. Adaptive model routing, a glassy Codex usage monitor, and reproducible benchmarks with receipts. | 20 d | 301 | Python | MIT | 2026-09-30 |
| 17. HRuiCcc/RuiC-oc-ferry 把 OpenCode 的免费模型摆渡给 WorkBuddy —— 隔离运行时、并发检测、判定缓存、一键导入、本机 OpenAI 兼容转发口 | 11 d | 300 | JavaScript | MIT | 2026-10-10 |
| 18. tswawa/WechatVibe 微信聊天分析工具，支持本地 Laya 与 API 模型，提供意图识别、情绪感知、人物画像、群聊画像、好感度分析和 MBTI 聊天推测。 | 28 d | 283 | Python | Apache-2.0 | 2026-09-23 |
| 19. Indras-Mirror/Xolotl Xolotl (formerly Strata-DSV4.1): DeepSeek-V4.1-Flash, all 552B, on one gaming PC (RTX 4090 + 90 GB RAM) at ~27 tok/s, plus GLM-5.3-Flash, Qwen3.8-Flash-Next and Qwen3.8-27B. A heavily extended fork of Strata: MoE experts tiered across VRAM, pinned RAM and NVMe; OpenAI + Anthropic API. | 14 d | 262 | C++ | MIT | 2026-10-06 |
| 20. MemTensor/MemT-1.0 Official repository for MemT-1.0 | 2 d | 251 | Python | Other | 2026-10-19 |
| 21. raullenchai/esp32-thinks A multi-model Muse: personal AI voice assistant on ESP32 - wake word or knob, any OpenAI-compatible brain (STT/chat/TTS mix-and-match), streaming replies, out of the box. Built on muse-gadget-sdk. | 8 d | 236 | Shell | MIT | 2026-10-12 |
AI launches on Show HN (3)
| Show HN post | Points | Comments | Posted |
|---|---|---|---|
| 1. Show HN: Open-source model routing for coding agents at Astra-level performance HN discussion · by adchurch | 123 | 42 | 2026-09-30 |
| 2. Show HN: Agentic CUDA Kernel Optimizer HN discussion · by bertaye | 37 | 10 | 2026-09-25 |
| 3. Show HN: Jevper – the Jev interface on top of any OpenAI-compatible model HN discussion · by czl_my | 11 | 4 | 2026-09-23 |