jundot /omlx

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

★ 23,653⑂ 2,134👁 113⑃ 2,570● 2,393▤ 138
PythonApache-2.0Created 2026-02-13omlx.ai
下载:0.8.0rc1v0.8.0rc1 ↓12026-11-12
+19
日增长
+325
周增长
+789
月增长
23,653
stars
2,134
forks
113
watchers
2,570
prs
2,393
issues
138
releases

README

oMLX
LLM inference, optimized for your MacContinuous batching and tiered KV caching, managed directly from your menu bar.


  
  
  



  [email protected] · https://omlx.ai/me



  Install ·
  Quickstart ·
  Features ·
  Models ·
  CLI Configuration ·
  Benchmarks ·
  oMLX.ai



  English ·
  中文 ·
  한국어 ·
  日本語


---


  


 Every LLM server I tried made me choose between convenience and control. I wanted to pin everyday models in memory, auto-swap heavier ones on demand, set context limits - and manage it all from a menu bar.

 oMLX persists KV cache across a hot in-memory tier and cold SSD tier - even when context changes mid-conversation, all past context stays cached and reusable across requests, making local LLMs practical for real coding work with tools like Claude Code. That's why I built it.

 Install

 macOS App

Download the .dmg from Releases, drag to Applications, done. The app includes in-app auto-update, so future upgrades are just one click. The macOS app also installs a lightweight ~/.omlx/bin/omlx CLI shim so terminal commands and Apple Shortcuts can control the app-managed server.

 Homebrew

bash
brew tap jundot/omlx https://github.com/jundot/omlx
brew install jundot/omlx/omlx --with-custom-kernel    With native custom kernels (needs full Xcode)
 No full Xcode? brew install jundot/omlx/omlx installs without the kernels

 Upgrade to the latest version
brew update && brew upgrade omlx

 Run as a background service (auto-restarts on crash)
omlx start

 Optional: MCP (Model Context Protocol) support, then restart to load it
/opt/homebrew/opt/omlx/libexec/bin/pip install mcp
omlx restart


 From Source

bash
git clone https://github.com/jundot/omlx.git
cd omlx
make install                 Editable install with the web UI and native custom kernels
 No full Xcode? make install-no-kernels installs without the kernels
make mcp                     Optional: MCP support, restart omlx serve to load it


Requires macOS 15.0+ (Sequoia), Python 3.11–3.13, and Apple Silicon (M1/M2/M3/M4/M5).

 Note on native custom kernels: GLM-5.2, MiniMax M3, and Qwen3.5 need them.
 Without them those families silently fall back to much slower generic paths --
 for GLM-5.2 the fused DSA prefill is roughly 30x faster with the kernels
 (measured 845 vs ~29 tok/s on an M3 Ultra), and the fallback also uses more
 memory (2137). make install builds them and checks that each one loads. That
 needs full Xcode with the Metal toolchain (xcodebuild -downloadComponent
 MetalToolchain); Command Line Tools alone do not provide it (xcrun: error:
 unable to find utility "metal"). Without Xcode, use the official DMG, which
 ships the kernels precompiled, or make install-no-kernels. A plain
 pip install -e . also skips them. make kernels rebuilds only the kernels
 from scratch. Homebrew builds them with --with-custom-kernel, which also needs
 full Xcode. To verify any install:

 bash
 python -c "from omlx.custom_kernels import native_kernel_status; print(native_kernel_st

Top issues

Discussions

Latest release

0.8.0rc1 v0.8.0rc1 2026-11-12
oMLX 0.8.0rc1

 This is a release candidate. Please report any bugs through GitHub Issues. If you need a stable release, please use oMLX 0.7.3.

The release candidate for oMLX 0.8.0 adds the Prefill Accelerator, which lets a PC with NVIDIA GPUs or a second Mac speed up prefill for a big model on your Mac, and makes DeepSeek V4.1 faster with many requests and on clusters.

Download: macOS 26 / 27 | macOS 15 Sequoia
 If you downloaded build 3727 right after release, the Cluster tab may not find a DGX Spark or NVIDIA PC, so please download it again.



 Prefill Accelerator with NVIDIA 

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON