Kimi K3 Model Runs on 8GB Memory Devices, Developers Open Source Lightweight C Inference Engine

Developers have open-sourced the project kimi-k3-in-c, aimed at enabling the Kimi K3 model, which has 2.78 trillion parameters, to run on devices with only 8GB of memory. The project is 176KB in size and is written entirely in C99, without relying on GPU, CUDA, PyTorch, or BLAS, completing model inference solely through the CPU. This solution leverages the MoE (Mixture of Experts) architecture feature of Kimi K3, where although the total parameter scale of the model reaches 2.78T, only 16 out of 896 experts in each layer are activated. Therefore, the complete model weights of approximately 1.56TB are not loaded into memory; instead, most expert weights are stored on NVMe drives and read in real-time according to inference needs. Additionally, some dense layers are also loaded in a layer-by-layer streaming manner. Under the 8GB memory mode, the model generates one token in about 32.7 seconds and requires nearly 1.7TB of high-speed storage space to support it. The developers state that this solution is currently more of an experimental exploration of optimizing the infrastructure for large model inference and does not possess practical production value, but it provides a new idea for running ultra-large-scale models at low cost in the future.