Colibri, Pure-C Inference Engine Running 2.8T MoE on Consumer Hardware

The inference engine “Colibri (Colibrì)” by JustVugg, featured on GitHub Trending, demonstrates an attempt to run MoE models ranging from hundreds of billions to trillions of parameters on consumer and heterogeneous hardware. Coverage of JustVugg on GitHub Trending highlights running frontier MoE models ranging from 744B to 2.8T parameters on a single engine written in pure C with no dependencies.

Tiny engine, immense model. Run frontier MoE models — 744B to 2.8T parameters — on consumer and heterogeneous hardware, in pure C with zero engine dependencies

The project centers on “AI memory multitiering,” which treats storage, RAM, and VRAM as a single inference hierarchy. It aims for a state where access to giant models does not depend on the availability of hyperscaler-grade hardware.

Design for Running Giant MoE on Consumer Machines

Colibri is positioned both as an inference engine and as an open research platform. Its goal is to pursue inference performance across the entire software-hardware boundary, from model formats, memory hierarchy, storage I/O, placement, scheduling, kernels, and speculative execution to CPU-GPU overlap.

A defining design feature is treating VRAM, RAM, and storage as one integrated multi-tier hierarchy. Even when fast memory is insufficient, speed may degrade, but the principle is to never implicitly change model accuracy or router semantics. Experimental optimizations are adopted only when they show benefits in reproducible end-to-end measurements.

This policy reflects an emphasis on end-to-end inference results rather than partial speedup metrics. It does not preserve existing methods merely because they are conventional, nor adopt new methods merely because they look fast on micro-benchmarks.

Running on consumer machines is tied to reducing dependence on dedicated infrastructure. Execution in heterogeneous environments assumes a wide range of configurations, from client devices to workstations. The spread of devices balancing power efficiency and performance, such as recent Linux tablets, provides fertile ground for such distributed execution. As shown in Juno Tab 4 Announces Two Linux Tablets with Core i3 and Core Ultra, inference demand is expanding even on low-power x86 platforms.

Implementing 8 Models in Single C Files

Colibri publicly supports eight model families: GLM-5.2/5.3 (744B), GLM-5.3-Flash (321B, vision-capable), Inkling (975B), Kimi K3 (2.8T), DeepSeek V4 Flash (284B), Qwen3.8-Flash-Next (125B + 51B n-gram), Qwen3.6 (35B-A3B), and OLMoE (7B).

Each is implemented in one C file per model. The frontend is shared, unified under three operation modes: coli chat, coli serve, and coli web. The ability to switch between chat execution, server serving, and web display with the same operations reduces the learning burden at introduction.

The pure-C, dependency-free implementation improves portability and ease of verification. By eliminating dependence on external engines, it appears intended to suppress behavioral variations due to environment differences. The single-file structure aligns with an approach that abstracts model differences while applying individual optimizations locally.

Supported scale ranges broadly from 7B to 2.8T. Handling everything from the lightweight OLMoE to the ultra-large Kimi K3 within the same framework allows cross-cutting verification of memory-hierarchy management and expert-routing control.

Inference Approach and Design Integrating the

Memory Hierarchy

At its core, AI memory multitiering is an inference approach that treats VRAM, RAM, and storage as a single hierarchy. Its operational scope includes how weights are represented and moved, which data resides in which tier, overlapping heterogeneous computation, reducing launch and synchronization overhead, exploiting sparsity and reuse, and validating new decoding algorithms.

Weight placement is premised not on static allocation but on dynamic management according to runtime usage. In mixture-of-experts (MoE), sparsity — where only a subset of experts is activated per input — is essential. Combining this sparsity with the memory hierarchy makes the efficiency of selecting and swapping resident experts decisive for performance.

Storage I/O optimization is also a key element. Reads from NVMe and SSDs are overlapped with inference to hide latency. The design overlapping CPU and GPU processing works to reduce synchronization points and curb kernel-launch overhead.

Given its nature as an open research platform, no service-level guarantees on speed are stated. Guarantees on semantics, on the other hand, are strict. Default behavior never changes model accuracy without permission. When quantization or routing semantics are changed for speed, they are treated as explicit settings.

For graphics and compute foundations, open driver layers are maturing. Efforts such as Mesa Rusticl Enables Mali Panfrost by Default broaden execution environments on heterogeneous GPUs, reinforcing the foundation for low-dependency engines like Colibri to deploy across diverse hardware.

Mechanism for Visualizing Expert Routing

Colibri includes a web platform for visualizing internal inference state. The dashboard launched via ./coli web showed 4 tokens per second, a time to first token (TTFT) of 1.6 seconds, and zero disk usage when running a 744B model. This was measured with full expert residency secured on six RTX 5090 GPUs.

The dashboard displays per-token metrics, per-turn time breakdowns, and VRAM/RAM/disk tier bars. A small brain model is placed in the corner of the screen, allowing intuitive tracking of inference progress.

On the Brain page, 19,456 experts are rendered as a single cortex. Color indicates the memory tier where an expert resides, and brightness indicates routing heat. Experts routed within a turn glow white. Hovering over an individual expert shows its measured topic affinity.

On the Atlas page, the measured expert map is expressed as a 3D galaxy. Clusters of 13,260 characterized experts and 1,041 duplicated experts gather by topic. Experts can be seen clustering in areas such as poetry, law, Chinese, and SQL. Placement is based not on learned embeddings but on measured routing affinity. The view can be rotated by dragging.

Visualization contributes not only to performance checking but also to understanding model behavior. Observing which experts activate for which topics directly informs validation of placement and duplication strategies.

Clear Positioning and Principles as a

Research Platform

Colibri is defined both as an inference engine usable today and as a venue for testing experimental ideas. Its operational mission is to change how weights are represented and moved, decide what to place where across VRAM, RAM, and storage, overlap heterogeneous computation, reduce launch and synchronization overhead, exploit sparsity and reuse, and try new decoding algorithms.

The criterion is end-to-end inference performance. Local speedups on micro-benchmarks are not adopted unless they contribute to overall inference time and quality. Conversely, even conventional practices are subject to review if they harm overall performance.

The emphasis on reproducible measurement relates to its reliability as a research platform. Ensuring experimental optimizations do not change default accuracy or routing semantics guarantees comparability of results. The principle of never implicitly changing the speed-accuracy tradeoff lets users evaluate performance and quality separately.

Execution on consumer hardware broadens the user base. Just as high-performance gaming peripherals have penetrated the general market, inference infrastructure has entered a stage where practicality on consumer machines is tested. Product rollouts such as ASUS Launches First Gaming Soundbar “ROG Gjallar” in the US illustrate the increasing performance and diversification of consumer hardware.

Performance and Resident Footprint Shown by

Run Examples

In the published run example, Colibri v1.10.2 runs GLM-5.2’s 744B MoE in int4. It completes preparation in 32 seconds in streaming CPU mode, with resident footprint held to 9.9GB. The small residency at startup reflects a design that leverages the memory hierarchy to load needed parts incrementally.

In the chat run example, an Italian greeting is answered immediately after launching ./coli chat. Conversational responsiveness depends not only on TTFT and token generation speed but also on initial load speed. Ready in 32 seconds is positioned as a practical startup time for a 744B-scale model.

Meanwhile, the web dashboard example using six RTX 5090s showed 4 tokens per second with full residency and zero disk usage. By keeping experts resident in VRAM, performance under conditions excluding storage I/O can be confirmed. Where experts are placed in the memory tiers shifts the balance between speed and resource use.

The two run examples show the same engine operating under different resource conditions. The ability to switch from lightweight CPU-centric residency to full residency across multiple GPUs with the same operation system eases development and validation iterations.

Editorial Opinion

In the short term, we see inference engines like Colibri, built in pure C and premised on multi-tier memory, expanding verification environments for researchers and small organizations. The ability to handle 744B to 2.8T-scale MoE on consumer and heterogeneous configurations could shift the focus from cloud-exclusive evaluation to hands-on reproducible verification. Within a 3 to 6 month horizon, we assess that reproducible experiments utilizing expert-routing visualization and placement control, as well as comparative validation of quantization and speculative decoding, will increase. In the long term, the design philosophy of treating VRAM, RAM, and storage as one unit may become incorporated as a standard option for inference infrastructure. As model scale continues to grow, configurations dependent on the absolute amount of fast memory will intensify resource constraints. Approaches that leverage sparsity premised on a multi-tier hierarchy are seen as sustainable from the perspectives of operating cost and power consumption. In 1 to 3 years, inference optimization is expected to shift from single-kernel speedups to systematic optimization including placement and scheduling. The remaining question is how strictly the separation of speed and semantics can be maintained. The guarantee of not changing accuracy or routing semantics by default supports reproducibility, while also narrowing room for performance gains.

References

• “JustVugg / colibri”, by JustVugg — GitHub Trending, 2026-09-14 (ARR)
• Source URL: https://github.com/JustVugg/colibri