quantization

Models and training scripts for "LSTMs for Keyword Spotting with ReRAM-based Compute-In-Memory Architectures" (ISCAS 2021).

a platform for monitoring the chip situation

Evidence-backed structural validation of Kimi K3 UD-IQ1_M and UD-Q4_K_XL split GGUF releases using OMIV.

Deploy Qwen3.6-35B-A3B (Q4_K_XL) + MTP speculative decoding on a single NVIDIA L4 24GB — GCP g2-standard-8 — via the official llama.cpp Docker image. Decode-optimized to ~91–99 tok/s (min ~91 chat, max ~99 math), lossless (full GPU residency + ECC-off).

GLM-5.2 QuantTrio (Unpruned ) on 4xDGX Spark (GB10): 327K-655K context or up to 5 concurrent agents, one receipe. TP4 + DCP + MTP

Whisper speech-to-text server installer for Ubuntu, Debian, AlmaLinux, Rocky Linux, CentOS, RHEL and Fedora. OpenAI-compatible transcription and translation APIs powered by faster-whisper. Supports all Whisper models, word-level timestamps, JSON/SRT/VTT output, SSE streaming and offline mode.

A pretty way to compress images

"Vitis-AI-YOLOv3-TF2-Quantization-Evaluation" Repo for quantization of YOLOv3 on Vitis-AI using TF2, aimed to deploy model on edge devices with limited resources. Includes training & quantization scripts and evaluation metrics. Experiment with different configurations.

Field-tested guide: multi-GPU vLLM tensor-parallel (TP=2/TP=4) on Intel Arc Pro B70 (Battlemage BMG-G31, Xe2) on Linux. Driver setup (xe force_probe=e223), bare-metal vLLM + oneAPI 2025.3, the compute-runtime multi-root USM + triton-xpu init_devices fixes, FP8/int4-AutoRound quant, root-cause error reports. AI-agent readable (AGENTS.md).

Qwen3.6-27B on one RTX 5090 — 200K context, 13.5K t/s prefill, 80–160 t/s decode (MTP spec-decode), ~2.6M tokens of restart-persistent tiered KV, SWE-Bench Verified 69.4%. W4A4 NVFP4 + vLLM + LMCache, benchmarked and reproducible.

Run PrismML Bonsai 27B (1-bit) LLM on NVIDIA Jetson Orin NX 16GB with CUDA-accelerated llama.cpp

Build, run, and setup scripts for the complete TensorRT-LLM pipeline on RTX A6000 Ada (SM89). Reproducible path from HuggingFace checkpoint to deployable .engine file, with FP16 baseline and FP8 quantization. Companion material to the 4-part blog series on ai-box.eu — in preparation for the NVIDIA TensorRT Edge-LLM ecosystem.

Coding assistant is a lightweight llama.cpp wrapper for quantized local SLM deployment

Optimized Ollama model variants with capped context (64K) — faster inference, lower RAM, minimal quality loss. Runs 9B models on 8-16 GB machines.

GGUF model for Cohere 30B A3B optimized for Tesla P40 deployment

Custom vLLM serving recipes, GPU-memory admission gating, and runtime mods for running LLMs on an NVIDIA DGX Spark (GB10/sm_121), layered on eugr/spark-vllm-docker.

Tools for 16gb NVIDIA benchmarks

LLM inference with 7x KV cache compression. Combines llama.cpp (production inference engine) with TurboQuant (KV quantization). Run 131K token context on 16GB VRAM. OpenAI-compatible API server. Supports 100+ model architectures.

Ultra-light Q1_S quant of GLM-5.2 MoE model optimized for Tesla P40 with llama.cpp

vLLM serving stack for Gemma 4 31B on RTX PRO 6000 Blackwell, with FP8 KV cache, MTP speculative decoding, and an async FastAPI logging proxy in front.