Tested 4B, 8B and 14B models. (14B seemed to handle this better than 4B but I’m posting 4B results)
Based on your description and the attached images, Project 42 achieves significant GPU KV-cache memory reduction (up to ~80x stored, ~34x peak live) compared to FP16, with only a minor drop in quality (mean score: 96.15 vs. 96.67) and a slight increase in mean query time, but higher generation throughput (20.40 vs. 18.36 tokens/s) on Qwen3-4B-Instruct-2507 with an RTX 5070. These results are self-reported and not yet independently validated.
No official vLLM documentation or benchmarking currently references “Project 42” or its specific KV-cache compression method, so its implementation and compatibility with vLLM are not confirmed in the provided data sources. For validated KV-cache quantization and memory reduction methods in vLLM (e.g., FP8, NVFP4, INT4), see official vLLM docs and recent PRs.
Would you like a detailed breakdown of vLLM’s supported KV-cache quantization methods and their benchmarked performance?
Sources:
Hint: Mention @Herald in the post for followups.