Does vLLM native KV Cache offloading to CPU work for gemm4?

Model Support
1

I am using vllm/vllm-openai:gemma image to run gemma4-31b-it inference on G4 machine in Google Cloud. I am trying to offload the KV Cache to CPU by using this configuration:

```
{“kv_connector”: “OffloadingConnector”,“kv_role”: “kv_both”,“kv_connector_extra_config”: {“cpu_bytes_to_use”: 128849018880, “blocks_per_chunk”: 8, “num_cpu_blocks”: 4468}}
```

The model server starts successfully. However, when i check the memory use of the container, it doesn’t indicate that the offloading worked. Here the snippet of free -h

```
total used free shared buff/cache availableMem:
Mem: 176Gi 9.0Gi 132Gi 3.8Gi 35Gi 162Gi
Swap: 0B 0B 0B
```

The log suggests that the KV configs are not even being passed while vLLM engine is being initialized.

```
(EngineCore pid=209) INFO 07-28 15:11:28 [core.py:105] Initializing a V1 LLM engine (v0.19.1.dev6+g6d4a8e6d2) with config: model=‘/root/.cache/vllm/assets/model_streamer/5fb74889’, speculative_config=None, tokenizer=‘/root/.cache/vllm/assets/model_streamer/5fb74889’, skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=28992, download_dir=None, load_format=runai_streamer, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend=‘auto’, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=‘’, reasoning_parser_plugin=‘’, enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=google/gemma-4-31b-it, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={‘mode’: <CompilationMode.VLLM_COMPILE: 3>, ‘debug_dump_path’: None, ‘cache_dir’: ‘’, ‘compile_cache_save_format’: ‘binary’, ‘backend’: ‘inductor’, ‘custom_ops’: [‘none’], ‘splitting_ops’: [‘vllm::unified_attention’, ‘vllm::unified_attention_with_output’, ‘vllm::unified_mla_attention’, ‘vllm::unified_mla_attention_with_output’, ‘vllm::mamba_mixer2’, ‘vllm::mamba_mixer’, ‘vllm::short_conv’, ‘vllm::linear_attention’, ‘vllm::plamo2_mamba_mixer’, ‘vllm::gdn_attention_core’, ‘vllm::olmo_hybrid_gdn_full_forward’, ‘vllm::kda_attention’, ‘vllm::sparse_attn_indexer’, ‘vllm::rocm_aiter_sparse_attn_indexer’, ‘vllm::unified_kv_cache_update’, ‘vllm::unified_mla_kv_cache_update’], ‘compile_mm_encoder’: False, ‘cudagraph_mm_encoder’: False, ‘encoder_cudagraph_token_budgets’: [], ‘encoder_cudagraph_max_images_per_batch’: 0, ‘compile_sizes’: [], ‘compile_ranges_endpoints’: [8192], ‘inductor_compile_config’: {‘enable_auto_functionalized_v2’: False, ‘size_asserts’: False, ‘alignment_asserts’: False, ‘scalar_asserts’: False, ‘combo_kernels’: True, ‘benchmark_combo_kernel’: True}, ‘inductor_passes’: {}, ‘cudagraph_mode’: <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, ‘cudagraph_num_of_warmups’: 1, ‘cudagraph_capture_sizes’: [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], ‘cudagraph_copy_inputs’: False, ‘cudagraph_specialize_lora’: True, ‘use_inductor_graph_partition’: False, ‘pass_config’: {‘fuse_norm_quant’: False, ‘fuse_act_quant’: False, ‘fuse_attn_quant’: False, ‘enable_sp’: False, ‘fuse_gemm_comms’: False, ‘fuse_allreduce_rms’: False}, ‘max_cudagraph_capture_size’: 512, ‘dynamic_shapes_config’: {‘type’: <DynamicShapesType.BACKED: ‘backed’>, ‘evaluate_guards’: False, ‘assume_32_bit_indexing’: False}, ‘local_cache_dir’: None, ‘fast_moe_cold_start’: True, ‘static_all_moe_layers’: []}"
```

When i run qwen3-32b with regular vllm image(not the one tagged with gemma4), the memory usage indicates that offloading has been established, see snippet below

```
total used free shared buff/cache available

Mem: 176Gi 9.3Gi 1.0Gi 128Gi 166Gi 37Gi
Swap: 0B 0B 0B
````

Also, the logs show that the KV Cache offloading configurations is being passed while vLLM engine is being initialized.

I am not sure if KV Cache offloading has been implemented for gemm4. Looking to get some feedback here.

Thanks

2

KV cache offloading for Gemma 4 models in vLLM is not fully supported or stable as of v0.19.1 and recent releases. Multiple users report that native CPU offloading (e.g., via OffloadingConnector or --kv-offloading-backend native) either fails to initialize, causes crashes, or does not actually offload memory for Gemma 4, even when the configuration is set correctly. This is a known limitation and not a misconfiguration on your part—see the discussion and crash logs in this issue.

In contrast, Qwen3 and other models do show working KV offloading, as confirmed by both memory usage and engine logs. For Gemma 4, the logs and memory profile you observe are expected, and there is no workaround documented in the official vLLM or Gemma 4 recipes. If you need working KV offload, you must use a model and vLLM image combination that is confirmed to support it. Would you like a step-by-step breakdown of the technical reasons and possible future workarounds?

Sources:

Hint: Mention @Herald in the post for followups.

3

yes, please provid emore clarification. @Herald

4

Certainly! Here’s a step-by-step clarification based on the data sources:

  1. Gemma 4 KV Offloading Status:
    Native CPU KV cache offloading (using OffloadingConnector or --kv-offloading-backend native) is not reliably supported for Gemma 4 models in vLLM as of v0.19.1 and later. Users have reported that enabling these features either causes the server to crash, does not offload memory as expected, or the configuration is ignored entirely. This is confirmed in this issue, where attempts to use offloading with Gemma 4 result in segfaults or no effect.

  2. Why It Works for Qwen3 but Not Gemma 4:
    For models like Qwen3, the same offloading configuration works as intended: memory usage drops and logs show the offloading connector is active. For Gemma 4, the vLLM engine does not pass the offloading configuration to the core engine, and memory usage remains high, indicating that the offloading buffer is not being used. This is a model-specific limitation, not a user error.

  3. No Documented Workaround:
    There is no official workaround for enabling KV offloading on Gemma 4 in the current vLLM releases. The Gemma 4 recipe does not mention offloading as a supported feature, and user reports confirm the limitation.

  4. What to Do:
    If you require KV offloading, you must use a model and vLLM version where this feature is confirmed to work (e.g., Qwen3). For Gemma 4, you may need to wait for future vLLM updates or track the relevant GitHub issues for progress.

Let me know if you need more technical details or guidance on alternative configurations.

Sources:

Hint: Mention @Herald in the post for followups.