(APIServer pid=17) INFO 08-22 18:39:39 [api_utils.py:345]
(APIServer pid=17) INFO 08-22 18:39:39 [api_utils.py:345] █ █ █▄ ▄█
(APIServer pid=17) INFO 08-22 18:39:39 [api_utils.py:345] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.26.0
(APIServer pid=17) INFO 08-22 18:39:39 [api_utils.py:345] █▄█▀ █ █ █ █ model /home/luozhong/model-DeepSeek-V4-Flash-0731
(APIServer pid=17) INFO 08-22 18:39:39 [api_utils.py:345] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=17) INFO 08-22 18:39:39 [api_utils.py:345]
(APIServer pid=17) INFO 08-22 18:39:39 [api_utils.py:273] non-default args: {‘enable_auto_tool_choice’: True, ‘tool_call_parser’: ‘deepseek_v4’, ‘model’: ‘/home/luozhong/model-DeepSeek-V4-Flash-0731’, ‘tokenizer_mode’: ‘deepseek_v4’, ‘trust_remote_code’: True, ‘dtype’: ‘bfloat16’, ‘max_model_len’: 65536, ‘reasoning_parser’: ‘deepseek_v4’, ‘block_size’: 256, ‘kv_cache_memory_bytes’: 6510182399, ‘kv_cache_dtype’: ‘fp8’, ‘offload_backend’: ‘uva’, ‘cpu_offload_gb’: 75.0, ‘max_num_batched_tokens’: 4096, ‘max_num_seqs’: 8, ‘enable_flashinfer_autotune’: False, ‘moe_backend’: ‘marlin’, ‘spec_method’: ‘dspark’, ‘spec_model’: ‘/home/luozhong/model-DeepSeek-V4-Flash-0731’, ‘spec_tokens’: 7}
(APIServer pid=17) WARNING 08-22 18:39:39 [envs.py:2096] Unknown vLLM environment variable detected: VLLM_FORCE_DSV4_DECODE_KERNEL
(APIServer pid=17) INFO 08-22 18:39:39 [config.py:776] Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM.
(APIServer pid=17) INFO 08-22 18:39:39 [model.py:623] Resolved architecture: DeepseekV4ForCausalLM
(APIServer pid=17) INFO 08-22 18:39:39 [model.py:1788] Using max model len 65536
(APIServer pid=17) INFO 08-22 18:39:40 [cache.py:285] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=17) INFO 08-22 18:39:40 [model.py:623] Resolved architecture: DeepSeekV4MTPModel
(APIServer pid=17) INFO 08-22 18:39:40 [model.py:1788] Using max model len 1048576
(APIServer pid=17) INFO 08-22 18:39:40 [speculative.py:1126] Overriding draft model max model len from 1048576 to 65536
(APIServer pid=17) INFO 08-22 18:39:40 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=4096.
(APIServer pid=17) INFO 08-22 18:39:40 [vllm.py:1109] Asynchronous scheduling is enabled.
(APIServer pid=17) INFO 08-22 18:39:40 [kernel.py:295] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=[‘native’], fused_add_rms_norm=[‘native’])
(APIServer pid=17) WARNING 08-22 18:39:40 [vllm.py:1718] max_num_scheduled_tokens is set to 4048 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens or max_num_seqs.
(APIServer pid=17) WARNING 08-22 18:39:40 [cuda.py:334] --cpu-offload-gb is enabled with CUDA graphs on WSL2. This combination requires pinned (page-locked) memory allocations. WARNING: Windows (WDDM) enforces a hard system-wide cap of roughly 50%% of physical RAM on pinned memory shared across ALL processes by default (limit can changed via %%USERPROFILE%%.wslconfig). Excessive use of page-locked memory can prevent Windows from reclaiming memory under load, which can cause the entire host OS to become unresponsive and may require a hard reboot to recover. Proceed at your own risk. To raise the WSL2 VM memory ceiling, increase the memory setting in %%USERPROFILE%%.wslconfig and run wsl --shutdown.
(APIServer pid=17) WARNING 08-22 18:39:40 [vllm.py:2228] Model Runner V2 does not yet support the thinking_token_budget request parameter. Set VLLM_USE_V2_MODEL_RUNNER=0 if this is required.
(APIServer pid=17) INFO 08-22 18:39:40 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=70) INFO 08-22 18:39:45 [core.py:116] Initializing a V1 LLM engine (v0.26.0) with config: model=‘/home/luozhong/model-DeepSeek-V4-Flash-0731’, speculative_config=SpeculativeConfig(method=‘dspark’, model=‘/home/luozhong/model-DeepSeek-V4-Flash-0731’, num_spec_tokens=7), tokenizer=‘/home/luozhong/model-DeepSeek-V4-Flash-0731’, skip_tokenizer_init=False, tokenizer_mode=deepseek_v4, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=deepseek_v4_fp8, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend=‘auto’, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=‘deepseek_v4’, reasoning_parser_plugin=‘’, enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode=‘warn’, jit_monitor_verbose=False), seed=0, served_model_name=/home/luozhong/model-DeepSeek-V4-Flash-0731, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={‘mode’: <CompilationMode.VLLM_COMPILE: 3>, ‘debug_dump_path’: None, ‘cache_dir’: ‘’, ‘compile_cache_save_format’: ‘binary’, ‘backend’: ‘inductor’, ‘custom_ops’: [‘+quant_fp8’, ‘none’, ‘+quant_fp8’], ‘ir_enable_torch_wrap’: True, ‘splitting_ops’: [‘vllm::unified_attention_with_output’, ‘vllm::unified_mla_attention_with_output’, ‘vllm::mamba_mixer2’, ‘vllm::mamba_mixer’, ‘vllm::short_conv’, ‘vllm::linear_attention’, ‘vllm::plamo2_mamba_mixer’, ‘vllm::qwen_gdn_attention_core’, ‘vllm::gdn_attention_core_xpu’, ‘vllm::olmo_hybrid_gdn_full_forward’, ‘vllm::kda_attention’, ‘vllm::sparse_attn_indexer’, ‘vllm::rocm_aiter_sparse_attn_indexer’, ‘vllm::deepseek_v4_attention’, ‘vllm::hpc_rope_norm_forward’, ‘vllm::unified_kv_cache_update’, ‘vllm::unified_mla_kv_cache_update’], ‘compile_mm_encoder’: False, ‘cudagraph_mm_encoder’: False, ‘encoder_cudagraph_token_budgets’: , ‘encoder_cudagraph_max_vision_items_per_batch’: 0, ‘encoder_cudagraph_max_frames_per_batch’: None, ‘compile_sizes’: , ‘compile_ranges_endpoints’: [4096], ‘inductor_compile_config’: {‘enable_auto_functionalized_v2’: False, ‘size_asserts’: False, ‘alignment_asserts’: False, ‘scalar_asserts’: False, ‘combo_kernels’: True, ‘benchmark_combo_kernel’: True}, ‘inductor_passes’: {}, ‘cudagraph_mode’: <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, ‘cudagraph_num_of_warmups’: 1, ‘cudagraph_capture_sizes’: [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], ‘cudagraph_copy_inputs’: False, ‘cudagraph_specialize_lora’: True, ‘use_inductor_graph_partition’: False, ‘pass_config’: {‘fuse_norm_quant’: True, ‘fuse_act_quant’: True, ‘fuse_attn_quant’: False, ‘enable_sp’: False, ‘fuse_gemm_comms’: False, ‘fuse_allreduce_rms’: False, ‘enable_qk_norm_rope_fusion’: False, ‘fuse_rope_kvcache_cat_mla’: False, ‘fuse_act_padding’: False, ‘fuse_qk_norm_rope_kvcache’: False}, ‘max_cudagraph_capture_size’: 128, ‘dynamic_shapes_config’: {‘type’: <DynamicShapesType.BACKED: ‘backed’>, ‘evaluate_guards’: False, ‘assume_32_bit_indexing’: False}, ‘local_cache_dir’: None, ‘fast_moe_cold_start’: False, ‘static_all_moe_layers’: }, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=[‘native’], fused_add_rms_norm=[‘native’]), enable_flashinfer_autotune=False, enable_cutedsl_warmup=True, enable_bf16x3_router_gemm=False, moe_backend=‘marlin’, linear_backend=‘auto’)
(EngineCore pid=70) INFO 08-22 18:39:45 [parallel_state.py:1615] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.35.229:60751 backend=nccl
(EngineCore pid=70) INFO 08-22 18:39:45 [parallel_state.py:1946] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=70) INFO 08-22 18:39:45 [gpu_worker.py:378] Using V2 Model Runner
(EngineCore pid=70) INFO 08-22 18:39:46 [model_runner.py:284] Loading model from scratch…
(EngineCore pid=70) INFO 08-22 18:39:46 [quant_config.py:75] DeepSeek V4 expert_dtype resolved to ‘fp4’
(EngineCore pid=70) INFO 08-22 18:39:46 [init.py:604] Selected DeepGemmFp8BlockScaledMMKernel for Fp8LinearMethod
(EngineCore pid=70) INFO 08-22 18:39:46 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(EngineCore pid=70) INFO 08-22 18:39:46 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(EngineCore pid=70) INFO 08-22 18:39:46 [deep_gemm.py:120] DeepGEMM E8M0 enabled on current platform.
(EngineCore pid=70) INFO 08-22 18:39:46 [attention.py:91] Using DeepSeek’s fp8_ds_mla KV cache format.
(EngineCore pid=70) INFO 08-22 18:39:47 [mxfp4.py:419] Using ‘MARLIN’ Mxfp4 MoE backend.
(EngineCore pid=70) INFO 08-22 18:39:47 [attention.py:694] Using FP8 indexer cache for Lightning Indexer.
(EngineCore pid=70) WARNING 08-22 18:39:48 [vllm.py:2353] torch.compile is turned on, but the model /home/luozhong/model-DeepSeek-V4-Flash-0731 does not support it. Please open an issue on GitHub if you want it to be supported.
(EngineCore pid=70) INFO 08-22 18:39:48 [weight_utils.py:869] Filesystem type for checkpoints: EXT4. Checkpoint size: 155.43 GiB. Available RAM: 168.86 GiB.
(EngineCore pid=70) INFO 08-22 18:39:48 [weight_utils.py:899] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (155.43 GiB) exceeds 90% of available RAM (168.86 GiB).(EngineCore pid=70) INFO 08-22 18:42:06 [default_loader.py:430] Loading weights took 138.00 seconds
(EngineCore pid=70) INFO 08-22 18:42:08 [mxfp4.py:1718] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=70) INFO 08-22 18:42:08 [mxfp4.py:1719] Using MarlinExperts
(EngineCore pid=70) INFO 08-22 18:43:24 [eagle3_utils.py:28] Using Eagle3 auxiliary layers from config: (41, 42, 43)
(EngineCore pid=70) INFO 08-22 18:43:24 [vllm.py:1109] Asynchronous scheduling is enabled.
(EngineCore pid=70) INFO 08-22 18:43:24 [kernel.py:295] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=[‘native’], fused_add_rms_norm=[‘native’])
(EngineCore pid=70) WARNING 08-22 18:43:24 [vllm.py:1718] max_num_scheduled_tokens is set to 4048 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens or max_num_seqs.
(EngineCore pid=70) WARNING 08-22 18:43:24 [cuda.py:334] --cpu-offload-gb is enabled with CUDA graphs on WSL2. This combination requires pinned (page-locked) memory allocations. WARNING: Windows (WDDM) enforces a hard system-wide cap of roughly 50%% of physical RAM on pinned memory shared across ALL processes by default (limit can changed via %%USERPROFILE%%.wslconfig). Excessive use of page-locked memory can prevent Windows from reclaiming memory under load, which can cause the entire host OS to become unresponsive and may require a hard reboot to recover. Proceed at your own risk. To raise the WSL2 VM memory ceiling, increase the memory setting in %%USERPROFILE%%.wslconfig and run wsl --shutdown.
(EngineCore pid=70) WARNING 08-22 18:43:24 [vllm.py:2228] Model Runner V2 does not yet support the thinking_token_budget request parameter. Set VLLM_USE_V2_MODEL_RUNNER=0 if this is required.
(EngineCore pid=70) INFO 08-22 18:43:24 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=70) WARNING 08-22 18:43:25 [vllm.py:2353] torch.compile is turned on, but the model /home/luozhong/model-DeepSeek-V4-Flash-0731 does not support it. Please open an issue on GitHub if you want it to be supported.
(EngineCore pid=70) INFO 08-22 18:43:25 [weight_utils.py:869] Filesystem type for checkpoints: EXT4. Checkpoint size: 155.43 GiB. Available RAM: 169.17 GiB.
(EngineCore pid=70) INFO 08-22 18:43:25 [weight_utils.py:899] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (155.43 GiB) exceeds 90% of available RAM (169.17 GiB).
Loading safetensors checkpoint shards: 0% Completed | 0/48 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 12% Completed | 6/48 [00:00<00:00, 51.20it/s]
Loading safetensors checkpoint shards: 25% Completed | 12/48 [00:00<00:03, 11.17it/s]
Loading safetensors checkpoint shards: 31% Completed | 15/48 [00:01<00:03, 9.02it/s]
Loading safetensors checkpoint shards: 35% Completed | 17/48 [00:01<00:03, 8.92it/s]
Loading safetensors checkpoint shards: 40% Completed | 19/48 [00:01<00:03, 8.94it/s]
Loading safetensors checkpoint shards: 44% Completed | 21/48 [00:02<00:02, 9.59it/s]
Loading safetensors checkpoint shards: 48% Completed | 23/48 [00:02<00:02, 9.29it/s]
Loading safetensors checkpoint shards: 52% Completed | 25/48 [00:02<00:02, 8.97it/s]
Loading safetensors checkpoint shards: 56% Completed | 27/48 [00:02<00:02, 9.14it/s]
Loading safetensors checkpoint shards: 58% Completed | 28/48 [00:02<00:02, 9.27it/s]
Loading safetensors checkpoint shards: 60% Completed | 29/48 [00:02<00:02, 8.97it/s]
Loading safetensors checkpoint shards: 62% Completed | 30/48 [00:03<00:02, 8.70it/s]
Loading safetensors checkpoint shards: 65% Completed | 31/48 [00:03<00:02, 8.15it/s]
Loading safetensors checkpoint shards: 67% Completed | 32/48 [00:03<00:02, 7.86it/s]
Loading safetensors checkpoint shards: 69% Completed | 33/48 [00:03<00:01, 7.88it/s]
Loading safetensors checkpoint shards: 71% Completed | 34/48 [00:03<00:01, 8.16it/s]
Loading safetensors checkpoint shards: 73% Completed | 35/48 [00:03<00:01, 8.38it/s]
Loading safetensors checkpoint shards: 81% Completed | 39/48 [00:03<00:00, 15.68it/s]
Loading safetensors checkpoint shards: 96% Completed | 46/48 [00:04<00:00, 9.36it/s]
Loading safetensors checkpoint shards: 100% Completed | 48/48 [00:05<00:00, 5.32it/s]
Loading safetensors checkpoint shards: 100% Completed | 48/48 [00:05<00:00, 8.25it/s](EngineCore pid=70) INFO 08-22 18:43:31 [dspark.py:457] DSpark draft model loaded: 96 params
(EngineCore pid=70) INFO 08-22 18:43:31 [default_loader.py:430] Loading weights took 5.84 seconds
(EngineCore pid=70) INFO 08-22 18:43:48 [model_runner.py:305] Model loading took 156.18 GiB and 242.504855 seconds
(EngineCore pid=70) INFO 08-22 18:43:48 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=70) INFO 08-22 18:45:57 [gpu_worker.py:479] Initial free memory 93.09 GiB, reserved 6.06 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM’ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly.
(EngineCore pid=70) INFO 08-22 18:45:57 [kv_cache_utils.py:2177] GPU KV cache size: 122,252 tokens
(EngineCore pid=70) INFO 08-22 18:45:57 [kv_cache_utils.py:2178] Maximum concurrency for 65,536 tokens per request: 1.87x
(EngineCore pid=70) INFO 08-22 18:45:57 [indexer.py:306] DSA indexer decode path: use_flattening=True (next_n=8, use_fp4_indexer_cache=False)
(EngineCore pid=70) INFO 08-22 18:45:57 [kernel.py:295] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=[‘native’], fused_add_rms_norm=[‘native’])
(EngineCore pid=70) INFO 08-22 18:46:00 [flashinfer_sparse_mla_warmup.py:233] Warming up DeepSeek V4 sparse MLA attention for mixed tokens=16.EngineCore pid=70) INFO 08-22 18:58:05 [model_runner.py:747] Graph capturing finished in 639 secs, took 0.00 GiB
(EngineCore pid=70) INFO 08-22 18:58:42 [jit_monitor.py:79] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=70) INFO 08-22 18:58:42 [core.py:347] init engine (profile, create kv cache, warmup model) took 894.26 s
(EngineCore pid=70) INFO 08-22 18:58:43 [kernel.py:295] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=[‘native’], fused_add_rms_norm=[‘native’])
(APIServer pid=17) INFO 08-22 18:58:43 [api_server.py:673] Supported tasks: [‘generate’]
(APIServer pid=17) INFO 08-22 18:58:43 [parser_manager.py:37] “auto” tool choice has been enabled.
(APIServer pid=17) WARNING 08-22 18:58:43 [model.py:1546] Default vLLM sampling parameters have been overridden by the model’s generation_config.json: {'temperature': 1.0, 'top_p': 1.0}. If this is not intended, please relaunch vLLM instance with --generation-config vllm.
(APIServer pid=17) INFO 08-22 18:58:43 [api_server.py:677] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:37] Available routes are:
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=17) INFO 08-22 18:58:43 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=17) INFO: Started server process [17]
(APIServer pid=17) INFO: Waiting for application startup.
(APIServer pid=17) INFO: Application startup complete.
Your vLLM DeepSeek-V4-Flash model is loading successfully with the specified configuration, but you are seeing several warnings. The most important ones are:
- torch.compile is turned on, but the model … does not support it: This means DeepSeek-V4-Flash currently does not support PyTorch’s torch.compile optimization. This is not an error, but you will not get the performance benefits of torch.compile. You can ignore this or set enforce_eager=True to silence the warning.
- WSL2 pinned memory warning: If you are running under WSL2, using --cpu-offload-gb with CUDA graphs can cause system instability due to Windows’ pinned memory limits. Consider increasing the memory limit in your %USERPROFILE%.wslconfig and restarting WSL, or reduce offload usage if you experience issues.
- Auto-prefetch is disabled: Since your checkpoint is on EXT4 and is larger than 90% of RAM, vLLM disables prefetching for safety. This is informational and not an error.
No critical errors are present; the model should work, but you may want to adjust settings for stability and performance as described above. Would you like more detail on any specific warning or configuration?
Sources:
Hint: Mention @Herald in the post for followups.