Float32 Qwen3.5 4B Load

General
1

When load Qwen3.5-4B with float32, faced below issue.



_llm_args = AsyncEngineArgs(

    model=_local_path,

    revision=_local_hash,

    gpu_memory_utilization=0.4,

    trust_remote_code=True,

    dtype=torch.float32,

    language_model_only=True,

    enforce_eager=True,

    logits_processors=["logit_processor:ForcedSeedLogitsProcessor"],

)

_vllm_engine = AsyncLLMEngine.from_engine_args(_llm_args)

Below is issue.

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085] Traceback (most recent call last):

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]   File "/root/reliquary-miner/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py", line 1071, in _warmup_prefill_kernels

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]     self.chunk_gated_delta_rule(

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]   File "/root/reliquary-miner/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]     return self._call_impl(*args, **kwargs)

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]   File "/root/reliquary-miner/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]     return forward_call(*args, **kwargs)

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]   File "/root/reliquary-miner/.venv/lib/python3.12/site-packages/vllm/model_executor/custom_op.py", line 136, in forward

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]     return self._forward_method(*args, **kwargs)

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]   File "/root/reliquary-miner/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py", line 250, in forward_cuda

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]     o, final_state = fi_chunk_gated_delta_rule(

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]   File "/root/reliquary-miner/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py", line 192, in fi_chunk_gated_delta_rule

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]     result = chunk_gated_delta_rule_fi(

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]              ^^^^^^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]   File "/root/reliquary-miner/.venv/lib/python3.12/site-packages/flashinfer/api_logging.py", line 2333, in _auto_dump_wrapper

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]     return _inner(*args, **kwargs)

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]            ^^^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]   File "/root/reliquary-miner/.venv/lib/python3.12/site-packages/flashinfer/gdn_prefill.py", line 465, in chunk_gated_delta_rule

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]     chunk_gated_delta_rule_sm90(

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]   File "/root/reliquary-miner/.venv/lib/python3.12/site-packages/flashinfer/gdn_kernels/delta_rule_dsl/delta_rule_sm90.py", line 2331, in delta_rule_prefill_dsl_sm90

(EngineCore pid=8978) WARNING 08-05 21:30:32 [qwen_gdn_linear_attn.py:1085]     raise RuntimeError(f"DSL kernel only supports fp16/bf16 inputs, got {q.dtype}")

I should use float32 for accurate rollout. How can I do this?

2

The error occurs because the Qwen3.5-4B model’s GDN (Gated Delta Net) attention kernel only supports fp16 or bf16 inputs, but you are using float32 (dtype=torch.float32). The kernel explicitly raises an error if the input dtype is not fp16 or bf16: “DSL kernel only supports fp16/bf16 inputs, got {q.dtype}” (source, API docs).

To resolve this, set dtype to either torch.float16 or torch.bfloat16 in your engine arguments. Using float32 is not supported for this model’s attention implementation.

Would you like more detail on why only fp16/bf16 are supported?

Sources:

Hint: Mention @Herald in the post for followups.

3

I should use float32 for accuracy. What are other way to do this?