# RTX PRO6000运行Qwen3-32B-FP8量化版本出错
**URL:** https://discuss.vllm.ai/t/rtx-pro6000-qwen3-32b-fp8/1005
**Category:** General
**Created:** [July 4, 2025, 6:03am UTC](https://discuss.vllm.ai/t/rtx-pro6000-qwen3-32b-fp8/1005 "2025-07-04T06:03:40Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@iseebrowser](https://discuss.vllm.ai/u/iseebrowser)
#### Post date: [July 4, 2025, 6:03am UTC](https://discuss.vllm.ai/t/rtx-pro6000-qwen3-32b-fp8/1005/1 "2025-07-04T06:03:40Z")
基本环境：
vllm 0.9.1 自己编译的版本，pytorch 2.7.1，cu128。机器信息：HP z6g5，192G内存，显卡RTX PRO 6000。操作系统：Ubuntu24.04。
启动命令：
VLLM\_ATTENTION\_BACKEND=FLASHINFER vllm serve ~/models/Qwen3-32B-FP8
–served-model-name Qwen3-32B
–api-key htxt-ai
–seed 3407
–disable-log-requests
–gpu-memory-utilization 0.90
–host 0.0.0.0 --port 6006
–max-model-len 64000
–dtype bfloat16
–trust-remote-code
–max-num-seqs 32
错误信息如下：
ERROR 07-04 13:17:38 [core.py:515] EngineCore failed to start.
ERROR 07-04 13:17:38 [core.py:515] Traceback (most recent call last):
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 506, in run\_engine\_core
ERROR 07-04 13:17:38 [core.py:515] engine\_core = EngineCoreProc(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 390, in **init**
ERROR 07-04 13:17:38 [core.py:515] super(). **init** (vllm\_config, executor\_class, log\_stats,
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 83, in **init**
ERROR 07-04 13:17:38 [core.py:515] self.\_initialize\_kv\_caches(vllm\_config)
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 141, in \_initialize\_kv\_caches
ERROR 07-04 13:17:38 [core.py:515] available\_gpu\_memory = self.model\_executor.determine\_available\_memory()
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/executor/abstract.py”, line 76, in determine\_available\_memory
ERROR 07-04 13:17:38 [core.py:515] output = self.collective\_rpc(“determine\_available\_memory”)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/executor/uniproc\_executor.py”, line 57, in collective\_rpc
ERROR 07-04 13:17:38 [core.py:515] answer = run\_method(self.driver\_worker, method, args, kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/utils.py”, line 2671, in run\_method
ERROR 07-04 13:17:38 [core.py:515] return func(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/\_contextlib.py”, line 116, in decorate\_context
ERROR 07-04 13:17:38 [core.py:515] return func(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu\_worker.py”, line 205, in determine\_available\_memory
ERROR 07-04 13:17:38 [core.py:515] self.model\_runner.profile\_run()
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu\_model\_runner.py”, line 2012, in profile\_run
ERROR 07-04 13:17:38 [core.py:515] hidden\_states = self.\_dummy\_run(self.max\_num\_tokens)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File "/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate\_context
ERROR 07-04 13:17:38 [core.py:515] return func(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu\_model\_runner.py”, line 1847, in dummy\_run
ERROR 07-04 13:17:38 [core.py:515] outputs = model(
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1751, in wrapped\_call\_impl
ERROR 07-04 13:17:38 [core.py:515] return self.call\_impl(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1762, in call\_impl
ERROR 07-04 13:17:38 [core.py:515] return forward\_call(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model\_executor/models/qwen3.py”, line 301, in forward
ERROR 07-04 13:17:38 [core.py:515] hidden\_states = self.model(input\_ids, positions, intermediate\_tensors,
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/compilation/decorators.py”, line 239, in **call**
ERROR 07-04 13:17:38 [core.py:515] output = self.compiled\_callable(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File "/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/dynamo/eval\_frame.py", line 655, in fn
ERROR 07-04 13:17:38 [core.py:515] return fn(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model\_executor/models/qwen2.py”, line 336, in forward
ERROR 07-04 13:17:38 [core.py:515] def forward(
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1751, in wrapped\_call\_impl
ERROR 07-04 13:17:38 [core.py:515] return self.call\_impl(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1762, in call\_impl
ERROR 07-04 13:17:38 [core.py:515] return forward\_call(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File "/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/dynamo/eval\_frame.py", line 838, in fn
ERROR 07-04 13:17:38 [core.py:515] return fn(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/graph\_module.py”, line 830, in call\_wrapped
ERROR 07-04 13:17:38 [core.py:515] return self.wrapped\_call(self, \*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/graph\_module.py”, line 406, in **call**
ERROR 07-04 13:17:38 [core.py:515] raise e
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/graph\_module.py”, line 393, in **call**
ERROR 07-04 13:17:38 [core.py:515] return super(self.cls, obj). **call** (\*args, \*\*kwargs) # type: ignore[misc]
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1751, in wrapped\_call\_impl
ERROR 07-04 13:17:38 [core.py:515] return self.call\_impl(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1762, in call\_impl
ERROR 07-04 13:17:38 [core.py:515] return forward\_call(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “\.130”, line 778, in forward
ERROR 07-04 13:17:38 [core.py:515] submod\_0 = self.submod\_0(l\_input\_ids, s0, l\_self\_modules\_embed\_tokens\_parameters\_weight, l\_self\_modules\_layers\_modules\_0\_modules\_input\_layernorm\_parameters\_weight, l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_qkv\_proj\_parameters\_weight, l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_qkv\_proj\_parameters\_weight\_scale\_inv, l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_q\_norm\_parameters\_weight, l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_k\_norm\_parameters\_weight, l\_positions, l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_rotary\_emb\_buffers\_cos\_sin\_cache); l\_input\_ids = l\_self\_modules\_embed\_tokens\_parameters\_weight = l\_self\_modules\_layers\_modules\_0\_modules\_input\_layernorm\_parameters\_weight = l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_qkv\_proj\_parameters\_weight = l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_qkv\_proj\_parameters\_weight\_scale\_inv = l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_q\_norm\_parameters\_weight = l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_k\_norm\_parameters\_weight_ = None
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/compilation/cuda\_piecewise\_backend.py”, line 111, in **call**
ERROR 07-04 13:17:38 [core.py:515] return self.compiled\_graph\_for\_general\_shape(\*args)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_dynamo/eval\_frame.py”, line 838, in \_fn
ERROR 07-04 13:17:38 [core.py:515] return fn(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_functorch/aot\_autograd.py”, line 1209, in forward
ERROR 07-04 13:17:38 [core.py:515] return compiled\_fn(full\_args)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_functorch/\_aot\_autograd/runtime\_wrappers.py”, line 328, in runtime\_wrapper
ERROR 07-04 13:17:38 [core.py:515] all\_outs = call\_func\_at\_runtime\_with\_args(
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_functorch/\_aot\_autograd/utils.py”, line 126, in call\_func\_at\_runtime\_with\_args
ERROR 07-04 13:17:38 [core.py:515] out = normalize\_as\_list(f(args))
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_functorch/\_aot\_autograd/runtime\_wrappers.py”, line 689, in inner\_fn
ERROR 07-04 13:17:38 [core.py:515] outs = compiled\_fn(args)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_functorch/\_aot\_autograd/runtime\_wrappers.py”, line 495, in wrapper
ERROR 07-04 13:17:38 [core.py:515] return compiled\_fn(runtime\_args)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_inductor/output\_code.py”, line 460, in **call**
ERROR 07-04 13:17:38 [core.py:515] return self.current\_callable(inputs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_inductor/utils.py”, line 2404, in run
ERROR 07-04 13:17:38 [core.py:515] return model(new\_inputs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/.cache/vllm/torch\_compile\_cache/24fba47138/rank\_0\_0/inductor\_cache/qv/cqvtomjpfvuxt5j6wimhfjiujwga4fgjjnth3npvhuixtbsqjpkg.py”, line 435, in call
ERROR 07-04 13:17:38 [core.py:515] buf3 = torch.ops.vllm.apply\_w8a8\_block\_fp8\_linear.default(buf2, arg4\_1, [128, 128], arg5\_1)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_ops.py”, line 756, in **call**
ERROR 07-04 13:17:38 [core.py:515] return self.\_op(\*args, \*\*kwargs)
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model\_executor/layers/quantization/utils/fp8\_utils.py”, line 141, in apply\_w8a8\_block\_fp8\_linear
ERROR 07-04 13:17:38 [core.py:515] output = w8a8\_blockscale\_func(q\_input, weight, x\_scale, weight\_scale,
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model\_executor/layers/quantization/utils/fp8\_utils.py”, line 39, in cutlass\_scaled\_mm
ERROR 07-04 13:17:38 [core.py:515] return ops.cutlass\_scaled\_mm(A,
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/\_custom\_ops.py”, line 717, in cutlass\_scaled\_mm
ERROR 07-04 13:17:38 [core.py:515] torch.ops.\_C.cutlass\_scaled\_mm(out, a, b, scale\_a, scale\_b, bias)
ERROR 07-04 13:17:38 [core.py:515] File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_ops.py”, line 1158, in **call**
ERROR 07-04 13:17:38 [core.py:515] return self.\_op(\*args, \*\*(kwargs or {}))
ERROR 07-04 13:17:38 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-04 13:17:38 [core.py:515] RuntimeError: Error Internal
Process EngineCore\_0:
Traceback (most recent call last):
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/multiprocessing/process.py”, line 314, in \_bootstrap
self.run()
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/multiprocessing/process.py”, line 108, in run
self.\_target(\*self.\_args, \*\*self.\_kwargs)
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 519, in run\_engine\_core
raise e
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 506, in run\_engine\_core
engine\_core = EngineCoreProc(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 390, in **init**
super(). **init** (vllm\_config, executor\_class, log\_stats,
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 83, in **init**
self.\_initialize\_kv\_caches(vllm\_config)
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 141, in \_initialize\_kv\_caches
available\_gpu\_memory = self.model\_executor.determine\_available\_memory()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/executor/abstract.py”, line 76, in determine\_available\_memory
output = self.collective\_rpc(“determine\_available\_memory”)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/executor/uniproc\_executor.py”, line 57, in collective\_rpc
answer = run\_method(self.driver\_worker, method, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/utils.py”, line 2671, in run\_method
return func(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/\_contextlib.py”, line 116, in decorate\_context
return func(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu\_worker.py”, line 205, in determine\_available\_memory
self.model\_runner.profile\_run()
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu\_model\_runner.py”, line 2012, in profile\_run
hidden\_states = self.\_dummy\_run(self.max\_num\_tokens)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate\_context
return func(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu\_model\_runner.py”, line 1847, in dummy\_run
outputs = model(
^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1751, in wrapped\_call\_impl
return self.call\_impl(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1762, in call\_impl
return forward\_call(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model\_executor/models/qwen3.py”, line 301, in forward
hidden\_states = self.model(input\_ids, positions, intermediate\_tensors,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/compilation/decorators.py”, line 239, in **call**
output = self.compiled\_callable(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/dynamo/eval\_frame.py", line 655, in fn
return fn(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model\_executor/models/qwen2.py”, line 336, in forward
def forward(
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1751, in wrapped\_call\_impl
return self.call\_impl(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1762, in call\_impl
return forward\_call(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/dynamo/eval\_frame.py", line 838, in fn
return fn(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/graph\_module.py”, line 830, in call\_wrapped
return self.wrapped\_call(self, \*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/graph\_module.py”, line 406, in **call**
raise e
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/graph\_module.py”, line 393, in **call**
return super(self.cls, obj). **call** (\*args, \*\*kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1751, in wrapped\_call\_impl
return self.call\_impl(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1762, in call\_impl
return forward\_call(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “\.130”, line 778, in forward
submod\_0 = self.submod\_0(l\_input\_ids, s0, l\_self\_modules\_embed\_tokens\_parameters\_weight, l\_self\_modules\_layers\_modules\_0\_modules\_input\_layernorm\_parameters\_weight, l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_qkv\_proj\_parameters\_weight, l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_qkv\_proj\_parameters\_weight\_scale\_inv, l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_q\_norm\_parameters\_weight, l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_k\_norm\_parameters\_weight, l\_positions, l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_rotary\_emb\_buffers\_cos\_sin\_cache); l\_input\_ids = l\_self\_modules\_embed\_tokens\_parameters\_weight = l\_self\_modules\_layers\_modules\_0\_modules\_input\_layernorm\_parameters\_weight = l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_qkv\_proj\_parameters\_weight = l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_qkv\_proj\_parameters\_weight\_scale\_inv = l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_q\_norm\_parameters\_weight = l\_self\_modules\_layers\_modules\_0\_modules\_self\_attn\_modules\_k\_norm\_parameters\_weight_ = None
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/compilation/cuda\_piecewise\_backend.py”, line 111, in **call**
return self.compiled\_graph\_for\_general\_shape(\*args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_dynamo/eval\_frame.py”, line 838, in \_fn
return fn(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_functorch/aot\_autograd.py”, line 1209, in forward
return compiled\_fn(full\_args)
^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_functorch/\_aot\_autograd/runtime\_wrappers.py”, line 328, in runtime\_wrapper
all\_outs = call\_func\_at\_runtime\_with\_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_functorch/\_aot\_autograd/utils.py”, line 126, in call\_func\_at\_runtime\_with\_args
out = normalize\_as\_list(f(args))
^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_functorch/\_aot\_autograd/runtime\_wrappers.py”, line 689, in inner\_fn
outs = compiled\_fn(args)
^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_functorch/\_aot\_autograd/runtime\_wrappers.py”, line 495, in wrapper
return compiled\_fn(runtime\_args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_inductor/output\_code.py”, line 460, in **call**
return self.current\_callable(inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_inductor/utils.py”, line 2404, in run
return model(new\_inputs)
^^^^^^^^^^^^^^^^^
File “/home/blackfog/.cache/vllm/torch\_compile\_cache/24fba47138/rank\_0\_0/inductor\_cache/qv/cqvtomjpfvuxt5j6wimhfjiujwga4fgjjnth3npvhuixtbsqjpkg.py”, line 435, in call
buf3 = torch.ops.vllm.apply\_w8a8\_block\_fp8\_linear.default(buf2, arg4\_1, [128, 128], arg5\_1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_ops.py”, line 756, in **call**
return self.\_op(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model\_executor/layers/quantization/utils/fp8\_utils.py”, line 141, in apply\_w8a8\_block\_fp8\_linear
output = w8a8\_blockscale\_func(q\_input, weight, x\_scale, weight\_scale,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model\_executor/layers/quantization/utils/fp8\_utils.py”, line 39, in cutlass\_scaled\_mm
return ops.cutlass\_scaled\_mm(A,
^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/\_custom\_ops.py”, line 717, in cutlass\_scaled\_mm
torch.ops.\_C.cutlass\_scaled\_mm(out, a, b, scale\_a, scale\_b, bias)
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/\_ops.py”, line 1158, in **call**
return self.\_op(\*args, \*\*(kwargs or {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Error Internal
[rank0]:[W704 13:17:40.856694377 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy\_process\_group() was not called before program exit, which can leak resources. For more info, please see [Distributed communication package - torch.distributed — PyTorch 2.7 documentation](https://pytorch.org/docs/stable/distributed.html#shutdown) (function operator())
Traceback (most recent call last):
File “/home/blackfog/miniconda3/envs/vllm/bin/vllm”, line 8, in
sys.exit(main())
^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py”, line 59, in main
args.dispatch\_function(args)
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py”, line 58, in cmd
uvloop.run(run\_server(args))
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/uvloop/ **init**.py”, line 109, in run
return \_\_asyncio.run(
^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/asyncio/runners.py”, line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/asyncio/runners.py”, line 118, in run
return self.\_loop.run\_until\_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “uvloop/loop.pyx”, line 1518, in uvloop.loop.Loop.run\_until\_complete
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/uvloop/ **init**.py”, line 61, in wrapper
return await main
^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api\_server.py”, line 1323, in run\_server
await run\_server\_worker(listen\_address, sock, args, \*\*uvicorn\_kwargs)
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api\_server.py”, line 1343, in run\_server\_worker
async with build\_async\_engine\_client(args, client\_config) as engine\_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/contextlib.py”, line 210, in **aenter**
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api\_server.py”, line 155, in build\_async\_engine\_client
async with build\_async\_engine\_client\_from\_engine\_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/contextlib.py”, line 210, in **aenter**
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api\_server.py”, line 191, in build\_async\_engine\_client\_from\_engine\_args
async\_llm = AsyncLLM.from\_vllm\_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/async\_llm.py”, line 162, in from\_vllm\_config
return cls(
^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/async\_llm.py”, line 124, in **init**
self.engine\_core = EngineCoreClient.make\_async\_mp\_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core\_client.py”, line 93, in make\_async\_mp\_client
return AsyncMPClient(vllm\_config, executor\_class, log\_stats,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core\_client.py”, line 716, in **init**
super(). **init** (
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core\_client.py”, line 422, in **init**
self.\_init\_engines\_direct(vllm\_config, local\_only,
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core\_client.py”, line 491, in \_init\_engines\_direct
self.\_wait\_for\_engine\_startup(handshake\_socket, input\_address,
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core\_client.py”, line 511, in \_wait\_for\_engine\_startup
wait\_for\_engine\_startup(
File “/home/blackfog/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/utils.py”, line 494, in wait\_for\_engine\_startup
raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
---
### Author:  [@RunLLM](https://discuss.vllm.ai/u/RunLLM)
#### Post date: [July 4, 2025, 6:04am UTC](https://discuss.vllm.ai/t/rtx-pro6000-qwen3-32b-fp8/1005/2 "2025-07-04T06:04:51Z")
根据你提供的报错信息，vLLM 在初始化时在 apply\_w8a8\_block\_fp8\_linear（FP8/W8A8 量化相关）自定义 CUDA kernel 处发生了 RuntimeError: Error Internal，导致 EngineCore 启动失败。这类报错在 vLLM 0.9.x 版本、PyTorch 2.7.x、CUDA 12.8 环境下，常见于新一代 RTX 50 系显卡（如 5070Ti、5090、PRO 6000 等）运行 FP8/W8A8 量化模型时，底层 cutlass\_scaled\_mm kernel 不支持该架构，或 CUDA/驱动/编译兼容性存在问题。该问题已被多次反馈，官方 issue 明确指出 RTX 50 系显卡目前对 FP8/W8A8 量化模型支持不完善，容易出现“no kernel image is available for execution on the device”或“Error Internal”等错误，且目前无官方修复方案，需等待后续 vLLM 或 PyTorch/驱动层支持完善【详见[相关 issue1](https://github.com/vllm-project/vllm/issues/19605)、[相关 issue2](https://github.com/vllm-project/vllm/issues/18244)、[相关 issue3](https://github.com/vllm-project/vllm/issues/19596)】。
建议：1）如必须使用 FP8/W8A8 量化模型，建议更换为 A100/H100/4090 等已验证支持的显卡；2）如仅需推理，可尝试使用非量化（FP16/BF16）模型规避该问题；3）可关注 vLLM 官方仓库后续对 RTX 50 系显卡的支持进展。你的环境和报错与上述 issue 完全一致，属于已知硬件兼容性问题，并非配置或代码错误。如需进一步排查或临时规避方案，请告知。
Sources:
- [Bug: RTX50xx GPU is not supported for running W8A8 FP8 quant models!](https://github.com/vllm-project/vllm/issues/19605)
- [Bug: vllm rtxt5090 Qwen3-14B-FP8](https://github.com/vllm-project/vllm/issues/18244)
- [Bug:rtx5060ti apply\_w8a8\_block\_fp8\_linear](https://github.com/vllm-project/vllm/issues/19596)