I have lmcache server up and running on a separate docker. With same version 0.5.2 I have also vllm server docker run command but it fails.
(Worker_TP0 pid=267) INFO 07-28 12:28:55 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/6e912a7ddd/rank_0_0/backbone for vLLM’s torch.compile
(Worker_TP0 pid=267) INFO 07-28 12:28:55 [backends.py:1148] Dynamo bytecode transform time: 20.65 s
(Worker_TP0 pid=267) INFO 07-28 12:29:12 [backends.py:378] Cache the graph of compile range (1, 11000) for later use
(EngineCore pid=223) INFO 07-28 12:29:35 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=267) INFO 07-28 12:29:41 [backends.py:393] Compiling a graph for compile range (1, 11000) takes 44.62 s
(Worker_TP0 pid=267) INFO 07-28 12:29:54 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/30784472f0ace49441aa2a95cc883fc3391067466f393af0ebfa1168b49b0ae9/rank_0_0/model
(Worker_TP0 pid=267) INFO 07-28 12:29:54 [monitor.py:53] torch.compile took 79.15 s in total
(EngineCore pid=223) INFO 07-28 12:30:38 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=223) INFO 07-28 12:31:41 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] WorkerProc hit an exception.
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] Traceback (most recent call last):
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] File “/opt/venv/lib/python3.12/site-packages/flashinfer/jit/cpp_ext.py”, line 368, in run_ninja
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] subprocess.run(
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] File “/usr/lib/python3.12/subprocess.py”, line 571, in run
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] raise CalledProcessError(retcode, process.args,
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’, ‘-C’, ‘/root/.cache/flashinfer/0.6.13/120f/cached_ops/fp4_gemm_cutlass_sm120’, ‘-f’, ‘/root/.cache/flashinfer/0.6.13/120f/cached_ops/fp4_gemm_cutlass_sm120/build.ninja’]’ returned non-zero exit status 9.
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004]
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] The above exception was the direct cause of the following exception:
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004]
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] Traceback (most recent call last):
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] File “/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py”, line 996, in worker_busy_loop
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] output = func(*args, **kwargs)
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] File “/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py”, line 124, in decorate_context
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004] return func(*args, **kwargs)
(Worker_TP0 pid=267) ERROR 07-28 12:31:46 [multiproc_executor.py:1004]
Your error is caused by a failure in the FlashInfer JIT compilation step, specifically when running ninja to build a CUDA kernel for fp4_gemm_cutlass_sm120. The error subprocess.CalledProcessError: ... returned non-zero exit status 9 indicates the build process failed, which is often due to insufficient shared memory or missing system dependencies in the Docker container. Additionally, repeated No available shared memory broadcast block found in 60 seconds messages suggest shared memory exhaustion or misconfiguration.
To resolve:
Ensure your Docker run command includes --ipc=host and a sufficiently large --shm-size (e.g., --shm-size=16g or higher).
Make sure the container has ninja and all CUDA build tools installed.
If running on a new GPU architecture, verify that the CUDA toolkit and drivers are compatible and that the container is built for your GPU’s compute capability.
If the error persists, try increasing the shared memory further or check for disk space issues in /root/.cache.
@herald
got now
File “/opt/venv/lib/python3.12/site-packages/lmcache/v1/platform/cuda/ipc_wrapper.py”, line 81, in to_tensor
storage = torch.UntypedStorage._new_shared_cuda( # noqa: SLF001
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: received sharable handle from a future version of torch that this version does not know how to handle
[2026-07-28 16:50:12,542] LMCache ERROR: Error handling request RequestType.REGISTER_KV_CACHE (mq.py:623:lmcache.v1.multiprocess.mq)
Using these:
docker pull lmcache/standalone:nightly-2026-07-28
docker pull lmcache/vllm-openai:nightly-2026-07-28
This error means your LMCache and vLLM containers are using mismatched PyTorch versions—one is newer than the other, so the CUDA IPC handle format is incompatible. Both containers must use exactly the same PyTorch version to share CUDA memory correctly (source).
Would you like detailed steps to align the PyTorch versions in your containers?
To align PyTorch versions between your LMCache and vLLM containers, ensure both are built or pulled with the same PyTorch version. If using prebuilt images, select tags that specify the same nightly or release date for both containers. If building from source, install the exact same torch, torchvision, and torchaudio versions in both environments (source).
For example, in both Dockerfiles or containers, run:
No, these images are not guaranteed to use the exact same PyTorch version or build environment, even if their tags match. Nightly tags may be built with different dependency versions or build order, leading to incompatibilities like the CUDA IPC handle error you saw (source).
You must verify the PyTorch versions inside both containers match exactly. Would you like instructions to check and synchronize the PyTorch versions in your running containers?
There is no official guarantee that lmcache/standalone:nightly-2026-07-28 and lmcache/vllm-openai:nightly-2026-07-28 use the exact same PyTorch version, even if the tags match. The only way to ensure compatibility is to check the PyTorch version inside each container after pulling, and if they differ, rebuild or override the PyTorch install so both use the same version (source).
Currently, the data sources do not specify a pair of lmcache Docker tags that are guaranteed to work together for CUDA IPC. You must check the torch version inside each container (e.g., python -c "import torch; print(torch.__version__)") and manually align them if needed.
Would you like a step-by-step guide to check and align the PyTorch versions inside your containers?