Does setting TRITON_CACHE_DIR break the bundled (AOT) static-autotuner cubin restore?

torch.compile integration
1

Context
vLLM 0.29.0 (pip install, not Docker), torch 2.13.0
NVIDIA GB10 (DGX Spark), single device, driver 580.173.02 / CUDA 13.0
Qwen/Qwen3.6-35B-A3B-FP8, --max-cudagraph-capture-size 8192, --max-model-len 262144, --kv-cache-dtype fp8, --gpu-memory-utilization 0.85
AOT compile cache in use (default path ~/.cache/vllm/torch_compile_cache/torch_aot_compile/<hash>/…)
Observation
On every server start I get 13 occurrences of

RuntimeError: ('Cubin file saved by TritonBundler not found at %s

each immediately followed by reconstructed serializable fn from standalone compile artifacts.
Affected kernels are e.g. triton_poi_fused_zeros_1/2/3, triton_poi_fused_mul_per_token_group_fp8_quant_permute_sigmoid_view_0, triton_red_fused__to_copy_add_fused_add_rms_norm_per_token_group_fp8_quant_permute_1.
Correctness is fine — torch/_inductor/triton_bundler.py catches the RuntimeError and continues, so those static autotuners are skipped and the kernels are recompiled at runtime. It looks like a warm-up cost only.
What the lookup actually resolved to
I wrapped os.path.exists to log failing *.cubin lookups (env + directory listing at that moment). One entry:

missing path            : $TRITON_CACHE_DIR/<hash>/triton_poi_fused_zeros_1.cubin
                          (parent directory does not exist)
TORCHINDUCTOR_CACHE_DIR : ~/.cache/vllm/torch_compile_cache/torch_aot_compile/<aot_hash>/inductor_cache
                          <- i.e. vLLM had already set this correctly for the artifact being restored
TRITON_CACHE_DIR (env)  : ~/.cache/triton

The same file does exist on disk with an identical hash directory name, only under a different prefix:

<TORCHINDUCTOR_CACHE_DIR>/triton/0/<hash>/triton_poi_fused_zeros_1.cubin
                                            ^ device level present

Mechanism as I read it
torch/_inductor/runtime/cache_dir_utils.py:

def triton_cache_dir(device: int) -> str:
    if (directory := os.getenv("TRITON_CACHE_DIR")) is not None:
        return directory                              # verbatim — no "/{device}"
    return os.path.join(cache_dir(), "triton", str(device))

triton_bundler.py read_and_emit() re-emits the bundled artifacts to os.path.join(triton_cache_dir(artifacts.device), artifacts.kernel_hash) (JSON payloads carry a [REPLACE] sentinel that is substituted with that directory).
triton_heuristics.py reload_cubin_path() recomputes os.path.join(triton_cache_dir(device), triton_hash_to_path_key(kernel.hash), f"{name}.cubin").
So emit and reload agree only when triton_cache_dir() resolves to the same directory in both cases. With TRITON_CACHE_DIR set it resolves without the device level, while the artifacts on disk sit under …/triton/0/… → miss.
Related: pytorch/pytorch#163336 (“Potential bug in _inductor/runtime/cache_dir_utils.py”) raised the “device id is not appended when TRITON_CACHE_DIR is set” part and was closed as intentional (one shared cache across ranks, atomic renames). I am not re-litigating that — my question is about the configuration and the emit/reload agreement.
Questions
Is setting TRITON_CACHE_DIR supported alongside vLLM’s AOT compile cache? I set it because on this host /tmp is wiped on every boot (systemd-tmpfiles rule D /tmp 1777 root root 30d), so torch’s default /tmp/torchinductor_$USER is always cold. Should users instead set only TORCHINDUCTOR_CACHE_DIR and leave TRITON_CACHE_DIR unset?
If it is not supported, would it make sense to document that — or to not depend on the global resolution when restoring a bundle?
Alternatively: is the 13-kernel rebuild per start simply expected, and the warning can be ignored?
Reproduction (to see the count on any setup using the compile cache):

journalctl -u <your-vllm-unit> | grep -c 'Failed to reload cubin'

Happy to share the full trace JSON (missing path, env, directory listings) or to run further

2

Setting TRITON_CACHE_DIR alongside vLLM’s AOT compile cache is not supported and leads to the exact issue you observed: Triton expects cubins in $TRITON_CACHE_DIR//…, but vLLM (and PyTorch Inductor) emit them under …/triton///… when TRITON_CACHE_DIR is unset. This mismatch causes the “Failed to reload cubin file” warnings and runtime recompilation, as described in your logs and confirmed by upstream discussions (pytorch/pytorch#163336) and vLLM documentation (vLLM cache directory security, compiler interface).

The recommended approach is to set only TORCHINDUCTOR_CACHE_DIR and leave TRITON_CACHE_DIR unset. vLLM will then manage both Inductor and Triton caches under its own cache root, ensuring emit and reload agree and avoiding unnecessary recompilation. The 13-kernel rebuilds per start are not expected if the cache is configured correctly; the warning can be ignored for correctness, but it does indicate a suboptimal cache setup (compiler interface, security docs). Would you like a step-by-step explanation of how vLLM manages these cache directories and why this mismatch occurs?

Sources:

Hint: Mention @Herald in the post for followups.