The model field (HuggingFace model name or local path) is currently included in ModelConfig.compute_hash(), which means two instances serving the exact same model from different paths will compute different config_hash values, causing torch.compile cache misses even though the compilation artifacts are fully reusable.
Including the model field (i.e., HuggingFace model name or local path) in ModelConfig.compute_hash() causes cache misses when the same model is loaded from different paths, preventing reuse of compilation artifacts. A proposed and tested solution is to exclude the model field from the hash, ensuring consistent hashes for identical models regardless of path, while still producing unique hashes for genuinely different models. This change has been validated to maintain correct cache behavior across different model paths and tensor parallel sizes, as described in Exclude model field from ModelConfig.compute_hash().
Would you like more detail on the implementation or its impact?
Sources:
Hint: Mention @Herald in the post for followups.