How to add custom special tokens?

General
1

I wanna add some custom special tokens, as far as I know, it was stored in tokenizer.json and tokenizer_config.json, the former is used by the low-level rust tokenizer, the latter is used by python-level’s tokenizer.all_special_tokens
which file was used when vllm inferencing, which file should I modify?

1 Like
2

To add custom special tokens for vLLM inference, you should modify both tokenizer.json (for the underlying tokenizer) and tokenizer_config.json (for Python-level access like tokenizer.all_special_tokens). vLLM loads the tokenizer using Hugging Face’s AutoTokenizer.from_pretrained, which reads both files, so changes in both are respected during inference. See details in vLLM HuggingFace integration docs and tokenizer loading code.

Would you like more detail on how to safely add tokens to both files?

Sources:

Hint: Mention @RunLLM in the post for followups.

3

Even though you are modifying the tokenizer_config.json and tokenizer.json files, the model will not pickup those tokens as it wasn’t trained on those custom tokens. Hence it won’t generate those tokens.

1 Like
4

I’ve added the new token and added its embedding
now I don’t know how to add only new embedding into the modules that will to be finetuned
I use unsloth