Tell me about the current status of the tokenize endpoint in vllm

General
5

Yes, the endpoint is /tokenize (not /v1/tokenize) for vLLM’s tokenizer API. This endpoint is used for tokenizing text or chat messages and is separate from the OpenAI-compatible /v1 routes to avoid confusion (PR #5054).

For examples, see the vLLM test file test_tokenization.py, which demonstrates POST requests to /tokenize with both prompt and chat-style inputs. Here is a minimal example:

import requests
response = requests.post(
    "http://localhost:8000/tokenize",
    json={"model": "meta-llama/Meta-Llama-3-8B-Instruct", "prompt": "test"}
)
print(response.json())

This returns token IDs, count, and max_model_len. For chat, you can send a messages array instead of prompt.

Sources:

Hint: Mention @RunLLM in the post for followups.