How can I pass kwargs to CustomLogitsProcessors?

General
1

Hi experts,

I have a setup where I can start a vllm server with some custom logits processors, like this:
–-logits-processors “custom_logit_processors:MyCustomLogitsProcessor” but I’m not able to pass any kwargs to the MyCustomLogitsProcessor in the class init method.

--additional-config does not pass kwargs to the logits processor class.

I’m currently passing init values to MyCustomLogitsProcessor via os.env which seems a bit crude. Any alternatives?

2

You can pass constructor arguments to custom logits processors in vLLM by using the JSON object syntax in the --logits-processors CLI argument. Each element can be either a qualified class name or a JSON object with qualname, args, and kwargs fields. For example:

vllm serve ... --logits-processors '[{"qualname": "custom_logit_processors:MyCustomLogitsProcessor", "args": [], "kwargs": {"my_arg": "value"}}]'

This approach allows you to specify keyword arguments directly for your logits processor, avoiding the need to use environment variables. For more details and examples, see RFC: Logits processor extensibility.

Would you like more detail or a Python example?

Sources:

Hint: Mention @RunLLM in the post for followups.