What is the functions of --enable-chunked-prefill and --max-num-batched-tokens? make inferen faster or something else？
--enable-chunked-prefill allows vLLM to split large prompt (prefill) requests into smaller chunks, batching them with decode requests. This improves GPU utilization and can increase throughput, especially under high concurrency, by overlapping compute-bound (prefill) and memory-bound (decode) operations. --max-num-batched-tokens sets the maximum number of tokens processed in a single batch, directly affecting batching efficiency, throughput, and latency. Tuning this value helps balance inter-token latency (ITL) and time to first token (TTFT) for optimal performance.
In summary, both options are for performance tuning: chunked prefill improves batching and resource utilization, while max-num-batched-tokens controls batch size for better throughput or latency, depending on your workload and hardware. For details and tuning recommendations, see Performance Tuning with Chunked Prefill and Optimization. Would you like more detail or code examples?
Sources:
Hint: Mention @Herald in the post for followups.
我的业务场景是用模型研判流量日志中是否存在威胁，流量日志加Agent上的工作流prompt，会涉及到很长的上下文。这种场景下，这两个参数有什么必要性的帮助？