I want to add a system of LLM observability data management, a component similar to Alibaba’s Druid. It exports runtime metrics, including request-lifecycle metrics such as block utilization, KV cache space, and read/write statistics that developers and testers care about, as well as metrics for each stage during service startup.
The goal is to consolidate performance data and fully exploit machine hardware capacity. To achieve this, relevant telemetry data needs to be exposed. The ultimate vision is to build not only observability capabilities but also built-in evaluation tooling. This eliminates the need to write various custom benchmark scripts, improves LLM maintenance and testing workflows, and drives advances in large language model technology.
The following implementation is my initial prototype.
Your prototype, llm-monitor, is well-aligned with vLLM’s observability goals. vLLM already exposes extensive runtime and request-lifecycle metrics (e.g., KV cache usage, request latency, token throughput, queue times) via Prometheus-compatible endpoints, and supports logging for ad-hoc analysis. These metrics cover block utilization, KV cache space, and request statistics, and can be visualized in Grafana dashboards for both real-time and historical analysis. Startup phase profiling and per-stage timing are also supported through logging and Prometheus metrics, but vLLM does not natively provide a Druid-like data warehouse or built-in evaluation tooling beyond metrics/logging and OpenTelemetry tracing integrations. For advanced telemetry, your approach of patching vLLM to export detailed metrics (including startup profiling and GPU stats) is compatible with vLLM’s design principles and can be activated with minimal intrusion, as shown in your repo and described in the vLLM metrics documentation.
If you want to consolidate and analyze these metrics at scale, you can use Prometheus for collection and Grafana for visualization, or extend your system to ingest vLLM’s /metrics endpoint data into a Druid-like backend for deeper analytics. For built-in evaluation tooling, vLLM currently relies on external scripts and dashboards, but your approach could fill this gap by providing a unified observability and benchmarking layer. For more detail on vLLM’s metrics and observability capabilities, see the official metrics documentation and production monitoring examples. Would you like a step-by-step breakdown of how to integrate your system with vLLM’s metrics and tracing infrastructure?
collect metrics throughout the lifecycle of each request , or time-series changes of KV cache or startup lifecycle ,or are workers responsible for load balancing ，a global health status.