# VLLM Engine Metrics

**URL:** <https://discuss.vllm.ai/t/vllm-engine-metrics/810>\
**Category:** Benchmarking\
**Created:** [June 11, 2025, 7:48pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810 "2025-06-11T19:48:03Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![g12megha](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/g12megha/32/381_2.png) [@g12megha](https://discuss.vllm.ai/u/g12megha)\
**Post date:** [June 11, 2025, 7:48pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/1 "2025-06-11T19:48:03Z")

</div>

Hi I m using VLLM 0.6.3 for RL training. I want to benchmark the performance of the token generation from the model with VLLM. I have setup $PROMETHEUS\_MULTIPROC\_DIR and i do see data files like counter\_2771563.db counter\_2771565.db gauge\_all\_2771563.db gauge\_all\_2771565.db histogram\_2771563.db histogram\_2771565.db.  
I executed:  
curl -i [http://127.0.0.1:8000/metrics](http://127.0.0.1:8000/metrics)  
HTTP/1.1 404 Not Found  
date: Wed, 11 Jun 2025 19:27:00 GMT  
server: uvicorn  
content-length: 22  
content-type: application/json  
Response received:  
{“detail”:“Not Found”}

I also setup a ray prometheus dashboard and fetching the metrics at localhost:9090/metrics endpoint but i m not observing any VLLM related metrics. Can you share documentation on changes to make to see the VLLM metrics?

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 11, 2025, 7:48pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/2 "2025-06-11T19:48:37Z")

</div>

To expose vLLM metrics at the /metrics endpoint, you must run the vLLM OpenAI-compatible API server (e.g., with vllm serve or python -m vllm.entrypoints.openai.api\_server). If you are running vLLM inside Ray Serve or another custom setup, you need to manually expose the /metrics endpoint and ensure the PrometheusStatLogger is active. For Ray Serve, you must add the RayPrometheusStatLogger to the engine using add\_logger, as the default PrometheusStatLogger is not automatically used in Ray deployments. The /metrics endpoint is not available if you are not running the API server directly or have not set up the logger correctly. See [this Slack thread](https://vllm-dev.slack.com/archives/C07QP347J4D/p1728410979.078409) and [official docs](https://docs.vllm.ai/en/latest/serving/metrics.html#print-main-content) for details.

If you see a 404 on /metrics, it means the endpoint is not being served—likely because the API server is not running, or the metrics middleware is not mounted. For Ray Serve, see the [RayPrometheusStatLogger API reference](https://github.com/vllm-project/vllm/blob/cdca8994bd856a234112875a92746c5782837768/vllm/engine/llm_engine.py#L1519) and ensure you add the logger and expose the endpoint in your Ray deployment. The Prometheus dashboard at :9090/metrics will not show vLLM metrics unless the vLLM server exposes them at /metrics and Prometheus is configured to scrape that endpoint.

Would you like a step-by-step example for Ray Serve or for the standard vLLM API server?

Sources:

- [vLLM Slack: Ray Serve Prometheus metrics](https://vllm-dev.slack.com/archives/C07QP347J4D/p1728410979.078409)
- [vLLM Metrics Documentation](https://docs.vllm.ai/en/latest/serving/metrics.html#print-main-content)

---

<div class="post-metadata">

**Author:** ![g12megha](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/g12megha/32/381_2.png) [@g12megha](https://discuss.vllm.ai/u/g12megha)\
**Post date:** [June 11, 2025, 8:00pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/3 "2025-06-11T20:00:04Z")

</div>

Yes step by step example of enabling the metrics endpoint within ray, configuring prometheus to scrap the endpoint will be helpful

---

<div class="post-metadata">

**Author:** ![g12megha](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/g12megha/32/381_2.png) [@g12megha](https://discuss.vllm.ai/u/g12megha)\
**Post date:** [June 11, 2025, 8:33pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/4 "2025-06-11T20:33:25Z")

</div>

I also see the .db files created under the PROMETHEUS\_MULTIPROC\_DIR . Is this expected? What kind of db is it? How can i parse it

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 11, 2025, 8:33pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/5 "2025-06-11T20:33:54Z")

</div>

It seems likely that the .db files under PROMETHEUS\_MULTIPROC\_DIR are expected when using Prometheus multiprocess mode with vLLM. These files are created by the prometheus\_client library to store metric data from each process, and their names (e.g., counter\ __.db, gauge\_all\__.db, histogram\_\*.db) correspond to different metric types. The format is internal to prometheus\_client and not intended for direct parsing; you should access metrics via the /metrics HTTP endpoint, not by reading these files directly. For more, see the [Prometheus multiprocess mode docs](https://prometheus.github.io/client_python/multiprocess/), which vLLM relies on as described in [vLLM metrics design](https://docs.vllm.ai/en/latest/design/v1/metrics.html).

Would you like more detail on how Prometheus aggregates these files or how vLLM interacts with them?

Sources:

- [vLLM metrics design](https://docs.vllm.ai/en/latest/design/v1/metrics.html)

---

<div class="post-metadata">

**Author:** ![g12megha](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/g12megha/32/381_2.png) [@g12megha](https://discuss.vllm.ai/u/g12megha)\
**Post date:** [June 11, 2025, 8:44pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/6 "2025-06-11T20:44:39Z")

</div>

Can i parse with promtool?

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 11, 2025, 8:45pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/7 "2025-06-11T20:45:11Z")

</div>

No, you cannot parse the .db files directly with promtool. These files are internal to the prometheus\_client Python library and are not in the Prometheus TSDB or text exposition format; promtool only works with Prometheus scrape targets or TSDB blocks, not these .db files. You should always access metrics via the /metrics HTTP endpoint, as described in the [Prometheus multiprocess mode docs](https://prometheus.github.io/client_python/multiprocess/) and [vLLM metrics documentation](https://docs.vllm.ai/en/latest/design/v1/metrics.html).

Would you like more detail on how to use promtool or how to expose metrics for scraping?

Sources:

- [vLLM metrics documentation](https://docs.vllm.ai/en/latest/design/v1/metrics.html)

---

<div class="post-metadata">

**Author:** ![g12megha](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/g12megha/32/381_2.png) [@g12megha](https://discuss.vllm.ai/u/g12megha)\
**Post date:** [June 11, 2025, 8:49pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/8 "2025-06-11T20:49:48Z")

</div>

Yes details on how to expose the metrics

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 11, 2025, 8:50pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/9 "2025-06-11T20:50:23Z")

</div>

To expose vLLM metrics for Prometheus when running with Ray, you must (1) add the RayPrometheusStatLogger to your engine, and (2) create an HTTP endpoint that serves the metrics in Prometheus format. The .db files are only for internal use; Prometheus scrapes metrics from an HTTP endpoint, not from these files directly. See the [vLLM Ray metrics API reference](https://docs.vllm.ai/en/stable/api/vllm/vllm.engine.metrics.html#rayprometheusstatlogger) and [Slack discussion](https://vllm-dev.slack.com/archives/C07QP347J4D/p1728410979.078409).

**Step-by-step:**

1. **Add RayPrometheusStatLogger to the engine:**

2. **Expose the /metrics endpoint:**  
You must create a FastAPI (or similar) HTTP server in your Ray deployment that, when /metrics is called, collects and returns the current metrics from RayPrometheusStatLogger. vLLM does not do this automatically in Ray Serve mode; you must implement this endpoint yourself.

3. **Configure Prometheus to scrape:**  
In your prometheus.yml, add:

For a full example and more context, see the [Slack thread](https://vllm-dev.slack.com/archives/C07QP347J4D/p1728410979.078409) and [vLLM metrics docs](https://docs.vllm.ai/en/latest/serving/metrics.html).

Would you like a code example for the FastAPI /metrics endpoint?

Sources:

- [vLLM Slack: Ray Serve Prometheus metrics](https://vllm-dev.slack.com/archives/C07QP347J4D/p1728410979.078409)

---

<div class="post-metadata">

**Author:** ![g12megha](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/g12megha/32/381_2.png) [@g12megha](https://discuss.vllm.ai/u/g12megha)\
**Post date:** [June 11, 2025, 8:56pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/10 "2025-06-11T20:56:07Z")

</div>

> [@RunLLM](#):
>
> RayPrometheusStatLogger

Do we need to setup PROMETHEUS\_MULTIPROC\_DIR

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 11, 2025, 8:56pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/11 "2025-06-11T20:56:58Z")

</div>

No, you do not need to set up PROMETHEUS\_MULTIPROC\_DIR when using RayPrometheusStatLogger. That environment variable is only required for the standard PrometheusStatLogger (which uses prometheus\_client in multiprocess mode). RayPrometheusStatLogger uses Ray’s own metrics system and does not rely on PROMETHEUS\_MULTIPROC\_DIR or the prometheus\_client multiprocess backend, as confirmed in the [vLLM Ray metrics API docs](https://docs.vllm.ai/en/latest/api/vllm/vllm.engine.metrics.html#rayprometheusstatlogger) and [source code](https://github.com/vllm-project/vllm/blob/main/vllm/v1/metrics/ray_wrappers.py).

Would you like more detail on how RayPrometheusStatLogger works or how to configure it?

Sources:

- [source code](https://github.com/vllm-project/vllm/blob/main/vllm/v1/metrics/ray_wrappers.py)

---

<div class="post-metadata">

**Author:** ![g12megha](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/g12megha/32/381_2.png) [@g12megha](https://discuss.vllm.ai/u/g12megha)\
**Post date:** [June 11, 2025, 9:03pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/12 "2025-06-11T21:03:24Z")

</div>

> [@RunLLM](#):
>
> RayPrometheusStatLogger

Why is PrometheusStatLogger not sufficient and we need RayPrometheusStatLogger

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 11, 2025, 9:03pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/13 "2025-06-11T21:03:44Z")

</div>

PrometheusStatLogger is designed for standard (single-process or multiprocessing) vLLM deployments and uses the prometheus\_client library, which does not natively support Ray’s distributed metrics collection. RayPrometheusStatLogger is required for Ray-based deployments because it uses Ray’s metrics API (ray.util.metrics), ensuring metrics are correctly aggregated and exposed in Ray clusters. This distinction is documented in the [vLLM metrics API](https://docs.vllm.ai/en/latest/api/vllm/vllm.engine.metrics.html#rayprometheusstatlogger) and [source code](https://github.com/vllm-project/vllm/blob/main/vllm/v1/metrics/ray_wrappers.py).

Would you like more detail on the technical differences or usage examples?

Sources:

- [source code](https://github.com/vllm-project/vllm/blob/main/vllm/v1/metrics/ray_wrappers.py)

---

<div class="post-metadata">

**Author:** ![g12megha](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/g12megha/32/381_2.png) [@g12megha](https://discuss.vllm.ai/u/g12megha)\
**Post date:** [June 11, 2025, 10:21pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/14 "2025-06-11T22:21:48Z")

</div>

I m not using RayServe. How can i expose the prometheus metrics ?

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 11, 2025, 10:22pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/15 "2025-06-11T22:22:34Z")

</div>

If you are not using Ray Serve, you can expose Prometheus metrics by running the vLLM OpenAI-compatible API server (e.g., with vllm serve or python -m vllm.entrypoints.openai.api\_server). The server will automatically expose metrics at the /metrics HTTP endpoint, which Prometheus can scrape. This is the standard and recommended way, as described in the [official vLLM metrics documentation](https://docs.vllm.ai/en/latest/serving/metrics.html) and [Prometheus/Grafana example](https://docs.vllm.ai/en/latest/getting_started/examples/prometheus_grafana.html).

No extra code is needed—just launch the server, and then configure Prometheus to scrape http://:8000/metrics.

Would you like a sample Prometheus configuration or a code snippet for launching the server?

Sources:

- [Prometheus/Grafana example](https://docs.vllm.ai/en/latest/getting_started/examples/prometheus_grafana.html)

---

<div class="post-metadata">

**Author:** ![g12megha](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/g12megha/32/381_2.png) [@g12megha](https://discuss.vllm.ai/u/g12megha)\
**Post date:** [June 11, 2025, 10:23pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/16 "2025-06-11T22:23:42Z")

</div>

yes code snippet would be useful

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 11, 2025, 10:24pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/17 "2025-06-11T22:24:08Z")

</div>

To expose Prometheus metrics with vLLM (not using Ray Serve), simply launch the OpenAI-compatible API server. Metrics will be available at [http://localhost:8000/metrics](http://localhost:8000/metrics) by default.

**Example code snippet:**

```bash
vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048 --disable-log-requests

```

Or, using Python:

```python
from vllm.entrypoints.openai.api_server import main

if __name__ == " __main__":
    main()

```

After starting the server, you can verify metrics are available:

```bash
curl http://localhost:8000/metrics

```

Prometheus can then be configured to scrape this endpoint. For more, see the [official example](https://docs.vllm.ai/en/latest/getting_started/examples/prometheus_grafana.html).

Sources:

- [Prometheus/Grafana example](https://docs.vllm.ai/en/latest/getting_started/examples/prometheus_grafana.html)

---

<div class="post-metadata">

**Author:** ![g12megha](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/g12megha/32/381_2.png) [@g12megha](https://discuss.vllm.ai/u/g12megha)\
**Post date:** [June 11, 2025, 10:34pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/18 "2025-06-11T22:34:24Z")

</div>

How does Prometheus server use the data from PROMETHEUS\_MULTIPROC\_DIR ? I do see .db files in this directory. is this generated by vllm?

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 11, 2025, 10:34pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/19 "2025-06-11T22:34:46Z")

</div>

Prometheus server does not directly use or read the files in PROMETHEUS\_MULTIPROC\_DIR. These .db files are generated by the prometheus\_client Python library (used by vLLM) when running in multiprocess mode. vLLM and prometheus\_client aggregate metrics from these files and expose them via the /metrics HTTP endpoint. Prometheus server then scrapes metrics from this endpoint, not from the .db files themselves. This behavior is described in the [Prometheus multiprocess mode documentation](https://prometheus.github.io/client_python/multiprocess/) and [vLLM metrics design](https://docs.vllm.ai/en/latest/design/v1/metrics.html).

Would you like more detail on how the aggregation works internally?

Sources:

- [vLLM metrics design](https://docs.vllm.ai/en/latest/design/v1/metrics.html)

---

<div class="post-metadata">

**Author:** ![g12megha](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/g12megha/32/381_2.png) [@g12megha](https://discuss.vllm.ai/u/g12megha)\
**Post date:** [June 11, 2025, 11:18pm UTC](https://discuss.vllm.ai/t/vllm-engine-metrics/810/20 "2025-06-11T23:18:20Z")

</div>

I m using this code in vllm engine. Can you confirm if this is right setting to enable the VLLM metrics? I m particularly interested in these metrics:

- `vllm:time_to_first_token_seconds` (Histogram)
- `vllm:time_per_output_token_seconds` (Histogram)
- `vllm:e2e_request_latency_seconds` (Histogram)

Will this code be able to generate the needed metrics :

```auto
                    "logging": LoggingStatLogger(local_interval=_LOCAL_LOGGING_INTERVAL_SEC),
                    "prometheus": PrometheusStatLogger(
                        local_interval=_LOCAL_LOGGING_INTERVAL_SEC,
                        labels=dict(model_name=model_config.served_model_name),
                        max_model_len=self.model_config.max_model_len,
                    ),
                }
```

[Next page](https://discuss.vllm.ai/t/vllm-engine-metrics/810.md?page=2)