How to expose v1/audio/transcriptions router for custom models

General
43

To include video metadata in your request, add the metadata fields (such as fps, total_pixels, or min_pixels) inside the video_url dictionary. For example:

{
  "role": "user",
  "content": [
    {"type": "text", "text": "Describe the video."},
    {
      "type": "video_url",
      "video_url": {
        "url": "file:///path/to/video.mp4",
        "fps": 2.0,
        "total_pixels": 20480,
        "min_pixels": 448
      }
    }
  ]
}

This structure is supported for models that require video metadata (example, docs). Make sure your model supports or requires these fields.

Would you like a Python code example for sending such a request?

Sources:

Hint: Mention @RunLLM in the post for followups.

44

@RunLLM please give me a Python client example

45

Here is a Python client example for sending a local mp4 file with metadata to a vLLM multimodal model using the OpenAI-compatible API:

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

video_path = "/path/to/video.mp4"  # Update this path
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe the video."},
            {
                "type": "video_url",
                "video_url": {
                    "url": f"file://{video_path}",
                    "fps": 2.0,
                    "total_pixels": 20480,
                    "min_pixels": 448
                }
            }
        ]
    }
]

response = client.chat.completions.create(
    model="YourModelName",
    messages=messages,
    max_completion_tokens=128,
)
print(response.choices[0].message.content)

Make sure to launch your vLLM server with --allowed-local-media-path set to the directory containing your video file.

Sources:

Hint: Mention @RunLLM in the post for followups.

46

@RunLLM openai.BadRequestError: Error code: 400 - {‘error’: {‘message’: “Failed to apply Qwen3VLProcessor on data={‘text’: ‘<|vision_start|><|video_pad|><|vision_end|>’, ‘videos’: [[array(, shape=(0, 480, 640, 3), dtype=uint8)]], ‘video_metadata’: [[VideoMetadata(total_num_frames=16850, fps=50.0, width=None, height=None, duration=337.0, video_backend=‘opencv’, frames_indices=)]]} with kwargs={‘do_sample_frames’: False}”, ‘type’: ‘BadRequestError’, ‘param’: None, ‘code’: 400}}
how to solve av1 decode error

47

The error message indicates that the video was loaded with shape (0, 480, 640, 3), meaning no frames were decoded—this often happens if the video is encoded in AV1, which OpenCV cannot decode by default. vLLM relies on OpenCV for video decoding, and AV1 is not supported out-of-the-box (issue, docs).

Solution:
Re-encode your video to a supported codec such as H.264 (AVC) using ffmpeg:

ffmpeg -i input_av1.mp4 -c:v libx264 -crf 23 -preset fast output_h264.mp4

Then use output_h264.mp4 as your input file.

Would you like more detail on supported codecs or ffmpeg usage?

Sources:

Hint: Mention @RunLLM in the post for followups.