# Embeddings with vLLM in Kubernetes **URL:** \ **Category:** General\ **Created:** [August 28, 2025, 10:11am UTC](https://discuss.vllm.ai/t/embeddings-with-vllm-in-kubernetes/1473 "2025-08-28T10:11:44Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 28, 2025, 10:11am UTC](https://discuss.vllm.ai/t/embeddings-with-vllm-in-kubernetes/1473/1 "2025-08-28T10:11:44Z")
How can I prepare a Kubernetes manifest to accommodate embeddings with vLLM image? We will use service and persistent volume for the weights. What other argumenets and environmental parameters will be needed? The weights will be downloaded from Hugging face ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 28, 2025, 10:13am UTC](https://discuss.vllm.ai/t/embeddings-with-vllm-in-kubernetes/1473/2 "2025-08-28T10:13:05Z")
To prepare a Kubernetes manifest for vLLM with embeddings, you need a Deployment (using the vLLM image), a PersistentVolumeClaim (PVC) for model weights, a Secret for your Hugging Face token (if required), and a Service to expose the API. For embeddings, set the `--task embed` argument in your container’s command or args. You must also mount the PVC to `/root/.cache/huggingface` and set the `HUGGING_FACE_HUB_TOKEN` environment variable if accessing gated models. Example arguments: `vllm serve --task embed`. A minimal manifest includes: - PVC for model cache - Secret for Hugging Face token - Deployment with: - vLLM image - args: `vllm serve --task embed` - env: `HUGGING_FACE_HUB_TOKEN` from Secret - volumeMounts for model cache - Service exposing port 8000 See the code example below and refer to [Kubernetes deployment docs](https://docs.vllm.ai/en/latest/deployment/k8s.html), [embedding task argument](https://docs.vllm.ai/en/latest/cli/index.html#-task), and [model config](https://docs.vllm.ai/en/latest/api/vllm/config/index.html#__span-0-261). ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: vllm-models spec: accessModes: [ReadWriteOnce] resources: requests: storage: 50Gi --- apiVersion: v1 kind: Secret metadata: name: hf-token-secret type: Opaque stringData: token: "" --- apiVersion: apps/v1 kind: Deployment metadata: name: vllm-server spec: replicas: 1 selector: matchLabels: app: vllm template: metadata: labels: app: vllm spec: containers: - name: vllm image: vllm/vllm-openai:latest command: ["/bin/sh", "-c"] args: ["vllm serve --task embed"] env: - name: HUGGING_FACE_HUB_TOKEN valueFrom: secretKeyRef: name: hf-token-secret key: token ports: - containerPort: 8000 volumeMounts: - name: llama-storage mountPath: /root/.cache/huggingface volumes: - name: llama-storage persistentVolumeClaim: claimName: vllm-models --- apiVersion: v1 kind: Service metadata: name: vllm-server spec: selector: app: vllm ports: - protocol: TCP port: 8000 targetPort: 8000 type: ClusterIP ``` Would you like a more detailed breakdown or explanation of each section? Sources: - [Kubernetes deployment docs](https://docs.vllm.ai/en/latest/deployment/k8s.html) - [embedding task argument](https://docs.vllm.ai/en/latest/cli/index.html#-task) - [model config](https://docs.vllm.ai/en/latest/api/vllm/config/index.html#__span-0-261) ---
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 28, 2025, 10:18am UTC](https://discuss.vllm.ai/t/embeddings-with-vllm-in-kubernetes/1473/3 "2025-08-28T10:18:21Z")
Currently I use this manifest as below. I want the vllm pod to download the model from hugging face. Should I use –model in args? spec: containers: - name: vllm image: vllm/vllm-openai:latest imagePullPolicy: IfNotPresent command: [“python3”, “-m”, “vllm.entrypoints.openai.api\_server”] args: - “–model” ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 28, 2025, 10:18am UTC](https://discuss.vllm.ai/t/embeddings-with-vllm-in-kubernetes/1473/4 "2025-08-28T10:18:57Z")
Yes, you should use the `--model` argument in `args` to specify the Hugging Face model name (e.g., `--model meta-llama/Llama-3-8B`). This tells vLLM to download the model from Hugging Face if it is not already present in the cache. Make sure to use standard double quotes (`"`) and double dashes (`--model`), not special or encoded characters, in your manifest. Example: ```yaml args: ["--model", "meta-llama/Llama-3-8B"] ``` For more, see [Kubernetes deployment docs](https://docs.vllm.ai/en/latest/deployment/k8s.html) and [Hugging Face integration](https://docs.vllm.ai/en/latest/design/huggingface_integration.html). Would you like a full manifest example or more details on other arguments? Sources: - [Kubernetes deployment docs](https://docs.vllm.ai/en/latest/deployment/k8s.html) ---
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 28, 2025, 10:20am UTC](https://discuss.vllm.ai/t/embeddings-with-vllm-in-kubernetes/1473/5 "2025-08-28T10:20:29Z")
provide complete manifest for embeddings using args –model ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 28, 2025, 10:21am UTC](https://discuss.vllm.ai/t/embeddings-with-vllm-in-kubernetes/1473/6 "2025-08-28T10:21:06Z")
Here is a complete Kubernetes manifest for running vLLM embeddings, using the `--model` argument to download from Hugging Face. This manifest includes a PersistentVolumeClaim, Secret for Hugging Face token, Deployment (with correct args and env), and Service. Replace `` and `` as needed. ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: vllm-models spec: accessModes: - ReadWriteOnce resources: requests: storage: 50Gi --- apiVersion: v1 kind: Secret metadata: name: hf-token-secret type: Opaque stringData: token: "" --- apiVersion: apps/v1 kind: Deployment metadata: name: vllm-server spec: replicas: 1 selector: matchLabels: app: vllm template: metadata: labels: app: vllm spec: containers: - name: vllm image: vllm/vllm-openai:latest command: ["python3", "-m", "vllm.entrypoints.openai.api_server"] args: ["--model", "", "--task", "embed"] env: - name: HUGGING_FACE_HUB_TOKEN valueFrom: secretKeyRef: name: hf-token-secret key: token ports: - containerPort: 8000 volumeMounts: - name: llama-storage mountPath: /root/.cache/huggingface volumes: - name: llama-storage persistentVolumeClaim: claimName: vllm-models --- apiVersion: v1 kind: Service metadata: name: vllm-server spec: selector: app: vllm ports: - protocol: TCP port: 8000 targetPort: 8000 type: ClusterIP ``` This manifest will download the model from Hugging Face and serve embeddings via the OpenAI-compatible API. For more, see [Kubernetes deployment docs](https://docs.vllm.ai/en/latest/deployment/k8s.html). Would you like a step-by-step explanation of each section? Sources: - [Kubernetes deployment docs](https://docs.vllm.ai/en/latest/deployment/k8s.html)