The new V1 way to ~--cpu-offload-gb

V1 Feedback
1

I attended the 8 April 2025 vLLM office hours. A lot of the talk sounded like topic(s) covered effectively supplant the need for v0’s --cpu-offload-gb param.

For my 16GB vRAM Nvidia RTX 5070ti and my dummy/n00b status, I get the idea of quantization and the idea that I should retrieve optimized models from RedHat on HuggingFace.

Beyond those two takeaways that I comprendo’ed as a n00b/dummy, mostly I felt like I was just listening to talking using words and phrases (vLLingo lol) so I can be accustomed to hearing & reading them without feeling scared by them.

Since a lot of the talk was over my head (for now) but i srsly need an alternative to --cpu-offload-gb RIGHT NOW, is there an office hours recap I can review, and if not, can the main points be placed here for how those of us with vRAM envy can make do with our tiny vRams?

I ask bc those of us with tiny vRAMs will (i bet) 99.9% overlap dummy-n00bs on the vLLM-user Venn diagram.

For example - I am pretty sure I launched a Phi-4 one time and it worked locally, but ever since that one time, I’ve not been able to make it start at all. KVCache for full sized KVs use up my whole 16GB vRAM. I can get it down to about 4GB VRAM but with 12GB being taken (with 4 of the 12 only being “reserved” but not used yet also not available), I need another 1G vRAM to … get past this “not enough memory” error and probably on to the next one.

So, not complaining!!! But can we get a recap/overview (or point me at it) for the office hours and how to work around limited GPU memory? Yeah the OH was about Terabyte sized models and sixteen of $1.5 million GPUs, but the memory management theory and even the params ought to be the same (excepting for distribution across GPUs and/or add’l machines) for working with 16gb vRAM and an 12+B param model (or with Phi 4 the model size isn’t that big but other things sure seem to be; and let me tell you, IME a significantly shorter context length makes for a very very dumb Phi-4 (coming from HF/MS, not HF/RedHat)

Thanks for any advice!

EDIT: I can re-watch the office hours video, but it will again just be words I am working toward learning to hear without them inducing anxiety. I’m more hoping for something akin to a hyperlink-annotated recap, like maybe the main points, very briefly whatabout each point, and hyperlink(s) to get myself lost in :slight_smile:

1 Like
2

--cpu-offload-gb should work in v1 after v0.8.3, after [V1] Fully Transparent Implementation of CPU Offloading by youkaichao · Pull Request #15354 · vllm-project/vllm · GitHub .

does this answer your question?

3

Oh!!! Yes, i think it does. Thank you!

I thought I’d read there wasn’t a plan to continue support for --gpu-offload-gb unless there was significant demand for that.

I considered being somebody to provide “demand for that” but I also figured there must be a new and/or improved approach that replacing using --GpuOffloadGb.

I have seen info about memory mangement, so I suppose I will learn what i can about that while continuing to use --GpuOffloadGb until I know how to work without.

Question, though: Is there a minimum VRAM (GRAM?) that vllm effectively expects you to have? During office hours the lowest i think i heard was an 80GB VRAM NV H100 that runs like $25000 (USD) and up. (OT The memory bus speed there seems to be the attraction, however, not the 80GB VRAM, bc 3xRTX5090 would get you more VRAM at around $6000 USD + a 3xpcie16 box.)

Or is vLLM moving toward “everybody just use cloud computing AI” and that’s why the GpuOffloadGB param was (i think) going to be deprecrated (but isn’t? or maybe i just read it wrong and GpuOffloadGB was never going to be deprecated?)

Thank you so much for helping me understand. I am very new to this world of LM inference, but I’m also very experienced in most everything else I.T., so I keep landing myself trying things that are ahead of where I probably should be working (on learning) in the realm of AI / LLMs.

4

didn’t see any reasons why we would deprecate --cpu-offload-gb , it should work and keep working.

5

Oh and if you need a guinea-pig with a literal, real 16gb vRAM GPU to check for explosions or anything, LMK! I have exactly one 16gb vRAM 5070ti. :crying_cat:

That said, I will be away for a week. Will be back at vLLM-ing on Tues April 22. Happy to contribute in any way that I possibly can (which at this early stage of my vLLM-ing would be downloading something you provide, pushing the button, waiting however long, and then telling you what happened / giving you the logfiles or whatever)

6

Oh! That is great (at least for me, lol). I am very glad that I read something wrong about that feature going away. I am very happy to hear it’s not really going away. :grin: Thank you so much for taking some of your time for helping me to understand things.