im am currently running a dual arc b60 pro setup. I got this for testing, and since I knew that I had large documents to process I went for the 48GB VRAM of that setup. I am serving with VLLM to an openwebui.
My test are very successful. BUT, with long contexts, the ttft is very long. It takes up to 10 minutes before the reasoning starts.
I am thinking about buying an Arc B70 Pro with 32GB VRAM for running Gemma4 31B it 4bit autround or awq (currently using autoround).
My question now is: Is the ttft increased by the PCIe communication between the two GPUs, and would the ttft significantly change with a single Arc B70 PRO GPU?
Yes, TTFT (time to first token) is often increased by PCIe communication overhead when using tensor or pipeline parallelism across multiple GPUs, especially for long contexts and large models. For smaller models or when the model fits on a single GPU, using one GPU (like a single Arc B70 Pro) generally results in lower TTFT compared to splitting across two GPUs, since inter-GPU communication is a major bottleneck and can outweigh the benefits of parallel compute for these workloads. This is a well-documented effect in vLLM and similar inference engines, particularly when GPUs are connected via PCIe rather than NVLink or similar high-bandwidth interconnects. For your use case, a single Arc B70 Pro should provide significantly better TTFT than a dual B60 setup for models that fit in its memory, as it avoids the communication penalty entirely (source, source, source, source, source).
Would you like a step-by-step breakdown of why PCIe communication increases TTFT and how single-GPU setups avoid this penalty?