Introduction
In the ever-evolving world of computer hardware, innovations never cease to amaze. Imagine for a moment: you own an already robust gaming PC, but you want to boost it for intensive computational tasks such as massive language model (LLM) inference. Instead of shelling out thousands for a brand-new GPU, how about an alternative solution? That's exactly what Oscar Molnar achieved by integrating a Tesla V100 datacenter GPU into his PC for just £200. Here's how he did it.
The Datacenter GPU: Tesla V100
The Tesla V100 SXM2 is a powerhouse designed for NVIDIA's DGX servers. With 16GB of HBM2 memory and 5120 CUDA cores, this GPU was a revolution in 2017 and remains incredibly performant today. Its memory bandwidth of 900 GB/s surpasses even some of the best recent products from Apple and AMD. The V100, though old, remains competitive against modern cards like the RTX 4080 or the RX 7900 XTX.
Why a Datacenter GPU?
For Oscar, the answer was simple: cost. Buying a modern card with 32GB of VRAM typically costs over £2000. The V100, though requiring an adapter to be compatible with desktop PCs, offers similar performance for a fraction of the price. With an unofficial SXM2-to-PCIe adapter, he was able to connect it to his PC and combine its power with his RTX 4080, totaling 32GB of VRAM.
The Connector Challenge
The main challenge was adapting the connector. Datacenter GPUs like the V100 use the SXM2 format, lacking a standard PCIe connector or display output. Fortunately, PCB adapters exist, though they are not officially supported. For about £50, Oscar was able to acquire this adapter and turn his dream into reality.
Performance and Usage
With this setup, Oscar was able to run a 27 billion parameter model at 32 tokens per second. For tasks where memory bandwidth is crucial, such as LLM inference, the V100 proved to be a wise choice. By comparison, even a card like the RTX 5090, with its 1,792 GB/s bandwidth, costs over £2000.
Conclusion
This adventure demonstrates that with a bit of creativity and resourcefulness, it's possible to push the limits of your hardware without breaking the bank. Of course, this isn't a solution for everyone. It requires a good level of technical expertise and a certain tolerance for risk. But if you're ready to take on the challenge, the rewards can be immense.
Let's discuss your project in 15 minutes.