CG Workstation – The Baby “Titan” Supercomputer

Configuration Options

Of course there are many many ways to configure any workstation. With the TYAN FT77A removing most of the guesswork around motherboard, case and PSU selection, our options our discussion will be brief and focused around CPU, RAM and GPU options.

CPU: There are quite a few E5-2600 series Xeons, ranging anywhere from 4C/9T to 8C/16T and $310~$1900+ a piece. Yes, you will be using two…
Depending on how you envision your workflow with this system, you might configure it to be almost 100% dedicated to GPU computing (in which case a couple of affordable E5-2609 2.4GHz Quad cores will be sufficient), to a fully capable CPU rendering workstation combined with 7-8 fast GPUs, in which case you should most likely opt for the fastest pair of 8-core E5 Xeons you can afford.
The 2.4GHz Octa-core E5-2665 is almost five times the price of the 2609 Quad, but since the 8C E5-2670 @ 2.6GHz is just 5% more, I would probably go for it should I decide to reach deep in my pockets. The best value in s2011 Xeons should be awarded to the E5-2620, the cheapest of the series 6C with Hyperthreading, yet probably people seriously considering a FT77A system for something more than a GPGPU platform, would opt for a 8C/16T.

Cooling: Most likely CPUs will be air-cooled. There is a large number of s2011 air coolers that can comfortably fit in a 4U case. With the FT77A, most likely you could even use a couple of 120mm AIO water coolers to be mounted behind the large intake fans between the HDDs and the motherboard bay. Thick 120mm radiator based units should be fair game, though most likely one of the supplied fans would be left out:

GPU(s): The FT77A was most likely designed with nVidia Tesla’s in mind. There is an onboard GPU chip that allows you to populate all 8 slots with head-less Tesla cards, should you wish to go for a remote GPGPU solution.

Of course those cards come with an impressive (or intimidating?) pricetag, with each K20X being in the $4,600 range, and the K20 around $3,500. With the introduction of the GTX Titan, users interested in the GK110 can have a card with as much ram as a K20X and faster clocks for around $1,000. It is perhaps the only scenario under which you could consider the GTX Titan a “bargain”.

All GTX Titans are based on the reference design and are sourced directly by nVidia.
In the US GTX Titans are importer exclusively by ASUS and EVGA.

Keep in mind that GTX Titans are in great demand and limited supply. The specialized “factory overclock” versions are kind of a gimmick, as all of the are sharing the same boards , VRMs and cooler. You can overclock all of them to similar levels just as well should you wish to. You could also mix and match Asus and EVGA cards without issue, or even mix any high power GTX card to an unlimited number of combinations.

Titans are far from the only viable scenario: 7-8x of any fast GTX card will be able to give impressive results. Any of the cards below would be a great choice, given those fit within our VRam buffer requirements, and budget:

  • GTX 780 3GB – 2304 CUDA Cores x 900*  MHz   ~ 88% of a GTX Titan 
  • GTX 680 4GB – 1536 CUDA Cores x 1058*MHz ~ 69% of a GTX Titan
  • GTX 670 4GB – 1344 CUDA Cores x 970*  MHz   ~ 55% of a GTX Titan

* Reference  speed: most available 680s and 670s with 4GB buffer are factory clocked higher for little or no price difference with the reference models. Unlike the Titan, customized boards and cooling solutions on those do yield better O/C results.

Keep in mind that under most loads, 2x GTX 680s or even 2x GTX 670s will be faster than a single Titan, while cost less. If you don’t have the money to invest into 6+ Titans or GTX 780s, a full rack of 8x 680s will be a faster choice with better value – given the 4GB max buffer is enough for the tasks you usually undertake. For tasks or GPU renderings that are not VRam limited, the new GTX 780 3GB, based on the same GK110 with a couple computation cluster disabled and ofc half the RAM. A GTX 780 with 2304 CUDA cores is within 10% of a GTX Titan’s performance.

Since GPGPU computing is not limited to an absolute max number of GPUs, you could even go for 8x GTX 690s for a total of 16x GK104 GPUs or more than 24,670 Kepler CUDA processing units in a single box! Unfortunately, unlike the label that mentions 4GB or VRam, only 2GB are available to each GPU, which might be too little for some demanding applications, while there is a high probability that 8x 690s under full load might surpass the available capacity the 2x 1200W PSUs can reliably provide for prolonged time.

 

FT77A_8x_Titans

 

A FT77A with a full rack of GTX Titan cards attached. 

Image source: http://www.soa-world.de/echelon/

Should you choose to use your FT77A as your primary workstation and not just a GPGPU compute server, you could consider your 8th card being a Quadro (or Firepro) workstation card that will give you fluid viewport performance, while 7x “gaming” GPUs will be dedicated to computation tasks.

Conclusion

This baby “Titan” supercomputer is a bit ahead of its time for the average CG Artist, and very few will be lucky to have a GPGPU monster like that to actually play with. Fully decked out, and without going crazy with system RAM, a FT77A with 8x Titans and a couple of E5-2665 Xeons can be above the $17,000 range, while with a more conservative approach using 8x 680s and vanilla E5-2605 Quads you could aim at around $10,000 as a realistic budget – before taxes/vat. For most interested in custom PC builds, this might sound insane, but once you venture into the pre-configured CG Workstation world of BOXX, Dell, HP and Lenovo, those figures are fair game for high-end workstations, far less potent than this one in the GPGPU and perhaps even for direct CPU computing.

Nevertheless, this blog was written mainly to serve as food for thought and help you understand the possibilities once devs for CG products start getting more involved with GPU acceleration, and not a pee-contest on how much money one can throw into a 4U rack.

Hardware and Software advancements have always been playing “chickens and eggs”, with either sides taking turns in making the next big step.

Disclaimer: some of the links contained within this post and resource pages have my referral ID, leading to a small commission for each actual sale  that originates directly from my material. This doesn’t mean I won’t strive to provide you with objective information and honest opinions that will potentially help you on educated decisions. If you like the information in this site, please support it by using the links below when you decide to buy.

10 thoughts on “CG Workstation – The Baby “Titan” Supercomputer

  1. Hi Dmitris, Once again I need you advice. I think to update my GTX 680 to Titan or 780ti EVGA is going to release GTX 780ti with 6Gb of RAM, but the question is do I really need floating point precision of Titan for VrayRT and Cycles? Thanks, Roman

    • No, I don’t think you will see major differences between the Titan (Black or not) and a 780Ti. It should be the same card as far as VRay RT / Cycles / Luxrender / Octane.

      I’ve seen very predictable performance scaling with all Kepler cards so far (e.g. you add 2 cards with the same cores * clocks aggregate as a Titan, you will get Titan-like performance, you add 2 Titans or 2* 770s = 2* performance etc)

    • I don’t believe OS GPU limitations are that relevant. As long as the GPGPU program detects a CUDA enabled device, should be able to use it, and you can do so even over LAN. Win OS might be limited to 7 or so GPUs driving monitors, but in this built one of the GTX cards or if you wish for an 8th dedicated viewport card is needed to do so. The rest is pure compute.

    • Do some research. Chances are that this chassis will be available in competitive prices. Now that Haswell-E is out and s2011 platform is obsolete, Tyan & its vendors will probably try to push the stock off the shelves. GPGPU performance won’t be affected either way. Using GTX980 – hopefully the 8GB version is not that far, should be ~$700ish – you will probably get better compute all-around, for less power consumption & probably noise. Important when we are talking 8 GPUs.

  2. Dmitris, Just got the info from guys over Octane’s forum that GTX 980 which is maxwell architecture apparently is not quite fast comparing to GTX 780 (keppler). I think i would stick with GTX 780 for now. I read from other forum , that somebody got problem with 8x GTX 780 on Tyan FT77A . it only run 7 GPUs. That make me wary a bit. link attached below: http://hashcat.net/forum/thread-3330.html

    • The performance of Maxwell cores – at least with OpenCL instructions – is far superior that that of Kepler, on par or better than R9 290X in some cases (which is lotsa faster than GK110). I guess Maxwell GPGPU engine will be revised in the future, but if 780s suffice for you atm, you should be fine. Especially if 3GB VRam will do it for you, there are lots of offers and rebates on 780s that need to “go” and make room for 9xx orders. 6GB 780s are in far shorter supply and I doubt you can find them @ great prices. As far as the 8 card limit goes, I think you can still install manually the 6th-8th card if Windows and/or NVidia driver fails to automatically detect them. It is a BIOS issue I think and not that of the OS. I know ppl using 8-GPU setups, tho I never had one setup myself. You should manually guide the GPU(s) the were not connected to find drivers from the location those are deployed (C:\nvidia for example) through the Device Manager > Install Drivers > Have Disk option.

  3. Seriously considering one of these for home use. When doing compute work, I’m sure it’ll be noisy, but curious if those 180mm fans spin down enough to be acceptable while sitting in the same room.

Leave a Reply to Dimitris Cancel reply

Your email address will not be published. Required fields are marked *