Why do people use GPUs for high-performance computation instead of a more specialized chip?

Question

From my understanding, people began using GPUs for general computing because they are an extra source of computing power. And though they are not a fast as a CPU for each operation, they have many cores, so they can be better adapted for parallel processing than a CPU. This makes sense if you already own a computer that happens to have a GPU for graphics processing, but you don't need the graphics, and would like some more computational power. But I also understand that people buy GPUs specifically to add computing power, with no intention to use them to process graphics. To me, this seems similar to the following analogy:

I need to cut my grass, but my lawn mower is wimpy. So I remove the cage from the box fan I keep in my bedroom and sharpen the blades. I duct tape it to my mower, and I find that it works reasonably well. Years later, I am the purchasing officer for a large lawn-care business. I have a sizable budget to spend on grass-cutting implements. Instead of buying lawn mowers, I buy a bunch of box fans. Again, they work fine, but I have to pay for extra parts (like the cage) that I won't end up using. (for the purposes of this analogy, we must assume that lawn mowers and box fans cost about the same)

So why is there not a market for a chip or a device that has the processing power of a GPU, but not the graphics overhead? I can think of a few possible explanations. Which of them, if any, is correct?

• Such an alternative would be too expensive to develop when the GPU is already a fine option (lawn mowers don't exist, why not use this perfectly good box fan?).
• The fact that 'G' stands for graphics denotes only an intended use, and does not really mean that any effort goes into making the chip better adapted to graphics processing than any other sort of work (lawn mowers and box fans are the same thing when you get right down to it; no modifications are necessary to get one to function like the other).
• Modern GPUs carry the same name as their ancient predecessors, but these days the high end ones are not designed to specifically process graphics (modern box fans are designed to function mostly as lawn mowers, even if older one weren't).
• It is easy to translate pretty much any problem into the language of graphics processing (grass can be cut by blowing air over it really fast).

EDIT:

My question has been answered, but based on some of the comments and answers, I feel that I should clarify my question. I'm not asking why everyone doesn't buy their own computations. Clearly that would be too expensive most of the time.

I simply observed that there seems to be a demand for devices that can quickly perform parallel computations. I was wondering why it seems that the optimal such device is the Graphics Processing Unit, as opposed to a device designed for this purpose.

Answer 1

It's really a combination of all your explanations. Cheaper and easier, already exists, and design has shifted away from pure graphics.

---

A modern GPU can be viewed as primarily stream processors with some additional graphics hardware (and some fixed-function accelerators, e.g. for encoding and decoding video). GPGPU programming these days uses APIs specifically designed for this purpose (OpenCL, Nvidia CUDA, AMD APP).

Over the last decade or two, GPUs have evolved from a fixed-function pipeline (pretty much graphics only) to a programmable pipeline (shaders let you write custom instructions) to more modern APIs like OpenCL that provide direct access to the shader cores without the accompanying graphics pipeline.

The remaining graphics bits are minor. They're such a small part of the cost of the card that it isn't significantly cheaper to leave them out, and you incur the cost of an additional design. So this is usually not done — there is no compute-oriented equivalent of most GPUs — except at the highest tiers, and those are quite expensive.

Normal "gaming" GPUs are very commonly used because economies of scale and relative simplicity make them cheap and easy to get started with. It's a fairly easy path from graphics programming to accelerating other programs with GPGPU. It's also easy to upgrade the hardware as newer and faster products are available, unlike the other options.

---

Basically, the choices come down to:

• General-purpose CPU, great for branching and sequential code
• Normal "gaming" GPU
• Compute-oriented GPU, e.g. Nvidia Tesla and Radeon Instinct These often do not support graphics output at all, so GPU is a bit of a misnomer. However, they do use similar GPU cores to normal GPUs and OpenCL/CUDA/APP code is more or less directly portable.
• FPGAs, which use a very different programming model and tends to be very costly. This is where a significant barrier to entry exists. They're also not necessarily faster than a GPU, depending on the workload.
• ASICs, custom-designed circuits (hardware). This is very very expensive and only becomes worth it with extreme scale (we're talking many thousands of units, at the very least), and where you're sure the program will never need to change. They are rarely feasible in the real world. You'll also have to redesign and test the entire thing every time technology advances - you can't just swap in a new processor like you can with CPUs and GPUs.

Answer 2

My favorite analogy:

• CPU: A Polymath genius. Can do one or two things at a time but those things can be very complex.
• GPU: A ton of low skilled workers. Each of them can't do very big problems, but in mass you can get a lot done. To your question, yes there is some graphics overhead but I believe it's marginal.
• ASIC/FPGA: A company. You can hire a ton of low skilled workers or a couple of geniuses, or a combination of low skilled workers and geniuses.

What you use depends on cost sensitivity, the degree to which a task is parallelizable, and other factors. Because of how the market has played out, GPUs are the best choice for most highly parallel applications and CPUs are the best choice when power and unit cost are the primary concerns.

Directly to your question: why a GPU over an ASIC/FPGA? Generally cost. Even with today's inflated GPU prices, it is still (generally) cheaper to use a GPU than designing an ASIC to meet your needs. As @user912264 points out, there are specific tasks that can be useful for ASICs/FPGAs. If you have a unique task and you will benefit from scale then it can be worth it to design an ASIC/FPGA. In fact, you can design/buy/license FPGA designs specifically for this purpose. This is done to power the pixels in high definition TVs for example.

Answer 3

Your analogy is bad. In the analogy, when you're buying equipment for a large lawn care business, you assume there are good lawn mowers available. This is not the case in the computing world - GPUs are the best tool readily available.

The R&D costs and possible performance gains for a specialized chip are likely too high to justify making one.

That said, I'm aware of Nvidia putting out some GPUs specifically for general purpose computing - they had no video outputs - a bit like selling box fans with the cages already removed.

Answer 4

Of course, you can use specialized chips, either for energy-efficiency or calculation speed. Let me tell you the history of Bitcoin mining:

• Bitcoin is new, geeks mine with their CPUs.
• Bitcoin is somewhat new, smart geeks mine with their GPUs.
• Bitcoin is now (kinda) famous, people buy FPGAs.
• Bitcoin is now famous (2013), even newbies buy ASICs ("Application Specific Integrated Circuits") to mine efficiently.
• Block reward drops (periodically), even old ASICs are not profitable anymore.

So no, there are no reasons to use a GPU instead of a specialized "giant calculator". The bigger the economical incentives, the more the hardware gets specialized. However, they are quite hard to design and infeasible to manufacture if you're not producing thousands at once. If it's not viable to design chips, you can buy one of those from the nearest Walmart.

TL;DR Of course you can use more specialized chips.

Answer 5

What you describe in your analogy is exactly what happened. Just as you grabbed your fan and sharpened the blades to try to use it as a mower, a group of researches realized "Hey, we have some pretty nice multi-core processing unit here, lets try to use it for general-purpose computations!".

The result was good and the ball started rolling. The GPU went from a graphics-only device to support general-purpose computation to assist in the most demanding situations.

Because anyway the most computationally demanding operation we expect from computers are graphics. Its enough to take a look at the stunning advances of how games look today as compared to how they did just a few years ago. This means that a lot of effort and money has gone into the development of the GPUs, and the fact that they could also be used to speed up a certain class of general-purpose computation (i.e. extremely parallel) just added to their popularity.

So in conclusion, the first explanation that you offer is the most accurate:

• Such an alternative would be too expensive to develop when the GPU is already a fine option.

GPUs where already there, they are readily available to everyone and they worked.

Answer 6

Specifically, GPUs are not "cores" in the sense of "task parallelism". For the most part, it is in the form of "data parallelism". SIMD is "single instruction multiple data". What this means is that you would not do this:

for parallel i in range(0,1024): c[i] = a[i] * b[i]

This would mean that you have 1024 instruction pointers all performing separate tasks progressing at different rates. SIMD, or "vector computing" will perform instructions across whole arrays all at once, more like this:

c = a * b

The "loops" are in the "*" and "=" instructions, rather than outside the instructions. The above would do this for all of the 1024 elements at the same time, at the SAME instruction pointer for all of them. It's like having three huge registers for a, b, and c. SIMD code is extremely constrained, and only works well for problems that are not excessively "branchy".

In realistic cases, these SIMD values are not quite as large as 1024 items. Imagine a variable that's a gang of int32 bound together. You can think of the multiply and assign as a real machine instruction.

int32_x64 c; int32_x64 b; int32_x64 a; c = b * a;

Real GPUs are more complicated than SIMD, but that's the essence of them. It's why you can't just throw a random CPU algorithm onto a GPU and expect a speedup. The more instruction branching the algorithm does, the less appropriate it is for a GPU.

Answer 7

The other answers here are pretty good. I'll throw in my 2 cents as well.

One reason that CPUs have become so pervasive is that they are flexible. You can reprogram them for an infinite variety of tasks. These days it's cheaper and faster for companies that produce products to stick a small CPU or microcontroller in something and program it's functionality than to develop custom circuitry to do the same task.

By using the same device as others, you can take advantage of the known solutions to problems using that same device (or similar). And as the platform matures, your solutions evolve and become very mature and optimized. The people coding on these devices also gain expertise and become very good at their craft.

If you were to create a new device type from scratch, some alternative to a GPU, it would take years for even the earliest adopters to actually get good at knowing how to use it. If you attach an ASIC to your CPU, how do you optimize offloading computation onto that device?

The computer architecture community has been abuzz with this idea for several years (obviously it's been popular before, but has recently seen a renaissance). These 'accelerators' (their term) have varying degrees of reprogrammability. The problem is, how narrowly do you define the scope of the problem that your accelerator can tackle? I've even talked to some people who were working creating an accelerator using analog circuits with op-amps to compute differential equations. Great idea, but extremely narrow scope.

After you have a working accelerator, economic forces are going to decide your fate. Market inertia is an incredible force. Even if something is a great idea, is it economically feasible to refactor your working solutions to use this new device? Maybe, maybe not.

GPUs are actually horrible for certain types of problems, so lots of people/companies are working on other types of devices. But GPUs are already so entrenched, will their devices ever become economically viable? I guess we'll see.

Edit: Expanding on my answer a bit, now that I'm off the bus.

A cautionary case study is the Intel Larrabee project. It started off as a parallel processing device that could do graphics in software; it had no specialized graphics hardware. I spoke with someone who worked on the project, and a major reason they said it failed and was canceled (besides horrible internal politics) was that they just couldn't get the compiler to produce good code for it. Of course it produced working code, but if the entire point of your product is maximum performance you better have a compiler that produces pretty optimal code. This hearkens back to my earlier comment about a lack of deep expertise in both hardware and software for your new device being a big problem.

Some elements of the Larrabee design made it into the Xeon Phi/Intel MIC. This product actually made it to market. It was entirely focused on parallelizing scientific and other HPC-type computations. It looks like it is a commercial failure now. Another person I spoke with at Intel implied that they just were not price/performance competitive with GPUs.

People have tried to integrate logic synthesis for FPGAs into compilers, so that you can automatically generate code for your FPGA accelerators. They don't work that well.

One place that seems to be really fertile soil for accelerators, or other alternatives to GPUs, is the cloud. The economy of scale that exists at these large companies like Google, Amazon, and Microsoft makes investing in alternative computation schemes worthwhile. Someone already mentioned Google's tensor processing units. Microsoft has FPGAs and other stuff throughout its Bing and Azure infrastructure. Same story with Amazon. It absolutely makes sense if the scale can offset your investment in time, money, and engineer tears.

In summary, specialization is at odds with a lot of other things (economics, maturity of the platform, engineering expertise, etc). Specialization can significantly improve your performance, but it narrows the scope to which your device is applicable. My answer focused on a lot of the negatives, but specialization has a ton of benefits too. It absolutely should be pursued and investigated, and as I mentioned many groups are pursuing it quite aggressively.

Sorry, edit again: I think your initial premise is wrong. I believe it was less a case of looking for extra sources of computing power, and more a case of people recognizing an opportunity. Graphics programming is very linear algebra heavy, and the GPU was designed to efficiently perform common operations like matrix-multiply, vector operations, etc. Operations that are also very common to scientific computing.

The interest in GPUs started just as people came to recognize that the promises given by the Intel/HP EPIC project were vastly overstated (late 90's early 2000's). There was no general solution to compiler parallelization. So rather than saying "where do we find more processing power, oh we could try the GPU", I think it was more "we have something that is good at parallel calculations, can we make this more generally programmable". A lot of the people involved were in the scientific computing community, who already had parallel Fortran code they could run on Cray or Tera machines (Tera MTA had 128 hardware threads). Perhaps there was movement from both directions, but I've only heard mentions of the origins of GPGPU from this direction.

Answer 8

An ASIC (custom silicon) is very fast, but it's very expensive to design and manufacture. ASIC's used to be purpose-specific, and the CPU was one approach which allowed computers to be "programmed" so computing tasks could be performed by software. Early CPU's gave people the ability to take advantage of the power of ASIC without the massive cost by programming the chip in the field. This approach became SO successful that it gave rise to the (very) fast computer you're using right now.

So why GPUs?

In the mid-90's, 3DFX realized that 3D-rendering tasks were so specific that a custom ASIC would perform MUCH better than a CPU. They created a computer co-processor that offloaded 3D rendering tasks from the CPU to this co-processor, which they dubbed a "GPU". Competition and market demand drove innovation in this space to a point where GPU's were performing calculations MUCH faster than CPU's, so the question arose, "Why can't I use the GPU to crunch my numbers instead of the CPU?" GPU manufacturers saw a demand and a way to make more money, so they started altering their platforms to allow developers to use their hardware. But the hardware hardware was so purpose-specific that there were, and still are, limitations in what you can ask the GPU to do. I won't go into specifics on why here.

So why wasn't there more purpose-specific silicon? Why JUST graphics?

Two reasons: 1) Price. GPU's had a good market, and could justify it, but even back then, it was a huge risk. No one really knew if 3DFX could make a profit (turns out, they couldn't actually, and went defunct). Even now, with the size of the GPU market, there are really only 3 competitors. 2) CPUs were actually meeting the need for "custom silicon" with instruction extensions. Think back to MMX - this was actually Intel's attempt to accelerate graphics in the CPU right as 3DFX was gaining speed. Since then, the x86 instruction set has grown to be quite massive with all of these custom extensions. Many of these extensions made sense at the time (like MMX), but are largely just dead-weight in the processor now. You can't remove them, though, because then it breaks existing software. It's actually one of the selling-points for ARM - ARM is a stripped down instruction set. There aren't as many instruction extensions, but this makes the silicon smaller and cheaper to manufacture.

Seems to me like you could make a lot of money if you could reduce the cost of custom silicon. Isn't anyone working on this?

There is a technology called FPGA - field programmable gate array, that's been around since the early days of computing. It's essentially a microchip you can design "in the field" using software. It's very cool technology, but all of the structure needed to make the chip programmable takes up a LOT of silicon and causes the chips to run at much lower clock speeds. FPGA's CAN be faster than CPU's, if you have enough silicon on the chip AND can effectively parallelize the task. But they're limited in how much logic you can put on them. All but the most expensive FPGA's were slower than GPU's for early bitcoin mining, but their ASIC counterparts effectively ended the profitability of GPUs mining. Other cryptocurrencies have used specific algorithms that cannot be parallelized, so FPGA's and ASIC's aren't better enough than CPU's and GPUs to justify the cost.

The main limiter with FPGA's is silicon size - how much logic can you fit on the chip? The second is clock speed, because it's hard to optimize things like hot spots, leakage, and cross-talk in an FPGA. Newer fabrication methods have minimized these issues, and Intel has teamed up with Altera to provide an FPGA that can be used by engineers to leverage the benefits of "custom silicon" as a co-processor in a server. So it's coming, in a sense.

Will FPGA's ever be replace CPU's and GPU's?

Probably not anytime soon. The latest CPU's and GPUs are MASSIVE and the silicon highly tuned for thermal and electrical performance. You can't optimize FPGA's in the same way you can a custom ASIC. Barring some ground-breaking technology, the CPU will likely remain the core of your computer with FPGA and GPU coprocessors.

Answer 9

Indeed there are specialized board for high-speed computing, e.g. Xilinx has a list of 178 PCI-e boards using their FPGAs, and about a third of these boards are "number crunchers" with one or several powerful FPGA chips and lots of on-board DDR memory. There are also high-performance DSP boards (example) aimed at high-performance computing tasks.

I guess the popularity of GPU boards stems from their aim at a wider customer group. You don't have to invest in special hardware to play with Nvidia CUDA, so by the time you do have a task which requires special hardware, Nvidia GPUs will have a competitive edge in that you already know how to program them.

Answer 10

I think the answer for your question depending on how to define high-performance computation.

In general, the high-performance computation is related to computation time. In that case, I like to share the link of high-performance computing cluster.

The link is specified the reason of usage of the GPU; The use of graphics cards (or rather their GPU's) to do calculations for grid computing is vastly more economical than using CPU's, despite being less precise.