From my understanding, people began using GPUs for general computing because they are an extra source of computing power. And though they are not a fast as a CPU for each operation, they have many cores, so they can be better adapted for parallel processing than a CPU. This makes sense if you already own a computer that happens to have a GPU for graphics processing, but you don't need the graphics, and would like some more computational power. But I also understand that people buy GPUs specifically to add computing power, with no intention to use them to process graphics. To me, this seems similar to the following analogy:

I need to cut my grass, but my lawn mower is wimpy. So I remove the cage from the box fan I keep in my bedroom and sharpen the blades. I duct tape it to my mower, and I find that it works reasonably well. Years later, I am the purchasing officer for a large lawn-care business. I have a sizable budget to spend on grass-cutting implements. Instead of buying lawn mowers, I buy a bunch of box fans. Again, they work fine, but I have to pay for extra parts (like the cage) that I won't end up using. (for the purposes of this analogy, we must assume that lawn mowers and box fans cost about the same)

So why is there not a market for a chip or a device that has the processing power of a GPU, but not the graphics overhead? I can think of a few possible explanations. Which of them, if any, is correct?

  • Such an alternative would be too expensive to develop when the GPU is already a fine option (lawn mowers don't exist, why not use this perfectly good box fan?).
  • The fact that 'G' stands for graphics denotes only an intended use, and does not really mean that any effort goes into making the chip better adapted to graphics processing than any other sort of work (lawn mowers and box fans are the same thing when you get right down to it; no modifications are necessary to get one to function like the other).
  • Modern GPUs carry the same name as their ancient predecessors, but these days the high end ones are not designed to specifically process graphics (modern box fans are designed to function mostly as lawn mowers, even if older one weren't).
  • It is easy to translate pretty much any problem into the language of graphics processing (grass can be cut by blowing air over it really fast).

EDIT:

My question has been answered, but based on some of the comments and answers, I feel that I should clarify my question. I'm not asking why everyone doesn't buy their own computations. Clearly that would be too expensive most of the time.

I simply observed that there seems to be a demand for devices that can quickly perform parallel computations. I was wondering why it seems that the optimal such device is the Graphics Processing Unit, as opposed to a device designed for this purpose.

  • 66
    Because they are specialized for this type of thing; it's basically the same type of math. And nVidia has built and sold GPU-only boards for people to do this type of massively parallel number crunching. – Heptite Jun 5 at 3:07
  • 7
    Keep in mind that we do have specialised "units" added to chips. AES is done in hardware (I think) on CPUs. AVX is implemented in hardware too. However, where do you stop? The Chipmaker does not know what you need and most people do not have the capabilities (technological or financial) to have their own chips designed for very specific tasks. Graphics cards are - as other said - one type of specialised architecture, which lends itself well to certain tasks. They aren't good for everything - but for certain specific tasks and thus used there. – DetlevCM Jun 5 at 7:13
  • 4
    A more accurate analogy would replace the box fans with 100-meter wide farming combines. – MooseBoys Jun 5 at 7:44
  • 6
    My PC already has a ready to use GPU, designing and producing a dedicated chip would set me back a couple of millions. – PlasmaHH Jun 5 at 8:43
  • 19
    Try another analogy. Suppose we have box fans, and we have helicopter rotors. In our hypothetical world, applications for box fans needed progresssively bigger fans running at higher speeds, until we ended up with 20m carbon-fibre-blade box fans, and mass-production made them cheap. Then someone realised that a 20m box fan is essentially just a helicopter rotor with a cage around it. It really is that similar. – Graham Jun 5 at 10:26

10 Answers 10

up vote 109 down vote accepted

It's really a combination of all your explanations. Cheaper and easier, already exists, and design has shifted away from pure graphics.


A modern GPU can be viewed as primarily stream processors with some additional graphics hardware (and some fixed-function accelerators, e.g. for encoding and decoding video). GPGPU programming these days uses APIs specifically designed for this purpose (OpenCL, Nvidia CUDA, AMD APP).

Over the last decade or two, GPUs have evolved from a fixed-function pipeline (pretty much graphics only) to a programmable pipeline (shaders let you write custom instructions) to more modern APIs like OpenCL that provide direct access to the shader cores without the accompanying graphics pipeline.

The remaining graphics bits are minor. They're such a small part of the cost of the card that it isn't significantly cheaper to leave them out, and you incur the cost of an additional design. So this is usually not done — there is no compute-oriented equivalent of most GPUs — except at the highest tiers, and those are quite expensive.

Normal "gaming" GPUs are very commonly used because economies of scale and relative simplicity make them cheap and easy to get started with. It's a fairly easy path from graphics programming to accelerating other programs with GPGPU. It's also easy to upgrade the hardware as newer and faster products are available, unlike the other options.


Basically, the choices come down to:

  • General-purpose CPU, great for branching and sequential code
  • Normal "gaming" GPU
  • Compute-oriented GPU, e.g. Nvidia Tesla and Radeon Instinct These often do not support graphics output at all, so GPU is a bit of a misnomer. However, they do use similar GPU cores to normal GPUs and OpenCL/CUDA/APP code is more or less directly portable.
  • FPGAs, which use a very different programming model and tends to be very costly. This is where a significant barrier to entry exists. They're also not necessarily faster than a GPU, depending on the workload.
  • ASICs, custom-designed circuits (hardware). This is very very expensive and only becomes worth it with extreme scale (we're talking many thousands of units, at the very least), and where you're sure the program will never need to change. They are rarely feasible in the real world. You'll also have to redesign and test the entire thing every time technology advances - you can't just swap in a new processor like you can with CPUs and GPUs.
  • 16
    ASICs also make sense when the computing literally pays for itself (crypto mining) – ratchet freak Jun 5 at 9:22
  • 4
    Actually, FPGA's are often worse than GPU's. The problem is that FPGA's are very flexible; they can implement many various operations. However, computation is generally a form of math, and in fact the bulk is just two operations : addition and multiplication (subtraction and division are variants of the above). GPU's are very, very good at those two operations, much more so than FPGA's. – MSalters Jun 5 at 11:20
  • 19
    You need to clarify more about FPGA's. The idea that there are a "step up" is a bit misleading. They are more of a step sideways. – Yakk Jun 5 at 14:07
  • 6
    As an example of the last one, Google has their own "Tensor processing units" for machine learning. To what degree they're customized is unclear, but are described as being ASICs. – mbrig Jun 5 at 20:21
  • 4
    @MSalters One of the main selling points of FPGAs over GPUs is performance/Watt, which is getting more important as data centres start to hit the power wall (FPGAs are generally more power efficient). As far as math, FPGAs are comparable to GPUs in fixed-point and integer arithmetic, and only lag in floating-point math. – wilcroft Jun 6 at 18:30

My favorite analogy:

  • CPU: A Polymath genius. Can do one or two things at a time but those things can be very complex.
  • GPU: A ton of low skilled workers. Each of them can't do very big problems, but in mass you can get a lot done. To your question, yes there is some graphics overhead but I believe it's marginal.
  • ASIC/FPGA: A company. You can hire a ton of low skilled workers or a couple of geniuses, or a combination of low skilled workers and geniuses.

What you use depends on cost sensitivity, the degree to which a task is parallelizable, and other factors. Because of how the market has played out, GPUs are the best choice for most highly parallel applications and CPUs are the best choice when power and unit cost are the primary concerns.

Directly to your question: why a GPU over an ASIC/FPGA? Generally cost. Even with today's inflated GPU prices, it is still (generally) cheaper to use a GPU than designing an ASIC to meet your needs. As @user912264 points out, there are specific tasks that can be useful for ASICs/FPGAs. If you have a unique task and you will benefit from scale then it can be worth it to design an ASIC/FPGA. In fact, you can design/buy/license FPGA designs specifically for this purpose. This is done to power the pixels in high definition TVs for example.

  • 7
    Comments are not for answering anyway, and this seems like a reasonable answer to me. – Raimund Krämer Jun 6 at 10:27
  • 1
    @BobtheMogicMoose But it might be orders of magnitude faster to use a custom FPGA designed for genomic analysis than to have the equivalent code in a GPU. When you're paying scientists to sit around waiting for the results, the faster FPGA pays for itself very quickly. – user912264 Jun 6 at 17:27
  • FPGAs are getting a lot more accessible to the common developer too - Microsoft for instance has a cloud AI solution using FPGAs (Project BrainWave). AWS has some offerings as well. Anyone can rent out some custom FPGAs for specialized tasks without having to build it themselves, not feasible for many use cases even a few years ago. – brichins Jun 7 at 23:28
  • Yeah, I think there are even FPGA hobby kits that are comparable to an arduino raspberry-pi. I still think programming FPGAs is far more costly that more developed architectures. – BobtheMagicMoose Jun 8 at 13:32

Your analogy is bad. In the analogy, when you're buying equipment for a large lawn care business, you assume there are good lawn mowers available. This is not the case in the computing world - GPUs are the best tool readily available.

The R&D costs and possible performance gains for a specialized chip are likely too high to justify making one.

That said, I'm aware of Nvidia putting out some GPUs specifically for general purpose computing - they had no video outputs - a bit like selling box fans with the cages already removed.

Of course, you can use specialized chips, either for energy-efficiency or calculation speed. Let me tell you the history of Bitcoin mining:

  • Bitcoin is new, geeks mine with their CPUs.
  • Bitcoin is somewhat new, smart geeks mine with their GPUs.
  • Bitcoin is now (kinda) famous, people buy FPGAs.
  • Bitcoin is now famous (2013), even newbies buy ASICs ("Application Specific Integrated Circuits") to mine efficiently.
  • Block reward drops (periodically), even old ASICs are not profitable anymore.

So no, there are no reasons to use a GPU instead of a specialized "giant calculator". The bigger the economical incentives, the more the hardware gets specialized. However, they are quite hard to design and infeasible to manufacture if you're not producing thousands at once. If it's not viable to design chips, you can buy one of those from the nearest Walmart.

TL;DR Of course you can use more specialized chips.

  • 1
    "Of course you can use more specialized chips" - but there are specialized chips for bitcoin ( SHA-256), then for litecoin(scrypt) and that is pretty much it. High-performance computing hardware for other problems doesn't exist. (That is, with performance higher than current high-end GPUs) – Agent_L Jun 7 at 10:20

What you describe in your analogy is exactly what happened. Just as you grabbed your fan and sharpened the blades to try to use it as a mower, a group of researches realized "Hey, we have some pretty nice multi-core processing unit here, lets try to use it for general-purpose computations!".

The result was good and the ball started rolling. The GPU went from a graphics-only device to support general-purpose computation to assist in the most demanding situations.

Because anyway the most computationally demanding operation we expect from computers are graphics. Its enough to take a look at the stunning advances of how games look today as compared to how they did just a few years ago. This means that a lot of effort and money has gone into the development of the GPUs, and the fact that they could also be used to speed up a certain class of general-purpose computation (i.e. extremely parallel) just added to their popularity.

So in conclusion, the first explanation that you offer is the most accurate:

  • Such an alternative would be too expensive to develop when the GPU is already a fine option.

GPUs where already there, they are readily available to everyone and they worked.

  • 5
    I have to disagree about "the most computationally demanding operation" being graphics, depending of course on exactly who "we" is. For general users, yes, but in the science & engineering community, there are many things more demanding than graphics. After all, acceptable graphics (as for games) can be done with a single mid-range PC and GPU combo. Significant problems often combine hundreds or thousands of such units to get performance in the petaflop range - and then problems still may take days or weeks of compute time. – jamesqf Jun 5 at 17:18
  • The most computationally demanding operation I expect from my computer is technically graphics, but structure-from-motion computations are not what most people (or GPU designers) think of when they hear the word "graphics". – Mark Jun 5 at 21:30

Specifically, GPUs are not "cores" in the sense of "task parallelism". For the most part, it is in the form of "data parallelism". SIMD is "single instruction multiple data". What this means is that you would not do this:

for parallel i in range(0,1024): c[i] = a[i] * b[i]

This would mean that you have 1024 instruction pointers all performing separate tasks progressing at different rates. SIMD, or "vector computing" will perform instructions across whole arrays all at once, more like this:

c = a * b

The "loops" are in the "*" and "=" instructions, rather than outside the instructions. The above would do this for all of the 1024 elements at the same time, at the SAME instruction pointer for all of them. It's like having three huge registers for a, b, and c. SIMD code is extremely constrained, and only works well for problems that are not excessively "branchy".

In realistic cases, these SIMD values are not quite as large as 1024 items. Imagine a variable that's a gang of int32 bound together. You can think of the multiply and assign as a real machine instruction.

int32_x64 c; int32_x64 b; int32_x64 a; c = b * a;

Real GPUs are more complicated than SIMD, but that's the essence of them. It's why you can't just throw a random CPU algorithm onto a GPU and expect a speedup. The more instruction branching the algorithm does, the less appropriate it is for a GPU.

The other answers here are pretty good. I'll throw in my 2 cents as well.

One reason that CPUs have become so pervasive is that they are flexible. You can reprogram them for an infinite variety of tasks. These days it's cheaper and faster for companies that produce products to stick a small CPU or microcontroller in something and program it's functionality than to develop custom circuitry to do the same task.

By using the same device as others, you can take advantage of the known solutions to problems using that same device (or similar). And as the platform matures, your solutions evolve and become very mature and optimized. The people coding on these devices also gain expertise and become very good at their craft.

If you were to create a new device type from scratch, some alternative to a GPU, it would take years for even the earliest adopters to actually get good at knowing how to use it. If you attach an ASIC to your CPU, how do you optimize offloading computation onto that device?

The computer architecture community has been abuzz with this idea for several years (obviously it's been popular before, but has recently seen a renaissance). These 'accelerators' (their term) have varying degrees of reprogrammability. The problem is, how narrowly do you define the scope of the problem that your accelerator can tackle? I've even talked to some people who were working creating an accelerator using analog circuits with op-amps to compute differential equations. Great idea, but extremely narrow scope.

After you have a working accelerator, economic forces are going to decide your fate. Market inertia is an incredible force. Even if something is a great idea, is it economically feasible to refactor your working solutions to use this new device? Maybe, maybe not.

GPUs are actually horrible for certain types of problems, so lots of people/companies are working on other types of devices. But GPUs are already so entrenched, will their devices ever become economically viable? I guess we'll see.

Edit: Expanding on my answer a bit, now that I'm off the bus.

A cautionary case study is the Intel Larrabee project. It started off as a parallel processing device that could do graphics in software; it had no specialized graphics hardware. I spoke with someone who worked on the project, and a major reason they said it failed and was canceled (besides horrible internal politics) was that they just couldn't get the compiler to produce good code for it. Of course it produced working code, but if the entire point of your product is maximum performance you better have a compiler that produces pretty optimal code. This hearkens back to my earlier comment about a lack of deep expertise in both hardware and software for your new device being a big problem.

Some elements of the Larrabee design made it into the Xeon Phi/Intel MIC. This product actually made it to market. It was entirely focused on parallelizing scientific and other HPC-type computations. It looks like it is a commercial failure now. Another person I spoke with at Intel implied that they just were not price/performance competitive with GPUs.

People have tried to integrate logic synthesis for FPGAs into compilers, so that you can automatically generate code for your FPGA accelerators. They don't work that well.

One place that seems to be really fertile soil for accelerators, or other alternatives to GPUs, is the cloud. The economy of scale that exists at these large companies like Google, Amazon, and Microsoft makes investing in alternative computation schemes worthwhile. Someone already mentioned Google's tensor processing units. Microsoft has FPGAs and other stuff throughout its Bing and Azure infrastructure. Same story with Amazon. It absolutely makes sense if the scale can offset your investment in time, money, and engineer tears.

In summary, specialization is at odds with a lot of other things (economics, maturity of the platform, engineering expertise, etc). Specialization can significantly improve your performance, but it narrows the scope to which your device is applicable. My answer focused on a lot of the negatives, but specialization has a ton of benefits too. It absolutely should be pursued and investigated, and as I mentioned many groups are pursuing it quite aggressively.

Sorry, edit again: I think your initial premise is wrong. I believe it was less a case of looking for extra sources of computing power, and more a case of people recognizing an opportunity. Graphics programming is very linear algebra heavy, and the GPU was designed to efficiently perform common operations like matrix-multiply, vector operations, etc. Operations that are also very common to scientific computing.

The interest in GPUs started just as people came to recognize that the promises given by the Intel/HP EPIC project were vastly overstated (late 90's early 2000's). There was no general solution to compiler parallelization. So rather than saying "where do we find more processing power, oh we could try the GPU", I think it was more "we have something that is good at parallel calculations, can we make this more generally programmable". A lot of the people involved were in the scientific computing community, who already had parallel Fortran code they could run on Cray or Tera machines (Tera MTA had 128 hardware threads). Perhaps there was movement from both directions, but I've only heard mentions of the origins of GPGPU from this direction.

  • By "accelerators" are you referring to custom made hardware or super clusters of low power computing nodes? Can you elaborate by providing reference to some example accelerator hardware. – manav m-n Jun 6 at 5:47
  • Sorry, I thought I made that clear from context. Accelerator is just an umbrella term for a coprocessor or offload card. Floating point was originally in a coprocessor and not the main CPU, and it would have been considered an accelerator. GPUs, DSPs, the Xeon Phi, FPGAs when they’re on a PCIe card or something similar, the analog differential equation thing I mentioned, there are devices that aid in virtualization, there is current research in neural network accelerators. Those are all examples of accelerators. – NerdPirate Jun 6 at 13:33

enter image description here

An ASIC (custom silicon) is very fast, but it's very expensive to design and manufacture. ASIC's used to be purpose-specific, and the CPU was one approach which allowed computers to be "programmed" so computing tasks could be performed by software. Early CPU's gave people the ability to take advantage of the power of ASIC without the massive cost by programming the chip in the field. This approach became SO successful that it gave rise to the (very) fast computer you're using right now.

So why GPUs?

In the mid-90's, 3DFX realized that 3D-rendering tasks were so specific that a custom ASIC would perform MUCH better than a CPU. They created a computer co-processor that offloaded 3D rendering tasks from the CPU to this co-processor, which they dubbed a "GPU". Competition and market demand drove innovation in this space to a point where GPU's were performing calculations MUCH faster than CPU's, so the question arose, "Why can't I use the GPU to crunch my numbers instead of the CPU?" GPU manufacturers saw a demand and a way to make more money, so they started altering their platforms to allow developers to use their hardware. But the hardware hardware was so purpose-specific that there were, and still are, limitations in what you can ask the GPU to do. I won't go into specifics on why here.

So why wasn't there more purpose-specific silicon? Why JUST graphics?

Two reasons: 1) Price. GPU's had a good market, and could justify it, but even back then, it was a huge risk. No one really knew if 3DFX could make a profit (turns out, they couldn't actually, and went defunct). Even now, with the size of the GPU market, there are really only 3 competitors. 2) CPUs were actually meeting the need for "custom silicon" with instruction extensions. Think back to MMX - this was actually Intel's attempt to accelerate graphics in the CPU right as 3DFX was gaining speed. Since then, the x86 instruction set has grown to be quite massive with all of these custom extensions. Many of these extensions made sense at the time (like MMX), but are largely just dead-weight in the processor now. You can't remove them, though, because then it breaks existing software. It's actually one of the selling-points for ARM - ARM is a stripped down instruction set. There aren't as many instruction extensions, but this makes the silicon smaller and cheaper to manufacture.

Seems to me like you could make a lot of money if you could reduce the cost of custom silicon. Isn't anyone working on this?

There is a technology called FPGA - field programmable gate array, that's been around since the early days of computing. It's essentially a microchip you can design "in the field" using software. It's very cool technology, but all of the structure needed to make the chip programmable takes up a LOT of silicon and causes the chips to run at much lower clock speeds. FPGA's CAN be faster than CPU's, if you have enough silicon on the chip AND can effectively parallelize the task. But they're limited in how much logic you can put on them. All but the most expensive FPGA's were slower than GPU's for early bitcoin mining, but their ASIC counterparts effectively ended the profitability of GPUs mining. Other cryptocurrencies have used specific algorithms that cannot be parallelized, so FPGA's and ASIC's aren't better enough than CPU's and GPUs to justify the cost.

The main limiter with FPGA's is silicon size - how much logic can you fit on the chip? The second is clock speed, because it's hard to optimize things like hot spots, leakage, and cross-talk in an FPGA. Newer fabrication methods have minimized these issues, and Intel has teamed up with Altera to provide an FPGA that can be used by engineers to leverage the benefits of "custom silicon" as a co-processor in a server. So it's coming, in a sense.

Will FPGA's ever be replace CPU's and GPU's?

Probably not anytime soon. The latest CPU's and GPUs are MASSIVE and the silicon highly tuned for thermal and electrical performance. You can't optimize FPGA's in the same way you can a custom ASIC. Barring some ground-breaking technology, the CPU will likely remain the core of your computer with FPGA and GPU coprocessors.

  • 1
    Many of these extensions made sense at the time (like MMX), but are largely just dead-weight in the processor now. 3D rendering is far from the only use-case for SIMD. Most of the "weight" of MMX is the execution units, and those can be shared with wider vector like SSE2, AVX2, and AVX512. Those are heavily used for high-quality video-encoding on CPUs, and many many other tasks, including high-performance computing. But also library implementations of memchr, strlen, and lots of other stuff. e.g. filtering an array more than 1 element at a time – Peter Cordes Jun 10 at 3:41

Indeed there are specialized board for high-speed computing, e.g. Xilinx has a list of 178 PCI-e boards using their FPGAs, and about a third of these boards are "number crunchers" with one or several powerful FPGA chips and lots of on-board DDR memory. There are also high-performance DSP boards (example) aimed at high-performance computing tasks.

I guess the popularity of GPU boards stems from their aim at a wider customer group. You don't have to invest in special hardware to play with Nvidia CUDA, so by the time you do have a task which requires special hardware, Nvidia GPUs will have a competitive edge in that you already know how to program them.

I think the answer for your question depending on how to define high-performance computation.

In general, the high-performance computation is related to computation time. In that case, I like to share the link of high-performance computing cluster.

The link is specified the reason of usage of the GPU; The use of graphics cards (or rather their GPU's) to do calculations for grid computing is vastly more economical than using CPU's, despite being less precise.

  • 2
    High-end GPGPUs have good throughput for 64-bit double-precision, not just single-precision 32-bit float. (Some regular GPUs skimp on HW for double). The major vendors all support IEEE FP math (I think even with denormals). So there's no precision loss unless you want to trade precision for performance, e.g. with 16-bit half-precision FP which has even better throughput on some hardware (and of course half the memory bandwidth). High-performance code on CPUs often uses 32-bit float as well, to get twice as many elements per SIMD vector and half the memory bandwidth. – Peter Cordes Jun 5 at 6:28
  • 1
    @PeterCordes I've seen some work in approximate computing that even goes down to eight-bit floating point, though I don't think many GPUs support that in hardware. – JAB Jun 5 at 6:43

protected by Community Jun 6 at 0:51

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.