12

Given the following CPUs and GeekBench scores:

  • Amazon EC2 z1d.large instance: Intel Xeon Platinum 8151 4061 MHz (1 cores) Single-Core Score: 1094, Multi-core score: 1300

  • Macbook Pro Laptop: Intel Core i5-8259U 2300 MHz (4 cores) Single-Core Score: 1002, Multi-core score : 4104

The Xeon is 9.1% faster in the single-threaded benchmark score.

However, when I compile javascript application code (single-threaded) on both devices, the Xeon completes the task 60% faster. Why? The benchmark score says the Xeon is only 9% faster.

They both have NVME drives, so that shouldn't be the bottleneck. I don't think there'd be a Mac vs Linux OS issue either, since Mac is linux based.

Is this because the Xeon is a server/desktop CPU? and running at 100% speed and power, whereas the Macbook Pro CPU is not running at full power and has to wait for the Intel Turbo Boost to ramp up?

18
  • 16
    What do you mean "compile" JavaScript? JavaScript is an interpreted language. I'm assuming you either mean compiling a Java program or running a JavaScript application. The answer here depends on what you're talking about. – Sam Forbis May 10 '20 at 3:28
  • 22
    Macs are not Linux based. The clocking/tdp and instruction sets of the CPUs likely play a big role. I also expect the Mac has a lot more stuff running in the background. – davidgo May 10 '20 at 3:58
  • 53
    "Mac is linux based" – Huh? The history of macOS is long and convoluted, and it is based on lots of bits and pieces from lots of different Operating Systems (e.g. MacOS, OpenStep, NeXTStep, FreeBSD, 386BSD, Mach, Nu Kernel, …), but Linux is actually one the few that it is not based on. – Jörg W Mittag May 10 '20 at 11:54
  • 5
    @SamForbis these days you actually "compile" js quite often. Some frameworks like Svelte are compile-only. And for the other cases transpiling, minifying and bundling is often referred to as "compiling". – Džuris May 10 '20 at 17:18
  • 8
    @SamForbis it's a build pipe-line. Compiles javascript using Babel: babeljs.io, processes CSS using postCss, bundles the assets using webpack, etc. – dandan May 10 '20 at 21:27
4

Given the task you describe, compiling a Bable project, and the CPUs involved I think I know the source of the difference in performance. I wanted to answer earlier but had to do a bit of research to confirm my hunch.

First, let's characterize the load you're putting on your system.

Babel.js is written as a single-threaded, single process compiler that mostly leverages asynchronous I/O for parallelism (at least nothing I've googled indicates it using worker threads). Since it is a compiler that compiles files from disk a large part of its execution involves waiting for data from disk. This gives us the following workload:

  1. Single-threaded so multiple cores or hyperthreading have no significant effect on compilation with one caveat:

  2. Node.js uses worker threads to handle disk I/O but beyond a two or four hardware threads there is no additional advantages to multiple cores (see: https://nodejs.org/en/docs/guides/dont-block-the-event-loop/)

  3. Most of the parallelism takes place at the I/O level. Babel will try to read as many files in parallel as possible.

Both the i5 and the Xeon are fairly comparable with regards to points 1 and 2. So let's look at how the CPU can handle point 3: servicing Babel's parallel file read request.

Here's the first big difference between the two systems:

  • The Core i5 8259 has 16 PCI lanes

  • The Xeon 8151 has 48 PCI lanes

So clearly the Xeon can handle more parallel I/O operations than the i5. When there's more I/O than the number of memory transfer lanes available the OS handles it the same way as when there's more tasks than number of hardware threads available: it queues them up and force them to take turns.

Next I wanted to know if NVME can actually use multiple lanes. This is where I hit another interesting fact. The NVME standard allows a card to use up to 4 PCI lanes (there is physically that many connections allocated) but some cards use only 2 while others use 4. So not all NVME cards are created equal. This alone will give you double the number of files Babel can copy to RAM in parallel at almost double the bandwidth.

It also depends how the NVME slot is connected to the CPU. The Core i5 having only 16 PCI lanes will no doubt be reserving at least 8 of them for the GPU. Leaving you only 8 to be shared among other devices. This means that sometimes your NVME card will have to share bandwidth with your Wifi or other hardware. This slows it down a bit more.

And your NVME may not even be connected directly to your CPU's PCI lanes. The Macbook may actually reserve all 16 lanes for the GPU and connect to your NVME via its south bridge (which may have additional PCI lanes). I don't know if the Macbook does this but this again may reduce performance a bit more.

In contrast, the large number of lanes that the Xeon has allow the motherboard designer much more freedom to create a really fast I/O platform. In addition the AWS server does not normally have a GPU installed so it does not need to reserve any lanes for GPU use. Again, I don't personally know the actual architecture of AWS servers but it's possible to create one that outperform a Macbook at compiling Babel projects.

So in the end the main factors that enables the EC2 instance to outperform the Macbook are:

  1. Number of PCI lanes directly supported by the CPU

  2. Number of PCI lanes supported by the NVME drive

  3. How the NVME lanes are connected to the CPU

Additional factors that may contribute include:

  1. The speed of the I/O bus (PCI2 vs PCI3 etc)

  2. The speed of RAM

  3. Number of DMA channels available (this alone requires a long answer so I sort of skipped it but the reasoning is similar to PCI lanes)

4
  • I'd like to note that none of the above factors affect gaming performance because if a game needs to hit the disk you will definitely notice a significant slowdown - so game developers work hard to have everything in RAM. What's good for gaming is not the same as what's good for compiling source code – slebetman May 11 '20 at 20:18
  • is there a reliable test that I can perform on both machines to test this? I've been trying with fio but MacOS doesn't have libaio, so I'm using ioengine=posixaio and that might be skewing the results. I read "Libaio has a huge performance benefit over the standard POSIX asynchronous I/O facility because the operations are performed in the Linux kernel instead of as a separate user process." Command I'm using is fio --name=fioTestFile --ioengine=posixaio --rw=randrw --bs=4k --numjobs=1 --size=1g --iodepth=1 --runtime=30 --time_based --end_fsync=1 – dandan May 11 '20 at 21:18
  • FYI - fio is showing read/write for Xeon at 143MB/s R & 16MB/s W and the Mac at 420MB/s R & 48MB/s W. I even tried with letting the linux test run with libaio and the mac test with posixaio ... I really do have the hunch that your theory is correct, I'm just having trouble proving it. Especially since I think the work Babel is doing isn't super computationaly heavy, mostly just concating some JS files and parsing stuff from form A to form B. – dandan May 11 '20 at 21:55
  • Again, you are trying to benchmark javascript performance by running non-javascript code. If Babel compilation is your target use Babel as your benchmark. I do have one suggestion though on making your benchmarks as similar as possible: boot your Macbook with Linux. Linux have famously fast I/O partly because around mid 2000s Linus Torvalds personally got obsessed with disk drivers and hand optimized a lot of the I/O code especially on x86. Windows file servers famously ran faster as a VM under Linux than on real hardware – slebetman May 11 '20 at 23:12
50

Benchmarks are vague handwaves to some very specific performance characteristics (peak instruction rate) that often do not take into account other factors in a system.

A non-exhaustive list of things that can make a big difference to programs but not peak instruction rates:

  • Memory. Type, bandwidth, channels. These all make a difference in how fast data can get to the CPU for it to do work. Servers typically have more channels of RAM, higher quantities and much higher peak bandwidth figures than desktop or laptop CPUs. Having a high single core instruction rate wins you nothing if you can't get data to the CPU fast enough to hit that rate.
    As a simple check I had a look and the 8180 Xeon (closest I could find) has 6 memory channels, while your laptop CPU would (hopefully) have 2 channels set up (or could have been poorly designed and only have one). The server has 3 times the memory bandwidth of your laptop. That will make a massive difference for memory intensive tasks.
  • Hard disk. Faster hard disks, SDDs and so on can make a big difference in getting data to the memory for the CPU to work on. An SSD is orders of magnitude faster seeking for small bits of data, and bulk transfer is also much higher too. NVMe is even faster again. Servers often use RAID for backup or can be set up for raw speed. While they may both be NVMe a server farm may well have enterprise class disks in a RAID 0 or 01 and be faster than your single disk, particularly likely on shared machines where minimal impact across VMs is desirable.
  • Thermal limiting. Benchmarks, especially on laptops and ultra-portable machines, only tend to last long enough to see the initial ramp-up of performance. Over time heat reservoirs become full as fans can't keep up with heat output, and that initial turbo-boost speed drops down to the "normal" peak clock frequency. This can skew benchmark results and make a laptop look a lot better than it will perform under long term loads. Servers tend to have over-specified (and loud) cooling systems to ensure performance, laptops are designed for quiet home comfort and the fans are far less powerful. What you see in a benchmark may not have the same thermal limiting as what you have in front of you, yours may not perform as well and may limit sooner.
  • Bottlenecks. Servers will have far more I/Os than laptops. More PCIe channels, more dedicated IO ports and much higher bandwidth to peripherals meaning more data in flight down uncontested paths. Multiple PCIe devices contending for time on a multiplexer connected to a 16-lane CPU will be slower than a CPU which has 40+ dedicated lanes.
  • Cores. Having more cores makes a difference to not only the task you are doing on one core, but means that the tasks are not fighting for time. The tradeoff is that is is easier to hit memory bandwidth limits with more cores vying for bus time.
  • Caches. Server CPUs tend to have much larger CPU caches. While this is more of an optimisation, larger caches do mean less time going to memory and allow the CPU to hit their peak performance more than a smaller cache. A single core benchmark is probably small enough to fit in most cache sizes and so tells you nothing about the rest of the system.
  • Graphics. Related to PCIe/memory bus contention, your laptop will be doing graphics work, most likely with an iGPU. That means your system memory is being used (and memory bandwidth stolen) in order to drive a graphical display. The server would likely have none of that, most likely being a headless node in a compute cluster. The server has far less graphical overhead.

Consumer class CPUs are indeed powerful, but server class has far more logic, control and bandwidth to the wider system. Generally though, that is fine. We don't expect a 15 watt processor to perform the same as a 10x more expensive CPU with a 140 watt power budget. That extra power budget gives a lot more freedom.

If server CPUs had the same performance as a desktop or laptop CPU, then there wouldn't be a distinction between the two.

Just to further nail home the point: a similar single core score just tells you that the cores are reasonably comparable under ideal conditions. They may be theoretically close in terms of performance, but it doesn't tell you anything about the wider system and what the CPU is capable of when tied to other components. Single core speed is artificially focused on one small point in the system, more so than most normal uses of a system will encounter.

For more information on why one system is "better" than another you need to look more at so-called "real world" benchmarks, which will show (still artificial but) more comparable system performance metrics and hopefully give some idea where bottlenecks might lie. Better yet you do the kind of test you did which shows that for that workload a server class system, with it's underlying architecture and components, is much better.

7
  • Idle thought, with absolutely no data to back it up… I wonder if an EC2 instance would 'lend' you some cycles from elsewhere if no-one else was busy on it at the time. I've known proprietary structures [totally different scenario, 3D world simulators running ostensibly 'one per core' but capable of this type of 'loan' if required] – Tetsujin May 10 '20 at 8:59
  • 1
    @Tetsujin EC2 has something kind of like that (spot instances), but it's at a much higher level than what you suggest. – Austin Hemmelgarn May 10 '20 at 16:33
  • 1
    On the last point about server versus desktop, there would still be a distinction because Intel insists that desktops and laptops don't need ECC RAM, so they refuse to provide support for it in anything but server and tight-embedded designs. – Austin Hemmelgarn May 10 '20 at 16:46
  • 7
    A good answer. In this case thermal throttling will be key in this question. I remember the "old" days of my PhD where my Ultrabook with an i7-4600U was happily 4 times as fast as my desktop with an i7-3770 running a code I wrote BUT only for about 5s. Once the CPU heated up, it was maybe half as quick or slower. -> Basically, the U-processors especially are designed for "burst use" where they perform well but are very much restrained by their thermal envelope and available cooling in reality. In contrast, Server and Desktop CPUs will be designed around more continuous use (including cooling). – DetlevCM May 11 '20 at 6:23
  • 1
    @Tetsujin - this definitely happens with many EC2 instances where they're spec'd as CPU "equivalents" - you can use cycles on that core that nobody else is using even beyond your purchased guarantee - but these particular z1d instances are sold specifically guaranteeing their extremely fast single core compute for people who are running compute-bound software licensed for a single core see here so that isn't the case for this OP's question. – davidbak May 11 '20 at 14:06
10

Adding to Mokubai's excellent answer:

  • Instruction Set Extensions. Some extensions, such as AVX-512, are available in server processors (such as the SKX processor mentioned in the question) but not (or only later) in consumer processors. The Coffee Lake consumer CPU from the question, for example, does not support AVX-512. I don't think that compilers are too heavily affected by this, but if you were to execute certain numeric tasks, including scientific computation or machine learning, this could cause a difference.

  • Core interconnects. Not relevant for single-threaded tasks, but when multiple cores are used, the type of interconnect has an influence on the "speed" with which cores can talk to each other. While the consumer processor uses a ring interconnect, the server processor is the first to use a mesh interconnect.

7

Intel Xeon Platinum 8151 Specs From Intel Corporation

Intel i5-8259U Specs From Intel Corporation

  • It appears the Xeon has 38.5 MB L3 Cache
  • It appears the Intel Core i5-8259U has only has 6 MB Intel® Smart Cache

A processor cache is where a processor stores recently written or read values instead of relying on main system memory.

  • Caches are designed in all sorts of shapes and sizes, but have several classic characteristics that make them easy to exploit. Caches typically have a low set associativity, and make use of bank selectors. Associative Caches
  • Inside a typical processor cache, a given physical (or logical depending on the design) address has to map to a location within the cache. They typically work with units of memory known as cache lines, which range in size from small 16-byte lines to more typical 64- and even 128-byte lines.
  • If two source addresses (or cache lines) map to the same cache address, one of them has to be evicted from the cache.
  • The eviction means that the lost source address must be fetched from memory the next time it is used. In a fully associated cache (also known as a completely associated memory or CAM), a source address can map anywhere inside the cache. This yields a high cache hit rate as evictions occur less frequently.
  • This type of cache is expensive (in terms of die space) and slower to implement. Raising the latency of a cache hit is usually not worth the minor savings in the cache miss penalties you would have otherwise... You can read more here

DDR4 at higher bus rate also helps increase speed. Not too mention that the Xeon has Transactional Synchronization Extensions whereas the i5 does not.

They are not in the same class of processor, but hopefully the information above helps you, and the links from intel corporation assist in the validity of my responses.

2
  • Are there any tests available where I could determine the impact of solely the cache and determine what performance impact it has? eg. do the number of PCI lanes have more of an effect? – dandan May 11 '20 at 21:26
  • @dandan are you referring to PCI or PCIe optionals mentioned within links above? Also are you referring to x2 x4 or 12x vs. 16x? There are many variables if we discuss outside of the mainboard and its sole components of only speed and its bare essentials to operate. Chipsets can vary as you see above in the option section. Please let me know if you are comparing onboard or off-board components so I can reply with a more detailed explanation as the word PCI escapes me. – Don-Pierre Halfaway Jun 21 '20 at 22:44
2

In addition to the other answers, I would add that instructions used in any given benchmark may not match the instructions used in your compiler. Basically, each processor may be faster at certain types of instructions, or one may be better-performing than the other in certain scenarios, for example, branch prediction failure.

The code of one is not guaranteed to be a predictor of the the performance of the other code. That's because they do different things, in different ways.

You could, for example, have a late-model Core2 processor like a Q9550, overclocked by 33% (quite doable), and it might match or exceed a lower-clocked 2nd gen i5 processor for many tasks, despite the latter being more recent.

But if you have a sequence of code which involves a lot of branching instructions with a high degree of randomness, likely the i5 would outperform the Core2 many times over due to the Core2 processor's poor performance in the event of a branch prediction failure.

This sort of thing occurs at all sorts of micro-levels, for all sorts of instructions and processing types. That's why one CPU might be better in a Cinebench benchmark (video encoding), but worse in a SunSpider benchmark (javascript).

2

You just invented one more benchmark - "building this particular project". And the build environment in Amazon is way better than your Mac AT THIS PARTICULAR BENCHMARK.

CPUs (and storage devices, and computers as a whole, and operating systems, and building environments) are not created equal. CPUs are made to fit in different constraints regarding available power, cooling, space, cost and available technology. So are all other components of your setup.

I wouldn't expect much difference because of the different OS (Linux, Mac OS or even Windows) or the underlying storage system because build tasks are CPU- and memory-intensive and don't load much of the filesystem or the process scheduler. Then again, I may be wrong because building a JS project may be different compared to C and Java projects I am familiar with.

The build tools in Linux and Mac OS can differ considerably in performance. They may be themselves built with different compilers, libraries, optimization options, etc and these may bring the whole difference that you see.

1

They both have NVME drives, so that shouldn't be the bottleneck. I don't think there'd be a Mac vs Linux OS issue either, since Mac is linux based.

Please backup that claim. MacOSX is indeed a Unix-like OS, with probably a lot of kernel code from BSD or SVR4 (Unix in the 1990s). But Unix predate Linux by more than two decades. Read the history of Linux (born in and the history of Unix. BTW, I used SunOS3.2 in 1987. The first Linux kernel was released in 1991. I used Linux in late 1993 (kernel 0.99.12) on a i486 PC.

But Linux has (in kernel land) AFAIK, few source code from that era.

Of course, both MacOSX and GNU/Linux can have some GNU software running (notably GNU bash).

At last 9% is within the noise margin. Did you consider, for example, recompile all your Linux distribution from its source code by systematically passing gcc -O3 -mtune-native -flto both at compile and at link time and using the latest GCC? You could try using some source Linux distribution like Gentoo and/or follow the LinuxFromScratch guidelines.

BTW, a server computer costs more US$ or € than your MacBook Pro. Look at DELL prices for them. I would expect more performance from the server. For exemple, server processors have more CPU cache and more cores, and that does make a difference. A typical server processor costs more than your entire MacBook Pro. For example, AND Ryzen Threadripper 2990WX costs 1 758€ in France, and you need to buy a motherboard, a watercooling, plenty of RAM, etc... The same reseller is selling an i5-8279U MAcBook Pro for 1 989€. The price tag of Dell PowerEdge R6525 Rack Server starts at US$ $2,689.00 (no idea if shipping is included).

You should run several benchmarks.

Such as SPEC benchmarks (they cost a few thousand US$). Or OpenBenchmarking. And run all of them -and more than once- on both laptop and server. Collectively they exercise different parts of your computers, and only then you have a better assessment of their performance.

6
  • Please read the question carefully. The Xeon is only 9% faster in benchmarks, but 60% faster in practice. It's true that it's more costly and you could expect it to be significantly faster just by looking at the price tag, but the actual question here is why isn't this performance difference reflected in benchmark results. – gronostaj May 11 '20 at 8:27
  • The actual question is can a single benchmark assess the performance of a computer. We all know this is false. – Basile Starynkevitch May 11 '20 at 8:31
  • 1
    Downvoters, explain? – Tobia Tesan May 11 '20 at 9:10
  • 2
    @TobiaTesan If my previous comment is insufficient: by volume this answer is 30% explanation that macOS is not Linux (true, but not really relevant), 30% suggestion to recompile Linux to get more believable benchmark results (misses the point), 30% "it's more performant because it costs more" (but why not in benchmarks?) and only 10% actual answer (benchmarks are only approximations of performance in some specific workload). – gronostaj May 11 '20 at 12:51
  • 9% is not within noise margin, unless your computer has an insane number of other processes running. In my benchmarking, 2% is minimum to be considered useful. – metamorphosis May 13 '20 at 4:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.