Samsung SSD 980 PRO capping out at 510MB/s

Question

I have a Samsung SSD 980 PRO (PCIe 4.0 NVMe M.2) which caps out at ~ 510 MB/s write speed (see test details below) which leads me to the conclusion that it is not attached via the NVMe interface but via the SATA interface with its 6 Gb/s limitation or PCIe 4.0 is not used. A friend of mine has the exact same SSD and he gets ~ 1150 MB/s with the same test so the drives potential is obviously not used in my system.

Mainboard: ASUS Rog Strix X570-F Gaming (https://rog.asus.com/motherboards/rog-strix/rog-strix-x570-f-gaming-model/spec)

The SSD is installed in the M.2_1 slot. According to the MB specs both M.2 slots are connected with the CPU via PCIe 4.0 x4 lanes no matter the number of other PCIe devices connected. Sadly I am unable to find detailed information about the actual lane distribution in this board. I have a graphics card installed in the X16_1 slot, an USB card in X16_3 and a WIFI card in X1_2. So far as I understand the manual, the X16_1 slot occupies the CPU direct lanes and the X16_3 and X1_2 slots are routed via the X570 chipset which should not impact the performance of the M.2 slots.

The nvme-cli tool shows me the drive, the BIOS lists the drive as NVMe and its device path is /dev/nvme0n1 so I guess the NVMe interface is actually used and not SATA.

Maybe it is not using PCIe 4.0 but I am unsure how to find that out. Are there any settings or configuration issues that I am missing?

PC specs (inxi -F):

System:
  Host: amdx1 Kernel: 5.13.0-27-generic x86_64 bits: 64 Desktop: Cinnamon 5.0.7 
  Distro: Linux Mint 20.2 Uma 
Machine:
  Type: Desktop Mobo: ASUSTeK model: ROG STRIX X570-F GAMING v: Rev X.0x 
  serial: <superuser/root required> UEFI: American Megatrends v: 4002 date: 06/15/2021 
CPU:
  Topology: 16-Core model: AMD Ryzen 9 5950X bits: 64 type: MT MCP L2 cache: 8192 KiB 
  Speed: 2196 MHz min/max: 2200/3400 MHz Core speeds (MHz): 1: 2202 2: 2573 3: 2800 
  4: 2240 5: 2212 6: 2238 7: 2200 8: 2217 9: 2198 10: 2193 11: 2199 12: 2196 13: 2199 
  14: 2197 15: 2196 16: 2213 17: 2237 18: 2242 19: 2213 20: 2464 21: 2198 22: 2223 
  23: 2200 24: 2199 25: 2230 26: 2203 27: 2200 28: 2198 29: 2210 30: 2233 31: 2239 
  32: 2214 
Graphics:
  Device-1: NVIDIA GP104 [GeForce GTX 1070] driver: nvidia v: 510.60.02 
  Display: x11 server: X.Org 1.20.13 driver: nvidia 
  resolution: 1920x1080~60Hz, 1920x1080~60Hz 
  OpenGL: renderer: NVIDIA GeForce GTX 1070/PCIe/SSE2 v: 4.6.0 NVIDIA 510.60.02 
Audio:
  Device-1: NVIDIA GP104 High Definition Audio driver: snd_hda_intel 
  Device-2: AMD Starship/Matisse HD Audio driver: snd_hda_intel 
  Device-3: Sunplus Innovation FHD Camera type: USB driver: snd-usb-audio,uvcvideo 
  Sound Server: ALSA v: k5.13.0-27-generic 
Network:
  Device-1: Realtek RTL8192EE PCIe Wireless Network Adapter driver: rtl8192ee 
  IF: wlp5s0 state: down mac: 5c:a6:e6:ee:9b:30 
  Device-2: Intel I211 Gigabit Network driver: igb 
  IF: enp6s0 state: up speed: 1000 Mbps duplex: full mac: f0:2f:74:de:91:5d 
  IF-ID-1: br-4733bfab680b state: down mac: 02:42:c8:df:41:69 
  IF-ID-2: docker0 state: down mac: 02:42:20:7a:52:0e 
  IF-ID-3: virbr0 state: down mac: 52:54:00:46:f4:49 
  IF-ID-4: virbr0-nic state: down mac: 52:54:00:46:f4:49 
Drives:
  Local Storage: total: 8.64 TiB used: 282.07 GiB (3.2%) 
  ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 980 PRO 1TB size: 931.51 GiB 
  ID-2: /dev/sda vendor: Seagate model: ST2000DM001-1ER164 size: 1.82 TiB 
  ID-3: /dev/sdb vendor: Samsung model: SSD 850 EVO 500GB size: 465.76 GiB 
  ID-4: /dev/sdc vendor: Seagate model: ST6000VX0023-2EF110 size: 5.46 TiB 
Partition:
  ID-1: / size: 915.40 GiB used: 282.04 GiB (30.8%) fs: ext4 dev: /dev/nvme0n1p2 
Sensors:
  System Temperatures: cpu: 52.8 C mobo: N/A gpu: nvidia temp: 47 C 
  Fan Speeds (RPM): N/A gpu: nvidia fan: 34% 
Info:
  Processes: 487 Uptime: 20m Memory: 62.71 GiB used: 3.37 GiB (5.4%) Shell: bash 
  inxi: 3.0.38

The command for the random write speed test:

sudo fio --bs=4k --ioengine=libaio --iodepth=32 --size=10g --direct=1 --runtime=60 --filename=/tmp/testfile.tmp --rw=randwrite --numjobs=1 --name=test

The output:

test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=519MiB/s][w=133k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3024: Tue May 10 13:16:09 2022
  write: IOPS=125k, BW=486MiB/s (510MB/s)(10.0GiB/21051msec); 0 zone resets
    slat (usec): min=2, max=781, avg= 7.10, stdev= 4.02
    clat (usec): min=12, max=2105, avg=249.37, stdev=33.39
     lat (usec): min=14, max=2162, avg=256.57, stdev=34.21
    clat percentiles (usec):
     |  1.00th=[  212],  5.00th=[  229], 10.00th=[  233], 20.00th=[  237],
     | 30.00th=[  239], 40.00th=[  243], 50.00th=[  245], 60.00th=[  247],
     | 70.00th=[  251], 80.00th=[  255], 90.00th=[  265], 95.00th=[  277],
     | 99.00th=[  388], 99.50th=[  400], 99.90th=[  519], 99.95th=[  685],
     | 99.99th=[ 1336]
   bw (  KiB/s): min=453504, max=556584, per=99.93%, avg=497758.93, stdev=19901.89, samples=42
   iops        : min=113376, max=139146, avg=124439.71, stdev=4975.46, samples=42
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=68.31%, 500=31.56%
  lat (usec)   : 750=0.08%, 1000=0.02%
  lat (msec)   : 2=0.03%, 4=0.01%
  cpu          : usr=7.28%, sys=59.19%, ctx=2702826, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2621440,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=486MiB/s (510MB/s), 486MiB/s-486MiB/s (510MB/s-510MB/s), io=10.0GiB (10.7GB), run=21051-21051msec

Disk stats (read/write):
  nvme0n1: ios=2612/2600827, merge=2170/51645, ticks=296/23740, in_queue=24070, util=99.59%

Answer 1

1. The advertised write speed is when writing to the NVMe's on-board DRAM cache (if any) or pseudo-SLC cache. The sustained speed is a lot slower. For example the drive could write at 4000MB/s for 10-20 seconds, but then switch to writing at 1400MB/s. The high end numbers can be a bit hard to get for real workloads, you may need to change the fio data block size and definitely make sure you are using sequential writes.

2. The process of write leveling can steal a lot of write bandwidth. The drive internally can be writing at full speed, but you only see 1/2 or even 1/4th as the drive is moving erase blocks around behind the scenes. Sequential writes should be faster, but your file system won't necessarily issue them as sequential writes to the drive, so you can get this issue even with sequential writes in some cases. You can try writing different block sizes (You can trace the block requests w/ blktrace or biosnoop).

Often with NVMe benchmarks you will see the drive switch between different write speeds:

• Write to cache speed e.g. 4000MB/s
• Sustain speed. e.g. 1000MB/s
• Garbage collection speed e.g. 300MB/s

Answer 2

Your SSD doesn't have a SATA mode. It can ONLY run on NVMe, which your motherboard supports in that M.2 slot with any combination of PCIe cards with a 3rd Gen Ryzen CPU (as you have).
(You can also tell from the inxi output: /dev/nvme0n1 means it is really NVMe)

So the apparent slowdown isn't due to SATA mode. Something else is going on.
But you already came to that conclusion yourself.

A couple of things to check:
First of all go through all Bios settings and make sure there is no PCIe fallback setting in place.
If you are over-clocking/over-volting/under-volting: Go back to standard and see if that makes a difference.
Make sure you have the latest Linux kernel. (I'm not up to date on the current Linux situation so I can't tell if 5.13 is reasonbly up to date.)

And last, but not least: Ask on Linux specific forums. You will find more experts there than here when it comes to Linux.