This is just a high level question about communicating with PCIe devices. Per my understanding it should be possible to do direct endpoint to endpoint communication with no Root Port middle man work as long as the Root Port (Linux kernel) sets up some initial info such as the BAR values. So let me explain what I've tried and maybe someone can see what I've missed.

First I boot the machine with my 2 devices plugged in. I record the BARs assigned to my target device (completer). To my understanding these BARs are the bus address which is a sort of physical address on the PCIe lines which the PCIe switches uses to route traffic. The second device is an FPGA (requester) so I remove the device from the kernel

echo 1 > /sys/devices/pci0000\:00/0000\:00\:03.0/0000\:04\:00.0/remove

Once the device is removed I flash the FPGA. Then I assign certain parameters in the PCI configuration space. Specifically I set byte 4 bits 0, 1, and 2. Which should be I/O Space, Memory space, and Bus Master. Specifically I know that last one is required if I want the device to be capable of requesting information on the PCIe bus. https://wiki.osdev.org/PCI

At this point I rescan my PCIe bus so the kernel can see what my device looks like.

echo 1 > /sys/bus/pci/rescan

At this point I check with lspci to see if my changes worked in the configuration space and all seems good to me. The kernel has set a BAR but my device is listed as a bus master.

sudo lspci -nn -k -x -d 0700:8038
04:00.0 Memory controller [0580]: Device [0700:8038]
    Subsystem: Xilinx Corporation Device [10ee:0007]
00: 00 07 38 80 07 01 10 40 00 00 80 05 08 00 00 00
10: 00 00 20 fb 00 00 00 00 00 00 00 00 00 00 00 00

From here I make a read request for BAR0(read from NVMe drive)+0x08 which should be the NVMe version according to the kernel code. https://elixir.bootlin.com/linux/latest/source/include/linux/nvme.h#L105

At this point I would expect the NVMe drive to produce a completion packet for my non-posted request but instead my PCIe controller times out on the FPGA implying that even the switch is not ackwnowledging my request (I think). It's possible my FPGA is just not correctly producing these requests but I don't have a way to scope the PCIe bus sadly.

So I'm checking to make sure I did not miss some over arching flag or bit, some miss understanding maybe with how BARs work or endpoints etc. If anyone sees a problem with what I'm trying to do please let me know. Based on this http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-1 it's possible that my motherboard is simply not designed to allow switching between the two devices but I have no idea how to even check for that.

This allows the peripheral to access the CPU’s memory directly (DMA) or exchange TLPs with peer peripherals (to the extent that the switching entities support that).

Thank you for any help.

  • Have you tried having your device talk to any other device besides the NVMe drive? – LawrenceC Jul 10 '19 at 20:36
  • No I haven't because sadly this machine only has 1 PCI slot and 1 M.2 slot (which is technically PCI but physically is different). I guess I could try talking to bus 0 which I believe is the root port. – arduic Jul 11 '19 at 11:07

For anyone experiencing a similar problem one rule about PCIe that I did not know (and still can't find specifics on exactly what rule this is since I don't own the standard). When reading from registers specified in the BARs. You must read the size of the data, you cannot read part of a register.

In my case I was trying to read a 64bit (2 dword) register but I was specifying a read request for only 1 dword. Apparently this had my NVMe drive simply refuse to respond. Up the dword count to 2 and poof I get data back all the sudden. I'm guessing this is device dependent but maybe it's related to how I addressed the data? I addressed as 64bit and 32bit with no change so I don't think that's it.

Does anyone know if there is a note in the standard somewhere about this or if this is strictly device specific? I'll checkmark anyone who can provide a specific reason for this.

  • 1
    I/O space can have arbitrary access rules. A requirement to access registers with their full width is one of the more common ones, because it would be a lot of effort to implement accesses to partial registers, with very little benefit. I/O space is not memory, every access is explicitly decoded, and if there is no matching decoding, no reply is generated. – Simon Richter Jul 11 '19 at 13:28
  • That makes sense. Saves a lot of resources on the drive if it can limit it's responses to certain widths and such. Just surprised to learn (from what you say) that this is "arbitrary" implying that each drive maker could setup a different set of access rules. One could require all reads to be 256bit aligned and writes to be 128bit aligned and that would be valid (although silly). – arduic Jul 11 '19 at 16:20
  • Generally this will follow some common sense pattern (64 bit registers require 64 bit aligned 64 bit accesses), and where the device follows a standard interface these rules would be part of the interface. – Simon Richter Jul 12 '19 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.