This is just a high level question about communicating with PCIe devices. Per my understanding it should be possible to do direct endpoint to endpoint communication with no Root Port middle man work as long as the Root Port (Linux kernel) sets up some initial info such as the BAR values. So let me explain what I've tried and maybe someone can see what I've missed.
First I boot the machine with my 2 devices plugged in. I record the BARs assigned to my target device (completer). To my understanding these BARs are the bus address which is a sort of physical address on the PCIe lines which the PCIe switches uses to route traffic. The second device is an FPGA (requester) so I remove the device from the kernel
echo 1 > /sys/devices/pci0000\:00/0000\:00\:03.0/0000\:04\:00.0/remove
Once the device is removed I flash the FPGA. Then I assign certain parameters in the PCI configuration space. Specifically I set byte 4 bits 0, 1, and 2. Which should be I/O Space, Memory space, and Bus Master. Specifically I know that last one is required if I want the device to be capable of requesting information on the PCIe bus. https://wiki.osdev.org/PCI
At this point I rescan my PCIe bus so the kernel can see what my device looks like.
echo 1 > /sys/bus/pci/rescan
At this point I check with lspci to see if my changes worked in the configuration space and all seems good to me. The kernel has set a BAR but my device is listed as a bus master.
sudo lspci -nn -k -x -d 0700:8038 04:00.0 Memory controller : Device [0700:8038] Subsystem: Xilinx Corporation Device [10ee:0007] 00: 00 07 38 80 07 01 10 40 00 00 80 05 08 00 00 00 10: 00 00 20 fb 00 00 00 00 00 00 00 00 00 00 00 00
From here I make a read request for BAR0(read from NVMe drive)+0x08 which should be the NVMe version according to the kernel code. https://elixir.bootlin.com/linux/latest/source/include/linux/nvme.h#L105
At this point I would expect the NVMe drive to produce a completion packet for my non-posted request but instead my PCIe controller times out on the FPGA implying that even the switch is not ackwnowledging my request (I think). It's possible my FPGA is just not correctly producing these requests but I don't have a way to scope the PCIe bus sadly.
So I'm checking to make sure I did not miss some over arching flag or bit, some miss understanding maybe with how BARs work or endpoints etc. If anyone sees a problem with what I'm trying to do please let me know. Based on this http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-1 it's possible that my motherboard is simply not designed to allow switching between the two devices but I have no idea how to even check for that.
This allows the peripheral to access the CPU’s memory directly (DMA) or exchange TLPs with peer peripherals (to the extent that the switching entities support that).
Thank you for any help.