I'm an absolute beginner starting to understand the PCI Express protocol and I need some clarifications about its mechanics.

I understood that a PCI Express endpoint device may have some memory BAR mapped in the system memory (is it always RAM the system memory we are talking about?). I need to understand what happens when I send from the CPU to the device (A) a memory read request, addressing a certain memory address (first memory BAR, offset 0).

Let's assume that my device doesn't have any in-device memory.

The root complex on behalf of CPU creates the TLP and forwards it to device (A) because the memory address of destination is assigned to (A).

A receives the TLP, unpacks it, and creates a completion TLP containing data coming from its internal application logic.

Now the TLP travels backwards to the Root Complex that unpacks it, gathers the interesting data and gives it back to the the CPU.

What is the system memory role during this communication? None?

Does the root complex communicate (in this case) with the physical memory of the system, other than sending and receiving packets to/from the endpoint?

To some people it may seem a silly question but for me this is crucial to understand the connection between the physical memory and the memory address BARs assigned to each PCI Express device.

  • What does BAR stand for? – user273580 Jun 13 '13 at 10:57
  • 1
    Base Address Register. – LawrenceC Jun 13 '13 at 11:38
  • Thank you very much Hennes, I really need to understand if the physical memory addressed by the memory BARs of a PCIe device resides in the device itself, is it part of the RAM, both, none? When we refer to memory as a whole, is it the sum of RAM + devices memories? the answer ultrasawblade gave seems to go in that direction, but I'm not sure – Jacopo Reggiani Jun 13 '13 at 16:11
  • "Memory as whole" in the way you are saying is simply a range of addresses. A better term is "CPU address space." When the CPU issues a MOV instruction to read or write from an address, it can be set up where things other than RAM respond. This was quite common on older systems, and I believe most ARM-based CPU's still mostly rely entirely on I/O devices appearing like and being addressable in the same way as RAM. – LawrenceC Jun 14 '13 at 16:33

When a CPU talks to a memory address, it isn't RAM that has to respond. It can be an I/O device. You can actually think of RAM as a "specialized memory-mapped I/O device" whose job is just to save and give back data, although with today's modern CPUs that have caching and such, it's not physically straightforward.

I am not too keen in the low level details of PCIe, but you seem to be wondering how the PCIe bus itself communicates with the CPU. It does so like anything else that communicates with the CPU:

  • Memory mapping - i.e. a device is "mapped" where reads and writes to a range of addresses don't go to RAM, but a device or controller.
  • DMA - an external device or controller reads/writes a section of RAM, without the CPU being involved at all.
  • I/O ports - this is just another address space (a feature of the Intel x86 family of CPUs) that has historically been dedicated to I/O devices. You'll never find RAM here, but it works like memory mapping. The main difference between I/O ports and memory mapping is that I/O port instructions always work serially, no "out of order" or "reordering" of operations will happen here as opposed to the CPU attempting to do that with accesses to main memory.
  • IRQs - an external device sends an interrupt to the CPU

The BARs and such are "mapped" into memory, and take the place of any RAM that may be "under" it. With the MMU I believe it's possible to "remap" the RAM beneath it to a different physical address.

  • In the case of Memory mapping yo wrote: – Jacopo Reggiani Jun 13 '13 at 14:21
  • In the case of Memory mapping yo wrote: >> "Memory mapping - i.e. a device is "mapped" where reads and writes to a range of addresses don't go to RAM, but a device or controller." If my device hasn't got an inboard memory, is it possible that its let's say 1 MB of Memory is mapped into RAM? If so, when I ask my software driver to read from an address linked to my device memory map, am I asking both data from the device and contextually writing it into the related address in RAM? – Jacopo Reggiani Jun 13 '13 at 14:27
  • 1
    It resides in the device. – LawrenceC Jun 14 '13 at 16:29
  • 1
    Memory mapped I/O does not refer to I/O ports. Memory mapped I/O is accessed by the CPU using the same instructions it does for reading/writing RAM. I/O ports are accessed by the CPU using I/O-port specific instructions. – LawrenceC Nov 10 '13 at 3:27
  • 1
    The PCI-E controller itself appears in the x86 I/O space on x86 and compatible architectures at well-known addresses. Now ... there are I/O BARs (which look like they are deprecated according to Wikipedia) so a PCI-E device can say it needs addresses in the I/O range as well as the memory range. If a PCI-E device has an I/O BAR then you will be using I/O ports after the device is configured. – LawrenceC Nov 10 '13 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.