I was wondering if CPUs read and write data (almost) always
following the hierarchy of storage:
CPU <- register <- CPU cache memory <- main memory <- secondary storage ?
No, that hierarchy is not always followed. That diagram you linked to is actually illustrating it not being followed, as it shows that the CPU can branch out to secondary/off-line storage directly (as opposed to through the southbridge). This concept is called MMIO (memory-mapped I/O), and maps these devices onto the system memory map.
What the diagram from Wikipedia is missing is another useful way for storage devices to place data into RAM directly through the northbridge. This is called DMA (direct memory access), and can greatly help performance. This allows your hard drive to place a program into memory without using the CPU (otherwise, the CPU would have to read one word at a time, and then place it into memory).
If without special treatment, can data, address or instructions in
secondary storage or main memory sometimes be read or written by
processing unit, skipping main memory, CPU cache memory and/or
Again, while a CPU can directly access data on a device, if it wants to perform any operation on this data, it must be moved into a register. You cannot manipulate data without moving it in/out of a CPU register. Doing this involves instructions, which are held in the CPU's cache (when possible).
Caching mechanisms can be influenced by the programmer, but since the CPU cache is much faster then even your RAM, it is unwise to "skip" using the CPU cache and hold everything in RAM or on disk, unless you have a specific reason to.
As an example, this is useful in some video encoding applications, where certain data is only needed for a single instruction. If the video data was constantly being placed into the CPU's cache, and was only needed for those few clock cycles, the likelihood of a cache miss would be greatly increased.