atop ... to get the most out of it you may have to enable some additional kernel patches (I/O accounting patches).
atop isn't an option then use
netstat -anp --inet (as root) to provide a listing of which TCP/UDP ports are in use by which processes (or possibly use
lsof for that). From there simply iterate over each process that's got an open socket and either attach to it using
ltrace -S or
strace to look at the reads, writes, sends and receives, or use
tcpdump with a filter specifying your local IP address(es) and the TCP/UDP ports that were listed.
atop is certainly the most convenient of these ... if you have it and the necessary kernel support installed. In the past I've had customers and employers set up special systems (different from their production images) for the sole purpose of supporting I/O profiling using
atop. However these other techniques will get you there.
I'm sure we could also do something using SystemTap ... but I don't know of any easy pre-cooked recipes for doing this. SystemTap is very much a programming analysis tool.