How to investigate I/O load on Linux?

Question

I'm trying to investigate I/O load my my Linux box (Centos6). I'm using iotop, and the top lines look like this:

 PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
  376 be/3 root          0.00 B      4.26 M  0.00 % 32.79 % [jbd2/vda1-8]
 1831 be/4 mysql       624.00 K    624.79 M  0.00 %  3.77 % mysqld ...
  375 be/4 root          0.00 B      8.25 M  0.00 %  0.75 % [flush-252:0]

It seems that ext4 journalling process (jbd2) takes the majority of the blame. Is it normal? I'm looking at the accumulating statistics, and it just doesn't make any sense to me (e.g. disk write versus IO%). Is there any other tool like iotop that I can use to look into disk activity on Linux?

Answer 1

I had a strong hunch MySQL on ext4 could be the problem and there are many links supporting the theory.

I have seen XFS recommended by multiple people for MySQL data. The first link suggest a remedy for the issue if you really want or must use ext4.

As to your question about investigating disk activity, have a look at systemtap which allows many things (for example printing call stack on any system call or even non-exported kernel functions)

http://sourceware.org/systemtap/examples/keyword-index.html#DISK

However, this is a kernel thread and you may not find the backtrace very helpful.

Also I would refrain from experimenting on production system as I have misused systemtap resulting in kernel panic.