I don't know if this is a linux problem or an xfce one,
Default installation (mint 7) came with Gnome and I don't remember this problem happening then. It only occured after I switched to Xfce,
What could be wrong?
I removed pulse audio (from synaptic) and the problem is fixed!
Based on suggestion (comment #2) from this bug report for ubuntu: xfce4-mixer master is always muted at reboot