I just installed Videolan on a Windows XP machine and all the videos have now its cone logo instead of a preview of the video in windows explorer.
How can I change that while keeping Videolan as the default video player?
You can install any number of different Codec packs that handle this - CCCP, K-Lite, etc.
It's worth noting that most of the codec packs will (by default, at least), install loads of extra stuff that you probably don't need. Be sure to choose custom install and just use what you need.