Is there any way to mute only one program while listening to an audio file in VLC (for example) on Windows 7?
Yes. Open up the volume mixer from the main audio control in the system tray.
When you click on it you'll get the master volume but there is an option for mixer. This lets you control each individual application (that is currently outputting audio) level.
Note it will remember these settings so if you mute program A, after you close it A won't be in the mixer (no current output), but next time you run A it will be running as muted.
To do it from the command-line, you can try nircmd: http://www.nirsoft.net/utils/nircmd2.html
nircmd muteappvolume vlc.exe 1
Yep, right-click the volume-control icon in the system tray and select "Open Volume Mixer" from the pop-up menu. You can set individual volume settings (including mute) for each program that is open.
Beware that Windows remembers these settings, so it might be polite to unmute the program when you're finished.