I am not an expert on this, but when you look at this:
And be reminded, that usally, when you use your digital output, like optical out, then you don't configure a speaker configuration.
Because, I would say, that it's sufficient, when you put out a digital signal to your 3.1 speakers. They will decode it to the channels like they can.
So if you would place a 5.1 system behind your digital output, it would be 5.1 sound, if 3.1 it will be 3.1 sound.
All theses Audio-Channel configurations I know only to be useful, when you are using the seperate channel connectors on your soundcard or mainboard.
So basically, have you tried to setup VLC to use SPDIF and then looked if the sound was right?
Here is a setup example i found on the net - "use S/PDIF when available" is one part of the magic - the other is to select the SPDIF-Audio track and NOT the 5.1 one.
Or here is a nice vid setting up the Zoom-Player: http://www.youtube.com/watch?v=Anz5FX2VlaI