The PC3200 (DDR-400) RAM should clock down to PC2700 (DDR-333) but I've personally dealt with boards that will take DDR-333, but not DDR-400 -- usually they just don't POST though.
PC3200 RAM does generally require slightly more power (~.1V -- 2.5v for PC2700, 2.6V for PC3200).
Generally PC2700 should work at a low-end voltage of about 2.3V, where as PC3200's low end is about 2.5V (see Wikipedia).
So if your motherboard is only providing 2.3V at the sockets, then it may be enough to power PC2700 RAM, but not PC3200.
Aside from that, have you tried removing one of the RAM sticks, see how it acts, and if it's still unstable, swap the sticks and try the other by itself? You might just have a faulty stick.
Unfortunately random reboots during install could be many things, not necessarily the RAM.
EDIT:
OK - according to Kingston (memory manufacturer) the highest supported RAM speed for that board (on the DDR side) is 266. Since it's old enough and weird enough to have both DDR and SDRAM slots, it's going to be very picky, and not too happy with newer DDR RAM.
Further to that I found this on ECS' site:
Question:
Does P4VXASD2+ v5.0 support DDR400?
Answer:
No, the P4VXASD2+ v5.0 only supports to DDR266 maximum.
...and this:
Question:
I installed DDR400 512 Mb next to a slower DDR 256 Mb. I’m having system stability problems, some EXE files/programs crashes when I try to open it and some times the systems crashes when finalising the booting process and auto reboots again.
Answer:
Maybe it's the DDR compatible issue with your M/B. I suggest you can use DDR266/333 512M 16 chips type DDR in P4VXASD2 5.0 M/B. Some DDR400 has new format DDR memory chip that could not be supported in some old M/B chipset.
So, unfortunately, it looks like your only bet is to find a couple old(er) DDR-266-specific sticks. :(
