My understanding is :
When you use VirtuaBox as a virtualization software, you are basically running the guest VM over the virtualization layer (i.e. VirtualBox) which itself runs on Windows 32 bit. So indirectly whatever restrictions that the host OS puts on Virtualbox will cause performance issues to its guest. One such limitation is memory.
However if you had used a true barebone virtualization software like Hyper V or VmWare ESXi, then the lowest layer is the virtualization layer and then comes the host followed by the guest. In this case the limitations of the hosts do not apply on the guest.
So my suggestion is either upgrade the host to x64 or switch to Hyper V /VMWare ESXi or XenApp Server