I downloaded BackTrack 5 R2; Image Type: VMWare.
I thought because it's name is VMWare it will look for the "Oracle VM VirtualBox" program but it wont work. so should I just download the iso version, or is there any way I can install the VMWare.
Download VMWare Player (http://www.vmware.com/products/player) and you should be able to run it.
Otherwise you can try to add it to virtualbox by creating a new guest client and using the VMWare image as the hard drive image for the new guest client (don't create a new hard drive).
If you wanted this to work in Virtualbox, there is a walkthrough on the backtrack wiki:
However, I'd recommend using VMWare Player as @richardhsu mentions, as the performance is better (IMO) and you have more functionality, such as drag and drop between host and guest.