Clearing Up Misconceptions
linux doesn't support Nvidia drivers out of the box because of legal issues
While this is technically true for completely separate reasons, Ubuntu (which is not just Linux; it is Linux plus a great many other (separate) things clobbered together) does ship the proprietary Nvidia and AMD graphics drivers as a default part of their distribution. In fact, it is fairly trivial to install them, and you don't have to do anything too technical to successfully do so. I am not sure what steps you took to try and install the Nvidia graphics drivers, but it sounds like you did not use the officially supported method.
linux thought I was using vmware
Nope. VMware actually contributes a lot of code to the Open Source Graphics Stack (OSGS for short). Because the creator of Mesa is a VMware employee (he wasn't when he started Mesa, but he is now), the name "VMware" often is seen in the OpenGL vendor string if you're using the OSGS. The OSGS actually consists of the following components:
Mesa - A userspace OpenGL implementation that contains both software-based and hardware-accelerated implementations of OpenGL. Also implements some other APIs recently, such as OpenCL, possibly Direct3D 9, OpenVG, etc. Basically, Mesa is what takes an open source kernel driver's exposed hardware and makes it draw 3d stuff. Within Mesa, there are hardware-specific implementations of OpenGL, which are either based on classic Mesa (currently, only the Intel GPU driver is based on classic Mesa), or on Gallium3D, which is a part of Mesa that provides a lot of shared code that can be used to relatively quickly bring up OpenGL support for new hardware.
The Linux kernel - The kernel contains the Direct Rendering Manager (acronym DRM, but no relation to Digital Restrictions Management), which contains the "kernel side" components of the OSGS for all supported hardware.
The Xorg display server - Also known as "X", "X11", or the organization behind it, "X.Org", this is a userspace component that provides basic input (mouse/keyboard/joystick) and display infrastructure. Programs connect to an Xorg display server using either TCP or UNIX domain sockets, and submit drawing commands to the server. The server then uses Display Device eXtensions (DDXes) to pass draw commands to the video hardware.
DDXes - DDXes implement the actual "backend" of drawing commands that are sent to the Xorg server. DDXes may use either using the generic (non-accelerated) VESA interface (which all graphics cards share in common), or a hardware-accelerated one, such as nouveau, intel, or radeon, which support Nvidia, Intel and AMD graphics processors, respectively, via the Direct Rendering Infrastructure (DRI) userspace component. DRI basically talks to the kernel's DRM to submit accelerated drawing commands to the hardware. Un-accelerated DDXes are painfully slow, so you should always use an accelerated one, if you can.
libdrm - The userspace component that is basically a thin wrapper around the Linux kernel's DRM API. Hardware-accelerated DDXes and Mesa both use libdrm.
So no, "Linux" (whatever that means; Mesa is not part of the Linux kernel) did not actually think that you are running under VMware. Rather, the most likely scenario is that your misconfiguration of the proprietary Nvidia graphics drivers caused the operating system to fall back to the llvmpipe Mesa backend, which is a best-effort attempt at software rendering on the CPU (but it is still painfully slow compared to hardware-accelerated drivers). Without you posting your full OpenGL version string, though, it's impossible to know what exactly you saw. You can display your OpenGL version string using the glxinfo command (you may have to install a package first).
as long as I don't enable "nomodeset" in my boot options, my nvidia card doesn't work.
Let me attempt to parse this confusing sentence by rewriting it:
IF I do enable "nomodeset" in my boot options, THEN my Nvidia card does work.
Is that true of your case, or did you mean something else?
If this is true, you should be aware that nomodeset actually tells the OSGS driver to enable User Modesetting, which mostly just breaks 3D acceleration.
To clarify:
There are two types of graphics driver stacks in common use on desktop GNU/Linux:
- Proprietary: Nvidia and AMD have developed fast graphics drivers, but they do not share the source code with anyone. These are collectively referred to as the proprietary drivers, since they have similar characteristics and integration problems.
- Open Source: The OSGS that I discussed above.
There are two types of modesetting in common use on desktop GNU/Linux:
User Modesetting: This is the only modesetting type supported by the proprietary drivers. In the past, this was also the only modesetting type supported by the OSGS. But the OSGS has moved increasingly toward Kernel modesetting in recent years, to such a degree that it is widely acknowledged that User Modesetting is an unsupported configuration, and is likely to break, when you are using the OSGS. However, User modesetting still works well with the proprietary drivers, provided that the OSGS is properly out of the way.
Kernel Modesetting: Supported by the OSGS only. If you intend to use the proprietary drivers, you cannot use them if the kernel component of the OSGS, the Direct Rendering Manager, has initiated the Kernel Modesetting. You would have to reboot the system to fix this.
Unfortunately, the proprietary graphics drivers are installed by trampling over the files that the OSGS consists of. For example, the file /usr/lib/libGL.so (I'm over-simplifying the exact file name here) is the system's default OpenGL implementation. A default installation will have this file "linked" to the OSGS -- Mesa, to be specific. Once you install a proprietary graphics driver, this file will instead be "linked" to the proprietary driver's specific libGL implementation, NOT Mesa.
The other part of the proprietary graphics installation procedure is to change some configuration files so that the kernel component of the OSGS does not attempt to load. If the kernel component of the OSGS loads, it may try to take control over the graphics hardware, which would prevent the proprietary graphics driver's own kernel module from loading successfully, and your system ends up in a "broken" state.
In fact, because of the componentized architecture of the graphics stack (both the proprietary and OSGS have a similar architecture), there are a great number of possible "broken" configurations, which either will not provide any display output at all, or will restrict you to the text-only terminal, or will load Xorg but not provide 3d acceleration.
Here is a non-exhaustive list of some example broken configurations:
You loaded the kernel DRM from the OSGS, but your libGL is from a proprietary graphics driver.
You loaded the proprietary graphics driver kernel module and libGL, but your Xorg DDX is from the OSGS.
You loaded the kernel DRM from the OSGS and the proprietary graphics driver kernel module, but the OSGS kernel DRM "wins" (gets exclusive control over the hardware). Meanwhile, your DDX and libGL are both from the proprietary graphics driver.
These types of broken configurations can occur when:
There is a bug in the proprietary graphics driver installation tool;
The proprietary graphics driver needs special customizations to work with your distribution of GNU/Linux, but you are installing the "vanilla" version of the driver (downloaded from nvidia.com or amd.com on the Internet) rather than installing it using your distribution's package manager. The distro-packaged version will have been tested by your Linux distribution, and is practically guaranteed to be functional if you install it the Right Way.
The proprietary graphics driver's installation is done in a way that expects a specific type of OSGS installation, but your OSGS is newer / older than expected, so the file(s) or configuration(s) are mismatched, leading to one of the broken scenarios above.
You are using too new or too old a version of the proprietary graphics driver. Typically, it is best to use a version of the proprietary driver that was released around the same time (within 6 months, at least) of the Linux distribution release you are running. For example, if you are running Ubuntu 10.04 (from April 2010), you would not want to install an Nvidia driver released in September 2014. On the other hand, you probably want to be running the latest release of Ubuntu to begin with; it is very inadvisable to run an old version because of all the enhancements that have been made in recent years.
Make sure you are running Ubuntu/Xubuntu 14.10 -- this is the latest release (2014 / 10 = October) as of this writing. For good reason, anyone who is serious about 3d acceleration on (x)Ubuntu will run the latest version. Things are changing rapidly and you will not have a good experience on an old version.
Advice
Judging by what I would perceive as your likely level of technical experience, I would suggest you do the following:
Wipe both Xubuntu and Ubuntu off of your system.
If you don't already have Ubuntu 14.10, download it. Get the 64-bit version. Do not bother with 32-bit.
Install it. It should "Just Work" and get you to a desktop.
Go to the "Additional Drivers" applet, like the other answer suggested, and grab the Nvidia proprietary drivers.
Reboot.
