Capture as vector? No of course not. There is no way for the capture program to know anything about the geometry of the shapes on screen without having some sort of hook into the program(s) doing the drawing.
What you can do however is to capture a raster image and then convert that to vector. There are tools like Inkscape that can perform this conversion (though you will usually have to manually do some tweaking, and even then, you will rarely be able to get pixel-perfect results).
As for the resolution, when you perform a capture, it cannot capture more information than is available. You can re-size the resulting image up, and if you use a good scaling algorithm, the interpolated pixels will blend fairly well, but there is no way to invent more pixel information than is present.
Theoretically, it could be possible to implement some sort of system in which everything is rendered internally at a higher resolution and then sent it to the screen at a lesser one, but that would require special support in both the OS and the drive, and is unlikely to be implemented since it has very limited use and would only be a waste of memory in most scenarios.
You may be able to use desktop-panning to achieve a higher resolution desktop than the screen supports, but again, that requires that your drivers support it.