The window size is controlled by the VM's screen resolution. If you manually resize the window on the host you'll only zoom on the image returned by the VM, so of course this gives a very bad image quality.
Change the screen's resolution in the VM's configuration, either via the GUI, or directly via the command line with
xrandr with no arguments, it'll display all outputs (in this case they're just the virtual outputs emulated by VirtualBox).
Screen 0: minimum 320 x 200, current 1024 x 768, maximum 8192 x 8192
VGA1 connected 1024x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
xrandr --output <output> --size <resolution> and replace "output" and "size" with the connected output that you got from the previous command (in this case
VGA1) and the resolution you want to set, I recommend setting it just a bit lower than your actual monitor to leave space for the taskbar and Virtualbox window title bar.
xrandr --output VGA1 --size 1920x1040.
Once you set the resolution in the VM, VirtualBox will automatically resize its window on the host.
By the way, here's the official Ubuntu documentation about changing the screen's resolution.