I feeling I have an opposite effect than most other people that install Ubuntu as a guest OS because I saw many were having trouble with the window being to small after the installation/restart. After trying install the virtual guest additions, I feel things have been much worse. The options for auto resize guest display and virtual screen is all greyed out. I also tried reinstalling virtualbox manager and creating the VM twice again. I just don't know what happen today that cause this issue as things were find in the past with no issues and I have never ever tried to install guest additions before too.
Are you running this on a laptop with a high-DPI display?
The display of the guest machine is pixelated, meaning that VirtualBox has scaled up the size of the guest machine so that it doesn't appear too small on the (high-DPI) screen. Unfortunately, it looks like this scale factor is too high and is causing it to exceed the size of the physical monitor. Go to View > Scale Factor and reduce the scale factor so that it fits on the display.
As for adjusting the resolution of the virtual display itself, you may want to try restoring (un-maximizing) the window and manually resizing it. Then again, I have little experience running Ubuntu on VirtualBox (I'm most familiar with openSUSE, which comes with the Guest Additions), so someone else may be able to help you better.
I got a okay solution to my problem. I made a clean install Virtualbox and got the newest Ubuntu iso. I lowered the scale factor to 125%. Now my guest OS was normal as expected and not small like I had in my other comment up above. I am able to resize my guest OS manually resizing it myself.
I feel that the issue was the scaling factor on my host OS somehow made an effect on the Virtualbox manager app and in turn my guest OS too.