I need to set some fancy resolution, like 700x500, 320x860, 400x400 etc. The problem is that the output of the device is not recognized, so it's set to default. But when I add a newmode to xrandr it warns that "Failed to get size of gamma for output default". And after I add the new mode to default output, xrandr fails to set the new resolution, responding "Configure crtc 0 failed". How can I fix this issue?
Not every mode is possible. Here you can get nice example of some possible custom modelines. Still every modeline is not supported by every video card or monitor.