I find this behavior annoying and would like to disable it. How do I do so?
Close Outlook and then delete the registry key named WindowPos at the path HKEY_CURRENT_USER\Software\Microsoft\Office\14.0\Outlook\Options\Reminders\ and restart Outlook.
Unfortunately, you may need to continually delete this registry key.
Here's a LINQPad query (as a GitHub Gist; save it on your computer and open it in LINQPad to run it) that will delete the registry key if it exists: