You are right, the DLLs of your application are loaded on-demand at run-time and they'll stay loaded unless your system runs out of memory.
So if you start the same program several times, the second run will be faster as a benefit from the pre-loaded libraries.
Please take a look:
Understanding Cold Startup and Warm Startup
Cold startup occurs when your application starts for the first time
after a system reboot, or when you start your application, close it,
and then start it again after a long period of time. When an
application starts, if the required pages (code, static data,
registry, etc) are not present in the Windows memory manager's standby
list, page faults occur. Disk access is required to bring the pages
Warm startup occurs when most of the pages for the main common
language runtime (CLR) components are already loaded in memory, which
saves expensive disk access time. That is why a managed application
starts faster when it runs a second time.