When I download something like Firefox, there is an icon on the desktop.
After installing it, the icon is still there. If I delete the icon, I have to go back to downloads and click on the .dmg and re-install it.
What am I missing here?
The .dmg is a disk image file. When you double-click on that, it mounts the disk image, which makes it show up as a virtual hard drive icon on your desktop. Open that virtual hard drive and you'll see either the app you want to install, or an installer package file (cardboard box icon, .pkg).
A: If you see an app, simply drag that app to your /Applications folder. This is a self-contained app that needs no installer; you just drag-copy it to wherever you want it, and that's it. Often, the mounted disk image already shows a link to the Applications folder. Just drag the application icon into that folder within the very same window:
B: If you see an installer package, double-click it to run the installer.
After either A or B, eject (unmount) the virtual hard drive icon, and drag the .dmg file to the trash. Then find your app in your /Applications folder and launch it by double-clicking it from there.
It sounds like you've been launching your apps from within the mounted disk images all the time, rather than copying them over to your /Applications folder first and launching them from there.