On your Mac, you have a bunch of applications: Safari, Microsoft Word, Address Book, what have you. Each one of these applications can open any number of windows: you might have one window open in Safari where you're reading this site, another where you're checking the news at http://www.bbc.co.uk, and so forth. At the same time, you might be editing your résumé in Microsoft Word in one window, and writing a memo in another.
The left half of the Dock is where applications sit. The compass in your Dock is Safari; a light underneath means that it's running. Like I said, Safari can open multiple windows—one at superuser, one at the BBC, and so on—but each of these windows is managed by Safari itself. If you use the yellow button, the window is minimized: it is effectively freeze-dried and placed in the right of your Dock. If you use the red button, the window is closed: it's gotten rid of and the application forgets everything about it. However, in both cases, the application will still be running: although it won't have to worry about managing superuser, it will still have to manage the page with BBC news. And if you close every window, the application will still be open, in case you want to open a new window or do something else with it.
Basically, the Mac is application-based, with each application containing multiple windows. The PC, on the other hand, is window-based, and each window typically corresponds to one application. This may be the source of your confusion.