The program Netbalancer looks to do something like what you are after. With the added bonus that it will also help to control internet usage, which to me would be your next question.
This is more "per application" monitoring, but from what I can tell you might be able to drill in to each application and see downloads "per site".
I've not used it myself but it is recommended by How To Geek which I would take as a good sign.
By the looks of it the program is primarily designed to limit and control downloads but the side effect is that you can monitor downloads from applications as well.
NetBalancer is an internet traffic control and monitoring tool designed for Windows XP/2003/Vista/Win7, both x86 and native x64.
There is a free version that has some limitations, but just for monitoring you should be fine. None of the free features are likely to be showstoppers for you at the moment.
*The Free version is limited to a maximum of 5 process priorities/limits and 5 rules at a time, has no separate network adapters management and no support for Network Grouping.
As you can see there are columns for current download rates as well as overall download totals per application on the right.
The HowToGeek article does mention a quirk with antivirus and downloading programs that you may need to be aware of.
Here we see which application is currently downloading data and it reveals a peculiar quirk about the way anti-virus software works. The real download is a copy of a Linux ISO we’re downloading in Chrome. The download, however, was kicked over from Chrome to the Avast anti-virus scanner. If we didn’t have Avast installed then the data would appear under the Chrome.exe directly.
Basically your antivirus may "catch" downloads then download them itself and pass on to the browser, but this should hopefully only be a minor annoyance and depends strongly on your antivirus package. You can get details of the download though so it should all work out in the end.
According to Rarst.net an alternative is NetLimiter which has a "monitoring" version.
According to Rarst:
Real time monitoring is nice but real power of NetLimiter is very detailed statistics. It remembers and can show what traffic specific process consumed in specific time period. Reports can be viewed in interface or exported in XML (manually or on schedule).
Free Monitor version only keeps stats but there are also advanced versions that allow to shape bandwidth by setting limits or guaranteed bandwidth for applications. That’s where Limiter part of name comes from actually.
Which may actually be closer to what you are after. Again I apologise that I have no real experience trying the software to be able to give real advice on how to use it.