Automatically collect internet (broadband) speed metrics over time

Question

(Before anyone jumps to the conclusion that this is the same question as Quick way to check my broadband speed, please read it through. Also I am looking for more comprehensive and detailed data than the poster in this question.)

Many years ago I used to use a tool called Line Speed Meter (or Monitor) by TCPIQ. This was a brilliant piece of software that would sit quietly in the background and several times a day (according to a selectable frequency) would reach out and test internet line performance against servers all over the world. In the course of a week (or more), thousands of servers would be tested and enough data would accumulate that one could analyze performance according to time of day, day of week, location of server, etc. Drill-down was possible to the point where you could see every server that was tested.

This data was invaluable in determining if one really was getting the performance being paid for. It was also infinitely more valuable than spot checks you perform at sites like http://speedtest.net or even the better-regarded ones like http://www.dslreports.com/speedtest or http://TestMy.net (both of which, incidentally, are biased toward USA users - which I am not one of anymore).

Unfortunately Line Speed Meter is no longer supported and the company that made it now has instead a tool called isposure that is nowhere near as flexible or transparent - it is geared to supplying data to ISPs rather than to the consumers. For example, according to their FAQ, it usually runs only once a day automatically - I'd like to start with running it every couple of hours for a week or two - and scale back once I have a good basis so that I can notice when performance changes. Also, the details of the tests are no longer visible to the consumer-user.

My question: is there a (preferably free) tool/service available today to automatically collect comprehensive line speed metrics so I can map out and analyze the performance I am getting on my Broadband internet connection?

EDIT 2012-02-13: I left something out of the above. In my country, each ISP sells a package that implies performance beyond the borders of my small country. They own (or lease) the lines going across the ocean. They have full control of the bandwidth leading from their servers to the backbones in the USA and EU. Peformance through the cloud is very much relevant and although not entirely, certainly largely under the local ISP's control, since traditionally, the bottleneck has been traffic out of the country. As an example, at the moment, my ISP is advertising widely their new transatlantic cable and how their performance now outstrips everyone else's. I haven't "felt" it - so I want a tool to quantify it.

Performance within my country is usually good. The problem is that almost all my work involves access to the Internet in North America, Europe, Pacific Rim and others. So I care about and want to document the end-to-end performance I am getting from my ISP - and if I switch, compare to what I get at a new ISP.

EDIT 2013-03-07: As @harrymc pointed out I neglected to mention OS. All the machines on my home LAN are (sadly) Windows (XP or 7). If A tool requires *nix, I have a couple ancient machines in storage I could probably re-purpose - but my preference is for a Windows-based tool. Thanks.

EDIT 2013-03-10: As a further clarification - a method to monitor speeds achieved during regular work provides half of what I'm looking for. What was nice about TCPIQ is that it would ferret out thousands of sites around the world and do brief d/l & u/l tests against them to check speed and then log the results. This gave a valuable statistical database for analysis of speed overall, as well as during times of day, days of week and according to geographical targets. That's the sort of functionality I'd like to achieve (again).

Answer 1

1. Not exactly an answer, but conceptually relevant:

Your ISP is selling you a speed from your house to their border router situated at the edge of The Internet. Your ISP has no control over what happens to your packets once they're out 'mongst the tubes.

The metrics you're describing are actually tracking latency between servers well outside your provider's network. Information like that is not at all relevant to the speed your provider is selling you, and can easily be obtained at places like internettrafficreport.com.

I assume the software you're describing was always meant for people managing networks, and not for end users who would confuse latency with last mile speed performance as you have.

2. Not a software solution, but still a way to get the information you want:

To test the health of your connection, run a tracert to some random server on the internet. Find the last hop on your provider's network: that's their border router, and the last point over which they have any control. Run a ping -t to that IP for up to a week: there's your real last mile performance metric.

If you're on a shared resource like cable, expect packet loss during peak hours (when everyone's online) and bursts of awesome performance when all your neighbors are at work or asleep. If you're on a private connection like DSL, expect a fairly uniform response over time.

3. A way to approach your provider with information they'll listen to:

If you think you're not getting the speed you're paying for, find your provider's own speed test (it will be on a server on their network and not out on the internet like the speed tests you mention). Perform this test ten or fifteen times over the course of a week. Calculate the average of all these tests.

Your final number should be roughly ten percent under the speed your provider sold you. (The missing 10% is protocol overhead.) If the end result is much lower, contact your provider and have them fix the problem.

Answer 2

The simplest bandwidth measuring tool is your router, if it supports a more advanced firmware such as DD-WRT or Tomato. Both firmwares automatically track bandwidth usage, so it's just a matter of knowing where to look.

See for example : How to use the DD-WRT firmware to monitor your bandwidth.

A free tool that I have found is NetWorx, that does much more than just bandwidth measuring.

(It would help to know your operating system when suggesting products.)

Answer 3

The best way to do this and the most common way I've seen it done by datacenters and ISP's is to use Cacti, which is a rrdtool-based graphing app.

http://www.cacti.net/

It can get data from any SNMP capable device (a surprising amount of devices). It will poll your devices every so often and collect data (bandwidth usage data in your case) and graph it for you. You can measure the 95th percentile of usage, total input in MB, GB, TB, output, etc. You can even graph memory usage of servers, etc when setup properly.

Great software, and it's free and open source. check it out.