How to visualize/graph a massive dataset?

Question

I have a CSV file of data collected using Performance Monitor. It collected CPU usage at 5 second intervals for two weeks.

I need to be able to graph or visual the data in a way that'll help me determine how often the CPU is maxed out versus idling.

Excel has a limit of 32,000 data points in a graph, and I have a lot more than that. So I either need a different way to graph the data, or a way to filter the data in Excel, such as averaging over 10-minute blocks of time.

Answer 1

If you are just looking to filter out the times when the data is maxing out a quick fix solution might be to just import the data 32,000 items at a time into multiple Excel sheets and filter each one of those out to find the time when the CPU maxes out.

For me, I would normally do this sort of thing with Python, Numpy and Matplotlib, but obviously if you don't have a preferred solution, it might be a bit much to learn something new to just plot the data once.

Answer 2

I would say you have to look into something like R for this purpose, not simply Microsoft Excel. I haven't used it myself yet but it was used a lot by students at my university, it should be able to handle such amounts.

Answer 3

You could sample over (say) a minute, bringing the data point from 241920 to 4032. To do this you need to add a column to your data, and create a function that mark just the start of the minute. Then the sample could be taken with a simple VLOOKUP(). If you want help with the sample function post details over the data format (expecially the date/time column).

Of course instead of sampling you could average or take max and/or min