Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I built a crawler for my own personal use that queries many pages with curl_multi_exec. It uses 98% CPU and memory for 10 seconds on my laptop. I'm only crawling around a 1000 pages, but I wanted to do more. Is there a boundary my laptop has to prevent it from crashing? Can I crawl a larger number of pages?

And for the main question, am I destroying my laptop a little everytime I do this?

share|improve this question
add comment

5 Answers

up vote 1 down vote accepted

A well designed operating system should handle any CPU and memory load.

You didn't mention what OS you're using, but both Linux and Windows should handle loads like that for longer periods of time. In extreme cases your process can be killed by OS, especially on Linux machines. Windows will rather try to limit its CPU/mem usage instead of killing it instantly.

For your second question: some people say silicon chips do wear out a bit. But there are many cases of CPUs running under 100% load 24/7 for years (for example servers). You don't have to worry about that as long as your cooling is right.

Still, maybe you should rethink the problem - splitting the job into separate smaller jobs is usually a good idea.

share|improve this answer
    
I split the job querying 10 pages at a time, and it still crashed my laptop. I had to do a system restore. I'm afraid to try again. –  John Smith May 3 '13 at 20:48
2  
curl_multi_exec seems to be a PHP function; consider whether you can add something like usleep(50000) somewhere within your main loop. I don't think PHP is smart enough for sleep(0) to work as a hint to the scheduler that the process can suck mud, but perhaps a 50ms sleep will be enough of a hint; failing that, sleep(1) should certainly keep the script from driving the CPU flat out, although it will most likely slow the crawl down a great deal. –  Aaron Miller May 3 '13 at 20:56
    
@AaronMiller, there's no reason to avoid using the CPU to its full ability, as this answer explains. –  psusi May 3 '13 at 23:51
    
@psusi Not in the general case, no. But if, as is detailed in other answers and comments on this page, running the CPU at 100% is guaranteed, thanks to lousy laptop cooling, to overheat it after a while and force a thermal protection shutdown, then I'd say that makes an excellent reason to throw some sleep() calls in there, don't you think? –  Aaron Miller May 4 '13 at 6:42
    
@AaronMiller, it is not guaranteed at all, quite the reverse. If it ever happens you have a broken laptop that you should RMA. A much better workaround is also to manually slow down the CPU rather than add sleeps to one program. –  psusi May 4 '13 at 16:52
add comment

Is there a boundary my laptop has to prevent it from crashing?

Modern processors will slow down or shut the computer down if they start to overheat, to prevent damage.

http://www.intel.com/support/processors/sb/CS-029426.htm

That's not to say that all consumer-grade low-cost laptops will be able to handle it.

I built a crawler for my own personal use that queries many pages with curl_multi_exec. It uses 98% CPU and memory for 10 seconds on my laptop.

"a program that consumes 100% CPU in pursuit of actually accomplishing something is hardly scorn-worthy; indeed it should be commended for efficiency!" - http://blogs.msdn.com/b/oldnewthing/archive/2010/12/03/10097861.aspx

(Author Raymond Chen is a knowledgable Microsoft employee).

I'm only crawling around a 1000 pages, but I wanted to do more. Is there a boundary my laptop has to prevent it from crashing? Can I crawl a larger number of pages?

The ideal is that it all fits in memory. If it goes outside memory, Windows will start using disk space as an extension to memory. This is, however, very much slower. If you run out of disk space, or have a fixed pagefile size, then your process will not be able to use more memory. If it's well written it will handle that, if not it will crash.

And for the main question, am I destroying my laptop a little everytime I do this?

No moreso than normal use. The ideal is that a processor rated for speed X can continuously run at that speed. It's not like a motor where they have a duty cycle of expected use 2-hours on 4-hours off, etc.

Of course, your specific laptop might have design flaws, poor tolerances, have a cooling system that can't cope with 100% full power, have component flaws, your software might have bugs, interactions of all the specific patches and versions and programs you have installed might cause odd behaviour...

share|improve this answer
    
Although it is only 3 years old, I think based on how much I've used it, it's cooling system has deteriated. Thank for the direct answers. –  John Smith May 3 '13 at 21:04
add comment

CPU Usage

  • When a processor executes instructions, it gradually gives off heat ("heat dissipation"). The more instructions it executes in a given amount of time, the more heat it dissipates. CPUs are designed in a way that they are limited in how many instructions they can execute within a given period of time; if they were unlimited, the CPU would get so hot that it would start to reach the melting point of the chemical compounds used to construct it, and might eventually catch fire.

  • The heat dissipation is handled in three ways: cooling, firmware control, and hardware/software limitations.

  • Cooling actively attempts to move heat away from the processor, which cools it down. Cooling in most computers is done by mounting a highly conductive piece of metal, called a heatsink, on the top of the CPU. The heatsink is effective because the metal used will "siphon" the heat off of the processor by physically touching the hot processor, and transferring the heat into the heatsink material at a rapid rate. The heatsink then has cool air blown over it by a fan, which cools down the heatsink by heating up the air. The air is then vented out of the case and into the environment. More advanced setups can use water or liquid nitrogen as a more extreme way of cooling than air, but heatsinks are almost universally used.

  • Firmware control is software implemented in the CPU and/or operating system that attempts to slow down the CPU if thermal monitoring (basically a thermometer) detects that the CPU is getting too hot. By slowing down the CPU, you may notice reduced application performance, but the CPU's hardware is kept in a stable, functional state by reducing the heat dissipation to a level that the cooling hardware can reliably dissipate. The firmware control software on modern CPUs (produced in the last 5 years) is extremely advanced, and can fine tune the CPU's execution speed so that it takes advantage of all the available processor power, without heating up the CPU to unsafe levels -- even if your cooling solution is sub-optimal; for example, if your ventilation duct is filled with dust, the air used to cool off the heatsink will be relatively hot, and will not do a great job of dissipating heat. In this scenario, the CPU would slow down to avoid overheating, thanks to the firmware doing its job.

  • Hardware/software limitations are physical limits of the device or the operating system that, by design, moderate how much current/voltage can be used by the CPU, and how many instructions it can execute per second. For example, the voltage regulator module (VRM) on the CPU or on the motherboard will determine how much voltage to let into the CPU. Other factors, such as mandatory synchronization (like waiting for a pipeline to empty, or cache line flushes) will slow down the CPU during the execution of certain software functions, such as system calls or thread synchronization primitives. These limitations are unavoidable, and are often viewed in a negative light because they limit how much performance the CPU can achieve, but these limitations also prevent the CPU from overheating.

In general, systems which are designed by well-regarded manufacturers are designed in such a way that, pending user modification of the hardware or extremely bad environmental conditions (a very hot room, vents clogged with dust, etc.), the hardware should be able to function at full speed at all times without being artificially slowed down. This is because the manufacturer will test the cooling solution "under load" and make sure that enough airflow is being done by the fans to keep the CPU cool while it's very busy, and make sure that the heat sink is transferring heat away from the CPU fast enough to keep up with the heat dissipation.

In short, when you are using 100% CPU on a well-designed system with sufficient cooling, you will not experience any adverse effects, and the CPU is actually designed to operate at this level; you can't force it to go "above" 100% because that 0 - 100% scale is determined by the CPU's clock speed and other factors of the microarchitecture which your application programs cannot modify. Note that you can go above 100% load, but load is not equated with actual throughput: the load is how much demand programs are placing on the CPU, but while demand can exceed throughput, that won't make the throughput somehow increase beyond its maximum of 100%.

RAM Usage

RAM usage is a completely different thing and has nothing to do with cooling (in general). Using up all your RAM with programs can have detrimental effects on the system, but this will mainly slow down the performance. It is pretty difficult to outright crash a system with a BSOD by using up all RAM these days, because the program will get Out Of Memory error and crash, before the operating system crashes. It's possible under extreme conditions, though. RAM usage has no ill effect on the hardware.

share|improve this answer
    
+1 my laptop screen just turned black and stayed there, so I eventually had to power off and system restore. –  John Smith May 3 '13 at 21:06
add comment

Run an anti-virus program and you will see higher than 98% utilization for more than 10 seconds... I don't see a problem.

share|improve this answer
    
but if I added a 1000 more to that, will the laptop crash? Is there a boundary for the CPU and memory? I've heard of "blue screens" and crashes from high memory use. There's no limit on curl_exec_multi, so it's querying pages simultaneously unmetered. –  John Smith May 3 '13 at 20:28
1  
The kernel will kill your program if it allocates so much memory that no more remains to satisfy its further allocation attempts, but you won't see a kernel panic (BSoD, &c., depending on platform) in that situation unless something is wrong with the underlying hardware. –  Aaron Miller May 3 '13 at 20:42
    
@AaronMiller Yeah, before you posted, I tried and my laptop crashed. I had to do a system restore. –  John Smith May 3 '13 at 20:45
1  
@JohnSmith Sorry to be the bearer of bad news, then. If it bluescreened, I'd start with Memtest86+ to find out whether it's your RAM that's bad, which is the most common cause I've seen for such mishaps. If it just powered itself down without so much as a by-your-leave, then I'd recommend investigating its cooling situation; modern CPUs include thermal protection and will shut themselves down before overheating to the point of physical damage, and laptops tend to have lousy cooling anyway. If it was already marginal, driving it at 100% for a while would certainly provoke a thermal shutdown. –  Aaron Miller May 3 '13 at 20:51
    
@AaronMiller Thanks aaron, on the plus side, it restored the laptop to a better point HAHAHA. I restored a lot of files I thought I had lost. Best day ever! –  John Smith May 3 '13 at 20:53
add comment

Assuming your CPU has adequate cooling, you can run it at 100% utilization forever, and it will not damage the computer in any way. The same is true of your memory chips.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.