Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have two systems. I have some log files on one system (A) that are being generated every hour. I want to parse those files from another system (B). One way is to copy the files and then parse them on (B). But this will consume a lot of bandwidth to copy and if I parse the files on original system (A) it will increase the CPU utilization, which I must avoid.

Can somebody suggest the most optimal way of doing this? In simple words, what I want is: Use system B to parse the files located on Remote System A so that I don't have much load on system A (regardless of load on system B) whilst also the load on the network is optimized.

share|improve this question
    
If the file is on B and you want to parse it on A, you will have to transfer the contents. What makes you think it's faster to "transfer" the files than to copy them? Aren't you moving the same data? –  Nicole Hamilton Mar 7 '13 at 6:43
    
how big are these files? copying text files should not consume much bandwidth. nor should parsing them consume much CPU for that matter, –  terdon Mar 7 '13 at 7:44

1 Answer 1

f they are just standard text files, you can probably get close to 90% compression rates if you compress them before sending. There will be some load while compressing them, but probably less than what analyzing would cost.

Edit: due to the comments, I should also suggest using HTTP compression and serving the log files to the remote machine for parsing. It will still generate load to compress the pages, but maybe your web server will do a better job at balancing the resources versus firing off 7-Zip in the background.

share|improve this answer
    
compression is bound to be more CPU intensive than parsing the files remotely. The OP wants to minimise cpu load on the remote system. –  terdon Mar 7 '13 at 7:41
    
-1 Wrong answer. This is almost certainly a much heavier load on both ends than simply pumping the data across a network. –  Nicole Hamilton Mar 7 '13 at 7:52
    
True, but he wants to minimize CPU load and network traffic. I guess parsing remotely with HTTP compression would be the best option. –  Louis Mar 7 '13 at 8:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.