Running multiple commands (or even computers) in parallel for more efficiency.

learn more… | top users | synonyms (1)

20
votes
7answers
16k views

Parallelizing rsync

I just moved and discovered after some trial and error that somewhere between my home and my remote server, there is some throttling going on...but the throttling is not very intelligent. It only ...
14
votes
5answers
16k views

Is a higher core count or higher clock speed more beneficial to a computer's performance? [closed]

With lowering silicon costs and rising consumer needs, manufacturers seem to be pushing one of two things: clock speed and/or core count. With the way things are going, it doesn't seem that clock ...
10
votes
6answers
5k views

Running commands in parallel with a limit of simultaneous number of commands

Sequential: for i in {1..1000}; do do_something $i; done - too slow Parallel: for i in {1..1000}; do do_something $i& done - too much load How to run commands in parallel, but not more than, for ...
9
votes
4answers
8k views

Parallel shell loops

I want to process many files and since I've here a bunch of cores I want to do it in parallel: for i in *.myfiles; do do_something $i `derived_params $i` other_params; done I know of a Makefile ...
7
votes
5answers
9k views

Running multiple concurrent background processes then wait for all of them to terminate, on Windows

Anybody know if there is cmd script that does that and/or a command line tool ready for the job? Thanks!
6
votes
3answers
4k views

Kill process after it's been allowed to run for some time

I want to limit the time a grep process command is allowed to run or be alive. For example. I want to perform the following: grep -qsRw -m1 "parameter" /var But before running the grep command I ...
5
votes
4answers
4k views

What's the best way to perform a parallel copy on Unix?

I routinely have to copy the contents of a folder on a network file system to my local computer. There are many files (1000s) on the remote folder that are all relatively small but due to network ...
5
votes
3answers
6k views

netcat as a multithread server

I use netcat to run a simple server like this: while true; do nc -l -p 2468 -e ./my_exe; done This way, anyone is able to connect to my host on port 2468 and talk with "my_exe". Unfortunately, if ...
5
votes
1answer
178 views

Read non-blocking from multiple fifos in parallel

I sometimes sit with a bunch of output fifos from programs that run in parallel. I would like to merge these fifos. The naïve solution is: cat fifo* > output But this requires the first fifo to ...
4
votes
1answer
497 views

Parallel processing slower than sequential?

EDIT: For anyone who stumbles upon this in the future: Imagemagick uses a MP library. It's faster to use available cores if they're around, but if you have parallel jobs, it's unhelpful. Do one of ...
4
votes
1answer
6k views

Differentiate CUDA Cores(NVIDIA) and Stream processor(ATI/AMD)

I think the question is answered here but I'm still wondering if what is the difference between the units it is referring to. For example, a GTX 570 has 480 CUDA cores, while the ATI equivalent HD ...
3
votes
4answers
643 views

Optimum number of threads while multitasking

I know similar questions have been asked but I think my case is a little bit diffrent. Let's say I have a computer with 8 cores and infinite memory with a Linux OS. I have a calculation software ...
3
votes
2answers
391 views

How can I run several parallel shell jobs and monitor all of them at once?

I have a large number of large files (hundreds of files, hundreds of MB each) which I need to pipe through a number of programs to filter and convert them. I'm taking advantage of multiple CPU cores ...
3
votes
1answer
400 views

Does anyone know of an antivirus that can scan multiple hard drives simultaneously?

I have 5 hard drives and am so very sick of waiting for 3 hours for the virus scanner to scan each drive one at a time. I see absolutely no reason that a virus scanner can't be threaded to scan each ...
3
votes
0answers
143 views

Jenkins realtime console output for parallel threaded fabric command

We have a Python fabric command that runs in parallel across several hosts, something like this: $ fab --hosts=prod1.server,prod2.server,prod3.server --parallel copy_cache This will copy cache to ...
2
votes
5answers
22k views

Is it possible to combine processing power of 2 computers?

Here are a few questions, I wish you could enlighten me. Is it possible to combine processing power of 2 computers? How do I do it?
2
votes
1answer
91 views

Parallel-Processing in Integrated GPU

I have an integrated GPU, meaning it works on the same RAM as my CPU. (ATI Radeon HD 4200, integrated on M4A785TD-V EVO) Does this also mean my GPU doesn't compute in parallel? see also: ...
2
votes
4answers
462 views

Intel Core 2 duo / AMD athlon X2 parallel processing capability

Does Intel Core 2 duo/ AMD athlon X2 really have 2 separate processors? i.e are they capable of doing real parallel processing? What I don't understand is the difference when somebody says Cores or ...
2
votes
2answers
52 views

Why does F@H not bind to more than one core on Windows?

I have been contributing to Stanford's Folding@Home project for some time with most of the computers I own. I just installed the Windows client on a new machine running Windows 7, but see that the ...
2
votes
3answers
837 views

Running threads on a non hyper-threading Intel processor

Intel Pentium Processor E5700 (2M Cache, 3.00 GHz, 800 MHz FSB) does not have hyper-threading but it has has 2 cores so I am assuming it has 2 threads Now if I write a heavy number crunching program ...
2
votes
1answer
758 views

How do the cores in an AMD Bulldozer module compare to Intel HTT virtual cores and two separate cores, in terms of multitasking performance?

This question is basically an extension of Hyper-Threading and Dual-Core, What's the Difference? The Bulldozer-based AMD FX processors are marketed as having four, six, or eight cores, but ...
2
votes
1answer
234 views

How can I share memory for parallel computing in Mathematica?

Is it possible to use RAM from other computers to assist with Mathematica Light Grid Server 7 parallel computing? If so, how?
2
votes
1answer
232 views

Simulink - running parallel simulations

Is it possible to run Simulink calculations in parallel mode on cluster? And if its possible then what actually Simulink do when number of processes that i need is more than cluster has nodes? Thank ...
1
vote
3answers
6k views

Have today's desktop PCs surpassed IBM's Deep Blue of 1997?

I was wondering if my PC has surpassed the processing power of Deep Blue which famously won against the human world champion Garry Kasparov. What was the processing speed of Deep blue as compared to ...
1
vote
2answers
616 views

gunzip multiple files and concatenate in parallel?

Is it possible to gunzip multiple files and concatenate them into one big file, but do it in parallel given a multicore machine? For example, right now I do: gunzip -c file1.gz > final gunzip -c ...
1
vote
1answer
754 views

Does a HyperThreaded Quad Core Processor Actually Have 8 “Cores” [duplicate]

I have an Early 2011 Quad Core i7 Mac. Due to hyperthreading, a lot of programs report 8 "cores". If I were to write a program that executes in parallel, could I kick off 8 instances of this program ...
1
vote
1answer
129 views

What kind of CPU/GPU integration is offered by APUs?

I'm truly fascinated by the idea of GPGPU and using the GPU for heavy processing. I'm seeing that also APUs (Accelerated Processing Units, CPU+GPU on the same chip) are gaining a consistent ...
1
vote
1answer
237 views

Modern CPU for highly concurrent server applications?

I am looking for a hardware, which must run about 256 computationally intensive real-time concurrent tasks in 24 hour mode. Each task takes about 40-50 MFLOPs, so all tasks require about 10 GFLOPs. ...
1
vote
1answer
341 views

Is it faster to run one process that spawns N threads or to run N processes?

I need to run ~1500 jobs on my lab's server. The program I am using (blastpgp if anyone cares) is capable of running in parallel mode, the "-a" option sets how many threads it will launch. The server ...
1
vote
0answers
43 views

What are some good options for small embedded parallel computing systems? [closed]

I've been trying to find an affordable and small system for parallel computing. I'm looking for something similar to the raspberry pi in that its a fully functional computer system. So far, the best ...
1
vote
0answers
65 views

Linux command-line utils with built-in support of multithreading

I would like to be able to compare performance of multicore computers from a point of view of a user who performs typical day-to-day tasks in Linux. My current goal is to collect a list of compute ...
1
vote
1answer
167 views

Is it normal to overworking of just one CPU when using openMP

When I use OpenMP in FORTRAN to parallelize a program on linux, ' top' command show me just one processor working ~%800 instead of ~%100 x 8 processors. Is it a expected situation ?
1
vote
1answer
107 views

Channelling processing power from Linux servers over a gigabit LAN to Macintosh system

As you may know Apple offers a program called Xgrid for sharing processing power between Macs. It also allows for easy clustering. Is there a way to channel processing power from Linux nodes over a ...
1
vote
0answers
73 views

Can I set up PVM in such way that master doesn't start own tasks?

We have a Perceus cluster (Warewulf) consisting of 4-core master and a number of 4-core nodes. I can start PVM on the master and then add all nodes - it works fine. However, there are other guys who ...
0
votes
2answers
183 views

Supercomputer via LAN [closed]

If I decide to parallel two computers via direct LAN connection would LAN (100 Mbits/s) bus bandwidth be enough for CPUs to communicate without performance loss? Or do I have to use some other bus ...
0
votes
2answers
72 views

What does MPICH stands for? [closed]

As we know that MPI is an abbreviation of Message Passing Interface. However, I am wandering about what does MPICH stands for? Is that just a brand name or does it has some terminology behind it? I ...
0
votes
1answer
24 views

How does multi-core architecture work with regard to parellelism?

I think I am correct in saying that scheduling is the/a main principle of the single core CPU, where the CPU is interrupted when switching from processing one set of instructions belonging to one ...
0
votes
1answer
43 views

The exact workflow of PVM (Parallel virtual machine)

These days I am learning PVM technology for concurrent processing. I go through so many blogs, tutorials to understand . Yes, I got some points, but no one gives me clear workflow of how it perform ...
0
votes
1answer
329 views

PBS batch jobs - the qalter command

I've got a giant computation running on a Scientific Linux cluster. At present I have over 600 jobs parked in the queue, waiting for processor time, while a few are running. I'm trying to use the ...
0
votes
1answer
40 views

How to find out whether libav is built on MPI/LAM?

Just what the title states ; I expect to be doing a touch of audio cleanup/format switching. My plan is to try to do this with libav. Apparently such processing is fairly intensive; hence I would like ...
0
votes
3answers
403 views

Size of L1 cache memory on T9600 CPU

I want to know the size of the l1 cache memory for my processor T9600 I visited Intel website for the specification of the processor but couldn't get the needed information. Any one help me thank you ...
0
votes
0answers
7 views

Is it possible to configure Linux OpenMPI v1.6.5 to use multiple working directories on a single node?

I am running some quantum chemical calculations on a refurbished PowerEdge rack server, with dual quad-core Xeons, 16GB RAM, and a single 500GB SATA HDD. OS is vanilla Debian Wheezy. For one of the ...
0
votes
0answers
19 views

Do threads/processes that create/delete files/directories under a shared parent dir interfere?

I was wondering: If I have a directory /a/b/c/ and under c X threads create subdirectories and files and delete them (they are temporary directories and files as intermediate steps of processing) do ...
0
votes
1answer
74 views

Is Virtual Machine capable of running as normal PC would?

Let's imagine that you have a super-powerful-server at home of size of desktop (just visualisation). 2048 physical cores each 3GHz, 64TB RAM, GTX 780 Ti Black Edition (SLIx4, (four graphics card ...
0
votes
2answers
61 views

fastest RAM configuration [closed]

I am planning to buy a workstation for doing heavy image processing parallel computation (for 3-dimensional electron microscopy) under linux. It will have 64 cores (4 AMD opterons of 16 cores each). ...
0
votes
0answers
37 views

How to parallelize crontab executions to increase user base for web app based on mongodb and mysql?

I have a symfony based web application that runs on mongodb and mysql backend. The principle of the application is that for each user there is a python script that runs 4-12 times a day on cronjobs ...
0
votes
0answers
26 views

How to accelerate a shell script for a mongoDB application?

I started shell scripting for my work, but I must admit I'm still far away from even being a rookie. Therefore I wanted to ask you for your help/advice. I build a script for a big data application ...
0
votes
0answers
65 views

How to extract multi volume cpio archive in parallel

I have a large set of files. How can I create multivolume archive using cpio in such a way that I can extract them in parallel? Currently I'm using: find mydir | cpio -ov | split -d -C 3840M - ...
0
votes
0answers
23 views

Is parallelism in computer science always directly tied to the number of processor cores?

Specific to the Java programming language, would it be accurate to say that two threads can be concurrent, but can only truly be run in parallel if there are two or more cores?
0
votes
0answers
46 views

Does OpenMp supports Real Time Task Computing?

OpenMP API provides a portable environment for modeling shared memory parallel applications in multi-Core platform. Since it works on the top of kernel level threads, my concern is can we use OpenMp ...