Running multiple commands (or even computers) in parallel for more efficiency.

learn more… | top users | synonyms (1)

24
votes
7answers
18k views

Parallelizing rsync

I just moved and discovered after some trial and error that somewhere between my home and my remote server, there is some throttling going on...but the throttling is not very intelligent. It only ...
14
votes
5answers
18k views

Is a higher core count or higher clock speed more beneficial to a computer's performance? [closed]

With lowering silicon costs and rising consumer needs, manufacturers seem to be pushing one of two things: clock speed and/or core count. With the way things are going, it doesn't seem that clock ...
10
votes
6answers
6k views

Running commands in parallel with a limit of simultaneous number of commands

Sequential: for i in {1..1000}; do do_something $i; done - too slow Parallel: for i in {1..1000}; do do_something $i& done - too much load How to run commands in parallel, but not more than, for ...
9
votes
4answers
9k views

Parallel shell loops

I want to process many files and since I've here a bunch of cores I want to do it in parallel: for i in *.myfiles; do do_something $i `derived_params $i` other_params; done I know of a Makefile ...
7
votes
5answers
10k views

Running multiple concurrent background processes then wait for all of them to terminate, on Windows

Anybody know if there is cmd script that does that and/or a command line tool ready for the job? Thanks!
6
votes
3answers
4k views

Kill process after it's been allowed to run for some time

I want to limit the time a grep process command is allowed to run or be alive. For example. I want to perform the following: grep -qsRw -m1 "parameter" /var But before running the grep command I ...
5
votes
4answers
5k views

What's the best way to perform a parallel copy on Unix?

I routinely have to copy the contents of a folder on a network file system to my local computer. There are many files (1000s) on the remote folder that are all relatively small but due to network ...
5
votes
3answers
7k views

netcat as a multithread server

I use netcat to run a simple server like this: while true; do nc -l -p 2468 -e ./my_exe; done This way, anyone is able to connect to my host on port 2468 and talk with "my_exe". Unfortunately, if ...
5
votes
1answer
215 views

Read non-blocking from multiple fifos in parallel

I sometimes sit with a bunch of output fifos from programs that run in parallel. I would like to merge these fifos. The naïve solution is: cat fifo* > output But this requires the first fifo to ...
4
votes
5answers
27k views

Is it possible to combine processing power of 2 computers?

Here are a few questions, I wish you could enlighten me. Is it possible to combine processing power of 2 computers? How do I do it?
4
votes
1answer
541 views

Parallel processing slower than sequential?

EDIT: For anyone who stumbles upon this in the future: Imagemagick uses a MP library. It's faster to use available cores if they're around, but if you have parallel jobs, it's unhelpful. Do one of ...
4
votes
1answer
8k views

Differentiate CUDA Cores(NVIDIA) and Stream processor(ATI/AMD)

I think the question is answered here but I'm still wondering if what is the difference between the units it is referring to. For example, a GTX 570 has 480 CUDA cores, while the ATI equivalent HD ...
3
votes
4answers
894 views

Optimum number of threads while multitasking

I know similar questions have been asked but I think my case is a little bit diffrent. Let's say I have a computer with 8 cores and infinite memory with a Linux OS. I have a calculation software ...
3
votes
2answers
419 views

How can I run several parallel shell jobs and monitor all of them at once?

I have a large number of large files (hundreds of files, hundreds of MB each) which I need to pipe through a number of programs to filter and convert them. I'm taking advantage of multiple CPU cores ...
3
votes
1answer
448 views

Does anyone know of an antivirus that can scan multiple hard drives simultaneously?

I have 5 hard drives and am so very sick of waiting for 3 hours for the virus scanner to scan each drive one at a time. I see absolutely no reason that a virus scanner can't be threaded to scan each ...
3
votes
0answers
290 views

Jenkins realtime console output for parallel threaded fabric command

We have a Python fabric command that runs in parallel across several hosts, something like this: $ fab --hosts=prod1.server,prod2.server,prod3.server --parallel copy_cache This will copy cache to ...
2
votes
3answers
8k views

Have today's desktop PCs surpassed IBM's Deep Blue of 1997?

I was wondering if my PC has surpassed the processing power of Deep Blue which famously won against the human world champion Garry Kasparov. What was the processing speed of Deep blue as compared to ...
2
votes
1answer
94 views

Parallel-Processing in Integrated GPU

I have an integrated GPU, meaning it works on the same RAM as my CPU. (ATI Radeon HD 4200, integrated on M4A785TD-V EVO) Does this also mean my GPU doesn't compute in parallel? see also: ...
2
votes
4answers
481 views

Intel Core 2 duo / AMD athlon X2 parallel processing capability

Does Intel Core 2 duo/ AMD athlon X2 really have 2 separate processors? i.e are they capable of doing real parallel processing? What I don't understand is the difference when somebody says Cores or ...
2
votes
2answers
53 views

Why does F@H not bind to more than one core on Windows?

I have been contributing to Stanford's Folding@Home project for some time with most of the computers I own. I just installed the Windows client on a new machine running Windows 7, but see that the ...
2
votes
3answers
967 views

Running threads on a non hyper-threading Intel processor

Intel Pentium Processor E5700 (2M Cache, 3.00 GHz, 800 MHz FSB) does not have hyper-threading but it has has 2 cores so I am assuming it has 2 threads Now if I write a heavy number crunching program ...
2
votes
1answer
841 views

How do the cores in an AMD Bulldozer module compare to Intel HTT virtual cores and two separate cores, in terms of multitasking performance?

This question is basically an extension of Hyper-Threading and Dual-Core, What's the Difference? The Bulldozer-based AMD FX processors are marketed as having four, six, or eight cores, but ...
2
votes
1answer
256 views

How can I share memory for parallel computing in Mathematica?

Is it possible to use RAM from other computers to assist with Mathematica Light Grid Server 7 parallel computing? If so, how?
2
votes
1answer
241 views

Simulink - running parallel simulations

Is it possible to run Simulink calculations in parallel mode on cluster? And if its possible then what actually Simulink do when number of processes that i need is more than cluster has nodes? Thank ...
1
vote
2answers
678 views

gunzip multiple files and concatenate in parallel?

Is it possible to gunzip multiple files and concatenate them into one big file, but do it in parallel given a multicore machine? For example, right now I do: gunzip -c file1.gz > final gunzip -c ...
1
vote
1answer
27 views

How can I force gnuparallel to carry out a command set sequentially?

Gnu parallel is a powerful tool that I use to run many independant BASH commands as a single set in parallel. I would like to be able to run the same commands SEQUENTIALLY without significant changes ...
1
vote
1answer
940 views

Does a HyperThreaded Quad Core Processor Actually Have 8 “Cores” [duplicate]

I have an Early 2011 Quad Core i7 Mac. Due to hyperthreading, a lot of programs report 8 "cores". If I were to write a program that executes in parallel, could I kick off 8 instances of this program ...
1
vote
1answer
139 views

What kind of CPU/GPU integration is offered by APUs?

I'm truly fascinated by the idea of GPGPU and using the GPU for heavy processing. I'm seeing that also APUs (Accelerated Processing Units, CPU+GPU on the same chip) are gaining a consistent ...
1
vote
1answer
238 views

Modern CPU for highly concurrent server applications?

I am looking for a hardware, which must run about 256 computationally intensive real-time concurrent tasks in 24 hour mode. Each task takes about 40-50 MFLOPs, so all tasks require about 10 GFLOPs. ...
1
vote
1answer
388 views

Is it faster to run one process that spawns N threads or to run N processes?

I need to run ~1500 jobs on my lab's server. The program I am using (blastpgp if anyone cares) is capable of running in parallel mode, the "-a" option sets how many threads it will launch. The server ...
1
vote
0answers
96 views

Linux command-line utils with built-in support of multithreading

I would like to be able to compare performance of multicore computers from a point of view of a user who performs typical day-to-day tasks in Linux. My current goal is to collect a list of compute ...
1
vote
1answer
173 views

Is it normal to overworking of just one CPU when using openMP

When I use OpenMP in FORTRAN to parallelize a program on linux, ' top' command show me just one processor working ~%800 instead of ~%100 x 8 processors. Is it a expected situation ?
1
vote
1answer
109 views

Channelling processing power from Linux servers over a gigabit LAN to Macintosh system

As you may know Apple offers a program called Xgrid for sharing processing power between Macs. It also allows for easy clustering. Is there a way to channel processing power from Linux nodes over a ...
1
vote
0answers
74 views

Can I set up PVM in such way that master doesn't start own tasks?

We have a Perceus cluster (Warewulf) consisting of 4-core master and a number of 4-core nodes. I can start PVM on the master and then add all nodes - it works fine. However, there are other guys who ...
0
votes
2answers
219 views

Supercomputer via LAN [closed]

If I decide to parallel two computers via direct LAN connection would LAN (100 Mbits/s) bus bandwidth be enough for CPUs to communicate without performance loss? Or do I have to use some other bus ...
0
votes
1answer
54 views

Clock speed vs. number of cores for parallelized computer simulation [duplicate]

My question is similar to some "speed vs. cores"-questions that have already been asked on this website. However, I am interested in some very specific and technical points. So I hope that my question ...
0
votes
1answer
27 views

How does multi-core architecture work with regard to parellelism?

I think I am correct in saying that scheduling is the/a main principle of the single core CPU, where the CPU is interrupted when switching from processing one set of instructions belonging to one ...
0
votes
1answer
45 views

The exact workflow of PVM (Parallel virtual machine)

These days I am learning PVM technology for concurrent processing. I go through so many blogs, tutorials to understand . Yes, I got some points, but no one gives me clear workflow of how it perform ...
0
votes
1answer
391 views

PBS batch jobs - the qalter command

I've got a giant computation running on a Scientific Linux cluster. At present I have over 600 jobs parked in the queue, waiting for processor time, while a few are running. I'm trying to use the ...
0
votes
1answer
41 views

How to find out whether libav is built on MPI/LAM?

Just what the title states ; I expect to be doing a touch of audio cleanup/format switching. My plan is to try to do this with libav. Apparently such processing is fairly intensive; hence I would like ...
0
votes
3answers
408 views

Size of L1 cache memory on T9600 CPU

I want to know the size of the l1 cache memory for my processor T9600 I visited Intel website for the specification of the processor but couldn't get the needed information. Any one help me thank you ...
0
votes
0answers
11 views

How to implement Intel MPI benchmarks for mpich2?

Many people have recommended the use of Intel MPI benchmarks for mpich2, however the installation is not very simple. Could anyone provide steps for the installation and sample test case for it? Which ...
0
votes
0answers
11 views

Scientific Linux 5.5 hangs at the beginning

We are taking theses days a parallel processing course in college, we are working on scientific linux 5.5 version with MPI, I just imported .ova on the virtual box but unfortunately when am trying to ...
0
votes
0answers
21 views

Is it possible to configure Linux OpenMPI v1.6.5 to use multiple working directories on a single node?

I am running some quantum chemical calculations on a refurbished PowerEdge rack server, with dual quad-core Xeons, 16GB RAM, and a single 500GB SATA HDD. OS is vanilla Debian Wheezy. For one of the ...
0
votes
0answers
21 views

Do threads/processes that create/delete files/directories under a shared parent dir interfere?

I was wondering: If I have a directory /a/b/c/ and under c X threads create subdirectories and files and delete them (they are temporary directories and files as intermediate steps of processing) do ...
0
votes
1answer
105 views

Is Virtual Machine capable of running as normal PC would?

Let's imagine that you have a super-powerful-server at home of size of desktop (just visualisation). 2048 physical cores each 3GHz, 64TB RAM, GTX 780 Ti Black Edition (SLIx4, (four graphics card ...
0
votes
2answers
91 views

fastest RAM configuration [closed]

I am planning to buy a workstation for doing heavy image processing parallel computation (for 3-dimensional electron microscopy) under linux. It will have 64 cores (4 AMD opterons of 16 cores each). ...
0
votes
0answers
55 views

How to parallelize crontab executions to increase user base for web app based on mongodb and mysql?

I have a symfony based web application that runs on mongodb and mysql backend. The principle of the application is that for each user there is a python script that runs 4-12 times a day on cronjobs ...
0
votes
0answers
35 views

How to accelerate a shell script for a mongoDB application?

I started shell scripting for my work, but I must admit I'm still far away from even being a rookie. Therefore I wanted to ask you for your help/advice. I build a script for a big data application ...
0
votes
0answers
91 views

How to extract multi volume cpio archive in parallel

I have a large set of files. How can I create multivolume archive using cpio in such a way that I can extract them in parallel? Currently I'm using: find mydir | cpio -ov | split -d -C 3840M - ...