Running multiple commands (or even computers) in parallel for more efficiency.

learn more… | top users | synonyms (1)

0
votes
1answer
9 views

The main difference between Superscalar and Parallelism

In superscalar processing, more than one instruction is executed in parallel at each core, so how would that differ from parallel processing then? Thank you.
0
votes
2answers
20 views

bash for loop in parallel with n cores

I have a bash loop that looks like: for i in $(seq 0 $max); do my_command $i done and I would like to run this in parallel on n cores. I know that I could do while [[ "$j" -le "$max" ]]; do ...
10
votes
0answers
677 views

Jenkins realtime console output for parallel threaded fabric command

We have a Python fabric command that runs in parallel across several hosts, something like this: $ fab --hosts=prod1.server,prod2.server,prod3.server --parallel copy_cache This will copy cache to ...
4
votes
4answers
10k views

Have today's desktop PCs surpassed IBM's Deep Blue of 1997?

I was wondering if my PC has surpassed the processing power of Deep Blue which famously won against the human world champion Garry Kasparov. What was the processing speed of Deep blue as compared to ...
0
votes
0answers
13 views

Transparently execute processes on another machine

Situation: At work, we have an in-house tool for data crunching. When a job is triggered, it starts multiple copies of itself in separate processes and communicates with them in order to crunch in ...
0
votes
1answer
20 views

How to run limited amount of instances of program?

For example, I have circa 14000 files I need to process Right now I'm doing it with for %%a in (*.OGG) do revorb.exe "%%a" This runs all revorbs one after one. I have quad processor, so I'd like to ...
11
votes
7answers
7k views

Running commands in parallel with a limit of simultaneous number of commands

Sequential: for i in {1..1000}; do do_something $i; done - too slow Parallel: for i in {1..1000}; do do_something $i& done - too much load How to run commands in parallel, but not more than, for ...
8
votes
5answers
13k views
1
vote
0answers
40 views

why is my master node idle when running gnu parallel remotely?

I'm using gnu-parallel to run a number of python scripts in parallel across a number of computers in my gigabit network. The files being processed are located on a tmpfs (RAM) folder on my master ...
0
votes
2answers
45 views

How to implement Intel MPI benchmarks for mpich2?

Many people have recommended the use of Intel MPI benchmarks for mpich2, however the installation is not very simple. Could anyone provide steps for the installation and sample test case for it? Which ...
1
vote
2answers
73 views

zero a large block device with parallelized dd

We have a sixty terabyte block device which we want to zero using dd. Is it possible to parallelize this operation in order to reduce the amount of time it takes? I've tried running the following ...
2
votes
1answer
116 views

Sort --parallel isn't parallelizing

I'm trying to make unique a set of lines pulled from a file with egrep with sort -u, then count them. About 10% of the lines(all 100 characters long from the alphabet [ATCG]) are duplicated. There are ...
2
votes
2answers
59 views

Will process run slower on one thread rather than one core

So many CPUs now come with 2/4 cores and 4/8 threads. There are also some CPUs that are 4 core & 4 threads. Would that mean that unparalleled process will run slower on the same clock cycle CPU ...
0
votes
1answer
567 views

PBS batch jobs - the qalter command

I've got a giant computation running on a Scientific Linux cluster. At present I have over 600 jobs parked in the queue, waiting for processor time, while a few are running. I'm trying to use the ...
8
votes
4answers
8k views

What's the best way to perform a parallel copy on Unix?

I routinely have to copy the contents of a folder on a network file system to my local computer. There are many files (1000s) on the remote folder that are all relatively small but due to network ...
0
votes
0answers
17 views

Scientific Linux 5.5 hangs at the beginning

We are taking theses days a parallel processing course in college, we are working on scientific linux 5.5 version with MPI, I just imported .ova on the virtual box but unfortunately when am trying to ...
-1
votes
1answer
258 views

Is it possible to use GPU to boost performance of computer automatically? [duplicate]

As we all know GPU is a sleeping beast. And we are also familiar with Intel's Turbo Boost technology which is really great. But, Intel's Turbo Boost technology works only on CPU by overclocking CPU ...
0
votes
1answer
324 views

Clock speed vs. number of cores for parallelized computer simulation [duplicate]

My question is similar to some "speed vs. cores"-questions that have already been asked on this website. However, I am interested in some very specific and technical points. So I hope that my question ...
3
votes
1answer
13k views

Differentiate CUDA Cores(NVIDIA) and Stream processor(ATI/AMD)

I think the question is answered here but I'm still wondering if what is the difference between the units it is referring to. For example, a GTX 570 has 480 CUDA cores, while the ATI equivalent HD ...
1
vote
1answer
76 views

How can I force gnuparallel to carry out a command set sequentially?

Gnu parallel is a powerful tool that I use to run many independant BASH commands as a single set in parallel. I would like to be able to run the same commands SEQUENTIALLY without significant changes ...
0
votes
0answers
34 views

Do threads/processes that create/delete files/directories under a shared parent dir interfere?

I was wondering: If I have a directory /a/b/c/ and under c X threads create subdirectories and files and delete them (they are temporary directories and files as intermediate steps of processing) do ...
15
votes
5answers
24k views

Is a higher core count or higher clock speed more beneficial to a computer's performance? [closed]

With lowering silicon costs and rising consumer needs, manufacturers seem to be pushing one of two things: clock speed and/or core count. With the way things are going, it doesn't seem that clock ...
0
votes
1answer
92 views

How to execute a script 20 at a time with different variables at a time in parallel?

I am trying to run 20 files at a time, using warc.sh or a better way to put it is 20 instances of warc.sh at a time with directory contents as the input. I am also trying to watch the PIDs in case bad ...
27
votes
7answers
22k views

Parallelizing rsync

I just moved and discovered after some trial and error that somewhere between my home and my remote server, there is some throttling going on...but the throttling is not very intelligent. It only ...
0
votes
1answer
252 views

Is Virtual Machine capable of running as normal PC would?

Let's imagine that you have a super-powerful-server at home of size of desktop (just visualisation). 2048 physical cores each 3GHz, 64TB RAM, GTX 780 Ti Black Edition (SLIx4, (four graphics card ...
0
votes
2answers
157 views

fastest RAM configuration [closed]

I am planning to buy a workstation for doing heavy image processing parallel computation (for 3-dimensional electron microscopy) under linux. It will have 64 cores (4 AMD opterons of 16 cores each). ...
1
vote
0answers
139 views

Linux command-line utils with built-in support of multithreading

I would like to be able to compare performance of multicore computers from a point of view of a user who performs typical day-to-day tasks in Linux. My current goal is to collect a list of compute ...
0
votes
1answer
39 views

losetup Find in Parallel / Concurrency

I'm writing a makefile that at some point builds image files. I'm using losetup -f in the file to find a free loop device when building out each image. Since the images are independent of each ...
-1
votes
1answer
230 views

MapReduce and Hadoop: what kinds of problems are they good at solving? [closed]

As I understand it already, it seems that both have similar functions (except that MapReduce is proprietary to Google, while Hadoop is open-source). I was wondering, not "how they work", but rather ...
5
votes
1answer
304 views

Read non-blocking from multiple fifos in parallel

I sometimes sit with a bunch of output fifos from programs that run in parallel. I would like to merge these fifos. The naïve solution is: cat fifo* > output But this requires the first fifo to ...
5
votes
5answers
44k views

Is it possible to combine processing power of 2 computers?

Here are a few questions, I wish you could enlighten me. Is it possible to combine processing power of 2 computers? How do I do it?
0
votes
0answers
28 views

Is parallelism in computer science always directly tied to the number of processor cores?

Specific to the Java programming language, would it be accurate to say that two threads can be concurrent, but can only truly be run in parallel if there are two or more cores?
0
votes
1answer
31 views

How does multi-core architecture work with regard to parellelism?

I think I am correct in saying that scheduling is the/a main principle of the single core CPU, where the CPU is interrupted when switching from processing one set of instructions belonging to one ...
-4
votes
3answers
328 views

Quad-CPU vs Dual-CPU AMD performance difference [closed]

important note: in this text - CPU is a CPU (processor, physical object) and CORE is a CORE (invisible, inside cpu)... Quad-CPU systems are expensive, and there is not so many information about them. ...
0
votes
2answers
341 views

Supercomputer via LAN [closed]

If I decide to parallel two computers via direct LAN connection would LAN (100 Mbits/s) bus bandwidth be enough for CPUs to communicate without performance loss? Or do I have to use some other bus ...
1
vote
1answer
1k views

Does a HyperThreaded Quad Core Processor Actually Have 8 “Cores” [duplicate]

I have an Early 2011 Quad Core i7 Mac. Due to hyperthreading, a lot of programs report 8 "cores". If I were to write a program that executes in parallel, could I kick off 8 instances of this program ...
0
votes
0answers
59 views

How to setup grid/clutster computers?

We have 3 labs and each lab have 30 computers having different OS's on them windows, Ubuntu mostly, so i was thinking if i can connect them using grid computing, so we can use them for our research ...
0
votes
0answers
41 views

How does niceness relate to CPU allocation on Solaris?

The wikipedia article for nice describes the relationship as follows: As a simple example, when two otherwise identical CPU-bound processes are running simultaneously on a single-CPU Linux system, ...
3
votes
4answers
1k views

Optimum number of threads while multitasking

I know similar questions have been asked but I think my case is a little bit diffrent. Let's say I have a computer with 8 cores and infinite memory with a Linux OS. I have a calculation software ...
7
votes
3answers
9k views

netcat as a multithread server

I use netcat to run a simple server like this: while true; do nc -l -p 2468 -e ./my_exe; done This way, anyone is able to connect to my host on port 2468 and talk with "my_exe". Unfortunately, if ...
1
vote
1answer
213 views

Is it normal to overworking of just one CPU when using openMP

When I use OpenMP in FORTRAN to parallelize a program on linux, ' top' command show me just one processor working ~%800 instead of ~%100 x 8 processors. Is it a expected situation ?
1
vote
1answer
243 views

Modern CPU for highly concurrent server applications? [closed]

I am looking for a hardware, which must run about 256 computationally intensive real-time concurrent tasks in 24 hour mode. Each task takes about 40-50 MFLOPs, so all tasks require about 10 GFLOPs. ...
0
votes
1answer
339 views

Kerrighed cluster does not use all nodes when running OpenMP

I've been given access to a recently built cluster that uses Kerrighed. I'm the first user of this cluster, so setup and configuration problems may be a possibility. I'm also totally new to ...
2
votes
1answer
112 views

Parallel-Processing in Integrated GPU

I have an integrated GPU, meaning it works on the same RAM as my CPU. (ATI Radeon HD 4200, integrated on M4A785TD-V EVO) Does this also mean my GPU doesn't compute in parallel? see also: ...
0
votes
1answer
52 views

The exact workflow of PVM (Parallel virtual machine)

These days I am learning PVM technology for concurrent processing. I go through so many blogs, tutorials to understand . Yes, I got some points, but no one gives me clear workflow of how it perform ...
2
votes
1answer
297 views

How can I share memory for parallel computing in Mathematica?

Is it possible to use RAM from other computers to assist with Mathematica Light Grid Server 7 parallel computing? If so, how?
1
vote
1answer
550 views

Is it faster to run one process that spawns N threads or to run N processes?

I need to run ~1500 jobs on my lab's server. The program I am using (blastpgp if anyone cares) is capable of running in parallel mode, the "-a" option sets how many threads it will launch. The server ...
2
votes
3answers
1k views

Running threads on a non hyper-threading Intel processor

Intel Pentium Processor E5700 (2M Cache, 3.00 GHz, 800 MHz FSB) does not have hyper-threading but it has has 2 cores so I am assuming it has 2 threads Now if I write a heavy number crunching program ...
1
vote
2answers
818 views

gunzip multiple files and concatenate in parallel?

Is it possible to gunzip multiple files and concatenate them into one big file, but do it in parallel given a multicore machine? For example, right now I do: gunzip -c file1.gz > final gunzip -c ...
3
votes
2answers
524 views

How can I run several parallel shell jobs and monitor all of them at once?

I have a large number of large files (hundreds of files, hundreds of MB each) which I need to pipe through a number of programs to filter and convert them. I'm taking advantage of multiple CPU cores ...