Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [parallel-processing]

Running multiple commands (or even computers) in parallel for more efficiency.

1
vote
0answers
10 views

MATLAB GPU-enabled function? [closed]

Is there a way for me to find out whether the functions I'm using in a MATLAB script are GPU-enabled? There does not seem to be a comprehensive list (though see this link for a list of image ...
0
votes
0answers
16 views

Cuda does not calculate big array [on hold]

I am new to CUDA, I have been working on matrix. In this program I want to add 2 matrices. It works very well in small matrix, but when I want to increase the number of row and col (for example more ...
3
votes
2answers
62 views

How does a computer sleep/wait without using all of its processing power?

Something I've never quite understood is how a computer can operate without using all of its processing power. (Likely this confusion stems from a dreadful OS class in college.) As far as I know, a ...
-1
votes
1answer
30 views

In Linux, is concept of Process similar to Container (in microservices?) [closed]

Process is executing an instance of an application. Every process got its State, ID, address space etc, and isolated to each other. It looks similar to Container that runs a specific service? In some ...
0
votes
0answers
206 views

How Can I run VLC Command lines within a Parallel Command?

MACOSX user... I have written a command to run VLC with specific parameters, as below: /Applications/VLC.app/Contents/MacOS/VLC /Users/Worm/Desktop/Work/WORKwork/2018/Resolutions/resolutionsmulti....
1
vote
1answer
396 views

What's the difference between a multiprocessor and a multiprocessing system?

I'm studying IPC for a simple test I have tomorrow. The book my classroom uses is not the best: it makes a lot of typos and its exercises have been proven wrong or terribly ambiguous more than once. ...
0
votes
1answer
361 views

Use bittorrent to download from China?

I have a server in China and a client in Europe and need to download large (50-100 GB) files as quickly as possible. The connections to China are wonky at best and very slow. A way to get around this ...
1
vote
0answers
214 views

Threads vs Processors for Excel [duplicate]

I'm writing a VBA module that uses a lot of computational power. To speed things up, I'm using all available processors on my machine. Excel gives me an option though to use a different number of ...
4
votes
1answer
148 views

Kill first command as soon as second exits

I've been looking for best practice for this scenario for a long time. If I have process2 that depends on process1 running beforehand. I set up an alias like: alias both='process1 & process2' ...
0
votes
1answer
403 views

How do you control thread affinity across multiple processes on Linux?

Question How do I run two benchmarks such that: They are both executing concurrently Both benchmarks have half their threads on each socket. No two threads share the same core. Background I have ...
2
votes
2answers
163 views

Create many tar files from a directory with 500000 files

I have a directory containing around 500k files, and want to slice them into t tar files. Put formally, let's call the files file_0, ..., file_{N-1}, where N around 500k. I want to create t tar files ...
0
votes
1answer
379 views

scp a file in parallel FROM multiple servers

Is there a script or something that can use scp to copy a single file from multiple servers/sources, each connection with its own chunk/chunks? I want something like how a torrent acts. I do not have ...
0
votes
2answers
701 views

How to gather multiple remote servers' CPU utilization info?

I need to test program in multiple remote servers, and at the same time need to gather the CPU utilizations information during the process. Here use one server (10.18.6.0) as the master server ...
-1
votes
1answer
39 views

How to deal with small time difference in multiple remote servers?

I use several remote servers to test a parallel system, the process will take about 0.4s each time. So when I run the program in several servers, the running time range from less than 0 to 1s. The ...
0
votes
1answer
649 views

How to use pssh to run multiple servers with different commands?

I need to test the distributed system with several remote servers, so I use pssh in master server to control others. Here need to run multiple server with different commands, for example In server ...
1
vote
0answers
131 views

remove slurm sacct command double entries: “extern”

Jobs currently running show two entries, one of them has an .extern suffix. Completed (or failed) jobs also have a third entry: .batch. Is there a way to remove (or not show these) from the sacct ...
0
votes
1answer
45 views

How to parallelize this image processing script?

I have this script that converts .CR2 images from my Canon camers to Jpegs. The images are named IMG_{NNNN}.CR2 where {NNNN} are the image sequence numbers from the camera. I'd like to parallelize it ...
2
votes
1answer
295 views

Parallel Programming in AMD based laptop

I've a Lenovo G50 series laptop with AMD A8-6410 @2Ghz, 8 GB RAM with a R5 230M and Radeon HD 8550M cards inside. In Windows 10 is it possible to run a CLI based program ( ffmpeg for video ...
5
votes
2answers
5k views

How to use Ctrl+C to kill all background processes started in a Bash script?

ArchLinux (Manjaro). I'm running one bash file. It runs 2 processes (commands), using &. But when I press Ctrl+C to stop the program - one process dies, and the other continues to work. How do I ...
1
vote
0answers
19k views

SSH error: Temporary failure in name resolution on mpi run

I have 6 computers (4 cores each and identical in hardwares) with static IPs and need to run NAMD computational chemistry software parallel using OpenMPI. So I installed SSH and OpenMPI (on mint 18)...
1
vote
0answers
35 views

Hardware Requirements Gap Analysis for Windows

I have an algorithm that needs to be repeated ('replication') a minimum of 30 times: each replication could be run in it's own core. The Ninja development stations feature the "Knight's Landing" (...
1
vote
1answer
450 views

How do I run multiple .bat files that are in subfolders

I'm new to supercomputers, I recently built an Amazon EC2 instance with 128 CPUs and Microsoft server 2012. I have one folder that has 100 subfolders, in each subfolder there is a .bat file. How do I ...
0
votes
1answer
62 views

Bash one-liner to open several processes and execute a python function in each - poor man's parallel processing

I have a python file like my_functions.py and within that file is a function def func(a,b): result = does_some_stuff(a,b) return result Is there a bash one-liner that would open N ...
0
votes
0answers
938 views

Using Workflows In Powershell

I have been using just a standard foreach loop, which executes as it should. However, I have an 8 core processor and 16GB of RAM and thought I would utilize such with a Parallel.foreach loop. Well, ...
64
votes
4answers
14k views

How can my computer use 300% PC processing power?

I'm playing around with a script and well I got carried away (it's still running right now and yet my computer has not yet died). I'm running 17 iterations of the script (with the intention of ...
1
vote
1answer
246 views

The main difference between Superscalar and Parallelism

In superscalar processing, more than one instruction is executed in parallel at each core, so how would that differ from parallel processing then? Thank you.
0
votes
2answers
1k views

bash for loop in parallel with n cores

I have a bash loop that looks like: for i in $(seq 0 $max); do my_command $i done and I would like to run this in parallel on n cores. I know that I could do while [[ "$j" -le "$max" ]]; do ...
0
votes
1answer
28 views

How to run limited amount of instances of program?

For example, I have circa 14000 files I need to process Right now I'm doing it with for %%a in (*.OGG) do revorb.exe "%%a" This runs all revorbs one after one. I have quad processor, so I'd like to ...
1
vote
0answers
82 views

why is my master node idle when running gnu parallel remotely?

I'm using gnu-parallel to run a number of python scripts in parallel across a number of computers in my gigabit network. The files being processed are located on a tmpfs (RAM) folder on my master ...
1
vote
2answers
985 views

zero a large block device with parallelized dd

We have a sixty terabyte block device which we want to zero using dd. Is it possible to parallelize this operation in order to reduce the amount of time it takes? I've tried running the following ...
8
votes
1answer
3k views

Sort --parallel isn't parallelizing

I'm trying to make unique a set of lines pulled from a file with egrep with sort -u, then count them. About 10% of the lines(all 100 characters long from the alphabet [ATCG]) are duplicated. There are ...
2
votes
2answers
285 views

Will process run slower on one thread rather than one core

So many CPUs now come with 2/4 cores and 4/8 threads. There are also some CPUs that are 4 core & 4 threads. Would that mean that unparalleled process will run slower on the same clock cycle CPU ...
0
votes
2answers
265 views

How to implement Intel MPI benchmarks for mpich2?

Many people have recommended the use of Intel MPI benchmarks for mpich2, however the installation is not very simple. Could anyone provide steps for the installation and sample test case for it? Which ...
-1
votes
1answer
4k views

Is it possible to use GPU to boost performance of computer automatically? [duplicate]

As we all know GPU is a sleeping beast. And we are also familiar with Intel's Turbo Boost technology which is really great. But, Intel's Turbo Boost technology works only on CPU by overclocking CPU ...
0
votes
1answer
1k views

Clock speed vs. number of cores for parallelized computer simulation [duplicate]

My question is similar to some "speed vs. cores"-questions that have already been asked on this website. However, I am interested in some very specific and technical points. So I hope that my question ...
2
votes
1answer
325 views

How can I force gnuparallel to carry out a command set sequentially?

Gnu parallel is a powerful tool that I use to run many independant BASH commands as a single set in parallel. I would like to be able to run the same commands SEQUENTIALLY without significant changes ...
0
votes
0answers
108 views

Do threads/processes that create/delete files/directories under a shared parent dir interfere?

I was wondering: If I have a directory /a/b/c/ and under c X threads create subdirectories and files and delete them (they are temporary directories and files as intermediate steps of processing) do ...
21
votes
0answers
3k views

Jenkins realtime console output for parallel threaded fabric command [closed]

We have a Python fabric command that runs in parallel across several hosts, something like this: $ fab --hosts=prod1.server,prod2.server,prod3.server --parallel copy_cache This will copy cache to ...
0
votes
1answer
1k views

Is Virtual Machine capable of running as normal PC would?

Let's imagine that you have a super-powerful-server at home of size of desktop (just visualisation). 2048 physical cores each 3GHz, 64TB RAM, GTX 780 Ti Black Edition (SLIx4, (four graphics card ...
0
votes
2answers
351 views

fastest RAM configuration [closed]

I am planning to buy a workstation for doing heavy image processing parallel computation (for 3-dimensional electron microscopy) under linux. It will have 64 cores (4 AMD opterons of 16 cores each). ...
2
votes
0answers
250 views

Linux command-line utils with built-in support of multithreading

I would like to be able to compare performance of multicore computers from a point of view of a user who performs typical day-to-day tasks in Linux. My current goal is to collect a list of compute ...
0
votes
0answers
43 views

Is parallelism in computer science always directly tied to the number of processor cores?

Specific to the Java programming language, would it be accurate to say that two threads can be concurrent, but can only truly be run in parallel if there are two or more cores?
0
votes
1answer
56 views

How does multi-core architecture work with regard to parellelism?

I think I am correct in saying that scheduling is the/a main principle of the single core CPU, where the CPU is interrupted when switching from processing one set of instructions belonging to one ...
-4
votes
3answers
550 views

Quad-CPU vs Dual-CPU AMD performance difference [closed]

important note: in this text - CPU is a CPU (processor, physical object) and CORE is a CORE (invisible, inside cpu)... Quad-CPU systems are expensive, and there is not so many information about them. ...
0
votes
2answers
840 views

Supercomputer via LAN [closed]

If I decide to parallel two computers via direct LAN connection would LAN (100 Mbits/s) bus bandwidth be enough for CPUs to communicate without performance loss? Or do I have to use some other bus (...
2
votes
1answer
3k views

Does a HyperThreaded Quad Core Processor Actually Have 8 “Cores” [duplicate]

I have an Early 2011 Quad Core i7 Mac. Due to hyperthreading, a lot of programs report 8 "cores". If I were to write a program that executes in parallel, could I kick off 8 instances of this program ...
0
votes
0answers
64 views

How does niceness relate to CPU allocation on Solaris?

The wikipedia article for nice describes the relationship as follows: As a simple example, when two otherwise identical CPU-bound processes are running simultaneously on a single-CPU Linux system, ...
0
votes
1answer
111 views

How to execute a script 20 at a time with different variables at a time in parallel?

I am trying to run 20 files at a time, using warc.sh or a better way to put it is 20 instances of warc.sh at a time with directory contents as the input. I am also trying to watch the PIDs in case bad ...
4
votes
4answers
2k views

Optimum number of threads while multitasking

I know similar questions have been asked but I think my case is a little bit diffrent. Let's say I have a computer with 8 cores and infinite memory with a Linux OS. I have a calculation software ...
0
votes
0answers
84 views

How to setup grid/clutster computers?

We have 3 labs and each lab have 30 computers having different OS's on them windows, Ubuntu mostly, so i was thinking if i can connect them using grid computing, so we can use them for our research ...