Questions tagged [parallel-processing]

Running multiple commands (or even computers) in parallel for more efficiency.

Filter by
Sorted by
Tagged with
0
votes
1answer
35 views

PowerShell: Script throwing up in parallel

When run in -Parallel, the following script does not pass the $basePath variable declared before the start of the parallel block. Instead, it is a null value. This is not a problem when the script is ...
0
votes
2answers
36 views

Why is folder deletion not parallelized?

When I Shift-delete a big folder, I see the following CPU usage: Not all boxes are used. Is this inefficient? This seems easily parallelizable to me, through divide-and-conquer. I tried manually ...
1
vote
1answer
49 views

GNU Parallel usage - how get currently passed string to parallel?

I am using GNU parallel and want to understand - how can I get the individual string passed to each parallel command? As an example, GNU Parallel documentation shows how to move files from the current ...
2
votes
2answers
101 views

Execute two or more commands simultaneously in shell script

It is often very helpful to run commands in parallel in a shell script, but I cannot find ways to do it. Is this possible? If so, how would I achieve that? I'm mainly interested in doing that for fish....
1
vote
1answer
15 views

bad parallel-processing gain on some linux machines, good on others

I'm measuring the run time for a complex signal-processing algorithm written in C on several different computers, all running linux. The code is partially parallelized by some of my own methods using ...
0
votes
0answers
47 views

Multi processor command queue

I'm working in Centos8. I'm generating timelapse videos from incoming photographs over hundreds of cameras spread throughout the country; the images from each site come in regularly (but randomly, ...
0
votes
0answers
16 views

Number of threads and number of processing elements in heterogeneous programming

For example, AMD A10 5800K APU has only 384 processing elements. But in kernel, we assign 2048 local workgroup size (i.e.number of working threads or work items). Each thread runs on processing ...
0
votes
2answers
567 views

How to download a single large file over sftp or http etc using multiple simultaneous connections

Is there a command line tool which supports both multiple simultaneous connections and multiple network protocols (ftp, http etc)? Here is a practical example: I have a large backup file on an ssh ...
0
votes
1answer
250 views

Are the Geekbench multicore scores done using embarrassingly parallel processes?

For each processor, the benchmarks at geekbench.com present both a single-core and multi-core score. For example, this 64-core processor has a single-core score of 1220 and a multi-core score of 23688....
0
votes
0answers
338 views

Guidance on Powershell Workflow foreach -parallel (){} functions

I just discovered Powershell workflows and I am experimenting but having some difficulties. I do not know if workflows even have the capabilities to do the following but if they do, I would appreciate ...
61
votes
12answers
39k views

Can I run my CPU at 100% usage for a long time?

I have a Dell XPS 15 9570, 32 GB ram, 2.2 ghz and 6 cores. When i use an application (The parsim command on MATLAB) (for more details) then my fan starts running fast , during this my PC doesn't seem ...
3
votes
3answers
375 views

Sequential parallel with wait

I want to run a multi-parallel task in one bash file like the example code below, for i in 1 2 3 4 5 6 7 8; do setsid python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 & done wait # ...
1
vote
1answer
218 views

How to split and join parallel pipes?

I'd like to pipe the output of 1 command into multiple pipes and pipe outputs from multiple commands into 1 single command. An example of split: # Single output to multiple pipes echo "This is a ...
0
votes
2answers
65 views

What is the optimal number of CPU-heavy data mining jobs to run in parallel? [closed]

I have a CPU heavy data mining job that's parallelizable. Each job run takes ~10 minutes. The job isn't heavy in IO, it's mostly heavy in CPU compute. I have a 32-core rig with 2 threads per core and ...
1
vote
0answers
323 views

Can I run chkdsk for different HDDs at the same time (external via USB)?

I want to run multiple chkdsk commands (for different HDDs) at the same time.  All of them are external via USB. I searched for this on Google, but there were few results that were really relevant to ...
1
vote
0answers
19 views

How can a processor execute more instructions per period than there are clock cycles? [duplicate]

This Wikipedia page lists several cases where the instructions per clock cycle per core is greater than 1. How can there be more instructions executed per core per period than there are clock cycles ...
0
votes
0answers
47 views

Eclipse PTP Parallel Debugging

I have a problem with Eclipse PTP 201906. I would like to perform debug in parallel with 4 cores using Eclipse. I adjust the debug configuration with Local connection Arguments: -np 4 and begin ...
1
vote
0answers
31 views

Ansys parallele job on Slurm Cluster stuck without error or exit message

I am working on a Slurm Cluster, executing Ansys (V18.2) jobs in a parallelized way. Large jobs (meaning large solver files) often stuck with out error message or exit message, the jobs keep running ...
1
vote
0answers
121 views

MATLAB GPU-enabled function? [closed]

Is there a way for me to find out whether the functions I'm using in a MATLAB script are GPU-enabled? There does not seem to be a comprehensive list (though see this link for a list of image ...
3
votes
3answers
790 views

How does a computer sleep/wait without using all of its processing power?

Something I've never quite understood is how a computer can operate without using all of its processing power. (Likely this confusion stems from a dreadful OS class in college.) As far as I know, a ...
-1
votes
1answer
33 views

In Linux, is concept of Process similar to Container (in microservices?) [closed]

Process is executing an instance of an application. Every process got its State, ID, address space etc, and isolated to each other. It looks similar to Container that runs a specific service? In some ...
1
vote
1answer
1k views

What's the difference between a multiprocessor and a multiprocessing system?

I'm studying IPC for a simple test I have tomorrow. The book my classroom uses is not the best: it makes a lot of typos and its exercises have been proven wrong or terribly ambiguous more than once. ...
1
vote
1answer
837 views

Use bittorrent to download from China?

I have a server in China and a client in Europe and need to download large (50-100 GB) files as quickly as possible. The connections to China are wonky at best and very slow. A way to get around this ...
1
vote
0answers
489 views

Threads vs Processors for Excel [duplicate]

I'm writing a VBA module that uses a lot of computational power. To speed things up, I'm using all available processors on my machine. Excel gives me an option though to use a different number of ...
3
votes
1answer
376 views

Kill first command as soon as second exits

I've been looking for best practice for this scenario for a long time. If I have process2 that depends on process1 running beforehand. I set up an alias like: alias both='process1 & process2' ...
0
votes
1answer
559 views

How do you control thread affinity across multiple processes on Linux?

Question How do I run two benchmarks such that: They are both executing concurrently Both benchmarks have half their threads on each socket. No two threads share the same core. Background I have two ...
2
votes
2answers
278 views

Create many tar files from a directory with 500000 files

I have a directory containing around 500k files, and want to slice them into t tar files. Put formally, let's call the files file_0, ..., file_{N-1}, where N around 500k. I want to create t tar files ...
0
votes
1answer
1k views

scp a file in parallel FROM multiple servers

Is there a script or something that can use scp to copy a single file from multiple servers/sources, each connection with its own chunk/chunks? I want something like how a torrent acts. I do not have ...
0
votes
2answers
1k views

How to gather multiple remote servers' CPU utilization info?

I need to test program in multiple remote servers, and at the same time need to gather the CPU utilizations information during the process. Here use one server (10.18.6.0) as the master server ...
-1
votes
1answer
41 views

How to deal with small time difference in multiple remote servers?

I use several remote servers to test a parallel system, the process will take about 0.4s each time. So when I run the program in several servers, the running time range from less than 0 to 1s. The ...
0
votes
1answer
848 views

How to use pssh to run multiple servers with different commands?

I need to test the distributed system with several remote servers, so I use pssh in master server to control others. Here need to run multiple server with different commands, for example In server ...
3
votes
1answer
519 views

remove slurm sacct command double entries: “extern”

Jobs currently running show two entries, one of them has an .extern suffix. Completed (or failed) jobs also have a third entry: .batch. Is there a way to remove (or not show these) from the sacct ...
0
votes
1answer
60 views

How to parallelize this image processing script?

I have this script that converts .CR2 images from my Canon camers to Jpegs. The images are named IMG_{NNNN}.CR2 where {NNNN} are the image sequence numbers from the camera. I'd like to parallelize it ...
2
votes
1answer
468 views

Parallel Programming in AMD based laptop

I've a Lenovo G50 series laptop with AMD A8-6410 @2Ghz, 8 GB RAM with a R5 230M and Radeon HD 8550M cards inside. In Windows 10 is it possible to run a CLI based program ( ffmpeg for video ...
16
votes
2answers
14k views

How to use Ctrl+C to kill all background processes started in a Bash script?

ArchLinux (Manjaro). I'm running one bash file. It runs 2 processes (commands), using &. But when I press Ctrl+C to stop the program - one process dies, and the other continues to work. How do I ...
2
votes
0answers
23k views

SSH error: Temporary failure in name resolution on mpi run

I have 6 computers (4 cores each and identical in hardwares) with static IPs and need to run NAMD computational chemistry software parallel using OpenMPI. So I installed SSH and OpenMPI (on mint 18)...
1
vote
0answers
46 views

Hardware Requirements Gap Analysis for Windows

I have an algorithm that needs to be repeated ('replication') a minimum of 30 times: each replication could be run in it's own core. The Ninja development stations feature the "Knight's Landing" (...
1
vote
1answer
2k views

How do I run multiple .bat files that are in subfolders

I'm new to supercomputers, I recently built an Amazon EC2 instance with 128 CPUs and Microsoft server 2012. I have one folder that has 100 subfolders, in each subfolder there is a .bat file. How do I ...
0
votes
1answer
87 views

Bash one-liner to open several processes and execute a python function in each - poor man's parallel processing

I have a python file like my_functions.py and within that file is a function def func(a,b): result = does_some_stuff(a,b) return result Is there a bash one-liner that would open N ...
0
votes
0answers
1k views

Using Workflows In Powershell

I have been using just a standard foreach loop, which executes as it should. However, I have an 8 core processor and 16GB of RAM and thought I would utilize such with a Parallel.foreach loop. Well, ...
64
votes
4answers
15k views

How can my computer use 300% PC processing power?

I'm playing around with a script and well I got carried away (it's still running right now and yet my computer has not yet died). I'm running 17 iterations of the script (with the intention of ...
1
vote
1answer
636 views

The main difference between Superscalar and Parallelism

In superscalar processing, more than one instruction is executed in parallel at each core, so how would that differ from parallel processing then? Thank you.
0
votes
2answers
3k views

bash for loop in parallel with n cores

I have a bash loop that looks like: for i in $(seq 0 $max); do my_command $i done and I would like to run this in parallel on n cores. I know that I could do while [[ "$j" -le "$max" ]]; do ...
0
votes
1answer
36 views

How to run limited amount of instances of program?

For example, I have circa 14000 files I need to process Right now I'm doing it with for %%a in (*.OGG) do revorb.exe "%%a" This runs all revorbs one after one. I have quad processor, so I'd like to ...
1
vote
0answers
87 views

why is my master node idle when running gnu parallel remotely?

I'm using gnu-parallel to run a number of python scripts in parallel across a number of computers in my gigabit network. The files being processed are located on a tmpfs (RAM) folder on my master ...
1
vote
3answers
2k views

zero a large block device with parallelized dd

We have a sixty terabyte block device which we want to zero using dd. Is it possible to parallelize this operation in order to reduce the amount of time it takes? I've tried running the following ...
14
votes
2answers
7k views

Sort --parallel isn't parallelizing

I'm trying to make unique a set of lines pulled from a file with egrep with sort -u, then count them. About 10% of the lines(all 100 characters long from the alphabet [ATCG]) are duplicated. There are ...
2
votes
2answers
337 views

Will process run slower on one thread rather than one core

So many CPUs now come with 2/4 cores and 4/8 threads. There are also some CPUs that are 4 core & 4 threads. Would that mean that unparalleled process will run slower on the same clock cycle CPU ...
0
votes
2answers
318 views

How to implement Intel MPI benchmarks for mpich2?

Many people have recommended the use of Intel MPI benchmarks for mpich2, however the installation is not very simple. Could anyone provide steps for the installation and sample test case for it? Which ...
0
votes
1answer
7k views

Is it possible to use GPU to boost performance of computer automatically? [duplicate]

As we all know GPU is a sleeping beast. And we are also familiar with Intel's Turbo Boost technology which is really great. But, Intel's Turbo Boost technology works only on CPU by overclocking CPU ...