2022 Developer Survey is open! Take survey.

Questions tagged [parallel-processing]

Running multiple commands (or even computers) in parallel for more efficiency.

Filter by
Sorted by
Tagged with
1 vote
0 answers
25 views

Connecting two desktops by an infiniband interconnect

I am a scientist working in a high performance computing project(the computational programming aspect) and I have no idea of networking or networking hardware required to answer my question. A ...
user avatar
  • 123
1 vote
1 answer
74 views

How can I stop Ubuntu 20.04 from restricting my cpu space?

After installation of Ubuntu 20.04 everything was working well. However, after installing and using some applications (Matlab and Matlab/Dynare) Ubuntu seems to limit my cpu usage. I use openMP and a ...
user avatar
  • 21
0 votes
1 answer
46 views

Can any program be run on multiple cores? Will the CPU decide that? Or can only certain programs run on multiple cores?

So I'm a little confused about the multicore CPUs - the impression I was initially under and what I thought I heard before was that some programs can't be run across multiple cores - those in which ...
user avatar
0 votes
1 answer
175 views

Parallel processing and order of execution with recursive loop

If i run a script that processes a path recursively using parallel processing to execute a command on the whole tree if items, am I guaranteed that the commands I use will be executed on all items in ...
user avatar
  • 317
0 votes
0 answers
327 views

How to use numactl --membind correctly?

I'm trying to run a script test.py for 1 instance bound to 4 cpus of numa-node-1. numactl --physcpubind=24-27 --membind=1 python -u test.py The lscpu shows CPUs 24-27 bound to numa-node-1. I expected ...
user avatar
  • 111
0 votes
1 answer
60 views

PowerShell: Script throwing up in parallel

When run in -Parallel, the following script does not pass the $basePath variable declared before the start of the parallel block. Instead, it is a null value. This is not a problem when the script is ...
user avatar
  • 481
0 votes
2 answers
37 views

Why is folder deletion not parallelized?

When I Shift-delete a big folder, I see the following CPU usage: Not all boxes are used. Is this inefficient? This seems easily parallelizable to me, through divide-and-conquer. I tried manually ...
user avatar
  • 101
1 vote
1 answer
66 views

GNU Parallel usage - how get currently passed string to parallel?

I am using GNU parallel and want to understand - how can I get the individual string passed to each parallel command? As an example, GNU Parallel documentation shows how to move files from the current ...
user avatar
2 votes
2 answers
498 views

Execute two or more commands simultaneously in shell script

It is often very helpful to run commands in parallel in a shell script, but I cannot find ways to do it. Is this possible? If so, how would I achieve that? I'm mainly interested in doing that for fish....
user avatar
  • 165
1 vote
1 answer
19 views

bad parallel-processing gain on some linux machines, good on others

I'm measuring the run time for a complex signal-processing algorithm written in C on several different computers, all running linux. The code is partially parallelized by some of my own methods using ...
user avatar
0 votes
2 answers
2k views

How to download a single large file over sftp or http etc using multiple simultaneous connections [closed]

Is there a command line tool which supports both multiple simultaneous connections and multiple network protocols (ftp, http etc)? Here is a practical example: I have a large backup file on an ssh ...
user avatar
0 votes
1 answer
391 views

Are the Geekbench multicore scores done using embarrassingly parallel processes?

For each processor, the benchmarks at geekbench.com present both a single-core and multi-core score. For example, this 64-core processor has a single-core score of 1220 and a multi-core score of 23688....
user avatar
0 votes
0 answers
482 views

Guidance on Powershell Workflow foreach -parallel (){} functions

I just discovered Powershell workflows and I am experimenting but having some difficulties. I do not know if workflows even have the capabilities to do the following but if they do, I would appreciate ...
user avatar
  • 481
61 votes
12 answers
47k views

Can I run my CPU at 100% usage for a long time?

I have a Dell XPS 15 9570, 32 GB ram, 2.2 ghz and 6 cores. When i use an application (The parsim command on MATLAB) (for more details) then my fan starts running fast , during this my PC doesn't seem ...
user avatar
  • 737
3 votes
3 answers
432 views

Sequential parallel with wait

I want to run a multi-parallel task in one bash file like the example code below, for i in 1 2 3 4 5 6 7 8; do setsid python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 & done wait # ...
user avatar
1 vote
1 answer
356 views

How to split and join parallel pipes?

I'd like to pipe the output of 1 command into multiple pipes and pipe outputs from multiple commands into 1 single command. An example of split: # Single output to multiple pipes echo "This is a ...
user avatar
0 votes
2 answers
101 views

What is the optimal number of CPU-heavy data mining jobs to run in parallel? [closed]

I have a CPU heavy data mining job that's parallelizable. Each job run takes ~10 minutes. The job isn't heavy in IO, it's mostly heavy in CPU compute. I have a 32-core rig with 2 threads per core and ...
user avatar
1 vote
0 answers
907 views

Can I run chkdsk for different HDDs at the same time (external via USB)?

I want to run multiple chkdsk commands (for different HDDs) at the same time.  All of them are external via USB. I searched for this on Google, but there were few results that were really relevant to ...
user avatar
  • 11
1 vote
0 answers
21 views

How can a processor execute more instructions per period than there are clock cycles? [duplicate]

This Wikipedia page lists several cases where the instructions per clock cycle per core is greater than 1. How can there be more instructions executed per core per period than there are clock cycles ...
user avatar
  • 1,164
0 votes
0 answers
58 views

Eclipse PTP Parallel Debugging

I have a problem with Eclipse PTP 201906. I would like to perform debug in parallel with 4 cores using Eclipse. I adjust the debug configuration with Local connection Arguments: -np 4 and begin ...
user avatar
  • 1
1 vote
0 answers
49 views

Ansys parallele job on Slurm Cluster stuck without error or exit message

I am working on a Slurm Cluster, executing Ansys (V18.2) jobs in a parallelized way. Large jobs (meaning large solver files) often stuck with out error message or exit message, the jobs keep running ...
user avatar
  • 11
1 vote
0 answers
126 views

MATLAB GPU-enabled function? [closed]

Is there a way for me to find out whether the functions I'm using in a MATLAB script are GPU-enabled? There does not seem to be a comprehensive list (though see this link for a list of image ...
user avatar
3 votes
3 answers
1k views

How does a computer sleep/wait without using all of its processing power?

Something I've never quite understood is how a computer can operate without using all of its processing power. (Likely this confusion stems from a dreadful OS class in college.) As far as I know, a ...
user avatar
-1 votes
1 answer
37 views

In Linux, is concept of Process similar to Container (in microservices?) [closed]

Process is executing an instance of an application. Every process got its State, ID, address space etc, and isolated to each other. It looks similar to Container that runs a specific service? In some ...
user avatar
1 vote
1 answer
2k views

What's the difference between a multiprocessor and a multiprocessing system?

I'm studying IPC for a simple test I have tomorrow. The book my classroom uses is not the best: it makes a lot of typos and its exercises have been proven wrong or terribly ambiguous more than once. ...
user avatar
  • 115
1 vote
1 answer
917 views

Use bittorrent to download from China?

I have a server in China and a client in Europe and need to download large (50-100 GB) files as quickly as possible. The connections to China are wonky at best and very slow. A way to get around this ...
user avatar
1 vote
0 answers
524 views

Threads vs Processors for Excel [duplicate]

I'm writing a VBA module that uses a lot of computational power. To speed things up, I'm using all available processors on my machine. Excel gives me an option though to use a different number of ...
user avatar
  • 13k
3 votes
1 answer
389 views

Kill first command as soon as second exits

I've been looking for best practice for this scenario for a long time. If I have process2 that depends on process1 running beforehand. I set up an alias like: alias both='process1 & process2' ...
user avatar
  • 133
0 votes
1 answer
574 views

How do you control thread affinity across multiple processes on Linux?

Question How do I run two benchmarks such that: They are both executing concurrently Both benchmarks have half their threads on each socket. No two threads share the same core. Background I have two ...
user avatar
2 votes
2 answers
339 views

Create many tar files from a directory with 500000 files

I have a directory containing around 500k files, and want to slice them into t tar files. Put formally, let's call the files file_0, ..., file_{N-1}, where N around 500k. I want to create t tar files ...
user avatar
  • 155
0 votes
1 answer
1k views

scp a file in parallel FROM multiple servers

Is there a script or something that can use scp to copy a single file from multiple servers/sources, each connection with its own chunk/chunks? I want something like how a torrent acts. I do not have ...
user avatar
0 votes
2 answers
1k views

How to gather multiple remote servers' CPU utilization info?

I need to test program in multiple remote servers, and at the same time need to gather the CPU utilizations information during the process. Here use one server (10.18.6.0) as the master server ...
user avatar
  • 87
-1 votes
1 answer
42 views

How to deal with small time difference in multiple remote servers?

I use several remote servers to test a parallel system, the process will take about 0.4s each time. So when I run the program in several servers, the running time range from less than 0 to 1s. The ...
user avatar
  • 87
0 votes
1 answer
878 views

How to use pssh to run multiple servers with different commands?

I need to test the distributed system with several remote servers, so I use pssh in master server to control others. Here need to run multiple server with different commands, for example In server ...
user avatar
  • 87
4 votes
1 answer
758 views

remove slurm sacct command double entries: "extern"

Jobs currently running show two entries, one of them has an .extern suffix. Completed (or failed) jobs also have a third entry: .batch. Is there a way to remove (or not show these) from the sacct ...
user avatar
0 votes
1 answer
61 views

How to parallelize this image processing script?

I have this script that converts .CR2 images from my Canon camers to Jpegs. The images are named IMG_{NNNN}.CR2 where {NNNN} are the image sequence numbers from the camera. I'd like to parallelize it ...
user avatar
2 votes
1 answer
481 views

Parallel Programming in AMD based laptop

I've a Lenovo G50 series laptop with AMD A8-6410 @2Ghz, 8 GB RAM with a R5 230M and Radeon HD 8550M cards inside. In Windows 10 is it possible to run a CLI based program ( ffmpeg for video ...
user avatar
18 votes
2 answers
17k views

How to use Ctrl+C to kill all background processes started in a Bash script?

ArchLinux (Manjaro). I'm running one bash file. It runs 2 processes (commands), using &. But when I press Ctrl+C to stop the program - one process dies, and the other continues to work. How do I ...
user avatar
  • 183
2 votes
0 answers
24k views

SSH error: Temporary failure in name resolution on mpi run

I have 6 computers (4 cores each and identical in hardwares) with static IPs and need to run NAMD computational chemistry software parallel using OpenMPI. So I installed SSH and OpenMPI (on mint 18)...
user avatar
1 vote
0 answers
48 views

Hardware Requirements Gap Analysis for Windows

I have an algorithm that needs to be repeated ('replication') a minimum of 30 times: each replication could be run in it's own core. The Ninja development stations feature the "Knight's Landing" (...
user avatar
  • 1,075
1 vote
1 answer
2k views

How do I run multiple .bat files that are in subfolders

I'm new to supercomputers, I recently built an Amazon EC2 instance with 128 CPUs and Microsoft server 2012. I have one folder that has 100 subfolders, in each subfolder there is a .bat file. How do I ...
user avatar
0 votes
1 answer
93 views

Bash one-liner to open several processes and execute a python function in each - poor man's parallel processing

I have a python file like my_functions.py and within that file is a function def func(a,b): result = does_some_stuff(a,b) return result Is there a bash one-liner that would open N ...
user avatar
0 votes
0 answers
1k views

Using Workflows In Powershell

I have been using just a standard foreach loop, which executes as it should. However, I have an 8 core processor and 16GB of RAM and thought I would utilize such with a Parallel.foreach loop. Well, ...
user avatar
  • 1,935
64 votes
4 answers
15k views

How can my computer use 300% PC processing power?

I'm playing around with a script and well I got carried away (it's still running right now and yet my computer has not yet died). I'm running 17 iterations of the script (with the intention of ...
user avatar
1 vote
1 answer
739 views

The main difference between Superscalar and Parallelism

In superscalar processing, more than one instruction is executed in parallel at each core, so how would that differ from parallel processing then? Thank you.
user avatar
  • 173
0 votes
2 answers
4k views

bash for loop in parallel with n cores

I have a bash loop that looks like: for i in $(seq 0 $max); do my_command $i done and I would like to run this in parallel on n cores. I know that I could do while [[ "$j" -le "$max" ]]; do ...
user avatar
  • 1,344
0 votes
1 answer
37 views

How to run limited amount of instances of program?

For example, I have circa 14000 files I need to process Right now I'm doing it with for %%a in (*.OGG) do revorb.exe "%%a" This runs all revorbs one after one. I have quad processor, so I'd like to ...
user avatar
  • 13
1 vote
0 answers
92 views

why is my master node idle when running gnu parallel remotely?

I'm using gnu-parallel to run a number of python scripts in parallel across a number of computers in my gigabit network. The files being processed are located on a tmpfs (RAM) folder on my master ...
user avatar
  • 375
1 vote
3 answers
2k views

zero a large block device with parallelized dd

We have a sixty terabyte block device which we want to zero using dd. Is it possible to parallelize this operation in order to reduce the amount of time it takes? I've tried running the following ...
user avatar
16 votes
2 answers
8k views

Sort --parallel isn't parallelizing

I'm trying to make unique a set of lines pulled from a file with egrep with sort -u, then count them. About 10% of the lines(all 100 characters long from the alphabet [ATCG]) are duplicated. There are ...
user avatar