Questions tagged [parallel-processing]
Running multiple commands (or even computers) in parallel for more efficiency.
107
questions
1
vote
0
answers
46
views
Powershell: script won't run in parallel [closed]
I have the following script. It will run without a problem when it is not in parallel. But when I add -Parallel it throws all kinds of errors. The primary error seems to relate to Cannot index into a ...
- 556
1
vote
1
answer
28
views
get parallel ssh output in order but use parallelism
With parellel-ssh -i -h hosts.ini "uptime" I can get the uptime from all hosts in hosts.ini. But they appear in order of the batches and first answers. I would like to have the result ...
- 397
0
votes
1
answer
35
views
Parallel processing implementation
I received an very old program in my company that need to be refactored. The code is a bunch of SQL instructions that are orchestred by a program written as shell commands with the objective of doing ...
1
vote
0
answers
154
views
Connecting two desktops by an infiniband interconnect
I am a scientist working in a high performance computing project(the computational programming aspect) and I have no idea of networking or networking hardware required to answer my question. A ...
- 123
1
vote
1
answer
174
views
How can I stop Ubuntu 20.04 from restricting my cpu space?
After installation of Ubuntu 20.04 everything was working well. However, after installing and using some applications (Matlab and Matlab/Dynare) Ubuntu seems to limit my cpu usage. I use openMP and a ...
- 21
0
votes
1
answer
663
views
Can any program be run on multiple cores? Will the CPU decide that? Or can only certain programs run on multiple cores?
So I'm a little confused about the multicore CPUs - the impression I was initially under and what I thought I heard before was that some programs can't be run across multiple cores - those in which ...
0
votes
1
answer
745
views
Parallel processing and order of execution with recursive loop
If i run a script that processes a path recursively using parallel processing to execute a command on the whole tree if items, am I guaranteed that the commands I use will be executed on all items in ...
- 319
0
votes
1
answer
374
views
PowerShell: Script throwing up in parallel
When run in -Parallel, the following script does not pass the $basePath variable declared before the start of the parallel block. Instead, it is a null value. This is not a problem when the script is ...
- 556
0
votes
2
answers
38
views
Why is folder deletion not parallelized?
When I Shift-delete a big folder, I see the following CPU usage:
Not all boxes are used. Is this inefficient?
This seems easily parallelizable to me, through divide-and-conquer. I tried manually ...
- 101
1
vote
1
answer
290
views
GNU Parallel usage - how get currently passed string to parallel?
I am using GNU parallel and want to understand - how can I get the individual string passed to each parallel command?
As an example, GNU Parallel documentation shows how to move files from the current ...
- 135
2
votes
2
answers
1k
views
Execute two or more commands simultaneously in shell script
It is often very helpful to run commands in parallel in a shell script, but I cannot find ways to do it. Is this possible? If so, how would I achieve that? I'm mainly interested in doing that for fish....
- 165
1
vote
1
answer
20
views
bad parallel-processing gain on some linux machines, good on others
I'm measuring the run time for a complex signal-processing algorithm written in C on several different computers, all running linux. The code is partially parallelized by some of my own methods using ...
- 11
1
vote
2
answers
3k
views
How to download a single large file over sftp or http etc using multiple simultaneous connections [closed]
Is there a command line tool which supports both multiple simultaneous connections and multiple network protocols (ftp, http etc)?
Here is a practical example:
I have a large backup file on an ssh ...
- 121
0
votes
1
answer
649
views
Are the Geekbench multicore scores done using embarrassingly parallel processes?
For each processor, the benchmarks at geekbench.com present both a single-core and multi-core score. For example, this 64-core processor has a single-core score of 1220 and a multi-core score of 23688....
- 129
0
votes
0
answers
690
views
Guidance on Powershell Workflow foreach -parallel (){} functions
I just discovered Powershell workflows and I am experimenting but having some difficulties. I do not know if workflows even have the capabilities to do the following but if they do, I would appreciate ...
- 556
61
votes
12
answers
53k
views
Can I run my CPU at 100% usage for a long time?
I have a Dell XPS 15 9570, 32 GB ram, 2.2 ghz and 6 cores. When i use an application (The parsim command on MATLAB) (for more details) then my fan starts running fast , during this my PC doesn't seem ...
- 737
3
votes
3
answers
1k
views
Sequential parallel with wait
I want to run a multi-parallel task in one bash file like the example code below,
for i in 1 2 3 4 5 6 7 8; do
setsid python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 &
done
wait # ...
- 185
2
votes
1
answer
766
views
How to split and join parallel pipes?
I'd like to pipe the output of 1 command into multiple pipes and pipe outputs from multiple commands into 1 single command.
An example of split:
# Single output to multiple pipes
echo "This is a ...
0
votes
2
answers
154
views
What is the optimal number of CPU-heavy data mining jobs to run in parallel? [closed]
I have a CPU heavy data mining job that's parallelizable. Each job run takes ~10 minutes. The job isn't heavy in IO, it's mostly heavy in CPU compute. I have a 32-core rig with 2 threads per core and ...
1
vote
0
answers
3k
views
Can I run chkdsk for different HDDs at the same time (external via USB)?
I want to run multiple chkdsk commands (for different HDDs)
at the same time.
All of them are external via USB.
I searched for this on Google, but there were few results
that were really relevant to ...
- 11
1
vote
0
answers
21
views
How can a processor execute more instructions per period than there are clock cycles? [duplicate]
This Wikipedia page lists several cases where the instructions per clock cycle per core is greater than 1. How can there be more instructions executed per core per period than there are clock cycles ...
- 1,194
0
votes
0
answers
68
views
Eclipse PTP Parallel Debugging
I have a problem with Eclipse PTP 201906. I would like to perform debug in parallel with 4 cores using Eclipse. I adjust the debug configuration with
Local connection
Arguments: -np 4
and begin ...
- 1
1
vote
0
answers
92
views
Ansys parallele job on Slurm Cluster stuck without error or exit message
I am working on a Slurm Cluster, executing Ansys (V18.2) jobs in a parallelized way. Large jobs (meaning large solver files) often stuck with out error message or exit message, the jobs keep running ...
- 11
1
vote
0
answers
130
views
MATLAB GPU-enabled function? [closed]
Is there a way for me to find out whether the functions I'm using in a MATLAB script are GPU-enabled? There does not seem to be a comprehensive list (though see this link for a list of image ...
- 11
3
votes
3
answers
2k
views
How does a computer sleep/wait without using all of its processing power?
Something I've never quite understood is how a computer can operate without using all of its processing power. (Likely this confusion stems from a dreadful OS class in college.)
As far as I know, a ...
- 259
-1
votes
1
answer
40
views
In Linux, is concept of Process similar to Container (in microservices?) [closed]
Process is executing an instance of an application.
Every process got its State, ID, address space etc, and isolated to each other.
It looks similar to Container that runs a specific service?
In some ...
1
vote
1
answer
2k
views
What's the difference between a multiprocessor and a multiprocessing system?
I'm studying IPC for a simple test I have tomorrow. The book my classroom uses is not the best: it makes a lot of typos and its exercises have been proven wrong or terribly ambiguous more than once.
...
- 115
1
vote
1
answer
976
views
Use bittorrent to download from China?
I have a server in China and a client in Europe and need to download large (50-100 GB) files as quickly as possible. The connections to China are wonky at best and very slow. A way to get around this ...
- 111
1
vote
0
answers
524
views
Threads vs Processors for Excel [duplicate]
I'm writing a VBA module that uses a lot of computational power. To speed things up, I'm using all available processors on my machine. Excel gives me an option though to use a different number of ...
- 13.2k
3
votes
1
answer
427
views
Kill first command as soon as second exits
I've been looking for best practice for this scenario for a long time.
If I have process2 that depends on process1 running beforehand. I set up an alias like:
alias both='process1 & process2'
...
- 133
0
votes
1
answer
583
views
How do you control thread affinity across multiple processes on Linux?
Question
How do I run two benchmarks such that:
They are both executing concurrently
Both benchmarks have half their threads on each socket.
No two threads share the same core.
Background
I have two ...
user736817
2
votes
2
answers
570
views
Create many tar files from a directory with 500000 files
I have a directory containing around 500k files, and want to slice them into t tar files.
Put formally, let's call the files file_0, ..., file_{N-1}, where N around 500k. I want to create t tar files ...
- 155
0
votes
1
answer
2k
views
scp a file in parallel FROM multiple servers
Is there a script or something that can use scp to copy a single file from multiple servers/sources, each connection with its own chunk/chunks? I want something like how a torrent acts. I do not have ...
- 249
0
votes
2
answers
2k
views
How to gather multiple remote servers' CPU utilization info?
I need to test program in multiple remote servers, and at the same time need to gather the CPU utilizations information during the process.
Here use one server (10.18.6.0) as the master server
...
- 97
-1
votes
1
answer
43
views
How to deal with small time difference in multiple remote servers?
I use several remote servers to test a parallel system, the process will take about 0.4s each time. So when I run the program in several servers, the running time range from less than 0 to 1s. The ...
- 97
0
votes
1
answer
958
views
How to use pssh to run multiple servers with different commands?
I need to test the distributed system with several remote servers, so I use pssh in master server to control others. Here need to run multiple server with different commands, for example
In server ...
- 97
5
votes
1
answer
1k
views
remove slurm sacct command double entries: "extern"
Jobs currently running show two entries, one of them has an .extern suffix. Completed (or failed) jobs also have a third entry: .batch. Is there a way to remove (or not show these) from the sacct ...
- 559
0
votes
1
answer
62
views
How to parallelize this image processing script?
I have this script that converts .CR2 images from my Canon camers to Jpegs. The images are named IMG_{NNNN}.CR2 where {NNNN} are the image sequence numbers from the camera. I'd like to parallelize it ...
- 111
2
votes
1
answer
506
views
Parallel Programming in AMD based laptop
I've a Lenovo G50 series laptop with AMD A8-6410 @2Ghz, 8 GB RAM with a R5 230M and Radeon HD 8550M cards inside.
In Windows 10 is it possible to run a CLI based program ( ffmpeg for video ...
- 21
24
votes
2
answers
21k
views
How to use Ctrl+C to kill all background processes started in a Bash script?
ArchLinux (Manjaro).
I'm running one bash file. It runs 2 processes (commands), using &.
But when I press Ctrl+C to stop the program - one process dies, and the other continues to work.
How do I ...
- 343
2
votes
0
answers
24k
views
SSH error: Temporary failure in name resolution on mpi run
I have 6 computers (4 cores each and identical in hardwares) with static IPs and need to run NAMD computational chemistry software parallel using OpenMPI. So I
installed SSH and OpenMPI (on mint 18)...
- 21
1
vote
0
answers
49
views
Hardware Requirements Gap Analysis for Windows
I have an algorithm that needs to be repeated ('replication') a minimum of 30 times: each replication could be run in it's own core.
The Ninja development stations feature the "Knight's Landing" (...
- 1,175
1
vote
1
answer
3k
views
How do I run multiple .bat files that are in subfolders
I'm new to supercomputers, I recently built an Amazon EC2 instance with 128 CPUs and Microsoft server 2012. I have one folder that has 100 subfolders, in each subfolder there is a .bat file. How do I ...
- 235
0
votes
1
answer
95
views
Bash one-liner to open several processes and execute a python function in each - poor man's parallel processing
I have a python file like my_functions.py and within that file is a function
def func(a,b):
result = does_some_stuff(a,b)
return result
Is there a bash one-liner that would open N ...
- 335
0
votes
0
answers
1k
views
Using Workflows In Powershell
I have been using just a standard foreach loop, which executes as it should. However, I have an 8 core processor and 16GB of RAM and thought I would utilize such with a Parallel.foreach loop. Well, ...
- 2,075
63
votes
4
answers
15k
views
How can my computer use 300% PC processing power?
I'm playing around with a script and well I got carried away (it's still running right now and yet my computer has not yet died). I'm running 17 iterations of the script (with the intention of ...
- 795
1
vote
2
answers
939
views
The main difference between Superscalar and Parallelism
In superscalar processing, more than one instruction is executed in parallel at each core, so how would that differ from parallel processing then?
Thank you.
- 173
0
votes
2
answers
5k
views
bash for loop in parallel with n cores
I have a bash loop that looks like:
for i in $(seq 0 $max); do
my_command $i
done
and I would like to run this in parallel on n cores. I know that I could do
while [[ "$j" -le "$max" ]]; do
...
- 1,454
0
votes
1
answer
38
views
How to run limited amount of instances of program?
For example, I have circa 14000 files I need to process
Right now I'm doing it with
for %%a in (*.OGG) do revorb.exe "%%a"
This runs all revorbs one after one. I have quad processor, so I'd like to ...
- 13
1
vote
0
answers
95
views
why is my master node idle when running gnu parallel remotely?
I'm using gnu-parallel to run a number of python scripts in parallel across a number of computers in my gigabit network.
The files being processed are located on a tmpfs (RAM) folder on my master ...
- 415