Questions tagged [parallel-processing]

Running multiple commands (or even computers) in parallel for more efficiency.

Filter by
Sorted by
Tagged with
-2
votes
0answers
28 views

Multiple Start and Wait in Batch File

i have a main.bat where i want to start multiple bat inside simultaneously. start test2.bat timeout 2 start test2.bat timeout 2 start test2.bat after all this bat is finished i want to run another ...
0
votes
1answer
52 views

Are the Geekbench multicore scores done using embarrassingly parallel processes?

For each processor, the benchmarks at geekbench.com present both a single-core and multi-core score. For example, this 64-core processor has a single-core score of 1220 and a multi-core score of 23688....
0
votes
0answers
68 views

Guidance on Powershell Workflow foreach -parallel (){} functions

I just discovered Powershell workflows and I am experimenting but having some difficulties. I do not know if workflows even have the capabilities to do the following but if they do, I would appreciate ...
55
votes
12answers
27k views

Can I run my CPU at 100% usage for a long time?

I have a Dell XPS 15 9570, 32 GB ram, 2.2 ghz and 6 cores. When i use an application (The parsim command on MATLAB) (for more details) then my fan starts running fast , during this my PC doesn't seem ...
2
votes
2answers
356 views

Sequential parallel with wait

I want to run a multi-parallel task in one bash file like the example code below, for i in 1 2 3 4 5 6 7 8; do setsid python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 & done wait # ...
1
vote
0answers
110 views

How to split and join parallel pipes?

I'd like to pipe the output of 1 command into multiple pipes and pipe outputs from multiple commands into 1 single command. An example of split: # Single output to multiple pipes echo "This is a ...
0
votes
2answers
36 views

What is the optimal number of CPU-heavy data mining jobs to run in parallel? [closed]

I have a CPU heavy data mining job that's parallelizable. Each job run takes ~10 minutes. The job isn't heavy in IO, it's mostly heavy in CPU compute. I have a 32-core rig with 2 threads per core and ...
1
vote
0answers
86 views

Can I run chkdsk for different HDDs at the same time (external via USB)?

I want to run multiple chkdsk commands (for different HDDs) at the same time.  All of them are external via USB. I searched for this on Google, but there were few results that were really relevant to ...
0
votes
0answers
43 views

Are ARM processors fundamentally inferior to Intel processors? [duplicate]

I have been toying with the idea of building a small computing cluster out of ARM-based single board computers (SBC), and while that looks like it may be infeasible for a number of reasons, I'm ...
1
vote
0answers
17 views

How can a processor execute more instructions per period than there are clock cycles? [duplicate]

This Wikipedia page lists several cases where the instructions per clock cycle per core is greater than 1. How can there be more instructions executed per core per period than there are clock cycles ...
0
votes
0answers
44 views

Deep learning studio only handling 5 images per second

I have a simple neural net that is running on Keras with Deep learning studio as an interface. The problem I'm seeing is that for 80% of the time, the python script is only using 1 core (and maxing it ...
0
votes
0answers
34 views

Eclipse PTP Parallel Debugging

I have a problem with Eclipse PTP 201906. I would like to perform debug in parallel with 4 cores using Eclipse. I adjust the debug configuration with Local connection Arguments: -np 4 and begin ...
1
vote
0answers
26 views

Ansys parallele job on Slurm Cluster stuck without error or exit message

I am working on a Slurm Cluster, executing Ansys (V18.2) jobs in a parallelized way. Large jobs (meaning large solver files) often stuck with out error message or exit message, the jobs keep running ...
1
vote
0answers
111 views

MATLAB GPU-enabled function? [closed]

Is there a way for me to find out whether the functions I'm using in a MATLAB script are GPU-enabled? There does not seem to be a comprehensive list (though see this link for a list of image ...
2
votes
3answers
220 views

How does a computer sleep/wait without using all of its processing power?

Something I've never quite understood is how a computer can operate without using all of its processing power. (Likely this confusion stems from a dreadful OS class in college.) As far as I know, a ...
-1
votes
1answer
33 views

In Linux, is concept of Process similar to Container (in microservices?) [closed]

Process is executing an instance of an application. Every process got its State, ID, address space etc, and isolated to each other. It looks similar to Container that runs a specific service? In some ...
1
vote
1answer
830 views

What's the difference between a multiprocessor and a multiprocessing system?

I'm studying IPC for a simple test I have tomorrow. The book my classroom uses is not the best: it makes a lot of typos and its exercises have been proven wrong or terribly ambiguous more than once. ...
1
vote
1answer
699 views

Use bittorrent to download from China?

I have a server in China and a client in Europe and need to download large (50-100 GB) files as quickly as possible. The connections to China are wonky at best and very slow. A way to get around this ...
1
vote
0answers
411 views

Threads vs Processors for Excel [duplicate]

I'm writing a VBA module that uses a lot of computational power. To speed things up, I'm using all available processors on my machine. Excel gives me an option though to use a different number of ...
3
votes
1answer
344 views

Kill first command as soon as second exits

I've been looking for best practice for this scenario for a long time. If I have process2 that depends on process1 running beforehand. I set up an alias like: alias both='process1 & process2' ...
0
votes
1answer
542 views

How do you control thread affinity across multiple processes on Linux?

Question How do I run two benchmarks such that: They are both executing concurrently Both benchmarks have half their threads on each socket. No two threads share the same core. Background I have ...
2
votes
2answers
237 views

Create many tar files from a directory with 500000 files

I have a directory containing around 500k files, and want to slice them into t tar files. Put formally, let's call the files file_0, ..., file_{N-1}, where N around 500k. I want to create t tar files ...
0
votes
1answer
609 views

scp a file in parallel FROM multiple servers

Is there a script or something that can use scp to copy a single file from multiple servers/sources, each connection with its own chunk/chunks? I want something like how a torrent acts. I do not have ...
0
votes
2answers
1k views

How to gather multiple remote servers' CPU utilization info?

I need to test program in multiple remote servers, and at the same time need to gather the CPU utilizations information during the process. Here use one server (10.18.6.0) as the master server ...
-1
votes
1answer
41 views

How to deal with small time difference in multiple remote servers?

I use several remote servers to test a parallel system, the process will take about 0.4s each time. So when I run the program in several servers, the running time range from less than 0 to 1s. The ...
0
votes
1answer
832 views

How to use pssh to run multiple servers with different commands?

I need to test the distributed system with several remote servers, so I use pssh in master server to control others. Here need to run multiple server with different commands, for example In server ...
3
votes
1answer
351 views

remove slurm sacct command double entries: “extern”

Jobs currently running show two entries, one of them has an .extern suffix. Completed (or failed) jobs also have a third entry: .batch. Is there a way to remove (or not show these) from the sacct ...
0
votes
1answer
60 views

How to parallelize this image processing script?

I have this script that converts .CR2 images from my Canon camers to Jpegs. The images are named IMG_{NNNN}.CR2 where {NNNN} are the image sequence numbers from the camera. I'd like to parallelize it ...
2
votes
1answer
435 views

Parallel Programming in AMD based laptop

I've a Lenovo G50 series laptop with AMD A8-6410 @2Ghz, 8 GB RAM with a R5 230M and Radeon HD 8550M cards inside. In Windows 10 is it possible to run a CLI based program ( ffmpeg for video ...
10
votes
2answers
11k views

How to use Ctrl+C to kill all background processes started in a Bash script?

ArchLinux (Manjaro). I'm running one bash file. It runs 2 processes (commands), using &. But when I press Ctrl+C to stop the program - one process dies, and the other continues to work. How do I ...
2
votes
0answers
23k views

SSH error: Temporary failure in name resolution on mpi run

I have 6 computers (4 cores each and identical in hardwares) with static IPs and need to run NAMD computational chemistry software parallel using OpenMPI. So I installed SSH and OpenMPI (on mint 18)...
1
vote
0answers
40 views

Hardware Requirements Gap Analysis for Windows

I have an algorithm that needs to be repeated ('replication') a minimum of 30 times: each replication could be run in it's own core. The Ninja development stations feature the "Knight's Landing" (...
1
vote
1answer
976 views

How do I run multiple .bat files that are in subfolders

I'm new to supercomputers, I recently built an Amazon EC2 instance with 128 CPUs and Microsoft server 2012. I have one folder that has 100 subfolders, in each subfolder there is a .bat file. How do I ...
0
votes
1answer
78 views

Bash one-liner to open several processes and execute a python function in each - poor man's parallel processing

I have a python file like my_functions.py and within that file is a function def func(a,b): result = does_some_stuff(a,b) return result Is there a bash one-liner that would open N ...
0
votes
0answers
1k views

Using Workflows In Powershell

I have been using just a standard foreach loop, which executes as it should. However, I have an 8 core processor and 16GB of RAM and thought I would utilize such with a Parallel.foreach loop. Well, ...
64
votes
4answers
14k views

How can my computer use 300% PC processing power?

I'm playing around with a script and well I got carried away (it's still running right now and yet my computer has not yet died). I'm running 17 iterations of the script (with the intention of ...
1
vote
1answer
488 views

The main difference between Superscalar and Parallelism

In superscalar processing, more than one instruction is executed in parallel at each core, so how would that differ from parallel processing then? Thank you.
0
votes
2answers
2k views

bash for loop in parallel with n cores

I have a bash loop that looks like: for i in $(seq 0 $max); do my_command $i done and I would like to run this in parallel on n cores. I know that I could do while [[ "$j" -le "$max" ]]; do ...
0
votes
1answer
34 views

How to run limited amount of instances of program?

For example, I have circa 14000 files I need to process Right now I'm doing it with for %%a in (*.OGG) do revorb.exe "%%a" This runs all revorbs one after one. I have quad processor, so I'd like to ...
1
vote
0answers
85 views

why is my master node idle when running gnu parallel remotely?

I'm using gnu-parallel to run a number of python scripts in parallel across a number of computers in my gigabit network. The files being processed are located on a tmpfs (RAM) folder on my master ...
1
vote
3answers
2k views

zero a large block device with parallelized dd

We have a sixty terabyte block device which we want to zero using dd. Is it possible to parallelize this operation in order to reduce the amount of time it takes? I've tried running the following ...
10
votes
1answer
6k views

Sort --parallel isn't parallelizing

I'm trying to make unique a set of lines pulled from a file with egrep with sort -u, then count them. About 10% of the lines(all 100 characters long from the alphabet [ATCG]) are duplicated. There are ...
2
votes
2answers
313 views

Will process run slower on one thread rather than one core

So many CPUs now come with 2/4 cores and 4/8 threads. There are also some CPUs that are 4 core & 4 threads. Would that mean that unparalleled process will run slower on the same clock cycle CPU ...
0
votes
2answers
307 views

How to implement Intel MPI benchmarks for mpich2?

Many people have recommended the use of Intel MPI benchmarks for mpich2, however the installation is not very simple. Could anyone provide steps for the installation and sample test case for it? Which ...
-1
votes
1answer
6k views

Is it possible to use GPU to boost performance of computer automatically? [duplicate]

As we all know GPU is a sleeping beast. And we are also familiar with Intel's Turbo Boost technology which is really great. But, Intel's Turbo Boost technology works only on CPU by overclocking CPU ...
0
votes
1answer
2k views

Clock speed vs. number of cores for parallelized computer simulation [duplicate]

My question is similar to some "speed vs. cores"-questions that have already been asked on this website. However, I am interested in some very specific and technical points. So I hope that my question ...
2
votes
1answer
552 views

How can I force gnuparallel to carry out a command set sequentially?

Gnu parallel is a powerful tool that I use to run many independant BASH commands as a single set in parallel. I would like to be able to run the same commands SEQUENTIALLY without significant changes ...
0
votes
0answers
113 views

Do threads/processes that create/delete files/directories under a shared parent dir interfere?

I was wondering: If I have a directory /a/b/c/ and under c X threads create subdirectories and files and delete them (they are temporary directories and files as intermediate steps of processing) do ...
22
votes
0answers
3k views

Jenkins realtime console output for parallel threaded fabric command [closed]

We have a Python fabric command that runs in parallel across several hosts, something like this: $ fab --hosts=prod1.server,prod2.server,prod3.server --parallel copy_cache This will copy cache to ...
0
votes
1answer
2k views

Is Virtual Machine capable of running as normal PC would?

Let's imagine that you have a super-powerful-server at home of size of desktop (just visualisation). 2048 physical cores each 3GHz, 64TB RAM, GTX 780 Ti Black Edition (SLIx4, (four graphics card ...