Questions tagged [distributed-computing]
The distributed-computing tag has no usage guidance.
50
questions
31
votes
9
answers
31k
views
Is there any way to distribute x264 encoding jobs across multiple computers (to increase the encoding speed)?
Does anyone know of a current, active solution to encoding x264 videos across many computers (via the network) to increase encoding FPS?
Brownie points for cross-platform and open source, but just so ...
5
votes
4
answers
2k
views
Distributed, decentralized data sharing and backup
Consider the following scenario: you have a team of a few people scattered around the globe that needs to work on the same data. Each member of the team has a copy of the data he needs. When someone ...
- 201
4
votes
2
answers
2k
views
How do distributed filesystems differ from cluster filesystems?
How do distributed filesystems differ from cluster filesystems?
Is it just the wording that I am getting tripped up on? Are there any significant differences between the two.
- 759
4
votes
1
answer
6k
views
Distributed vs clustered vs networked file system
What is the differrence between a distributed file system, clustered file system and a network file system ?
Bandicoot
3
votes
1
answer
2k
views
Force most compilation to a remote host with distcc
The distcc FAQ states that there is currently no scheme for balancing or prioritizing hosts. They recommend to simply list the fastest host on top or the list.
The situation here is that the remote ...
- 11.3k
3
votes
2
answers
3k
views
Distributed Compiling on Windows
I read about distcc for distributed compiling on Linux systems. Since I have to compile a big project quite often, I was wondering if there are methods on Windows on how to distribute that onto ...
- 4,067
3
votes
0
answers
75
views
Manage processes on an distributed windows computer system
I want to manage the startup / re-start and shutdown of a multi module software system running on a set of windows machines.
For example:
[some machines booted, some still doing so]
run the whole ...
- 31
2
votes
1
answer
70
views
Donate disk and CPU capacity to a third party? [closed]
Due to an impasse with one of our providers, we have come to have a non-trivial amount of CPU, and disk capacity that will stand unused for a number of months.
How can I find services that will, ...
- 125
2
votes
1
answer
617
views
How to put idle CPU processing power to good use? [closed]
I just got a new computer that has far more processing power than I need 90% of the time that I'm using it. To what ought I to devote that power?
I know of SETI @ Home and Protein Folding. What ...
2
votes
1
answer
388
views
What are "multikernel computers"?
Which does the term 'modern multikernel computers' refer to? Distributed systems or multi-core computers?
I saw it in a one-line question, so no context to help! Google didn't help neither.
- 131
2
votes
2
answers
539
views
DFS for a several small clusters over WAN
My friends and I all have TBs on our system(s). None of us have any full backups which are geographically distributed however, because at that amount of data, solutions such as Dropbox, S3, et al. are ...
user16511
2
votes
0
answers
62
views
how to keep a distributed software up to date on Linux machines
I have my own distributed software running on few RasPi (20+) in different geographical locations, but every time i change something in my app, i have to update the excuteble in all Pi's manually, ...
- 131
1
vote
5
answers
755
views
Is it possible to "farm" out graphics processing to a comparatively more powerful machine?
I have a computer with comparatively weak graphics, and one with comparatively strong graphics. I would like to run a graphics-intensive application on the weaker machine, using the graphics power of ...
- 119
1
vote
2
answers
3k
views
Open source Distributed computing tool
I want to set up distributed computing on my Local Area Network consisting a bunch of PCs. Say for the time being each one has the same OS - Windows 7.
Is there any opensource tool available so that ...
1
vote
1
answer
675
views
How does distributed computing work?
I heard a 3D modeller say that when he renders a scene he uses multiple machines to do it because simply the hardware is not powerful enough and he is using the collective computational power of more ...
- 169
1
vote
4
answers
2k
views
Problems when loop over a series of ssh-ed commands
I have a series of server machines which I want to run the same command on. Each command takes hours and (even though I am running the commands using nohup and setting them to run in the background) ...
- 124
1
vote
4
answers
120
views
Distributed website server redundancy [closed]
Assume a website infrastructure is very complicated and is fully distributed (probably like most large web companies).
Am I right in thinking that although there are all these extra web servers to ...
- 11
1
vote
1
answer
3k
views
Is it possible to use 2 Mac minis to do a distributed computing system
I need some advice on connecting and using 2 Mac minis as a distributed computing system. The situation is that, my current Mac mini went bad on discrete graphics card and is almost usable only in ...
- 133
1
vote
1
answer
895
views
Run Folding@Home inside VMs on a single physical host, or just on the parent OS?
From a pure efficiency standpoint, would running 8 VMs on a 8 CPU core system, one in each of 8 1 CPU VMs be more efficient at processing the F@H data, or just running one instance on a dedicated ...
- 10.2k
1
vote
1
answer
39
views
What is the correct way to turn a second computer into a simple expansion for the first? [closed]
I have two computers side-by side in the same large case. One is a gaming rig, one is currently just managing case fans and doesn't even have an OS.
I have a truly insane number of peripherals. ...
- 1,206
1
vote
1
answer
716
views
Delete logs after consumption: logstash
I have logstash watching a directory on the host. Every time it sees a log that matches the path I specified in the logstash config it starts to import the data into my elasticsearch cluster. Does ...
- 133
1
vote
1
answer
1k
views
Distributed File System using NFS?
Is it possible to setup NFS to perform as a parallel distributed file system? I'd like to have to NFS servers containing the same file data and serving file ops request for load balancing and to ...
- 11
1
vote
0
answers
48
views
I'm following the tutorial for setting up tf_jobs as given in Kubeflow documentation and I run into error in ks apply stage
I'm following this instruction for setting up TF JOBs.
These are instructions that work just fine:
CNN_JOB_NAME=mycnnjob
VERSION=v0.4.0
This is my init code:
ks init ${CNN_JOB_NAME}
cd ${...
- 11
1
vote
0
answers
131
views
MATLAB GPU-enabled function? [closed]
Is there a way for me to find out whether the functions I'm using in a MATLAB script are GPU-enabled? There does not seem to be a comprehensive list (though see this link for a list of image ...
- 11
1
vote
0
answers
80
views
How to parallel windows pc in same network?
I have some windows pc in my local network, each of them have 8 cores cpu.
I have program that takes lot of cpu resource, so I want to parallel it on all of the pc.
Right now I running my program ...
- 11
1
vote
2
answers
562
views
Running BOINC on Debian
I run Debian unstable. I've installed boinc-client and boinc-manager packages. When i run boincmgr, it first says "Retrieving current status" and then "Unable to connect to the core client". http://...
- 1,394
0
votes
1
answer
596
views
What is a partitioned network? [closed]
I've learned what a partitioned hard disk is, but how does partitioning work when referring to distributed systems and computer networks?
- 759
0
votes
1
answer
109
views
Using distcc across two ssh servers
For those who do not know, "distcc" is a program that provides an infrastructure for remote compilation. I compile the source code for a software called FreeCAD, a lot. It takes about 4 ...
- 101
0
votes
2
answers
75
views
What's currently the fastest way to get data in memory of one computer into memory of other computer?
What's currently the fastest way of making data in memory of one server available in the memory of another server?
Of course I I can use regular networking - but I could imagine there being faster ...
0
votes
1
answer
1k
views
running a simulation software on a remote server
Some simulation software packages take so much time and processing power to finish the job. How can I run these kinds of software on a remote server? (e.g. a finite element method software such as ...
- 1,086
0
votes
1
answer
111
views
How large can a VMware datastore be? Name of VMWare software that monitors single server/multiple servers?
I had an interview and was asked the following questions about VMWare. I'm eager to know the answers, please help me :
How large can a VMware datastore be?
What is the name of the VMware software ...
- 115
0
votes
1
answer
663
views
fastest non volatile memory type available for High Performance Computing/ Cloud computing? [closed]
I am working on parallel computing and distributed programs and noticed that file i/o is quite a bottleneck. It seems quite clear this particular aspect is major issue in many high performance ...
- 149
0
votes
1
answer
88
views
The exact workflow of PVM (Parallel virtual machine)
These days I am learning PVM technology for concurrent processing. I go through so many blogs, tutorials to understand . Yes, I got some points, but no one gives me clear workflow of how it perform ...
- 101
0
votes
1
answer
676
views
Free software/open source remote process execution and management system for linux? [closed]
I have the following requirements for a remote process execution/management system and thought I'd ask if anyone had seen something similar available under free software/open source licensing terms ...
- 401
0
votes
1
answer
649
views
Cloud Computing - Multiple Physical Computers, One Logical Computer
I know that you can set up multiple virtual machines per physical computer. I'm wondering if it's possible to make multiple physical computers behave as one logical unit?
Fundamentally the way I ...
- 268
0
votes
2
answers
782
views
torque in Cluster [closed]
I am learning and have some questions about using PBS to submit jobs in a cluster.
(1) If a job is parallelized by MPI and run by mpirun without using qsub, it cab be running on different nodes. So ...
- 17.3k
0
votes
0
answers
79
views
Dynamically checking and allocating SLURM nodes within a python script
I have a computationally expensive simulation function I am looking to distribute accross a multi-node cluster. The code looks something like this:
input_tasks = [input_0, input_1, ..., input_n]
for i ...
- 109
0
votes
0
answers
52
views
Computing configuration for nvidia geforce rtx 2080 ti
I am trying to build a small station with nvidia GeForce RTX 2080 TI which can be used to perform AI computing related study and practice.
I installed the CUDA driver. However, I have below problems:
...
- 121
0
votes
1
answer
41
views
Running a scripted program between a rapsberry pi and a laptop
What I have: A laptop(CPU: i7 6500U, RAM:8GB, OS: Windows 8.1) and a RPi Zero
What I know: Distributed computing is something that requires a lot of time and effort using special software such as ...
- 103
0
votes
0
answers
369
views
Using multiple computers as a processing and data storage cluster
I have about 17 or so old computers that by themselves can't do much, but I was wondering if there was any way to combine them so that they could run multiple VMs in a distributed way. For example, ...
- 63
0
votes
0
answers
369
views
Can we use Ceph or Gluster FS to avoid IO bottleneck to start many VMs?
AIM: I want to start 50s of VM on s single server or on a set of PCs.
Scenerio: I tried to boot 15 VM on a single server with enough computing power(processor) and RAM but disk is a bottleneck in ...
- 136
0
votes
1
answer
1k
views
Difference between "network RAM" and "distributed shared memory" (DSM)
What is the difference between "network RAM" and "distributed shared memory" (DSM)?
Basically, I know that both of them provide a shared storage on RAM among all the systems in a cluster. So, what is ...
- 160
0
votes
0
answers
127
views
Distributed data storage for slow network with high node churn?
What distributed data storage shall I use for reliably storing data over set of nodes with poor networking between them?
I want something that:
Does not expect that all nodes are reachable now, ...
- 17.2k
0
votes
0
answers
62
views
connecting a raspberry pi to a fedora and a windows 7 machine
I am not sure where i would post this question, so please do not delete this question and just guide me to where i can post this question.
My question is that by buying a switch would I be able to ...
Sacke Tacked
0
votes
1
answer
81
views
Distribute list of processes to many windows machines
I have a list of independent tests which run a common executable with different command lines and in different working directories.
At the moment i simple run these one after the other in a batch ...
- 101
0
votes
1
answer
88
views
Are there any solutions where spare disk on workstations can be used as a distributed NAS?
With a windows based workstation heavy infrastructure, we have a lot of spare disk space.
I am look about to see if there are any solutions where this space can be used as a sort of distributed NAS.
...
- 123
0
votes
1
answer
395
views
SimpleBackUp with multiple hard drives
I have a lot of hard drives from old computers (6-7 years old). i want to get an external enclosure with USB 3.0 and use them for back up.
To my understanding SimpleBackup copies everything the first ...
- 1,318
0
votes
3
answers
2k
views
Linux live-cd for distributed computing projects?
I have had this dream for years now... but sadly, even to this day, it remains just a dream.
I want to find a live-cd that I can pop into any computer I have permission to use, and it boots up and ...
- 15.7k
-1
votes
1
answer
42
views
In Linux, is concept of Process similar to Container (in microservices?) [closed]
Process is executing an instance of an application.
Every process got its State, ID, address space etc, and isolated to each other.
It looks similar to Container that runs a specific service?
In some ...
-1
votes
1
answer
289
views
Distributed systems: how does their architecture allow them to do what is needed? [closed]
I was just wondering how these processes work, in an overarching sense, such as "distributed grep", "distributed compiling", "distributed sort", "distributed file systems", "distributed _______"
Are ...
- 759