For the last while I’ve been exploring cloud computing and distributed systems, toying around with different problems that exist and can possibly be solved. When starting out from zero I couldn’t find a really short and sweet summary or article on the basics of distributed computing, so I decided to combine my notes and create the ELI5 I wish I had when I started my research.
What is Distributed Computing?
Distributed computing is a concept that revolutionized the way we use computer systems, and changed many of the ways we look at approaching computational tasks and problems.
Traditionally computation is performed on just one machine. You give the machine an input or series of inputs, it processes and computes those inputs, and once it’s done all the calculations it spits out outputs for you. When we’re on our laptops browsing different applications, running code for a python script we created or using our phones to browse the internet this is happening all the time in the background.
Typically using one computer is fine, but when you get to computationally intensive tasks like trying to solve huge math problems, running experiments like simulating protein scaffolding or trying to run a huge neural network the processing power of just one computer might not be enough for you to solve the problem.
This is where the concept of distributed computing comes in: “A collection of independent computers that appear to the users of the system as a single computer,” - Andrew Tanenbaum.
Distributed computing is the idea of splitting the tasks of one really gigantic computationally intense problem, into smaller pieces that multiple different computers can handle. As a result, you have more processing power split up across a larger number of devices so you can achieve your output much faster.
In order for a system of computers to be distributed it must have 3 Characteristics:
Computers operate concurrently (at the same time)
Computers fail independently (if one goes down the rest of the computers can still run)
The computers do not share a global clock (this is a harder concept to comprehend, but in short each computer must have it’s own global clock. The different clocks may be operating at a similar time but in order for a system to be truly distributed all computers clocks cannot be perfectly synced)
In theory, this sounds pretty simple, but in reality, it’s much harder to execute. Designing distributed systems and figuring out how one huge computational task can be split up among multiple computers is a challenging problem in itself. If the task being performed requires the output from a previous task, or the problem needs to be solved in sequential order it might not be as effective to use distributed computing.
The best problems to run on a distributed computing system, are problems that contain multiple sub-tasks that can all be completed independently of one another. This way tasks can be run concurrently, and not be slowed down by the wait for other tasks to be completed.
The way distributed computing is done is by setting up a host computer that has the program running and is going to be the computer where you run your main task. Along with the host computer, you also set up an assortment of other computers that are going to help with performing the computation.
The host computer has a few special jobs to perform, it’s in charge of splitting the huge computational problem into multiple smaller tasks and then distributing it to the available computers. Each of the computers then performs the computation, and once they receive an output it is sent back to the host computer.
Finally, the host computer receives all these outputs of the smaller tasks and puts them all together to generate the final result. These tasks are often done over a network, and the computers used to perform the smaller tasks are just regular computers. Which means that any device that has reasonable computation power can be used in this network, and if you wanted to you could even create a distributed computing network yourself by using all your and your friend's MacBooks together.
What makes me excited about distributed computing is the potential of applications like crowdsourced computing. Using your computer you could become part of a distributed computing network, and use your computers processing power with performing tasks. You might see this with the view that applications are pretty limited, but it opens up room for some crazy possibilities.
If you had an old computer with really low processing power and wanted to play Fortnite which is more graphically intensive. You could theoretically get someone to use an idle gaming computer they have at home to loan you computing power that you could then use to play Fortnite on your old computer. (In fact, there are companies
already doing this)
The future impact of technology?
As the world has more and smarter devices coming online, that’s more available computing power that we have access to. McKinsey predicts that by 2020 we’ll have 25 Billion+ IoT devices that are connected to the internet.
Everything from your fridge to your car will have an abundance of computational power, but not all of it will be leveraged all the time. If you think about how often devices are actually used like your car, your car is are often idle or parked most of the time.
After imagining the new opportunities that can be locked by leveraging the idle computing power sitting in 25 Billion+ devices, I can definitely see the inception of a global marketplace for idle computing power. Tasks like simulations and renderings that require intense computing power, can tap into the idle compute of billions of devices worldwide to speed up computation times.
There are projects that currently exist to leverage and make use of additional compute power, like Golem and the Fold At Home Project from Stanford. But rather than use centralized marketplaces and tools, it’s possible that a global machine language is developed for the automatic use of device to device communication and sharing of distributed computing power.
It'll definitely be interesting to see how distributed computing evolves, and takes advantage of the growing amount of idle computing power.