<parallel> (Or "multiprocessing") The simultaneous use of more than one
computer to solve a problem. There are many different kinds of parallel computer
(or "parallel processor"). They are distinguished by the kind of interconnection
between processors (known as "processing elements" or PEs) and between
processors and memory. Flynn's taxonomy also classifies parallel (and serial)
computers according to whether all processors execute the same instructions at
the same time ("single instruction/multiple data" - SIMD) or each processor
executes different instructions ("multiple instruction/multiple data" - MIMD).
The processors may either communicate in order to be able to cooperate in
solving a problem or they may run completely independently, possibly under the
control of another processor which distributes work to the others and collects
results from them (a "processor farm"). The difficulty of cooperative problem
solving is aptly demonstrated by the following dubious reasoning:
If it takes one man one minute to dig a post-hole
then sixty men can dig it in one second.
Amdahl's Law states this more formally.
Processors communicate via some kind of network or bus or a combination of both.
Memory may be either shared memory (all processors have equal access to all
memory) or private (each processor has its own memory - "distributed memory") or
a combination of both.
Many different software systems have been designed for programming parallel
computers, both at the operating system and programming language level. These
systems must provide mechanisms for partitioning the overall problem into
separate tasks and allocating tasks to processors. Such mechanisms may provide
either implicit parallelism - the system (the compiler or some other program)
partitions the problem and allocates tasks to processors automatically or
explicit parallelism where the programmer must annotate his program to show how
it is to be partitioned. It is also usual to provide synchronisation primitives
such as semaphores and monitors to allow processes to share resources without
Load balancing attempts to keep all processors busy by allocating new tasks, or
by moving existing tasks between processors, according to some algorithm.
Communication between tasks may be either via shared memory or message passing.
Either may be implemented in terms of the other and in fact, at the lowest
level, shared memory uses message passing since the address and data signals
which flow between processor and memory may be considered as messages.
The terms "parallel processing" and "multiprocessing" imply multiple processors
working on one task whereas "concurrent processing" and "multitasking" imply a
single processor sharing its time between several tasks.
See also cellular automaton,symmetric multi-processing.
Usenet newsgroup: comp.parallel.
Parallel Pascal « parallel port « Parallel Presence
parallel processing » parallel processor »
parallel random access machine » parallel reduction