Python Programming for Biology: Bioinformatics and Beyond

Using the ‘

multiprocessing

’ module

Below  we  give  a  couple  of  simple  examples  that  illustrate  how  multiprocessing  can  be

used.  The  examples  are  simple  in  the  sense  that  the  calculation  jobs  are  completely

separate and each parallel process doesn’t need to communicate much, other than to accept

input arguments and pass back the result. However, this is not an uncommon situation in

biological computing; for example, things like molecular structure calculations generate a

range (an ensemble)  of  separate  solutions  to  a  problem,  and  in  many  circumstances  jobs

can be run separately on independent input data.

---

For  our  examples  we  make  three  imports,  one  for  the  Process  class,  which  is  the

simplest  way  of  generating  a  parallel  job,  another  for  Queue,  which  allows  data  to  be

shared  between  jobs,  and  Pool,  which  provides  a  convenient  way  of  matching  a  fixed

number of processing cores with a variable number of sub-tasks; the pooling mechanism

collects jobs to be run and then allocates them to the processor when there is a free slot.

from multiprocessing import Process, Queue, Pool