Parallel Processing in Python

Today we have a guest post by Ian Crossfield (UCLA) on parallel computing with python.

When analyzing astronomical data, one often finds oneself repeating the same tasks over and over again (e.g. fitting a model to a PSF, measuring the width of a spectral feature, etc.). Sometimes the results of one computation influence the next, in which case there’s no choice but to compute everything sequentially.  In many cases, though, the computations are independent and (with sufficient computing power) could all be performed simultaneously. You like your laptop or workstation (and don’t want to worry about the hassle of learning about your local computing cluster), but aren’t sure how to take advantage of the multiple CPU cores available on even mid-range modern computers.  What’s an astronomer to do?

Python has many and various options to enable this sort of parallel processing; I’ve only tried two. One, “Parallel Python,” seemed to require a high degree of specificity about module dependencies and other issues I would rather not have to deal with; in constrast, getting the “PProcess” module to run is simplicity itself, and I recommend it to anyone who has need of parallel-processing capabilities.

Just download the “pprocess” package and put it in your Python path.  Then it’s simply a matter of using pprocess’ slick “Map” wrapper functionality; essentially this creates a wrapper function that you call with exactly the same arguments you normally use for your function; see the example below.  You can spawn as many simultaneous computations as you’d like with the “limit” option, but you won’t see much benefit once you set this to the number of CPU cores you have at your disposal.

First, here is a schematic example

# Just replace this (calling one function twice, in series):
desired_values = [function(args) for args in [args1,args2]]
 
# ... with this (calling one function twice, in parallel):
import pprocess
nproc = 2  	# maximum number of simultaneous processes desired
results = pprocess.Map(limit=nproc, reuse=1)
parallel_function = results.manage(pprocess.MakeReusable(function))
parallel_function(args1)
parallel_function(args2)
desired_values = results[0:2]

Now for a concrete example:

import pprocess
import time
import numpy as np
 
# Define a function to parallelize:
def takeuptime(ntrials):
    """A function to waste CPU cycles"""
    for ii in range(ntrials):
        junk = np.std(np.random.randn(1e5))
    return junk
 
list_of_args = [500, 500]
 
# Serial computation:
tic=time.time()
serial_results = [takeuptime(args) for args in list_of_args]
print "%f s for traditional, serial computation." % (time.time()-tic)
 
# Parallel computation:
nproc = 2  	# maximum number of simultaneous processes desired
results = pprocess.Map(limit=nproc, reuse=1)
parallel_function = results.manage(pprocess.MakeReusable(takeuptime))
tic=time.time()
[parallel_function(args) for args in list_of_args];  # Start computing things
parallel_results = results[0:3]
print "%f s for parallel computation." % (time.time() - tic)

Do you parallelize your analysis?  Why or why not, and (if so) what do you use?

Information on other parallel computing packages can be found on the AstroBetter wiki.