For anyone who uses python and ds9 to visualize their FITS files, I think pyds9 is now a must-have. It is officially written and developed through SAOImage ds9 so it will be supported for the long haul. Here are the primary links to get going:

• TARball for installation: Source
• Documentation for installation and use: Docs

The documentation is pretty basic so I put together a very small tutorial of a way I might typically use it. Most of the controls work through the XPA interface, which seems very flexible, if a little tricky to figure out exactly how to make it work for specific use-cases. If you have samples of how you use pyds9 or ds9 xpa, post them in the comments or on this wiki page.

from scipy import stats
import numpy as np
import ds9
 
# Make a 2D gaussian image that is stored in a 2D numpy array
x = np.arange(-3, 3, 0.1)
xx, yy = np.meshgrid(x, x)
gauss2d = stats.norm.pdf(xx) * stats.norm.pdf(yy)
 
# Now open ds9 (this assumes no ds9 instance is yet running)
d = ds9.ds9()
 
# Load up our 2D gaussian
d.set_np2arr(gauss2d)
 
# Zoom to fit
d.set('zoom to fit')
 
# Change the colormap and scaling
d.set('cmap bb')
d.set('scale log')
 
# Add a label
d.set('regions command {text 30 20 #text="Fun with pyds9" font="times 18 bold"}')
 
# Now you can play in ds9 to your heart's content.
# Check back to see what the current color scale is.
print d.get('scale')
 
# Finally, save your completed image (including regions or labels)
d.set('saveimage png my_pyds9_img.png')

Interpolation

Interpolation: Filter profiles may be reported as transmission vs. wavelength data points. To resample the filter profiles to a different (regularly spaced) wavelength grid, I used scipy's interpolation package.

Interpolation is a used for many astronomical applications. Interpolation is required to combine sub-pixel dithered images or spectroscopy, sample grids of stellar evolution or stellar atmosphere models, calculate extinction from observed extinction curves, and many many more applications. The scipy.interpolate package in python has some nice built-in interpolation functions and I have gathered a few links describing the capabilities (in addition to the documentation). I would recommend using splrep/splev over interp1d for speed.

• Scipy cookbook on interpolation
• A slideshow outlining the interpolation capabilities in Scipy
• A thread explaining terms in spline interpolation/smoothing

Integration

Within Scipy, there is an integrate package with several different functions that perform definite or indefinite integrals. I have only played with these briefly; however, I tested both the scipy.integrate.quad() and scipy.integrate.romberg() functions. These all work in roughly the same way by taking a user-defined function, and the upper and lower boundaries of the integral. Here is a brief example:

from scipy import integrate
def myfunc(x, a, b):
return (x**b) + a
 
# These are the arguments that will be passed as a and b to myfunc()
args = (1.0, -2.0)
 
# Integrate myfunc() from 0.5 to 1.5
results = integrate.quad(myfunc, 0.5, 1.5, args)
print 'Integral = ', results[0], ' with error = ', results[1]

The well-established Fortran QUADPACK integration code lies underneath quad() while romberg() appears to be a python only implementation (??). While I would much rather use quad() for its robustness, it only supports user-functions that accept single value floats; while romberg() will pass in vectors of values. For some of my applications, myfunc() is computationally intensive, so there is an order of magnitude speed gain by having a properly vectorized myfunc() used in vector-form by the integration code. Only with romberg() have I matched the speed of IDL’s qpint1d (from Markwardt library, also based on QUADPACK, but vectorized) for identical myfunc().

There are many applications for taking fourier transforms of images (noise filtering, searching for small structures in diffuse galaxies, etc.). I wanted to point out some of the python capabilities that I have found useful in my particular application, which is to calculate the power spectrum of an image (for later separation of the distribution of stars from the PSF and the noise; see Sheehy et al. 2006).

Below I have posted an example snippet of code, which you can also find on the wiki. But first, some figures to show what we are doing —  an astronomical image (left), the 2D power spectrum of the image (middle), and the azimuthally averaged 1D power spectrum (right).


The bulk of the heavy lifting can be done using SciPy’s fftpack. I also wrote radialProfile.py and it is very crude at the moment.
[code lang="python"]
from scipy import fftpack
import pyfits
import numpy as np
import pylab as py
import radialProfile

image = pyfits.getdata('myimage.fits')

# Take the fourier transform of the image.
F1 = fftpack.fft2(image)

# Now shift the quadrants around so that low spatial frequencies are in
# the center of the 2D fourier transformed image.
F2 = fftpack.fftshift( F1 )

# Calculate a 2D power spectrum
psd2D = np.abs( F2 )**2

# Calculate the azimuthally averaged 1D power spectrum
psd1D = radialProfile.azimuthalAverage(psd2D)

# Now plot up both
py.figure(1)
py.clf()
py.imshow( np.log10( image ), cmap=py.cm.Greys)

py.figure(2)
py.clf()
py.imshow( np.log10( psf2D ))

py.figure(3)
py.clf()
py.semilogy( psf1D )
py.xlabel('Spatial Frequency')
py.ylabel('Power Spectrum')

py.show()
[/code]

If you have suggestions for the above examples or other related code snippets or helper functions, let me know or comment below. I have posted this example on the wiki and will keep it updated with your suggestions.

I recently needed to do some simple 3D plotting in python. The strongest choice as discussed on the web is Mayavi, which is part of the Enthought Python distribution. However, I have a previous python distribution already installed (via scisoft) with all of my favorite packages updated to the latest and greatest versions (e.g. matplotlib, PyEphem, slalib for python, etc.). I wanted to install Mayavi into my existing pythong distribution. In theory this should have been straightforward as there are eggs (a python package zip-type file). However, no matter how hard I tried, I could not get Mayavi, and all its required software, installed properly. Part of this may have been that I was doing a local install without admin privileges.

After spending an entire day with Mayavi issues, just in my attempt to make a very simple 3D scatter plot, I ran across a much easier option. The latest edition of matplotlib (0.99.1) comes with a toolkit called mplot3d. After looking over examples here and here, it took me about 5 minutes to download and install the latest matplotlib version and I was ready to go! Simple 3D figures are a snap to put together and you can interact in order to zoom and rotate around. The toolkit uses matplotlib on the backend, so if you are familiar with all of the plotting options there, then mplot3d follows on logically.

My plot from matplotlib.mplot3d

There are still a few bugs/features that I haven’t quite figured out. Most notably, in wire or surface plots, I can’t seem to set the rstride and cstride (row and column segment step sizes) to anything less than 1. But one of my favorite things about python is that I should be able to go into the source code for mplot3d and figure out what the issue is.

If you have experience with either Mayavi or mplot3d, let us know in the comments when you would need to use Mayavi over the simpler functionality of mplot3d or if you have run into other issues with either.

One of the most popular posts on this blog is Kelle’s reposting of Greenfield and Jedrzejewski’s IDL vs Python earlier this year. I am an avid Python user and have never installed IDL on my computer. Recently, I developed IDLSave, a package to read IDL ‘save’ files into Python, after being sent a few such files by collaborators. In this package, scalars are converted to their respective Python types, while arrays and structures are mapped to Numpy arrays and recarrays respectively. Most importantly, IDLSave does not require an installation of IDL.

With this package, there will be no need to bug your collaborators who send you IDL save files to resend them in a different format. This also means that if you are an IDL user holding back from switching over to Python because you have a large number of legacy IDL save files, you can now read these files directly into Python without first re-converting them to another format.

Using IDLSave is easy. Let’s say you (or a collaborator) writes a save file with

IDL> save, wavelength, flux, file='spectrum.sav'

you will now be able to open this file in Python using

>>> import idlsave
>>> s = idlsave.read('spectrum.sav')

after which you will be able to access the values using s.wavelength and s.flux.

The IDLSave homepage includes download and installation instructions. If you encounter problems reading in save files, don’t give up – please submit a bug report through the bug tracker, and I will look into it! One of the main limitations worth noting is that while it is possible to read save files into Python, it isn’t (currently) possible to write save files for reading back into IDL. It also is not possible to read special variables such as pointers or system variables from save files.

Enjoy!

This is a guest post by Roban Hultman Kramer, who is currently an astronomy grad student at Columbia University and will be starting as a Zwicky Fellow at ETH Zurich in January.

CosmoloPy is a package of basic cosmology routines for Python, designed for use with NumPy and SciPy. Features currently include: calculation of cosmological distance measures, perturbation-theory-based calculation of the matter power spectrum and halo collapse fraction (courtesy of Eisenstein & Hu’s transfer function code), and integration of simple reionization models and the resulting electron scattering optical depth due to reionization. I hope it will prove useful to the increasing number of astronomers turning to Python. Read on to find out more.

Using CosmoloPy is as simple as setting up a dictionary of cosmological parameters (some WMAP5 parameters are provided for convenience in cosmolopy.parameters) and passing it to the functions you want to use. For instance, we could quickly calculate the comoving distance to redshift 6 like this:

>>> import cosmolopy.distance as cd
>>> cosmo = {'omega_M_0' : 0.3,
... 'omega_lambda_0' : 0.7,
... 'h' : 0.72}
>>> d_co, err = cd.comoving_distance(z=6., **cosmo)
>>> print "Comoving distance to z=6 is %.1f Mpc" % (d_co)
Comoving distance to z=6 is 8017.8 Mpc


A few more lines of code, and you can produce a plot like the one shown above.
CosmoloPy is released under the MIT software license, which allows it to be freely used and modified. So please download it, use it, and request or add features you’d like to see included (and point out bugs we need to fix). Anyone can access the git repository, so please edit the code and send in useful changes and additions. All questions about use or development can be posted to the developer discussion group.

Today APLpy 0.9.3 is out! It is a Python plotting package made to generate publication-quality plots in multiple formats such as EPS, PDF, PS, PNG, and SVG.

APLpy was created by Thomas Robitaille and Eli Bressert, who come from a Fortran and IDL background. With Python’s ease of use, portability, and programming they decided to make a plotting package for astronomers. APLpy’s objective is easy usage with great looking publication plots. In other words, more bang for the buck. Here’s an example of what an APLpy plot looks like:

A color generated APLpy plot from FITS files with grid lines.

To see more examples and how to make the plots in Python check out this link.

APLpy has quite a few features (some listed below – from the APLpy site) and is continuing to expand. Fortunately, there’s good documentation to accompany the features.

• Make plots interactively or using scripts
• Show grayscale, colorscale, and 3-color RGB images of FITS files
• Generate co-aligned FITS cubes to make 3-color RGB images
• Overlay any number of contour sets
• Overlay markers with fully customizable symbol
• Plot customizable shapes like circles, ellipses, and rectangles
• Overlay coordinate grids
• Customize the appearance of labels and ticks
• Hide, show, and remove different contour and marker layers
• Pan, zoom, and save any view as a full publication-quality plot
• Save plots as EPS, PDF, PS, PNG, and SVG

The plotting package is under active development and there has been extensive interaction between the users and developers. We should be seeing some exciting features added to APLpy in the next few releases.

For those of you who use scisoft, IRAF, PyRAF, Python, etc. there has been an update of the Scisoft package for Macs. This package is a very nice way to install some of the most commonly used astronomy software tools (at least in optical/IR). The Urania blog as a breakdown of some of the updates:

Urania Blog – Scisoft OSX 2009.5.2 Released

Here is a really nice listing of the pros and cons of IDL and Python for astronomers. It’s an Appendix of the Python tutorial Using Python for Interactive Data Analysis by Greenfield and Jedrzejewski at STScI. Data and scripts for the examples and exercises in the tutorial are available at scipy.org.

Update 28 Jan 2010: We’ve transferred this article to the wiki so that it can be updated as Python evolves.

Why would I switch from IDL to Python (or not)?

by Greenfield and Jedrzejewski

We do not claim that all, or even most, current IDL users should switch to using Python now. IDL suits many people’s needs very well and we recognize that there must be a strong motivation for starting to use Python over IDL. This appendix will present the pros and cons of each so that users can make a better informed decision about whether they should consider using Python. At the end we give a few cases where we feel users should give serious consideration to using Python over IDL.

Pros and Cons are addressed below in a comparative sense. Attributes that both share, e.g., that they are interpreted and relatively slow for very simple operations, are not listed.

Pros of IDL

• Mature many numerical and astronomical libraries available
• Wide astronomical user base
• Numerical aspect well integrated with language itself
• Many local users with deep experience
• Faster for small arrays
• Easier installation
• Good, unified documentation
• Standard GUI run/debug tool (IDLDE)
• Single widget system (no angst about which to choose or learn)
• SAVE/RESTORE capability
• Use of keyword arguments as flags more convenient

Cons of IDL

• Narrow applicability, not well suited to general programming
• Slower for large arrays
• Array functionality less powerful
• Table support poor
• Limited ability to extend using C or Fortran, such extensions hard to distribute and support
• Expensive, sometimes problem collaborating with others that don’t have or can’t afford licenses.
• Closed source (only RSI can fix bugs)
• Very awkward to integrate with IRAF tasks
• Memory management more awkward
• Single widget system (useless if working within another framework)
• Plotting:
  • Awkward support for symbols and math text
  • Many font systems, portability issues (v5.1 alleviates somewhat)
  • not as flexible or as extensible
  • plot windows not intrinsically interactive (e.g., pan & zoom)

Pros of Python

• Very general and powerful programming language, yet easy to learn. Strong, but optional, Object Oriented programming support
• Very large user and developer community, very extensive and broad library base
• Very extensible with C, C++, or Fortran, portable distribution mechanisms available
• Free; non-restrictive license; Open Source
• Becoming the standard scripting language for astronomy
• Easy to use with IRAF tasks
• Basis of STScI application efforts
• More general array capabilities
• Faster for large arrays, better support for memory mapping
• Many books and on-line documentation resources available (for the language and its libraries)
• Better support for table structures
• Plotting
  • framework (matplotlib) more extensible and general
  • Better font support and portability (only one way to do it too)
  • Usable within many windowing frameworks (GTK, Tk, WX, Qt…)
  • Standard plotting functionality independent of framework used
  • plots are embeddable within other GUIs
  • more powerful image handling (multiple simultaneous LUTS, optional resampling/rescaling, alpha blending, etc)
• Support for many widget systems
• Strong local influence over capabilities being developed for Python

Cons of Python

• More items to install separately
• Not as well accepted in astronomical community (but support clearly growing)
• Scientific libraries not as mature:
  • Documentation not as complete, not as unified
  • Not as deep in astronomical libraries and utilities
  • Not all IDL numerical library functions have corresponding functionality in Python
• Some numeric constructs not quite as consistent with language (or slightly less convenient than IDL)
• Array indexing convention “backwards”
• Small array performance slower
• No standard GUI run/debug tool
• Support for many widget systems (angst regarding which to choose)
• Current lack of function equivalent to SAVE/RESTORE in IDL
• matplotlib does not yet have equivalents for all IDL 2-D plotting capability (e.g., surface plots)
• Use of keyword arguments used as flags less convenient
• Plotting:
  • comparatively immature, still much development going on
  • missing some plot type (e.g., surface)
  • 3-d capability requires VTK (though matplotlib has some basic 3-d capability)

Specific cases where using Python provides strong advantages over IDL

• Your processing needs depend on running a few hard-to-replicate IRAF tasks, but you don’t want to do most of your data manipulation in IRAF, but would rather write your own IDL-style programs to do so (and soon other systems will be accessible from Python, e.g., MIDAS, ALMA, slang, etc)
• You have algorithms that cannot be efficiently coded in IDL. They likely won’t be efficiently coded in Python either, but you will find interfacing the needed C or Fortran code easier, more flexible, more portable, and distributable. (Question: how many distributed IDL libraries developed by 3rd parties include C or Fortran code?) Or you need to wrap existing C libraries (Python has many tools to make this easier to do).
• You do work on algorithms that may migrate into STSDAS packages. Using Python means that your work will be more easily adapted as a distributed and supported tool.
• You wish to integrate data processing with other significant non-numerical processing such as databases, web page generation, web services, text processing, process control, etc.
• You want to learn object-oriented programming and use it with your data analysis. (But you don’t need to learn object-oriented programming to do data analysis in Python.)
• You want to be able to use the same language you use for data analysis for most of your other scripting and programming tasks.
• Your boss makes you.
• You want to be a cool, with-it person.
• You are honked off at ITT Space Systems/RSI.

Obviously using a new language and libraries entails time spent learning. Despite what people say, it’s never that easy, especially if one has a lot of experience and code invested in an existing language. If you don’t have any strong motivations to switch, you should probably wait.

Reproduced with permission. Copyright 2007, Association of Universities for Research in Astronomy, Inc (AURA).

Update 28 Jan 2010: We’ve transferred this article to the wiki so that it can be updated as Python evolves.