Options for Saving in Python

Question: 

What is the best way to store data that is easily readable and writable in Python?

Answer:

There isn’t a single correct answer as it will largely depend on how large your data set is, how fast you have to read/write it, and whether it needs to be readable by other applications or languages. Below I have compiled a few suggestions and example code snippets.

astropy

For most tabular data, this is probably your best bet. Astropy is the future “do-everything” python package for astronomy. It contains the astropy.table subpackage, which will read in data in many different formats (ASCII, FITS, HDF5, SQL, etc.) and write it out in many different formats. As a general rule of thumb, FITS (or another binary file format) is a better option than ASCII for saving tabular data. (Just trust us on this one, the justification is worthy of its own post.) The columns are numpy arrays, so they are easy to work with. These formats are relatively compact and very quick to read compared to ASCII. This should replace all uses of atpy (see below).

import astropy.table
import astropy.units as u
import numpy as np
 
# Create table from scratch
ra = np.random.random(5)
t = table.Table()
t.add_column(table.Column(name='ra', data=ra, units=u.degree))
 
# Write out to file
t.write('myfile.fits')  # also support HDF5, ASCII, etc.
 
# Read in from file
t = table.Table.read('myfile.fits')

numpy

If your data is all numbers and not too big, then you can save to a numpy (*.npy) file. For example, a 2D image stored as a numpy array. This method also supports record arrays (arrays with column names). One caution is that future computer upgrades that might change the default size of a python “float” might require special handling if you read in old numpy save files.

import numpy as np
 
# Create column (or array)
ra = np.random.random(size=(1000,1000,5))
 
# Save to file
np.save(filename, ra)
 
# Read from file
ra = np.load(filename)

pickle and cPickle

If you have something more complicated than a table or a numpy array, than you probably want to pickle it. An excellent pickle tutorial for intermediate users (e.g. pickle the “right way”) is presented Doug Helman’s Blog.

import pickle  # or import cPickle as pickle
 
# Create dictionary, list, etc.
favorite_color = { "lion": "yellow", "kitty": "red" }
 
# Write to file
f_myfile = open('myfile.pickle', 'wb')
pickle.dump(favorite_color, _myfile)
f_myfile.close()
 
# Read from file
f_myfile = open('myfile.pickle', 'rb')
favorite_color = pickle.load(f_myfile)  # variables come out in the order you put them in
f_myfile.close()

atpy

This used to be my preferred package; but it has been merged into astropy as astropy.table (described above) and is no longer supported. I have included an example here for completeness. We wrote up an earlier AstroBetter Blog Post with a quick tutorial on atpy.

import atpy
import numpy as np
 
# Create table from scratch
ra = np.random.random(5)
t = atpy.Table()
t.add_column('ra', ra, unit='deg')
 
# Write table to file
t.write('myfile.fits')  # also supports HDF5, ASCII, etc.
 
# Read table from file
t = atpy.Table('myfile.fits')

Which method do you prefer and why? If you’re not sure which one to use, ask in the comments!