Read IDL ‘save’ files into Python

by Tom on November 24, 2009

python-logo-new2One of the most popular posts on this blog is Kelle’s reposting of Greenfield and Jedrzejewski’s IDL vs Python earlier this year. I am an avid Python user and have never installed IDL on my computer. Recently, I developed IDLSave, a package to read IDL ‘save’ files into Python, after being sent a few such files by collaborators. In this package, scalars are converted to their respective Python types, while arrays and structures are mapped to Numpy arrays and recarrays respectively. Most importantly, IDLSave does not require an installation of IDL.

With this package, there will be no need to bug your collaborators who send you IDL save files to resend them in a different format. This also means that if you are an IDL user holding back from switching over to Python because you have a large number of legacy IDL save files, you can now read these files directly into Python without first re-converting them to another format.


Using IDLSave is easy. Let’s say you (or a collaborator) writes a save file with

IDL> save, wavelength, flux, file='spectrum.sav'

you will now be able to open this file in Python using

>>> import idlsave
>>> s = idlsave.read('spectrum.sav')

after which you will be able to access the values using s.wavelength and s.flux.

The IDLSave homepage includes download and installation instructions. If you encounter problems reading in save files, don’t give up – please submit a bug report through the bug tracker, and I will look into it! One of the main limitations worth noting is that while it is possible to read save files into Python, it isn’t (currently) possible to write save files for reading back into IDL. It also is not possible to read special variables such as pointers or system variables from save files.

Enjoy!

{ 13 comments… read them below or add one }

1 Diego Bermejo April 27, 2012 at 10:16 am

Hi,
thanks a lot for the tool you’ve developed for reading IDL sav files.
I would like to ask you how do you manage to concatenate two or more output variables of the idlsave procedure preserving the field names. Using the numpy functions “r_” or “concatenate” the field names are lost.
Looking forward to your answer and thanks again.
Diego

Reply

2 Tom April 27, 2012 at 10:24 am

@Diego: Do you mean specifically variables that are structured arrays? (i.e. structures from IDL). If so, then this might be helpful: http://stackoverflow.com/questions/5355744/numpy-joining-structured-arrays (in particular the last answer on that page)

Reply

3 Diego Bermejo April 27, 2012 at 3:37 pm

@Tom: yes, I mean IDL structures. My question is about how to join 2 (or more) IDL structures, which have the same fields, so that the resulting variable keep the same fields (and their names) of the original IDL structure. The link you suggest could be useful, however, the functions in there try to add the fields of every IDL structure as new fields, instead of joining those fields which have the same name. The output sends an error message as: “the are two fields with the same name”.
Example:
What I have:
var1={field1 (string1), field2 (array1)}
var2={field1 (string2), field2 (array2)}
What I want:
var=[var1, var2]
var.field1=[string1, string2] ; var[0].field1= string1; var[1].field1=string2
var.field2=[array1, array2] ; var[0].field2= array1; var[1].field2=array2
What it seems to happen:
var={field1, field2, field1, field2} –> subsequent error: “field1 (first)” has the same name of “field1” (second).

Thanks for your help.
Diego

Reply

4 Tom April 29, 2012 at 3:32 pm

I just wrote a function to easily merge two structured arrays: https://gist.github.com/2552867 – can you try it out?

Reply

5 Diego Bermejo May 3, 2012 at 6:38 am

@Tom: that’s right, your function works perfect. I’m sorry the delay in my answer, but I had some vacation days in Spain and I didn’t check the mail. I’m new in Python and I might ask silly questions. I see from your function that the key was in the ‘resize’ function. Now, I’m going to learn how to create attributes, so that, given the structured array:
a = array(zip([1,2,3],[‘a’,’b’,’c’]), dtype=[(‘col1’, int), (‘col2’, ‘S1’)])
I can type: print a.col1
and get:
[1,2,3]

thanks a lot Tom.
Diego

Reply

6 Tom May 3, 2012 at 8:38 am

@Diego – here’s how to convert a structured array into a record array (which allows attributes):

a = np.array(zip([1,2,3]), dtype=[(‘a’,int)])
b = a.view(np.recarray)
print b.a

Reply

7 Diego Bermejo May 4, 2012 at 5:02 am

@Tom: thanks for your very useful tutorial.

8 Diego Bermejo May 4, 2012 at 5:02 am

@Tom
thanks for your very useful tutorial. Diego

Reply

9 David June 18, 2012 at 12:54 pm

Tom,

Is there a way to be able to write to an IDL save file from Python?

Reply

10 Tom June 19, 2012 at 9:48 am

No, there is no support for writing IDL save files, and I don’t have plans to implement it. However, the format is documented, so it should be possible if anyone wants to give it a shot (if you do, please let me know!)

11 Max October 16, 2014 at 9:03 am

Thanks a lot for this!
However. I’ve got an error when tried to read a *.sav file: UNKNOWN RECTYPE 20
After adding to idlsave.py RECTYPE_DICT[20] = “UNKNOWN” it works fine. It looks that I can read all the data I need.

Reply

12 Parke October 16, 2014 at 7:29 pm

There is also a SciPy module for this now: scipy.io.readsav. It creates a dictionary from the .sav file instead of an object like Tom’s code. Although we are all indebted to Tom for generously making his code available some five years ago, the SciPy option might be preferable just for portability.

http://docs.scipy.org/doc/scipy/reference/generated/scipy.io.readsav.html#scipy.io.readsav

Reply

13 Vikash September 7, 2017 at 11:05 pm

How can we read huge binary files, when the size is in tens of GB?

Reply

Leave a Comment

Previous post:

Next post: