Mike Data Reduction Cookbook v 1.6; December 2005 (MC) : Installation & Setup Notes for Mac OSX 10.6.7 system

I. Mike Suggested Calibrations

  1. 10 Bias (0s) frames
  2. 10 trace flat images (per setup and side)

    • Note: It is ideal to take a set of Arcs right before and/or after creating the trace flat images
  3. 10 Milky Flats (per setup and side)
  4. Arc Calib

    • One pair taken at the same time as the trace flat
    • One image taken right before (or after) each science exposure
  5. One Standard Star (each slit size)

II. Setup: Pre-Reduction

  1. Computer Resources

    • >300 MHz Processor
    • 5G disk space / night
    • 750M RAM (1G on Solaris)
    • Linux or Solaris
  2. Solaris WARNINGS (These don't apply to Linux)

    • Solaris + IDL does not release memory until IDL is exited
    • Consider exiting IDL occasionally and monitor memory usage
    • IDL + Solaris freezes up on the CPU intermittently

      • Possibly bug in licensing software
      • Forced to kill IDL process and restart
  3. Software

    • gcc or cc
    • gfortran

      • I just chose a generic download for mac like this one, followed the instrutions, and simply put it in my path (ex: /usr/local/bin/gfortran/) 
    • IDL v5.4 or higher

      • Downloading IDLUTILS, IDLSPEC2D, XIDL

        •  Go here and grab the 'MacOSX 10.6; 64bit processor: TARBALL 240Mb; February 11, 2011' 
      • Downloading MIKE

        • Go here and grab the most up to date .tar.gz file 
      • Setting up IDLUTILS, IDLSPEC2D, XIDL, and MIKE

        • Move both .tar.gz(s) into your main IDL directory (eg  ~/Code/IDL/)
        • Untar the files
        • Set up environmental variables necessary for future use of MIKE. I added mine right to my .cshrc file. the specific path will vary from user to user but mine were: 
        1. $setenv IDL_DIR "/Applications/itt/idl71"          
        2. $setenv IDLUTILS_DIR "/Users/emilylemonier/Code/IDL/idlutils"
        3. $setenv IDLSPEC2D_DIR "/Users/emilylemonier/Code/IDL/idlspec2d"
        4. $setenv XIDL_DIR "/Users/emilylemonier/Code/IDL/xidl"
        5. $setenv MIKE_DIR "/Users/emilylemonier/Code/IDL/MIKE
        6. $setenv IDL_PATH "/Applications/itt/idl71/lib/:/Applications/itt/idl71/examples/:$IDL_DIR/Default/:+$IDLUTILS_DIR/pro/:$IDLUTILS_DIR/lib/:+$IDLUTILS_DIR/goddard/pro/:+$IDLSPEC2D_DIR/pro/:+/Users/emilylemonier/Code/IDL/MIKE/pro/:/Users/emilylemonier/Code/IDL/MIKE/doc/:+/Users/emilylemonier/Code/IDL/xidl/:$IDL_DIR/lib/obsolete/"IDL_DIR/Default/ $IDLUTILS_DIR/pro/ $IDLSPEC2D_DIR/pro/ $MIKE_DIR" 
        • Building the Code (IDLUTILS, IDLSPEC2D, XIDL)
    • $cd $IDLUTILS_DIR
    • $bin/evilmake clean       #clean directory
    • $bin/evilmake all           #detect operating system
    • $cd $IDLSPEC2D_DIR
    • $IDUTILS_DIR/bin/evilmake clean
    • $IDUTILS_DIR/bin/evilmake all
  3. XIDL
    • cd $XIDL_DIR
    • $bin/evilmake clean      
    • $bin/evilmake all
III. Setup: Initial (Repeat for each night data was collected)
  1. Create a new directory for the night (eg MIKE_060704) in the folder within your main IDL directory  (eg  ~/Code/IDL/Default). Enter the new directory.
  2. Create a 'Raw/' directory and put all the raw data in it
  3. If the files do not read mb(r)####.fits or b(r)####.fits it is quite likely the code will be unhappy

    1. r signifies red, b signifies blue
    2. mine read r####.fits and it worked fine
  4. gzip the data (gzip *.fits)
  5. Launch idl in the directory above 'Raw/' (I used 7.1.1 64-bit idl, though MC says to use 32-bit) 
  6. MIKE_STRCT - Create MIKE Structure and organize the night's data

    • Note:

      • Here is the pro file which defines the structure and lists the tags 
    • Calling Sequence: IDL> mike_strct, name

      • 'name', or whatever you put after 'mike_strct,' is the variable to which strct.fits is written, and can be called whatever you would like
    • Output:

      • An IDL structure in memory with whatever name you choose (eg name)
      • A fits version of the structure - 'mikestrct.fits'
      • An ASCII version of the fits file which lists some of the tag values - 'mike.list'
    • You know the code is working propperly when:

      • It runs through the completion after checking each Raw file
      • It launches the edit structue GUI (unless supressed)
      • The majority of files are correctly identified by type. The code won't get them right all of the time however, and you might have to edit the structure on the command line or through making a script
  7. MIKE_EDITSTRCT - Modify (if necessary) some MIKE Structure tags with GUI

    • Note:

      • I found the GUI to be unreliable. Instead I used mike_editstrct to view the main tags and then edited the structure by writing a simple script
    • Calling Sequence: IDL> mike_editstrct, name
    • Output: n/a
    • You know the code is working properly when: the GUI opens up
    • Obvious tags to modify are:

      • Obj - Object name (this propagates into the final spectra and should haveno spaces)
      • flg_anly - Include in analysis (default is 1 for yes, but if you do not want to use a data file for analysis, set its flg_anly to 0) 
      • type

        • ARC = ThAr exposure
        • TFLT = Trace flat (ideally with twilight light NOT internal)
        • MFLT = Milky flat (i.e. with diffuser)
        • TWI = Twilight exposure (not used)
        • OBJ = Object exposure
        • STD = Standard Star
        • ZRO = Bias Frame
      • Setup - Unique integer for each instrument setup including slit width (ex: 1, 2, etc)
  8. MIKE_UPDSTRCT - Append new files to current MIKE Structure

    • Note:

      • This is mainly useful at the telescope as you are taking new data, I personally didn't use it
    • Calling Sequence: IDL> mike_updstrct, name
    • Output: Searches the Raw directory for new files, and adds them to the structure
    • You know the code is working properly when: it finds the right fiels
  9. MIKE_WRSTRCT  - (Re)Save the structure in fits form and (Re)Write MIKE ASCII file

    • Note:

      • Useful for when you modify tag values
      • Alternative Call sequence: IDL> mike_wrstrct, name, FITS = 'mike_name.fits'  -  this saves the structure to a fits file other that the default of 'struct.fits', but I'm not sure of the benefits in doing so 
    • Calling Sequence: IDL> mike_wrstrct, name
    • Output: Updated fits & ASCII forms of MIKE structure ('strct.fits' & 'mike.list')
    • You know the code is working properly when:
  10. MIKE_AR - Read MIKE Structure from disk into a variable (ex: name, mike, etc)

    • Note:

      • If you quit or crash IDL, you will need to use mike_ar() to read the structure into a variable
    • Calling Sequence: IDL> name = mike_ar('strct.fits') or IDL> name = mike_ar('mike_name.fits')
    • Output: MIKE structure will be written into a variable
    • You know the code is working properly when: 'IDL> help, name, /str' lists the structure
  11. MIKE_SETUP - This routine examines the mike structure and looks for calibration files associated with the various setups (a unique setup is defined by the slit width, binning, and angles of the echelle and prism, and obviously-each setup requires its own set of calibration files). It also groups together exposures with idential Obj names (mike.obj) and sets the obj_id tags accordingly

    • Before running the routine: The user must specify a setup number (>0) for each file to be analyzed.

      • If you have observed with only one setup during the night, you may just do: $IDL> mike.setup  = 1 (this is what I did)
      • Or if you observed with more than one setup during the night you can specify ranges in the mike structure: $IDL> mike20:30.setup = 2 (sets the 11 frames 0-indexed from 20-30 to setup 2)
    • Note:

      • This program is useful to run if you happen to crash IDL before saving the MIKE structure
    • Calling Sequence: $IDL> mike_setup, name
    • Output:

      • A summary of the MIKE exposures is put in 'mike_summ.txt'
    • You know the code is working properly when: you check the output file, 'mike_summ.txt' and everything is properly setup. You may modify the setup and obj_id tags and rerun mike_setup as necessary

IV. Calibrations

  1. MIKE_MKBIAS  - Create Bias (Zero) Frames: (I didn't do this step)
  2. MIKE_SETGAIN  - Loops on the Milky Flats and performs statistics (in mike_gain) to determine th gain and apply it to the gain tag

    • Note: n/a
    • Calling Sequence: $IDL>mike_setgain, name, setup, side
    • Output: the value of the gain is store in name.gain
    • You know the code is working properly when:

      • the routine loops through all of the milky flat files and reports gain values within the ballpark of the published values
      • You can also print out the gain tags after completion: $IDL>print, mike.gain
  3. Process Flats - these routines process the Milky & Trace flats to create a response image (pixel by pixel variations) and to determine the curvature of the orders on the CCD. There are two methods to doing this:

    1. MIKE_ALLFLAT - identifies and processes the milky flats (runs mike_mkmflat, mike_mktflat, and mike_edgeflat all together)

      • Note: Before you call this routine, you should read the documentation for the second option of running each command individually, and make sure you understand each routine
      • Calling Sequence: $IDL> mike_allflat, name, setup, side, /INTER
      • Output: The same as the three individual routines if mike_mkmflat, mike_mktflat, mike_edgeflat
      • You know the code is working properly when: use the same checks as the individual routines below
    2. Run each command individually:

      1. MIKE_MKMFLAT - Creates a stacked, normalized milky flat to correct pixel resopnse variations

        • Note:

          • If your milky flats have many absorption features in them (not recommended) then one should call this routine with the /SMOOTH keyword
        • Calling Sequence: $IDL> mike_mkmflat, name, setup, side, /SMOOTH
        • Output: 'Flats/Flat_B_01_M.fits' and/or 'Flats/Flat_R_01_M.fits'  
        • You know the code is working propperly when:

          • you see the routine loop through the various milky flats
          • Check the flat with: $IDL> xatv, 'Flats/Flat_B_01_M.fits'
      2. MIKE_MKTFLAT - Combines the series of trace flats (currently twilight flats) to create one S/N image for order and slit tracing

        • Note: Because of the thermal gradients, it is highly recommended that one use ony a coeval set of trace flats
        • Calling Sequence: $IDL> mike_mktflat, name, setup, side
        • Output: 'Flats/Flat_B_01_T.fits' and/or 'Flats/Flat_R_01_T.fits'   
        • You know the code is working propperly when:

          • you see the routine process and stack the various trace flat files
          • Check the trace flat with: $IDL> xatv, 'Flats/Flat_B_01_T.fits'
      3. MIKE_EDGEFLAT - Traces the trace flat created in mike_mktflat to determine the order curvature and return a smooth fit

        • Note:

          • At present this is one of the weaker spots in the automated pipeline. This is especially true given that there is no 'standard' prescription for creating trace flats with MIKE. The main failing is that the code may miss or misidentify the slit edges. As such, we highly recommend you run this routine with the /INTER and (or) /CHK keywords to interactively check the order edges.
        • Calling Sequence: $IDL> mike_Edgeflat, name, setup, side, /INTER, /CHK
        • Output: 'QA/Flats01/qa_trcflt_01B.ps' and/or 'QA/Flats01/qa_trcflt_01R.ps' 
        • You know the code is working propperly when:

          • 'QA/Flats01/qa_trcflt_01B.ps' and/or 'QA/Flats01/qa_trcflt_01R.ps' (1) have a small reduced_chi

            2 (<.005) and (2) the XO & PCA0 values should vary significantly but be well fit while the PCA1-4 values should have minor variations
          • You can also check the tracing with MIKE_CHKTRCFLAT

            • $IDL> 'xatv, 'Flats/Flat_B_01_T.fits'
            • $IDL>mike_chktrc, name, setup, 1, /NOSTOP, /FIT
  4. Arc Images - Processing the Arcs

      • There are two ways you can run through the Arc Processing Routines:
      1. You can run through all 7 routines in 1 by using MIKE_ALLARC
      2. You can run through the 7 arc routines individually
      • Option 1 is highly recommended unless you need to debug - it is even possible that some procedures won't run on their own at the present time.
    1. MIKE_ALLARC - runs through all the Arc processing routines in one go

      • Note: You can run the routines on individual arcs or process them all together (the latter is recommended)
      • Calling Sequence:

        • All Arcs: $IDL>mike_allarc, name, setup, (side)
        • One Arc: $IDL> mike_allarc, name, index, /INDX

          • The user inputs the indice(s) of the arc(s) to process 
      • Output: The same outputs as listed below in the individual routines in Option 2
      • You know the code is working properly when: It runs through the code without stopping

        • You can also use the 'you know the code is working properly when....' for the indiviudal routines listed below in Option 2 after you run mike_allarc
    2. Run each routine individually

      1. MIKE_SETARCM - Measures the curvature of the arc lines

        • Note: n/a
        • Calling Sequence: $IDL> mike_setarcm, raw_fil, setup, side

          • where raw_fil is the name of an Arc file, eg. 'Raw/mb0428.fits'
        • Output: n/a
        • You know the code is working propperly when: it runs to completion
      2. MIKE_PROCARC - Bias subtracts and flat fields the arc image

        • Note: n/a
        • Calling Sequence:

          • All arcs: $IDL> mike_procarc, name, setup, obj_id, side
          • One arc: $IDL> rslt = mike_procarc_sngl('raw####.fits', setup, side)
        • Output: fits files in the 'Arcs/' directory
        • You know the code is working properly when:

          • it runs to completion
          • You can check the fits files by calling: $IDL>xatv, 'Arcs/Arc_####.fits.gz'
      3. MIKE_ARCALIGN - Finds the offset between the Arc line profies and the trace flat profiles

        • Note: n/a
        • Calling Sequence: $mike_arcalign, name, setup, side, /CHK
        • Output: the offset between the arc line & trace line profiles are written to the arc_xyoff tag
        • You know the code is working properly when:

          • the output to the screen shows a reasonable boxcar profile and he fractional shifts (indicated y the vertical blue line) are smaller than .2 or so
          • The total shifts should by on the order of 2 unbinned pixels for the blue and less than 1 unbinned pixel for the red
      4. MIKE_FITARC - Extracts a 1D spectrum down the center of each order and determines the wavelength solution

        • Note:

          • It is common for the code to fail for the very bluest orders of the blue side (at least with the original blue CCD)
          • There are three levels of interaction with this routine

            • The most interaction is to use /INTER which prompts the user to identify and fit the Arc lines
            • The next is to use /PINTER which has the program attempt to identify a set of lines in each order. The user than interactively fits the lines using the routine 'x_identify' which calls 'x1dfit'
            • Normal full auto mode, where the routine is just allowed to be called normally
        • Calling Sequence:

          • All arcs: $IDL> mike_fitarc, name, setup, obj_id, side, /INTER, /PINTER, /PSOUT
          • One arc: $IDL> rslt = mike_fitarc_work('arc_file', setup, side)
        • Output: QA file(s) (eg: 'QA/Arcs01/qa_arcfit_mb0038.ps.gz') which shows the residuals to the order by order fits including the RMS (in pixels)
        • You know the code is working properly when:

          • The QA files show an RMS of the fits wto each order which are generally less than .1 pixel. There will be several exceptions per full exposure
      5. MIKE_FIT2DARC - Fits a 2D solution to the wavelengths as a function of their y pixel value and their order number

        • Note: n/a
        • Calling Sequence:

          • All Arcs: mike_fitarc, name, setup, obj_id, (side)
          • One Arc: rslt = mike_fit2darc_work('arc_file', setup, side)
        • Output: QA file(s) (eg: 'QA/Arcs01/qa_arcfit_mb0038.ps.gz') whih shows the 2D solution and lists the RMS
        • You know the code is working properly when:

          • the RMS of the 2D fit is less than .1 pix or so
      6. MIKE_TRACEARC - Traces the lines in the curved order frame of the original image and then fits a straight line to each arc line

        • Note:

          • the slope and centroid of the arc line is recorded to a file for later use
        • Calling Sequence:

          • All Arcs: $IDL> mike_tracearc, name, setup, obj_id, (side)
          • One Arc: $IDL> rslt = mike_tracearc_work('arc_fil, setup, side)
        • Output: QA files (eg: 'QA/Arcs01/qa_tracearc_mb0034.ps.gz'
        • You know the code is working properly when:

          • The arc lines and the fits in the QA files it produces look reasonably straight
          • You can also check the output by using MIKE_CHKARCTRC

            • $IDL>xatv, arc_img
              $IDL>mike_chkarctrc, mike, index
      7. MIKE_FITTRCARC - fits the slope of the arc lines as a function of order number and y position on the CCD, which is then used to construct a 2D wavelength image

        • Note: n/a
        • Calling Sequence:

          • All Arcs: $IDL> mike_fittrcarc, mike, setup, obj_id, (side), /CLOBBER, /CHK 
          • One Arc: $IDL> rslt = mike_fittrcarc_work('arc_file', setup, side, XOFF=)
        • Output: QA files (eg: 'QA/Arcs01/qa_fittrcarc_mb0038.ps.gz')
        • You know the code is working properly when:

          • the QA files shows the RMS3 is less than .01
      8. MIKE_MKAIMG - Given the 2D solution for the slope of the lines as a funciton of position, this code creates a wavelength image (ie assigns a unique wavelength to each pixel in each orer)

        • Note: n/a
        • Calling Sequence:

          • All Arcs: $IDL> mike_mkaimg, name, setup, obj_id, (side), /CHK
          • One Arc: $IDL> mike_mkaimg_work('arc_file', setup, side, XOFF=)
        • Output: a series of fits files, (eg: 'Arcs/Arc_####I.fits')
        • You know the code is working properly when:

          • The code doesn't complain about an order not being single-valued in wavelength
          • The BAD_ANLY parameter should be 0 upon completion
  5. MIKE_SLITFLAT -  determines the slit profile for each order

    • Note:

      • The procedure must be run after the arc calibrations as the slope of the arc lines is a necessary input
      • If you have observed the twilight sky, you will probably want to turn the /NODETILT switch on (the default mode is to flatten the counts along the slit)
    • Calling Sequence: $IDL> mike_slitflat, name, setup, (side), (/chk), (/nodetilt)
    • Output: QA files (eg: 'QA/Flats01/qa_slitflat_01B.ps.gz')
    • You know the code is working properly when: the slit profiles on the QA files look reasonably well behaved and the chi2 values are around .5

V. Extraction

  • Note:

    • The following routines all apply to a single object, i.e. multiple exposures of that object will be reduced together
    • Most of the following routines take the mike structure, setup, side, and obj_id tags
  • There are two options for performing the extraction:

    1. MIKE_ALLOBJ - runs the processing, CR rejection, tracing, sky subtraction, and extraction routines in one.

      • Note:

        • Only processes one object (with or without multiple exposures),
        • does not allow for fluxing
      • Calling Sequence: $IDL> mike_allobj, name, 1, /procall, /nocr
      • Output:
      • You know the code is working properly when:
    2. Run each routine individually

      1. Process the Image

        1. MIKE_PROC - Bias subtracts and flat fields the Raw image

          • Note: n/a
          • Calling Sequence:

            • All exposure for 1 object: $IDL> mike_proc, name, SETUP= setup, OBJ= obj_id 
            • 1 exposure for 1 object: $IDL> rslt = mike_proc_sngl('rawfil', (side))
          • Output: A flattened flux and inverse variance fitz file (one gzipped fits image with two extensions perfile). 'Final/f_name.fits'
          • You know the code is working properly when:

            • You check the output of the fits file with: xatv, 'Final/f_name.fits' and make sure the image looks reasonable
        2. MIKE_OBJCR - compares two or more images to identify cosmic rays

          • Note: Optional, recommended with caution

            • You should NOT use use this on multiple exposures of the same object if the object has moved along the slit or if thermal expansion has resulted in a large shift in the data
          • Calling Sequence: $IDL> mike_objcr, name, setup, obj_id, side, (exp), (/NOCHK)
          • Output: n/a
          • You know the code is working properly when: you run the code with the /CHK keywords and the code is not identifying excessive CR's along the object profile
      2. Identify and Trace the Object

        1. MIKE_FNTOBJ - identifies the object in each order and than traces it

          • Note:
          • Calling Sequence: $IDL> mike_fntobj, mike, setup, obj_id, side, /CHK
          • Output:

            • Creates the object structure 'mikeobstruct' that contains the trace and will contain 1D extractions (eg: 'Extract/Obj_name.fits') which holds information relating to the trace and extraction of the spectra

              • The code which lists the tags used is outlined in: objstruct_define.pro
          • You know the code is working properly when:

            • you run with the /CHK keyword and the fit to the position in the slit fraction is nearly constant and well fit as a function of trace number & The image with the trace overplotted should look sensible
            • The QA output (eg: 'QA/Obj01/qa_fntobj_mr0022.ps.gz') should have:

              • XO & PCA0 coefficients which are well fit
              • the remaining PCA coefficients should show small variation
              • the reduced chi

                2 should be <.1 (blue) or <.01 (red)
      3. Sky Subtraction

        1. MIKE_SKYSUB - pretty self explanitory...performs sky subtraction

          • Note:
          • Calling Sequence: $IDL> mike_skysub, mike, setup, obj_id, side, (/CHK), (/FCHK)
          • Output:

            • Sky subtracted 2D image appended to the unsubtracted flux and variance 2D images (eg: 'Final/f_name.fits')

              • You can examine the final product (and get wavelength info) by: $IDL> xatv, 'Final/f_name.fits', getsky=2, WVIMG='Arcs/Arc_nameI.fits'
            • Bspline coefficients are written to 'Sky/sky_name.fits'
          • You know the code is working properly when:

            • you can run with /FCHK and the final image shows 'random' noise in places where sky lines used to be
            • you can run with /CHK to examine the order by order fits
            • you can examine the QA file (eg: 'QA/Obj01/qa_skysub_mr0022.ps.gz') and it shows a zoom in on key sky lines in the red and several patches in the blue
      4. Extraction

        1. MIKE_BOX - This routine does many things, I would recommend reading over the secifics in the IDL Help for Mike (click on MIKE_BOX). Generally this routine: Extracts 1D spectra from 1D images, and extracts the object's flux

          • Note: n/a
          • Calling Sequence: $IDL> mike_box, name, setup, obj_id, side, /RESCHK
          • Output:

            • 1D spectral tags in the object structure (defined in objstruct_define.pro): .box_fx, .box_var, .box_wv, .fx, .var, .wave
          • You know the code is working properly when:

            • Check the obj_id tags and mae sure they appear to hold reasonable values
            • Examine the final product: $IDL> mike_specplt, (/BOX)

              • the reduced chi2 values are 1.2 or lower (a bit higher for very bright objects)
      5. Standard Star (Optional, and recommended) .....I didn't complete this step

VI. Flux Calibration and Coding

  1. Flux

    1. MIKE_FLUX - Fluxes the spectrum using a standard star calibration

      • Note:

        • The default standard star to use is the one in CALIBS, which is probably good enough for relative fluxing but not good enough for absolute
        • However you can change the efault by setting the FLUXFIL keyword
      • Calling Sequence: $IDL>mike_flux, name, setup, obj_id, side, (FLUXFIL=), (/STD)
      • Output: The object structure tags .flux and .sig are filled
      • You know the code is working properly when:

        • You canexamine the final product using $IDL>mike_specplt, /FLUX
  2. Combine

    1. MIKE_COMBSPEC - Combines multiple exposures of the same object

      • Note:

        • This routine should be run even on an object with a single exposure, to obtain 1d spectra
        • The documentation for this routine should definitely be checked
      • Calling Sequence: $IDL> mike_combspec, name, setup, obj_id, side, (exp_id)
      • Output:

        • An ESI FSpec structure saved in a .fits file (eg: 'FSpec/name_ech.fits')
      • You know the code is working when:

  3. 2d to 1d


1. For more info on the specific modules, go to the IDL Help for Mike page on the MIKE website

2. For keywords you can use with the routines, click on the routine name, it will link you to a page with further info about each command

  • Notes:
  • brackets, keywords
  1. IDL_PATH may need to be updated in the future. The one presented above is my final path, which differs greatly, due to the amount I had to add, while setting up the data reduction
  2. To hopefully help a few of you avoid unecessary headaches:
    • I was having issues with my code and replaced '~' with 'Users/name' and a ' ' with ':' in my path and for some reason it fixed them...so if youre having trouble with your code, you might want to try that
Page last modified on Wednesday 20 of July, 2011 13:47:31 EDT