Better ways to make large image mosaics?

Given the useful responses to Jane’s question about spectral line analysis, here’s another query for the community. This one is about making large image mosaics and it comes from Adam Ginsburg, a grad student at the University of Colorado, Boulder.

I want to make a large-scale mosaic of the Galactic Plane covering 90-180 degrees x a few degrees. There are a few ways to do this:

  • IRAF’s imcombine with offset=wcs is a quick and dirty path to a mosaic. It is very fast, but very inaccurate over large scales and it sometimes does things flat-out wrong.
  • IDL’s hastrom and related astrolib utils generally work, but for large arrays (96000×4000 pixels) I’ve seen IDL just crash.
  • IRSA’s Montage works well, but is slow and consumes a lot of hard drive space in the process of making mosaics. The setup is also kind of a pain; you need to make a lot of subdirectories and .tbl files in the process.
  • SSC’s mopex must work quite well, but I just discovered it today and, like Montage, it has a learning curve.

So, I’m curious, what do others use? For example, what did GigaGalaxy Zoom use? If you’re trying to build a large near-IR finder chart using 2MASS, do you just go to montage? Does anyone have convenient wrapper scripts written up? Let’s hear it in the comments.

22 comments… add one
  • Tim Oct 21, 2009 @ 11:09

    SWarp?

    http://astromatic.iap.fr/software/swarp

    I haven’t used this for science per se, but to visualize what a field looked as I did analysis on several separate images that spanned a wider field.

  • Eilat Oct 21, 2009 @ 12:28

    swarp! for sure. i use it all the time and it works great. not a very steep learning curve — short manual. If you are familiar with SExtractor then it is even easier to pick up. If your images have standard WCS header keywords then it is a piece of cake.

  • Jane Rigby Oct 21, 2009 @ 13:55

    I used to sit on the IRSA users group. If folks have comments on how to improve Montage, I would like to pass them on to IRSA.

    I have used the Montage web interface to effortlessly make big 2MASS and SDSS mosaics. Perfect for my findercharts, and a 3rd color for Warm Spitzer pretty RGB pictures. Took me 5 min as a complete newbie.

    Like Adam, I’ve casually tried running Montage on my own computer to make mosaics from my own data, since it’s extremely powerful and produces wonderful results. Like Adam, I was also frustrated by the setup complexity and lack of a start-to-finish cookbook. Can others comment on their use of Montage, or compare it to other mosaickers?

  • Gus Muench Oct 21, 2009 @ 15:44

    my experience is that swarp is designed for coadding and does not have the background matching capabilities of montage/mopex. montage seems to scale fine in big pipelines e.g. swire/glimpse.

    what i’ve wanted is a python api to montage…

  • Thomas Robitaille Oct 21, 2009 @ 15:54

    I use Montage all the time. I wrote a small python function that takes care of setting up the directories and cleaning up afterwards, so that all one needs to specify is a directory containing the input files to mosaic, the header to use (or let it choose the header automatically), and the output filename. Maybe I should make this available?

    • Jessica Lu Aug 22, 2013 @ 15:25

      Tom, we can put a link to a git repository on the Python wiki page.

  • Adam Ginsburg Oct 22, 2009 @ 1:45

    Thomas, based on what I’ve seen of APLpy, you should definitely make your montage wrappers available.

    I was completely unaware of SWarp – thanks guys!

    re: Jane – The quick cutouts are fine, and the big, large-scale projects are also fine, but I have very often been in situations where I just want to stitch stuff together quickly. Also, “big” is a relative term… 1 square degree isn’t much when you’re looking to compare to galactic plane surveys. I’ve successfully produced ~6 square degree 2MASS fields with MONTAGE but it took a couple days per image on my really fast computer. Also, if you want your output image bigger than the inputs, I’ve found you have to hack around it by passing a blank image of the desired size as one of the images to co-add.

  • Adam Ginsburg Oct 23, 2009 @ 11:40

    A little follow-up….
    swarp is actually more along the lines of what I’ve been hoping for: a single command-line program with sensible defaults but very extensible via configuration file. It also defaults to removing unnecessary temp files.

    montage has some great features, but is split up into a lot of separate commands. I’ve particularly had problems (today!) with making a mosaic that matches the input header exactly (i.e. cropping), while in the past I’ve had difficulty but figured out how to do the opposite (make larger).

    Are there more arguments in favor of montage over swarp, or is it just a matter of preference, perhaps based on side of the atlantic?

    Sort of related – does either program work well with data cubes? Or are there any good data-cube-mosaicing tools? (STARLINK does OK on this for JCMT data, at least)

    • Son Aug 20, 2013 @ 17:12

      Hi,

      I need some help here. I’m to start a mosaic scene at school of about 640 cm length and 13 feet high. I’hd like to create a large scale of the scene on the computer. How do I do this?

    • John Good Aug 20, 2013 @ 23:42

      This is in reply to the question below. Montage can certainly make mosaics this size or larger and if the data source is 2MASS, WISE, SDSS or DSS it includes the data retrieval tools as well. However, it is unclear exactly what data is desired, what resolution, etc. I can’t really comment without more detail.

    • Emanuela Pompei Aug 26, 2014 @ 11:19

      Well, swarp is just perfect. Montage has a nice web pages, but no one will help you if you run into a problem. They list what kind of error you might get, but no reply if you ask help to them.

  • Bruce Berriman Nov 6, 2009 @ 9:19

    John Good (Montage architect) and myself (Montage project manager) are posting this note to clarify some of the comments made on this page. Montage is actively supported, and we welcome suggestions for upgrades and improvements. Please, contact us with requests and suggestions at the Montage Help desk at

    http://irsa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=Montage&projmail=montage@ipac.caltech.edu

    On-request Mosaics
    There is an on-request mosaic service at http://hachi.ipac.caltech.edu:8080/montage

    This service returns mosaics of 2MASS, SDSS and DSS images. In the case of SDSS it is the currently the only place you can get photometrically-scaled Sloan date (including from the project itself). You simply select the parameters of the mosaic you want on a web form, and the service will notify you when the mosaic is done. You can enter a number of requests and monitor progress on a web page. In Spring 2010, we plan to extend this service to new data sets, as well as to extend functionality to, e.g., deliver 3-color images. We welcome feedback from users on new functionality and which new data sets may be most useful. For many users, this is the simplest way of getting mosaics.

    Montage vs. Swarp
    Swarp and Montage are very different products and satisfy different use cases. Swarp is a fine product and is best suited for a fast, browse quality mosaic, or if you want to a co-add a collection of images (as was pointed out on this web page). Beware that the output is not strictly flux-conserving. Montage, by design, preserves the flux and astrometric fidelity of the input images. And it rectifies the background radiation in the input images to a common level, which is essential interpreting the mosaics in regions with extended emission.

    Speed
    Part of the price that Montage pays for a “science grade” mosaic is speed – it takes a lot of crunching to generate such mosaics, especially if projections such as Aitoff are involved. They are computed by a general algorithm that applies to all WCS projections. We have sped up Montage by more than an order of magnitude in cases where only tangent plane projections are involved. Montage avoids costly plane-to-sky-to-plane deprojection/projection calculations in these cases using a fast “plane-to-plane library,” a modification on the algorithm that MOPEX uses.

    Usability
    Montage is designed as a toolkit rather than a single executable. This design has many benefits:
    – it allows end users to adapt the code for their own use cases; a number of Spitzer Legacy teams have, for instance, installed it into their image processing pipelines.
    – it makes Montage scalable across large-scale parallel systems
    – the many files it produces are necessary to do proper background matching and assure photometric and astrometric accuracy. The files are a very powerful QA tool and we recommend users examine them.

    Now, there is an “executive” (mExec) which you can point at an
    existing set of image files and which will do the end-to-end
    processing for you. See

    http://montage.ipac.caltech.edu/docs/mExec.html

    It tries to minimize storage, deleting unneeded files as it goes. This is a good solution for users who may want occasional mosaics. We do recommend that regular users examine the intermediate files and understand what the processing does to the images.

    There is an end-to-end cookbook tutorial at

    http://montage.ipac.caltech.edu/docs/m101tutorial.html

    If this does not satisfy your needs, please let us know what further tutorials and documentation will be valuable.

    – John Good and Bruce Berriman (IPAC, Caltech)

  • Loren Anderson Dec 4, 2009 @ 4:35

    An important note on Montage is that it does not conserve flux when changing the image resolution. For example, if you want to rebin the image to a pixel size three times larger, the flux of all point sources will decrease by a factor of 9 (3^2). Multiplying by the ratio of the pixel areas, (new_pixel_size/old_pixel_size)^2, will get you back to the original flux. This can be done via the “-x” flag in Montage’s mProject. A “conserve flux” flag in Montage, as in MIDAS, would be nice though.

  • John Good Dec 6, 2009 @ 11:31

    In response to Loren Anderson’s comment: Montage is very careful to conserve flux in how it redistributes pixel content (in fact the method used for the slower reprojection algorithm does so using spherical angles even at the pixel scale). The problem arises in not being able to know whether the units of the data represent a flux or a flux density. If the former, she is correct but most of the datasets we have encountered in practice have been flux densities, where such scaling would be inappropriate.

    However, if there are enough people that need rescaling like this, we would be happy to include it. It’s a trivial addition; we just try to avoid too many extraneous flags. Anyone desiring additional functionality can write to me directly.

    By the way, whenever possible you should try to use mProjectPP, as it is 25-30 times faster than mProject. The former can handle any projection pair; the latter is more restrictive but for small areas can often be used even with some of the odder projections through a little mathematical trickery (creating a “distorted” tangent-plane projection that fits the image locally).

  • Adam Ginsburg Nov 29, 2010 @ 12:24

    This post (and comments on it) are mentioned in http://arxiv.org/abs/1011.5294.

  • Bruce Berriman Dec 31, 2010 @ 18:51

    As a result of the comments in this thread, and subsequent e-mails, we have re-organized the Montage web site (http://montage.ipac.caltech.edu) to assist newcomers by making tutorials and scripts more visible. And we have released a blog at http://montageblog.wordpress.com to report news more quickly than we can do with the web page. We will be blogging about how Montage is used in scientific analysis and generating new science products, showing images derived with it, posting tips for getting the most out of the toolkit, entertaining suggestions for new functionality, and discussing image mosaics in general . We welcome comments and guest blog posts from users.

  • Jane Rigby Mar 10, 2011 @ 13:44

    I’m thrilled by the Montage group’s response to this thread. They could have been defensive (“Jane you dummy, there is too a tutorial!”) but instead they took comments to heart, as well as clarifying what they felt we were misunderstanding (fluxing). I’m also pleased to see that there’s a Montage blog with examples and published results.

  • Bruce Berriman Jul 31, 2012 @ 12:03

    Tom Robitaille and I are interested in finding out how Python users are using the Montage image mosaic engine (as this may impact future directions for the package).

    [Note that if you use APLpy, you may be making use of Montage, so
    please read on!]

    If you indeed use or have used Montage before, please take a couple of
    minutes to email thomas.robitaille AT gmail.com or gbb AT ipac.caltech.edu and let us know:

    – whether you use Montage via Python scripts, and if so whether you
    use it via the python-montage package, or via APLpy’s make_rgb_cube
    and north=True functionality (which requires Montage), or whether you
    have developed your wrappers or just call the commands directly with
    subprocess or os.system (or other ways I haven’t mentioned!)

    – what kind of projects you use Montage for (broadly)

    – whether you have developed any packages that use Montage or python-montage

    – whether you use Montage, but not from Python, and if so why not?

    Your replies, however short, would be very helpful!

    Thanks!
    Tom and Bruce

  • Ana Vudragović May 24, 2018 @ 7:14

    There is also a great IRAF package: MSCRED
    And a great tutorial: http://www.ifa.hawaii.edu/~rgal/science/lfcred/lfc_red.html#fixpix

  • Kris Stern Sep 17, 2018 @ 14:38

    I personally prefer using IRSA Montage to perform mosaicking tasks for FITS files. It is easy-to-use, fast, and reliable. But the thing is that due to legacy issues the software in 2018 can be a pain to install… (Although I was successful in installing it in the end, I did need to hack into my own laptop in a “deep” way, which is not what I expected in the first place.) I tried IRAF’s imcombine task with “wcs” as the offset, but it did something completely wrong, so I gave up on it.

    • Bruce Berriman Sep 17, 2018 @ 18:55

      Hello Kris,
      I am the PI of Montage and I am replying to your post. I am concerned about the comment “…ue to legacy issues the software in 2018 can be a pain to install.. ” Can you send me details please? If there are issues we need to fix them.

      Also, we are deploying in October Python binary extensions of the Montage modules, so you will be able to run Montage with compiled speeds under Python 3.x. The package is in an advanced test state now, but if you wish to use it on a shared risk basis, we will be happy to give you access.

      Hope this helps,
      Cheers

      Bruce Berriman

  • Adam Ginsburg Sep 17, 2018 @ 16:52

    Kris: You might consider using astropy reproject (https://reproject.readthedocs.io/en/stable/), which uses the same algorithm as MONTAGE but is distributed as a python package.

Leave a Reply

Your email address will not be published. Required fields are marked *