Topcat: Leader of the Catalogue Manipulation Gang

by Guest on September 20, 2010

This is a guest post—featuring our first screencast!—from Niall Deacon who studies brown dwarfs and white dwarfs in the Pan-STARRS survey at the University of Hawai`i. Niall also blogs about astronomy at weareallinthegutter.

TopcatIf you are drowning in search windows from different data archives and wasting time writing code to plot graphs of the simplest datasets then maybe it’s time to look at Topcat, a nifty little Java application to make life easier.

There’s a huge amount of astronomical data on the web and Topcat can give you easy access to a big chunk of it. Take for example large surveys like 2MASS and SDSS, these can be queried from the cone search (if you want all the objects in a particular area) or the multiple cone search (for extracting photometry for a list of objects). Want to check your candidates against the literature? Just search for any data table in Vizier and cross reference with it. There is also a spectroscopic database to query against. Want to quickly eyeball a couple of candidates? The activation actions can be used to bring up finders from 2MASS, SuperCOSMOS or SDSS.

Topcat was developed by Mark Taylor at Bristol as part of the UK’s VO program (Astrogrid). While it can be used with other VO tools, I use it as a stand-alone application because it can do everything I want it to do on its own. It’s a Java application which I’ve run on both my Mac desktop and Ubuntu laptop without any issues.

Without an internet connection Topcat is also a useful tool for basic data manipulation and vetting. Take for example a table of candidates you’ve extracted from an archive. Using the scatter plot you can quickly plot up the data, select out a subset that looks interesting and examine it further. It also provides table matching so you could check the dataset against a table of objects you’ve already observed (either by positional matching or by matching a catalogue number). I find this incredibly useful for doing basic data quality and consistency checks.

Rather than just waffle on about how useful it is, here’s a video of me doing a short science project looking for bright members of the Pleiades using Topcat:

One word of warning, like all applications Topcat has memory limits. It displays how much memory it has taken up but it’s a good idea to save your session regularly just in case. As I mentioned before I only use Topcat as a stand-alone tool, anyone who has experiences using it with other VO tools is welcome to use the comments thread to show me what I’m missing out on.

Niall babbles about astronomy with a few other selfish transients (or postdocs as some call them) at weareallinthegutter. He studies brown dwarfs and white dwarfs in the Pan-STARRS survey at the University of Hawai`i and was using his phone voice for the video.

{ 10 comments… read them below or add one }

1 Tigran Khanzadyan September 20, 2010 at 3:58 pm

Thanks for presenting the TOPCAT here. Maybe this post should go into the wiki section for our future reference. I have been using TOPCAT intermediately for various projects and found it to be quite a power tool to sort enormous tables like WFCAM photometry tables. Couple of times I’ve managed to work with whole Astrogrid suite connecting TOPCAT and ALADIN – nice idea but since everything is on java it’s gradually slows down the whole system.

Using TOPCAT separately I found that by increasing the Java’s heap memory things are manageable. The command for the quick reference would be “topcat -Xmx256M” or “java -Xmx256M -jar topcat-full.jar”. Yes I know 256M is nothing so in my case I tend to use 2048 at least from my 8G RAM. Conclusion would be to run this beast on powerful machines otherwise things could become quite frustrating when dealing with java-written software.

Reply

2 David Rodriguez September 20, 2010 at 7:39 pm

I’ve used TOPCAT quite a bit myself and love it. I only have memory issues with the default settings when dealing with large tables (10k+ rows). In those cases I tend to either increase the memory as described above, or split up the table by selecting interesting objects and saving those as a separate (smaller) table.

I think one of the coolest features these VO tools have is the ability to communicate with each other. If you load up Aladin and Topcat and set up an internal SAMP hub (done automatically in Aladin; with Topcat you use the Interop menu) you can send targets back and forth between both programs.
You can set the Activation Action so that when you click on an object in Topcat it transmits the RA/Dec to any VO program. Selecting an object in a color-magnitude diagram, for example, will focus the Aladin display on that object (you need to have an appropriate image loaded up, of course, though old (PLASTIC) versions of Aladin would automatically do this for you- I’ve written a script to work around this). If you haven’t tried this out, go do so: it’s fun!

One thing I wish for is being able to edit and annotate the Topcat graphs. The graphs are nice looking and very easy to make, but I rely on IDL or Python to draw lines or labels on selected objects. There are ways to ‘trick’ Topcat to do this, but it’s usually too time-consuming.

Reply

3 Gus September 20, 2010 at 9:25 pm

Passing the “-disk” parameter also helps with large tables; I think it is basically a disk cache and I pass it by default.

See (yes, the manual): Large Tables

History: TOPCAT was begotten of Starlink, continued by Astrogrid and now lives on via ?.

While there are a number of things TOPCAT users wish they could do to improve TOPCAT’s graphing capabilities, one can achieve figure creation in a scripting manner using TOPCAT’s “sister package” (Mark’s words) STILTS command line toolkit. This alleviates my primary problem with TOPCAT figure creation, namely, reproducing the graph to tweak it later on.

Reply

4 Raffaele D'Abrusco September 20, 2010 at 10:00 pm

I am a long-time TOPCAT enthusiast (thank you, Mark Taylor, I owe you much!), and recently I have taken advantage of its amazing VO capabilities for a demo (meant for a general public) involving other tools (DS9 and CSCview) to showcase the interoperability of VO tools. The loose narrative of the demo is ‘Exploring X-ray point sources in M42′, and this is the list of the steps of the demo:
1. We start with Google Earth, and display optical (Hubble, Sloan) images of M42;
2. From Google Earth we connect to the Chandra Data Archive (CDA) and display
CSC images and metadata on top of the optical images;
3. With WebChaser, weconnect to the ADS and do a literature search related to the
X-ray data selected;
4. We search the same region of the sky (Orion) in the CSC with CSCView;
5. We display tabular information in CSCView and visualize (Browse) previews of
the images;
6. We transmit the tabular data to TOPCAT using SAMP, select parameters and
produce 2-d and 3-d plots;
7. We select interesting objects in TOPCAT plot and transmit the information to
CSCView with SAMP;
8. We select and retrieve the X-ray images, light-curves, and spectra of the
selected sources from CSCView;
9. We trasmit and display the images in DS9 using SAMP
10. We transmit and display light curves and spectra in TOPCAT
11. From TOPCAT we query the VO registry and select optical (2MASS) data set
12. From TOPCAT we perform a cone search around the CSC sources and find stellar
counterparts by querying the VO registry and selecting optical (2MASS) data set.

Reply

5 Ivan Zolotukhin September 21, 2010 at 6:54 am

Great tool, indeed. I’ve been using it since ancient times. It’s a base of several of my public VO science case tutorials presented in many research institutions around the globe (to appear at http://www.euro-vo.org/pub/fc/workflows.html which is a nice collection of other VO-based demos, including ones with TOPCAT). There will be another one at forthcoming ADASS meeting. TOPCAT served as a base for my online web-site, http://ocl.sai.msu.ru, devoted to online scientific exploration of some unstudied open clusters datasets. So, many thanks to Mark again, and come join his maillists if you have any recipes, suggestions or thoughts on TOPCAT to share inside the lively community around the author.

Reply

6 Mark Taylor September 21, 2010 at 8:05 am

Thanks Niall for the post and video, and the rest of you for your comments. It’s very nice to hear people’s enthusiasm!

Concerning large data sets, it’s true that java presents some issues with memory management, but TOPCAT works hard to get round these, and it should be possible to work with fairly large tables even on modest hardware – as a rule of thumb a million rows, even with a lot of columns, should be quite feasible. Configuring it so that this is the case can sometimes be a bit of a fine art, but I’ve tried to improve this over the last year or so, and with recent versions you may find you don’t need to up the “-Xmx…” heap memory (setting it too high can itself cause problems) or supply any other non-default options at all for large-table work. As Gus points out, ways to get the most out of the tool are discussed in the manual, but the number one rule for dealing with big tables is, use (uncompressed) FITS files – these can be accessed much faster and in less resource-intensive ways than other formats (you can use TOPCAT or STILTS to convert from your existing table formats to FITS if necessary).

Any feedback, good or bad, is always welcome, here or on the mailing list or direct to me – TOPCAT is very much the product of user experience.

Reply

7 Jessica September 23, 2010 at 12:18 pm

I have added a link back to this blog post on the wiki under

Analysis Tools
http://www.astrobetter.com/wiki/tiki-index.php?page=Analysis+Tools

Reply

8 Alyssa Goodman October 19, 2010 at 11:08 pm

I liked the video so much it inspired me to update a YouTube Channel http://www.youtube.com/user/astronomy999 with more. Now I’m hoping someone “better”(!) will take over that channel…

Reply

9 Kelle October 20, 2010 at 8:25 am

@Alyssa,

I did indeed make an astrobetter channel youtube channel (http://www.youtube.com/user/astrobetter) but I didn’t have the vision you did to actually make it a repository of relevant videos organized by category…like the wiki, but for videos. nice.

ooo, I’ve never heard of VisIVO before…that seems worth looking into further…

Reply

10 Jessica October 21, 2010 at 4:01 pm

@Alyssa and @Kelle,

I added a link on the Wiki home page to astronomy999@YouTube under Other Resources. Thanks!

Reply

{ 2 trackbacks }

Leave a Comment

Previous post:

Next post: