Filtergraph: A Web-based Data Visualization Application

Dan Burger is a Computer Science graduate student at Vanderbilt University. He has been working with the Astroinformatics Group on how to dynamically handle and visualize large volumes of data.

Datasets.

If you work in astronomy, chances are you have them.

They are hardly the romantic vision that you had when you decided to go into astronomy: staring through a telescope in the middle of nowhere, surrounded by the stillness of the night sky and the sound of crickets in every direction, and updating your status on Facebook every time you find a new exoplanet.

They are the kind that arrive in your inbox as an ASCII file with thousands of lines of data. How are you going to visualize and make sense of it? Sure, you could fire up IDL, read in the data, and issue some plotting commands. But what if you wanted to quickly filter the data according to various criteria across multiple variables? What if you need to do some transformations on the data and then visualize the result? What if you want to share these data with your collaborators for on-the-fly visualization and discussion, and not just keep forwarding that big ASCII file around?

The filtergraph portal for one of our datasets.

Fortunately for you there’s Filtergraph: a web application that allows you to upload data from your project and instantly generate a web portal that can build graphs and tables on-the-fly based on your dataset. Filtergraph can handle datasets of up to 3 million lines and is flexible enough to build sophisticated graphs from such large datasets in seconds. If you want to create a scatter plot of Teff versus flux, Filtergraph can do that. If you wanted something more complex, say, a scatter plot of Teff vs. log(flux), color-coded by [Fe/H] and symbol size scaled to the distance, for the stars that are between 8 and 18 hrs in RA. Filtergraph can do that as well, with just a few clicks. There is no need to write many lines of IDL code full of where commands.

The plotting is dynamic and can be modified on the fly. You can even select the point in certain part of the plot with your mouse.  You can output the data (HTML, ASCII) and save the plots (PS, PDF, PNG, JPG, GIF) when you are done.

Feel free to play around with the datasets that are already uploaded: KELT-S, the SLoWPoKES and WD+dM binaries, and the SDSS spectroscopic M dwarf and L dwarf catalogs.

Not only is it easy to share your web portal with others, Filtergraph has plenty of interactive features such as zooming, 3D support, interactive tables, and file exporting.  Plots are created using Python’s web2py framework and plotted using gnuplot.  Of course, Filtergraph will not make your discoveries for you, but it sure will make the process a lot easier. Try it out!

8 comments… add one
  • Dan Burger Oct 15, 2012 @ 12:38

    Thanks for the post Saurav!

  • Mark Stacey Oct 15, 2012 @ 16:39

    3 million rows is a pretty big limitation – I much prefer PowerPivot for doing data analysis, and it’s free for use on the Desktop (http://www.powerpivot.com) – great features for anyone familiar with excel, and ability to handle 100 million rows with interactive response is awesome.

    I have some videos I’ve recorded on using PowerPivot specifically for asto data – if you’re interested, tweet me on @MarkGStacey and I’ll send them to you.

    You are of course basically constrained to Excel visualisations, but if you want to use any of the JS visualisation libraries, such as D3, Roland Bourman has a library for querying XML/A type data sources (of which the tabular model inside PowerPivot is one) from Javascript: http://code.google.com/p/xmla4js/

    • Dan Burger Oct 17, 2012 @ 11:53

      Thanks for the feedback. I would like to continue to expand Filtergraph’s capacity to handle more data. As our own needs have grown from hundreds of data points to millions of data points, we have made plenty of tweaks to make sure Filtergraph is able to handle large datasets. We anticipate making more of these tweaks in the future.

  • Ana Vudragovic Oct 15, 2012 @ 16:42

    Nice one, but also check out TOPCAT (http://www.star.bris.ac.uk/~mbt/topcat/). It’s fast, pretty, intuitive and quite interactive. Also can handle many table formats, import online catalogs, make cross-matches etc.

    • Jorge Oct 17, 2012 @ 19:57

      I just *love* topcat. It’s particularly great for visualizing the output of very complex datasets (e.g., GAVO simulations, etc.). Its quick and pretty plots can inspire you for further studies with more sophisticated codes. It’s quite a treat!

  • Peter Teuben Oct 18, 2012 @ 11:45

    Apart from commenting on the beauty of such dynamic queries (Schneiderman et al.), applications such as xgobi and ggobi, and the new python effort by the Harvard group called “glue” , see
    http://projects.iq.harvard.edu/seamlessastronomy/software/glue,
    i’d like to rant about something i’ve ranted about before , and usually get the response “get real, this is the future”.
    I am worried that we’ve done ourselves a disservice with provided web services for everything. It has come partially out of our own mistakes: we have windows, mac and linux, so writing an app on all three major ones is a pain. call in the web., they say is the solution. I already have an overloaded machine with chrome running 50 different tabs and flash apps and ads running amok in half of those, and I can’t get my mouse from left to right within 20 seconds. Now how am i going to load my fits file into ds9 on the local machine. there goes another 10 second of pure wait.
    You can see the money makers see the writing on the wall. We don’t have to give them a CD anymore, we control them via an app, so we can track everything they do, and if they’re bad, we’ll just shut them down. Big data are here to stay, they just get bigger. We do need solutions for this, and webapps (e.g. see CyberSKA) are good avenues, but web application builders better keep in mind that they should not develop this for remote access alone. “configure; make install” should make it run on my laptop.

  • BH Oct 19, 2012 @ 16:41

    I love this as a tool for quick visualization and playing around with different axes. I would certainly install a standalone version if one is ever available. Great job!

Leave a Reply

Your email address will not be published. Required fields are marked *