VisIVO: a VO enabled tool for Scientific Visualization and Data Analysis

Abstract:

Key words
Astronomical data, Virtual Observatory,Visualization, Numerics

Introduction

Data represent a critical issue for scientists and, in particular, astronomers. Observational instruments (telescopes, satellites...) produce enormous quantities of images and information. Computers and numerical simulations generate huge amount of data. All these data must be stored, managed and analyzed. These steps require a great human effort, large scale facilities and efficient, powerful tools.

We present VisIVO, a software for the visualization and analysis of astrophysical data, which can be retrieved from the Virtual Observatory framework. VisIVO is VO standards compliant and supports the most important (and popular) astronomical data formats such as FITS, HDF5 and VOTables. Data can be retrieved directly connecting to an available VO service (like, for example, Vizier [Ochsenbein 2000]) and loaded in the local computer memory where they can be further selected, visualized and manipulated.

VisIVO can deal with both observational and simulated data and it is particularly effective in handling multidimensional datasets (e.g. catalogues, computational meshes...). It is completely open source and the first release can be downloaded from the web site (http://VOmaf.cineca.it). At present the software is fully tested only for Windows XP platforms, the porting for Linux is in progress.

This work is only one of many different applications that the Italian astronomical community is developing in the VO framework. This effort is led and coordinated by the National Institute for Astrophysics (INAF), in collaboration with other institutions like, CINECA, the largest Italian academic supercomputing centre. This partnership allows you to collect and blend the necessary heterogeneous competencies (astronomy, data management, grid services, software development, visualization) and infrastructures. A further application, which involves again INAF and CINECA, is given by the set up of a VO service represented by an archive of data coming from numerical simulations (at present in a definition phase).

VisIVO

VisIVO is a data visualization and analysis tool, specifically oriented to astrophysical problems. It provides an effective and intuitive way of managing, visualizing and analysing the large amount of data produced by observations and numerical simulations. The software is specifically designed to deal with multidimensional data. Catalogues or numerical simulations, rather than 2D images, represent the basic target of VisIVO. Different quantities can be visualised and treated at the same time. Data can be both retrieved from the VO framework and loaded from local files.

VisIVO exploits the expertise and technology developed in two different research projects financed by the European Union in the Fifth Framework: Cosmo.Lab and Multimod.

The Cosmo.Lab project involved CINECA, the Astrophysical Observatory of Catania (OACT), the Institute of Radio Astronomy of Bologna (IRA), the Institute of Cosmic Physics G. Occhialini of Milan (IFCTR) and the Netherlands Foundation for Research in Astronomy (NFRA). The outcome of Cosmo.Lab has been AstroMD [Gheller 2004], an open source visualization software for astronomical data, which is the basic core of VisIVO. The new software in fact, even using a more sophisticated technology (see later), implements most of the functionalities and uses instruments developed and tested for AstroMD. AstroMD was presented in several international workshops (Adass 2002, Eurographics 2003...) and can be downloaded from the web site http://cosmolab.cineca.it together with detailed information about the Cosmo.Lab project.

The Multimod project is the result of the ongoing collaboration of six European institutions: CINECA Istituti Ortopedici Rizzoli (Italy's largest research and healthcare orthopaedic centre), the University of Luton (UK), Université Libre de Bruxelles, Vicon Motion Systems Ltd. (motion capture) and ESI Software (numerical simulation). The Multimod project was initially intended to provide biomedical researchers and professionals with a software tool for virtual representation and analysis of musculoskeletal structures. However, the fast-paced development and the diverse, yet interconnected, efforts involved in the project have led to a more ambitious outcome: a multimodal application framework providing high and low-end users with the necessary tools for the development of applications in the field of scientific visualization.

The Multimodal Application Framework

The Multimodal Application Framework (MAF) [Viceconti 2004] is an Open Source software development environment entirely written in C++. It focuses on scientific visualization thus making extensive use of the Visualization ToolKit (VTK), a powerful Open Source visualization library. VTK is the true foundation of MAF: its powerful class architecture is enforced throughout the framework. MAF is constructed upon three layers; each layer provides a different level of abstraction (hence depth of access to the core of the framework). The layers are MFL (Multimod Foundation Layer), LAL (Low Abstraction Layer) and HAL (High Abstraction Layer).

The lowest of the three architectural layers, MFL, may be thought of as an extension of VTK. It contains some ad hoc visualization libraries that are not included within VTK, thus adding some important extra functionalities to MAF. In particular, one of the fundamental elements incorporated within the MFL is the Virtual Medical Entity (VME). Although its name explicitly refers to medical data (as this was the initial focus of Multimod), this data structure has been extended to provide a powerful representation unit for scientific data. The employment of VMEs for the visualization and analysis of astronomical data in VisIVO proved efficient and successful.

The middle layer of the architecture, LAL, implements the bulk of the framework services which are likely to be utilised by most applications in the framework. In other words, it provides all the classes necessary for the development of an application under MAF. It is at this level that framework services such as Operations (high level procedures that modify existing VMEs) and Views (interactive visualizations of VMEs) are taken care of. This layer is also responsible for the user interface which is handled via integration of a portable Open Source GUI library called WxWindows.

The top layer, HAL, incorporates the final structure of each specialised program developed under the MAF. In the HAL, an application is assembled using the elements provided in the MFL and LAL together with the classes specific to the application. In other words, the High Abstraction Layer is where the actual behaviour and functioning of the MAF application gets translated into a set of customized classes that co-operate with the underlying architectural layers.

To summarize, the lowest layer, MFL, deals with the fundamental visualization services and is to be extended by developers that wish to add some important VTK-like functionalities to the MAF. The LAL contains the basic framework services and is targeted on programmers that wish to add important services (such as Views or Operations) throughout the framework (so that they are available in all MAF applications). The HAL exposes the interface that all programmers use to create "vertical" applications in the framework. The development of VisIVO was mainly carried out at this layer, although some work led to important extensions to both the MFL and LAL layers.

VisIVO architecture

The architecture of VisIVO strictly reflects the structure of a typical scientific application built upon the MAF framework. As it is a piece of software mainly developed in the High Abstraction Layer, VisIVO exploits, wherever possible, the standard visualization services, views, operations and interface structures provided by the framework and implements all the elements that are specific to the visualization and analysis of astronomical data.

The overall behaviour of VisIVO is handled by a class called Logic which communicates with the four managers: VmeManager, OperationManager, ViewManager and InterfaceManager. Each manager is totally independent of the others and provides hardwired behaviour to its respective framework element (namely VMEs, Operations, Views and Interface elements). Communication takes place following a command-event model that ensures that child modules do not have to know their parent module in the architecture. Hence, modules of an application do not know about each other; it is the class Logic that coordinates and controls the communication among modules. This architecture guarantees the highest level of modularity and extensibility.

VisIVO embodies a select object, apply operation utilization metaphor which is somehow similar to that of many graphical commercial applications (such as Adobe PhotoShop). The usage of VisIVO generally begins with an operation of data loading. Data types that are currently supported by the application are VOTables, FITS, HDF5, Tipsy, VTK as well as binary and ascii raw data (dump of memory). Data importing is performed by launching a special type of operation, handled by the OperationManager, which takes a data file as input and generates a VME as its output. It is important to highlight the fact that a VME (or a set of VMEs) is the standard data structure produced upon loading/importing/modifying any type of data. This is represented in the VME tree (in the contextual menu) as a VME node. The VME tree is a hierarchical structure handled by the VMEManager which links all VMEs currently loaded in the program.

In order to display data loaded in the VME tree, it is necessary to instantiate a View and associate it with the chosen VMEs. A View is a rendering window that gives a particular representation of the VME tree. VisIVO currently supports the following Views: Points, Volume Rendering, Isosurface, Glyph, Stereo, Histogram. Each view displays data according to a specialised visualization pipeline. It is possible to instantiate several Views and each View can display a different subset of the VME tree. This is a point of strength of VisIVO as many specialised views, displaying different (or same) data, can be instantiated and data can be manipulated and analysed independently of the views displayed. The properties of the selected view (e.g. camera perspective, axes, logarithmic scale) can be adjusted via the contextual menu.

In order to analyse and modify data loaded in the VME tree, a set of operations are available to the user. Operations are mutually exclusive and act solely on the VME selected in the tree (provided such VME allows that type of operation). Operations are instantiated by the Operation Manager which takes a VME as input and generates a VME (or a set of VMEs) as output. There are operations that simply modify the data, operations that perform some statistical analysis on the data and operations that do both. All of these generate output VME nodes that can be displayed according to the type of output (e.g. a VME output that represents statistical 2D data can then be displayed in a Histogram View).

Every element instantiated throughout the use of the program may have an interface for communication with the user. GUI services are provided by the InterfaceManager. There are interfaces that are always visible (e.g. toolbar, contextual menu) and interfaces that are generated/destroyed upon an event. The InterfaceManager constantly communicates with Logic to be instructed upon the current status of the program (i.e. selected VME, selected View and running operation) in order to manage the GUI accordingly.

Data access and manipulation

VisIVO is expected to load data both from a local disk and the Virtual Observatory. The use of the local file is standard. VisIVO has different data importers which can handle various file formats. At present, FITS, Tipsy, VOTables, ASCII and Raw data formats are supported. HDF files are partially supported (but work is in progress to extend their usage). Each supported data format is associated to a specific reader. The reader allows you to handle and present meta information associated to local data, select the fields of interest and convert the selected data in the internal VisIVO format (VME). Data could also be retrieved from the Virtual Observatory. VisIVO will be able also to deal with such environment. Presently, this functionality is not implemented yet. However, the the VisIVO team is already experimenting the technologies that will enable the software to interact with the VO.

Data Readers

VisIVO supports various common file formats.

VOTables

The possibility of displaying and analyzing VOTable data is a fundamental feature of VisIVO. In order to facilitate the process of reading VOTables, VisIVO uses VOTableParser, a set of C++ libraries developed by VOIndia. These libraries are based on more popular Xalan/Xerces libraries which provide a reliable interface for performing parsing and writing procedures of XML files. VOTableParser implements a set of useful classes and methods that are specific to the characteristic XML format of VOTable files and are suitable for large datasets.

The content of a VOTable is made up of the header, which contains a description of the data, and the actual data, which can be an ASCII table, a binary table or a FITS file. At present, VisIVO only displays ASCII tables, but it will be soon extended to read the other formats. The fact that data in VOTables are so nicely separated into data and meta-data allows VisIVO to break the reading procedure into two separate steps:

1. first, the header of the VOTable is parsed and its content is prompted to the user; this includes information such as table name, name of the fields present in the table, unit of the fields, description of the fields and so on. The user can now select the fields that he/she wants to plot onto the three axes (x, y, z) and the associated scalars (if any).
2. once the fields that are going to be loaded have been selected, VisIVO can then proceed to parse the actual data. Obviously, only the necessary fields will be read, minimising time consumption. The data read will be then encapsulated into the usual VME data structure, to allow visualization and analysis.

This approach has a number of important advantages:

• Processing speed. VisIVO minimizes processing times whenever possible. Parsing of the header is fast as it is totally independent of the size of the actual data. Data loading, on the other hand, may be more time consuming as it is proportional to both the number of rows in the table and the number of scalars selected. However, only the selected fields are read avoiding whole datasets to be loaded into memory.
• Coordinate conversion. A problem often associated with visualization of astronomical datasets, especially observational data, concerns coordinate conversion. As VisIVO is extremely flexible, allowing the user to select and display any numerical data field from the table, it is extremely important to consider the unit and coordinate system in which the dataset is expressed, to produce a correct plot. An advantage of VOTable files is that fields are generally described in detail in the header section, so that the unit (hence coordinate system) may be deducted. VisIVO can then act accordingly, making the necessary unit conversion and transforming the data into Cartesian coordinates. If the selected fields have no unit description, VisIVO assumes data are already in Cartesian coordinates and plots the values as (x, y, z).
• Data loading. VOTables are often generated dynamically following a query to database services (e.g. Vizier). Although the overall structure of a VOTable is regular and well-defined, the composition of tabledata only relies on the authenticity of the actual data present in the VO. Therefore, we cannot expect to have VOTables that have field values for all the rows. Empty fields are no exception when handling catalogue data. This could lead to mistakes when a user chooses to plot a field (e.g. the redshift), but this field is available only for a subsection of the table. A solution could be to set the missing fields to a default value, but this may cause in serious flaws when statistical analysis is performed upon the data. Our solution is to simply skip the table rows that have missing required fields; in other words, it is better to have a smaller but solid dataset rather than a larger unreliable one. When VisIVO is forced to skip some rows, it prompts the user with a warning that displays the number of rows skipped.

FITS

FITS stands for `Flexible Image Transport System' and is the standard astronomical data format endorsed by both the NASA and IAU. FITS is much more than an image format (such as JPG or GIF) and is primarily designed to store scientific data sets consisting of multi-dimensional arrays (1-D spectra, 2-D images or 3-D data cubes) and 2-dimensional tables containing rows and columns of data. The FITS format is similar to the VOTable one. However FITS files do not use XML to handle metadata. The description of the file content is a binary header itself. The FITS format is often more efficient than the VOTables one, even though it is much less flexible and extensible. We have used the standard CFITSIO (see http://heasarc.gsfc.nasa.gov/docs/software/fitsio/fitsio.html) library to build the VisIVO FITS reader.

HDF5

The Hierarchical Data Format (HDF) is still partially supported by VisIVO. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic objects, one can create and store almost any kind of scientific data structures, such as images, arrays of vectors, structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. HDF5 is very flexible and emphasizes storage and I/O efficiency. For instance, it can accommodate data in a variety of ways, such as compressed or chunked. And the library is tuned and adapted to read and write data efficiently on parallel computing systems. Many applications (in particular simulation codes) use this format to handle their outputs.

Other formats

VisIVO can also deal with other, non-standard, file formats. The Tipsy reader allows you to handle data from particle-based cosmological simulations. The other two formats, ASCII tables and Raw files, should allow all the users with non-standard format, to use VisIVO, with a simple conversion data format.

Interacting with the VO

Presently, the only possible interaction of VisIVO with the VO is off-line. Data can be dowloaded from a web service (e.g. Vizier from CDS) in a standard format like VOTables or FITS.

However, finally, VisIVO will be able to query directly the web service and to retrieve data from it.

VisIVO functionalities

Data visualization

VisIVO supports several visualization techniques for the data. Unstructured points are visualizad as pixels. The user can set the transparency of the pixels and their color. It is possible to color each pixel differently, according to an associated quantity. For example, gas particles can be colored by their temperature (e.g. blue for cold particles, red for hot particles). Points can be also described with glyphs (solid geometrical shapes), whose shapes and size can be parametrized to some physical quantities associated with data.

Mesh based data can be visualized with two different techniques, isosurfaces and volume rendering. Isosurfaces are surfaces characterized by a given fixed value of the plotted quantity. They separate regions with higher values from regions with lower values. Different isosurfaces can be calculated and visualized at once. They can be characterized by different colors, according to the contour value. Using the volume rendering technique, different values of the quantity are represented by different colors and different transparencies. The overall effect is a cloud appearance.

Two dimensional visualization, graphs and curves, and HTML file browsing are included. In both cases the implementation is quite simple and basic. Furthermore, stereographic view is supported.

Data Analysis

The main following implemented functionalities are specifically directed at cosmological simulation data outputs. Some of them are statistical tools that one can deploy in order to discriminate among different cosmological models.

Particles Mass Density and Gravitational Field Calculations - By using an eight-point Cloud-in-Cell smoothing algorithm [Hockney 1988] , VisIVO computes the mass density field associated with the particle distribution by spreading the mass of each particle over the computational mesh. The gravitational field, calculated solving Poisson's equation:

where is the gravitational potential, is transformed into its space image by using an FFT function, adequately generated and inserted in the VTK structure. The equation is thus reduced to a simpler algebric operation:

where and are the Fourier images of the potential and of the density, respectively, and is the square module of the wavenumber. Finally, the potential is transformed into the physical space using an inverse FFT.

Power Spectrum and Correlation Function - The quantity is used to calculate the Power Spectrum of the matter distribution, which is defined as the average value of the square norm of :

The spatial distributions shown in figures 2.2 and 2.2 (top) are about the visualization of the output data from the FLY code, for a box having size of Mpc and with 262144 bodies, randomly extracted from a file with particles. By using a Nearest grid point algorithm, the code distributes the masses of all particles on a 3-dimensional grid whose dimensions (number of nodes per side) are user-defined. Then, for each node of the grid, , is computed and a 3-dimensional FFT of the scalar values in the real parts is executed.
The Power Spectrum is a powerful measure of the statistical properties of the distribution, together with the associated Correlation Function . If one defines the Correlation Function in terms of the density field , one obtains:

passing to the limit , we can see that the Power Spectrum is the Fourier Transform of the Correlation Function (Wiener-Khintchine theorem):

Several estimators have been used in the literature to measure, in particular, the two-point Correlation Function, that is defined in terms of probability of finding a point in a randomly-chosen volume and a point in another volume separated by a distance . For a comparison among most of the estimators of this statistical quantity, see Pons-Borderia et al (1999).
The two-point Correlation Function of VisIVO is based on the three-dimensional counterpart the Peebles & Hauser estimator (1974):

where is the number of points of an auxiliary random sample, is the number of all pairs of points with separation inside the interval , and is the number of pairs between the data and the random sample with separation in the same interval. It was considered the same cosmological model used before at two different redshifts.

Minkowski Functionals The Minkowski Functionals (MFs) provide a tool to characterize the Large Scale Structure of the Universe. They describe the Geometry, the Curvature and the Topology of a point-set [Platz 1995].
In a three-dimensional Euclidean space, these functionals have a direct geometric interpretation. In particular, VisIVO computes the reduced values of the Minkowski Functionals, with , that are the ratio of the MFs of the actual distribution to the MFs of the same number of disjoint convex bodies. So their values start always from unity because, for small radii, all covering balls are disjoint. In the 3-rd functional , the first zero provides an estimate ofthe percolation threshold [Mecke 1991]. A

Friend-of-Friend Algorithm Dynamical studies of groups of galaxies are an important method for estimating galaxy masses. The most common group-finding algorithm is known as Friend-of-Friend (FoF). This technique was first used by Huchra & Geller. The outputs obtained with VisIVO and those with the Buchert code [Huchra 1982] are in good accordance.

Randomizer Data that the user wants to visualize and analize may be too large. Moreover, data files may be tidy sequences of coordinates (for increasing values of x y z) and the user cannot visualize and/or analyze the first elements but he has to extract a significative sub-set.
VisIVO contains a randomizer tool to extract a significant representative of the whole sample that can be visualized and analyzed with tools inside the package.

Future development and conclusions

VisIVO represents the first experience of an immersive Visualisation and Data Analysis Tool in astrophysics. The internal structure is based on the VTK library, written in C++, and all the analysis tools consist in specific modules that we have implemented in an ad hoc manner.
VsIVO can read data in VOTable standard format and will be integrated in the VO framework with the capability to have direct interaction with web services using SAOP/XML direct interaction. It will also use the grid services to execute external tools in a dataset and can be integrated into a VO theoretical data center to obtain a preview and a pre-analysis of data that users want to consider.
Acknowledgments: we wish to thank the VisIT group of CINECA (http://www.cineca.it/HPSystems/Vis.I.T) and in particular Cinzia Zannoni, Marco Petrone, Paolo Quadrani and Silvano Imboden for the help and support continuously provided during the developemnt of the VisIVO software.
The authors wish to thank Mrs. Luigia Santagati for English revision of the text.

Bibliography

Berger 1989

Berger M.J., Colella P., 1989, J. Comput. Phys., 82, 64

Gheller 2004

Gheller C., Rossi E., Becciani U. et al , 2004, AstroMD 3.1 User Guide ISBN 88-86037-33-3

Hockney 1988

Hockney R and Eastwood J. Computer Simulation using particles., 1988 New York, Mac Graw Hill HE.

Huchra 1982

Huchra J and Geller M. Groups of Galaxies I. Nearby Groups., 1982 ApJ 257 423.

Mecke 1991

Mecke K and Wagner H. 1991, J. Stat. Phys., 64, 843.

Ochsenbein 2000

Ochsenbein F., Bauer P., Marcout J., 2000, A&AS 143, 221

Platz 1995

Platzöder M, and Buchert T. Applications of Minkowski-functionals to the Statistical Analysis of Dark Matter Models. astro-ph/9509014.

Viceconti 2004

Viceconti M., Astolfi L., Leardini A., Imboden S., Petrone M., Quadrani P., Taddei F., Testi D., Zannoni C. The Multimod Application Framework IEEE Proceedings of IV2004, pp. 15-20.