VO-compliance and data exploration tools

1 AstroGrid and Virtual Observatories

1.1 Background

Purpose

This document discusses the implications of Virtual Observatory compliance for data mining and visualisation tools. The tools considered are mostly those that are already specific to the astronomy community.

The document covers the philosphy and need for the Virtual Observatory, the relationship between the various organisations involved in its construction and protocols and standards. Readers who are already familiar with the VObs and simply wish to integrate applications with the Euro-VO infrastructure should consult the "Integrating Applications into the Euro-VO infrastructure" section.

Virtual Observatory

'Virtual Observatories' refer to web services, particularly data querying services, that allow astronomers to access and process data from their desks. The Virtual Observatory refers to the single global Virtual Observatory being grown by the International Virtual Observatory Alliance. This consists of web services that implement certain interfaces and protocols defined by the IVOA. However as these interfaces are not yet complete enough to support a full VO most 'IVOA-compatible' implementations have their own extensions. Some of these extensions are proposed IVOA standards.

AstroGrid

AstroGrid is the UK effort to develop a fully-functioning Virtual Observatory. At the moment [Sep 2005] the implementation consists of a set of services to provide online storage, publish data, run remote astronomy applications, register and find online services and a skeleton security framework for authenticating and authorising access.

In particular it includes two aspects that are relevent to Visualisation and Mining Tool providers: a 'Common Execution Architecture' that defines how astronomy-related web services can communicate with each other and an 'Astro Client Runtime' that helps connect desktop-based clients with the Virtual Observatory. These are described in more detail later.

AstroGrid is used in this document both to refer to the consortium of collaborating institutions and also the actual software produced by the organisation.

EuroVO

The EuroVO is the European part of the International Virtual Observatory. It includes the development of several so far independent tools and datasets that will be incorporated into the VO. Some of these tools are listed and discussed later in this document. AstroGrid provides the grid middleware that enables the EuroVO tools to work together. In the remainder of this document the terms AstroGrid and Euro-VO infrastructure are used synonymously.

1.2 General principles of the VO

The Virtual Observatory is driven by the enormous data sets that are emerging in the astronomy community large enough (TBs -> PBs) that they cannot be copied to a user's desktop. The nature of the community is distributed, fragmented and frequently competitive. Thus Virtual Observatory services tend to be located at research centres and tend to be implemented individually.

Even though the VO is data-oriented, certain applications will need to be located 'close' (in network bandwidth) to the data and thus remote from the user. Gateways to processor-grids are being introduced.

The mantra is 'move the application to the data, not the data'. When considering visualisation and mining tools we must consider that the datasets cannot necessarily be moved to a client machine. We will need to look at sampling, grouping and remote processing.

1.3 How the VO Works

The Virtual Observatory is a set of web services each of which is entered into a VO 'Registry'. The web services might be applications or algorithms for running, datasets for querying or filestores for keeping results 'close' to the services that will be processing them.

To publish your data or algorithm to the VO you first need to 'wrap' it up as a web service and publish it to the web. The standard interfaces and protocols for the web services vary but include http-post (SIAP) for image datasets, http-post/SOAP (SkyNode) for catalogue datasets and http-post/SOAP (CEA) for applications and algorithms, details below.

Once a web service is available it should be 'Registered' with a VO-compatible Registry service. This involves creating a suitable XML document that describes the service and where it can be found and submitting it to the Registry. The VO Registries 'harvest' from each other so the document will spread throughout the whole VO until all Registries are aware of the new service.

Client software can then interrogate the Registry, ideally with a question as vague as "Give me all the X-ray datasets that cover this patch of the sky", and the Registry will return a list of all the services that might have the relevent information.

While AstroGrid has a framework for securely accessing services, including a means of registering users within communities, a suitable practical global secure certification system has not yet been identified or implemented. However it has allowed AstroGrid to implement a virtual remote storage system based around the idea of a user's 'homespace'.

1.4 AstroGrid Components

The AstroGrid VO-implementation consists of a small number of collaborating components. These components are distributed, communicate using SOAP, are written in Java and are deployed to Tomcat web servers. Of particular interest to tool writers are: MySpace, Registry, and the CEA. Each component is described in brief below.

A list of the currently deployed components is available at http://wiki.astrogrid.org/bin/view/Deploy/DeployedComponents or astronomers and developers can browse the Registries directly to see which services are available. These will include services and components from other development groups, such as the National Virtual Observatory in the USA.

Registry

AstroGrid has an IVOA-compliant Registry component built on an eXist XML database. If any component can be considered the heart of AstroGrid it is the Registry, since this is used to locate (almost) all other components.

Items are located in the Registry using keys (or URNs) called IVORNS (IVO Resource Names) of the form ivo://org.astrogrid/foo Each IVORN can be split into three parts:

• ivo:// - indicates this is an IVORN
• org.astrogrid is the authority id
• foo is the resource id

Occasionally one sees IVORNs of the form ivo://org.astrogrid/foo#bar. In this case only the substring preceding the # is actually a registered IVORN, and the remainder of the string is interpreted by another component such as MySpace.

The authority id is a string unique to each "organisation" that wishes to publish to a Registry. Within each institution publishing under that authority id, the resource id is unique. Taken together, the IVORN is a globally unique key to a particular resource.

Each authority id is managed by exactly one Registry, while a Registry may manage several authority ids. For example, the main AstroGrid Registry (galahad) manages the authority Ids uk.ac.le.star, uk.ac.port.dsg, uk.ac.man.jb, roe.ac.uk, org.astrogrid, eurovo.workshop. However, browsing the galahad Registry will yield many other authority ids - these are resources that have been harvested from other Registries. Galahad regularly harvests other Registries throughout the IVOA, storing any new or updated resources that might be managed elsewhere. Galahad can only mirror these resources - it is forbidden to change them on galahad.

The resources are stored in the Registry as <VOResource> xml documents. These documents contain standard meta-data about the resource: the contact details of the person who entered it, a human-readable name and description, which other resources it is related to etc. In addition, there are a number of standard extensions to the document schema to contain meta-data for a particular kind of resource, for example input and output parameters for an application resource. The VOResource schema is currently being refined and extended by the IVOA, and the reader should consult http://www.ivoa.net for the latest standard.

Further information about the AstroGrid Registry can be found at http://www.astrogrid.org/maven/docs/HEAD/registry/index.html

The Common Execution Architecture (CEA)

The CEA is an architecture for publishing server-side applications to the VO. Implementations of this architecture are known as Common Execution Connectors. AstroGrid provides 3 variants of a CEC: for wrapping pure Java applications, existing http web services, and legacy commandline applications. This component will be of interest to data mining application writers and so is discussed in some detail in the second part of this document.

The interface is used to bind AstroGrid's Job Execution Service with its applications, so all AstroGrid application servers provide this interface. It is a proposed IVOA standard, but other Virtual Observatory services do not generally implement it yet.

For more information see: http://www.astrogrid.org/maven/docs/HEAD/applications/index.html

The Job Execution Service (JES)

The JES orchestrates CEA-services. The user supplies the JES with a workflow document and the JES calls the CEA services in the correct order. Furthermore, the JES allows the user to manipulate the results of CEA services using a sophisticated scripting engine based on Groovy.

For more information see: http://www.astrogrid.org/maven/docs/HEAD/jes/index.html

Community

Authentication and authorization are handled in AstroGrid by Community components. This is a decentralised system for managing users, groups and permissions. Each user on the AstroGrid system has a username and a Community name. The Community name is the IVORN (see Registry) of the Community server that manages that user. Logging into AstroGrid is a multistage process: the user enters his username, password and Community name. A query is made on the central Registry with the Community name, which returns the VOResource document describing the Community. The actual URL of the Community server is extracted from the VOResource document, and the username and password passed to that server for authentication.

For more information see: http://www.astrogrid.org/maven/docs/HEAD/community/index.html

MySpace: Astrogrid's remote storage, homespace

AstroGrid's 'MySpace' is based on linux's 'home' directory or Windows 'My Documents' folder. It gives the user a way of specifying a file in a virtual area based around the user; the software works out where the file is actually stored.

Files can be specified using a URN string. 'For historical reasons' this URN starts with an 'ivo' scheme. For example, "ivo://roe.ac.uk/mch#Samples/SomeFile.txt" identifies a file "SomeFile.txt" in the folder "Samples" in the homespace of the user 'mch' who is registered with the Community identified globally by 'roe.ac.uk'. Note that though this form 'looks' like an IVORN (or IVO-ID) that can be resolved by the Registry, it is not. In the future AstroGrid may switch to a more 'correct' scheme such as 'homespace:mch@roe.ac.uk/Samples/SomeFiles.txt' that does not confuse the two concepts.

MySpace actually consists of two distinct components: FileStore, which is responsible for storing the files themselves, and FileManager, which stores the meta-data about the files (name, MIME-type, FileStore etc). A user will typically have a single FileManager (his "homespace") but will have files stored in various FileStores. The FileManager presents a unified view of the files to the user so that the actual location of the file is transparent to the user. The FileStore interface is currently undergoing IVOA standardisation as VOStore, and it anticipated that a FileManager will soon be able to manage files stored in a wide range of VOStore implementations such as SRB and other VO groups' file storage systems. Note that accessing files in MySpace requires access to the appropriate Community service (to locate the services that support that Community's users), a Registry (to resolve access to the Community and the MySpace components) and the relevent FileManager service and FileStore service. This means not only that the service is somewhat delicate but also that the process of resolving a homespace reference is not trivial. See 'connecting to the VO' below.

Data Set Access (DSA)

There are now several standard protocols for accessing data in the VO. Furthermore this data is stored in many different types of database. The AstroGrid DSA component allows data owners to plug their database into the grid, and expose it using the standard IVOA protocols with a minimum of configuration.

IVOA-standard data access protocols include:

SIAP; publishing and accessing image datasets

SIAP is an IVOA standard for accessing images through http-post web services.

SkyQuery; Publishing and Accessing Catalogues

SkyQuery, or ADQL (Astronomical Data Query Language), is an XML form of SQL with some special-to-astronomy additions.

SkyNodes define the SOAP interfaces that dataset publishers should provide as part of their web services. Unfortunately AstroGrid dataservers don't yet provide these complete; they do provide several other ways of carrying out queries using ADQL, including the CEA interface (see below) and http-post.

The Portal

The Portal is the AstroGrid user interface to the Virtual Observatory services. The Portal is a Cocoon-based web portal giving access to the other AstroGrid services. The following portals are currently available:

http://www.astrogrid.org/release	Leicester
http://vo.roe.ac.uk:8080/astrogrid-portal	Edinburgh

For more information see http://www.astrogrid.org/maven/docs/HEAD/portal/index.html

The Astro Client Runtime (ACR)

The ACR is a desktop application that provides access to AstroGrid and external IVOA resources. It duplicates many of the functions of the portal, but since it is Java- rather than html- based it provides a much richer interface. The ACR also allows application access to AstroGrid and so it is discussed in detail in the second part of this document.

1.5 AstroGrid Architecture

AstroGrid's architecture is difficult to describe in a single diagram since every component interacts with almost every other component. The diagram below shows the major relationships, simplified to show only one instance of each component.

The complexity of the interactions between the components is best illustrated by a sequence diagram for a simple Use Case. The following diagram shows what happens when a user logs into the AstroGrid portal, creates a workflow and submits it. This workflow consists of a single step: a call to a CEA application that processes some data held in the user's home MySpace. Almost every operation requires several calls to the Registry. For example, to load data from the user's home MySpace, the Registry is first queried to locate the user's Community. This Community provides the IVORN of the user's homespace. The Registry is then queried again to locate the corresponding FileManager server, and the FileManager then queried for the FileStore where the data is actually held.

1.6 Connecting to the Virtual Observatory

The IVOA's effort is to develop standards for the service interfaces, interchange formats and service descriptions ('metadata') mostly in XML. Some are fairly well settled now, such as many of the service descriptions and the database query language (ADQL). A few prototypes have been developed that are rapidly becoming permanent standards by default. Effort is now underway to construct a global model of all astronomical data everywhere, which unfortunately is delaying the development of practical models. VO-compliant tools must be able to accept and produce these forms where appropriate.

There are however large gaps in the standards and few have 'matured'.

Standard file formats

By far and away the most important IVOA-compliance feature is to be able to read and write VOTables. These are XML documents that associate limited metadata, such as units and terms from an astronomical dictionary (UCDs), with the values in the tables.

Mark Taylor, ex of Starlink, has created and maintains a table reading and writing Java library (STIL) that includes VOTables. The Dataset Access Layer library (DSA) also includes a class for streaming VOTables.

Other common astronomical formats include FITS, HDS, etc. but astronomical tools will already be able to read or write these where appropriate.

Registry entries/Metadata

We expect to create Registry entries for the data mining and visualisation tools, but we do not yet have a standard form for this.

Mining and Visualisation tools will need to use the Registries to resolve access to datasets and supporting processes that might be used for mining, as well as to resolve MySpace entries to users' data files which is detailed below.

Describing data and their relationships is extremely important for some aspects of mining and visualisation. Some progress has been made (UCDs) and it might be useful for some tools to consider using these, particularly when combining data from several different sources. However it is an extremely difficult subject, and the astronomy community has not yet been able to tackle 'ontologies' properly.

Accessing MySpace

The protocol for accessing MySpace is not trivial as it involves looking up the user on a Community server, several resolving calls to a Registry and connecting to a FileManager and FileStore using SOAP.

However, applications wrapped as described in the section Integrating Applications can take advantage of libraries that hide much of the complexity from the application author. The 'ACR' (see below) is a localhost service that can provide access to MySpace files for any language.

There is also an AstroGrid Java Library for Java apps (Slinger - download the UberJar as that includes the classes needed to access the Registries and Community services).

Connecting to AstroGrid Services using CEC

AstroGrid application servers include a special SOAP interface called the 'Common Execution Architecture' interface. This interface specifies more information about the parameters (units, etc) than standard WSDL, and has standard methods for specifying 'listening' services that wish to monitor the application's progress. The interface's parameters, application names, etc, are more fully described by the VOApplication resource entry in the Registry. Therefore connecting to a CEA service is a two step process; the Registry entry must be found, and then the suitable SOAP call made.

Alternatively, all the features of a CEA service are available through the ACR, so it's easier to use that if it's available.

Querying Data using ADQL

AstroGrid DSA dataservers include http-post and http-get access to the data so tools can submit either simple cone queries or full ADQL query documents straight-forwardly.

The ACR also provides a simple route to accessing DSA dataservices through its CEC SOAP interface without having to muck about with SOAP.

2 Integrating Applications into the Euro-VO infrastructure

This section provides a "getting started" guide for application writers who wish to integrate their code into the Euro-VO infractructure, access AstroGrid Registries, MySpace servers or external IVOA services such as SIAP servers. Extensive documentation on this topic is available elsewhere, so we provide a gentle introduction here and refer the reader to the original documents for the details.

2.1 Types of Application

Most applications that wish to connect to the Euro-VO infrastructure will fall into one of two categories which we shall refer to as Client and Service applications. This division is made according to how the application needs to interact with the infrastructure rather than where the application is located. In most cases the former will be "desktop" applications and latter "server" applications but there are exceptions. In the following sections we give guidelines on how you can decide into which camp your application falls and how to wire it up to the Euro-VO infrastructure in each case.

2.2 Service Applications

We use the term Service Application to describe a program that makes available a service to the Euro-VO infrastructure. Such applications are passive, waiting to be invoked by another process and are usually run as batch processes rather than interactively. Typically they will be used as steps in a workflow and will be registered in a VO Registry � an example is the source extraction code SExtractor. The majority of data-mining applications will fall into this category. Almost always the application will run on a remote server. The AstroGrid infrastructure does not (yet) support making code running on a user's desktop available for use in workflows.

The Common Execution Architecture (CEA)

Service applications are integrated into the Euro-VO infrastructure using the Common Execution Architecture (CEA). The CEA is a set of interfaces for asynchronous (web) services that is currently undergoing IVOA-standardization [Harrison]. The CEA offers a number of features over those offered by WSDL-described interfaces including call backs for asynchronous services, more comprehensive parameter metadata and a uniform interface to all VO-applications. For a thorough discussion of the motivation and origins of the CEA see the above IVOA document and also the proposal for a Universal Worker Service [Rixon].

For a service application to be visible to other components in the Euro-VO infrastructure such as the Job Execution Service and Workflow Builder it must offer a CEA interface and have an entry in the Registry. A component that implements the CEA interface is known as a Common Execution Connector (CEC).

Registration

An application that is published to the Euro-VO using the CEA requires at least two Registry entries. The first entry describes the application itself in a non-location-specific way: it includes the usual VOResource metadata plus metadata about the parameters and interfaces offered by the application. A typical entry is that for sExtractor:

        <ns135:ApplicationDefinition xmlns:ns135="http://www.ivoa.net/xml/CEAService/v0.2" xmlns:vm="http://www.ivoa.net/xml/VOMetadata/v0.1" xmlns:vt="http://www.ivoa.net/xml/VOTable/v0.1">
            <ns135:Parameters>
                <ns135:ParameterDefinition accept-encodings="" name="CATALOG_NAME" sub-type="" type="VOTable">

                    <ns136:UI_Name xmlns:ns136="http://www.astrogrid.org/schema/AGParameterDefinition/v1">Derived output catalogue</ns136:UI_Name>
                    <ns137:UI_Description xmlns:ns137="http://www.astrogrid.org/schema/AGParameterDefinition/v1">The catalogue of images derived from the input images</ns137:UI_Description>
                    <ns138:UCD xmlns:ns138="http://www.astrogrid.org/schema/AGParameterDefinition/v1"/>
                    <ns139:DefaultValue xmlns:ns139="http://www.astrogrid.org/schema/AGParameterDefinition/v1"/>
                    <ns140:Units xmlns:ns140="http://www.astrogrid.org/schema/AGParameterDefinition/v1"/>
                </ns135:ParameterDefinition>
                <ns135:ParameterDefinition accept-encodings="" name="GAIN" sub-type="" type="double">
                    <ns141:UI_Name xmlns:ns141="http://www.astrogrid.org/schema/AGParameterDefinition/v1">Gain</ns141:UI_Name>

                    <ns142:UI_Description xmlns:ns142="http://www.astrogrid.org/schema/AGParameterDefinition/v1">detector gain in e-/ADU</ns142:UI_Description>
                    <ns143:UCD xmlns:ns143="http://www.astrogrid.org/schema/AGParameterDefinition/v1"/>
                    <ns144:DefaultValue xmlns:ns144="http://www.astrogrid.org/schema/AGParameterDefinition/v1">0.0</ns144:DefaultValue>
                    <ns145:Units xmlns:ns145="http://www.astrogrid.org/schema/AGParameterDefinition/v1"/>
                </ns135:ParameterDefinition>
 ........
            </ns135:Parameters>
            <ns135:Interfaces>
                <ns291:Interface name="Galactic" xmlns:ns291="http://www.astrogrid.org/schema/CommonExecutionArchitectureBase/v1">

                    <ns291:input>
                        <ns291:pref maxoccurs="1" minoccurs="1" ref="PARAMETERS_NAME"/>
                        <ns291:pref maxoccurs="1" minoccurs="1" ref="config_file"/>
                        <ns291:pref maxoccurs="1" minoccurs="0" ref="SEEING_FWHM"/>
                        <ns291:pref maxoccurs="1" minoccurs="1" ref="DetectionImage"/>
                        <ns291:pref maxoccurs="1" minoccurs="1" ref="IMAGE_BAND"/>
                        <ns291:pref maxoccurs="1" minoccurs="0" ref="FILTER_NAME"/>
                        <ns291:pref maxoccurs="1" minoccurs="0" ref="MAG_ZEROPOINT"/>
                        <ns291:pref maxoccurs="1" minoccurs="1" ref="PhotoImage"/>

                        <ns291:pref maxoccurs="1" minoccurs="0" ref="ANALYSIS_THRESH"/>
                    </ns291:input>
                    <ns291:output>
                        <ns291:pref maxoccurs="1" minoccurs="1" ref="CATALOG_NAME"/>
                    </ns291:output>
                </ns291:Interface>
                <ns292:Interface name="simple" xmlns:ns292="http://www.astrogrid.org/schema/CommonExecutionArchitectureBase/v1">
                    <ns292:input>
                        <ns292:pref maxoccurs="1" minoccurs="1" ref="PARAMETERS_NAME"/>

                        <ns292:pref maxoccurs="1" minoccurs="1" ref="config_file"/>
                        <ns292:pref maxoccurs="1" minoccurs="1" ref="DetectionImage"/>
                        <ns292:pref maxoccurs="1" minoccurs="1" ref="IMAGE_BAND"/>
                        <ns292:pref maxoccurs="1" minoccurs="0" ref="FILTER_NAME"/>
                        <ns292:pref maxoccurs="1" minoccurs="1" ref="PhotoImage"/>
                        <ns292:pref maxoccurs="1" minoccurs="0" ref="ANALYSIS_THRESH"/>
                    </ns292:input>
                    <ns292:output>
                        <ns292:pref maxoccurs="1" minoccurs="1" ref="CATALOG_NAME"/>

                    </ns292:output>
                </ns292:Interface>
            </ns135:Interfaces>
        </ns135:ApplicationDefinition>

where the standard VOResource metadata has been omitted for clarity. This Registry entry describes the application, defines its input and output parameters and provides parameter metadata such as units and default values that can be used by a client to populate a GUI to the application. The Registry entry defines one or more interfaces to the application, specifying different combinations of parameters. Thus the GUI can offer the user a choice of simplified or full interfaces.

Note that the above Registry entry does not provide any information about the location of the application. Instead there is a separate Registry entry for the CEC itself:

        <interface xmlns:vm="http://www.ivoa.net/xml/VOMetadata/v0.1" xmlns:vt="http://www.ivoa.net/xml/VOTable/v0.1" xsi:type="vs:WebService">
            <accessURL use="full">http://astrogrid.jb.man.ac.uk:8080/astrogrid-cea-commandline/services/CommonExecutionConnectorService</accessURL>
        </interface>

        <cea:ManagedApplications xmlns:vm="http://www.ivoa.net/xml/VOMetadata/v0.1" xmlns:vt="http://www.ivoa.net/xml/VOTable/v0.1">
            <cea:ApplicationReference>ivo://org.astrogrid/Galaxev</cea:ApplicationReference>
            <cea:ApplicationReference>ivo://org.astrogrid/Pegase</cea:ApplicationReference>
            <cea:ApplicationReference>ivo://org.astrogrid/SExtractor</cea:ApplicationReference>
            <cea:ApplicationReference>ivo://org.astrogrid/BPZ</cea:ApplicationReference>
            <cea:ApplicationReference>ivo://org.astrogrid/annz</cea:ApplicationReference>

            <cea:ApplicationReference>ivo://org.astrogrid/HyperZ</cea:ApplicationReference>
            <cea:ApplicationReference>ivo://org.astrogrid/CrossMatcher</cea:ApplicationReference>
            <cea:ApplicationReference>ivo://org.astrogrid/Starburst99</cea:ApplicationReference>
            <cea:ApplicationReference>ivo://org.astrogrid/CrossMatcherV2</cea:ApplicationReference>
        </cea:ManagedApplications>

Again, much of the standard VOResource metadata has been omitted. This Registry entry describes the actual location of the CEC (http://astrogrid.jb.man.ac.uk:8080/astrogrid-cea-commandline/services/CommonExecutionConnectorService) and which applications the CEC will run. A CEC can offer several applications. Equally, an application can be offered by several CECs. This allows the user to select an application without needing to know where it will execute and leaving the Job Entry System to load balance by selecting the most appropriate CEC to actually invoke. More details about the Registry entries required for a CEA-application can be found at http://www.astrogrid.org/maven/docs/HEAD/applications/multiproject/astrogrid-cea-commandline/REGISTER.html .

The AstroGrid CEA components described below provide some assistance in registering applications.

Using AstroGrid CEA components to publish an application to the Euro-VO

AstroGrid has created Java components to act as adaptors and offer CEA interfaces to legacy applications. In most cases the quickest way to VO-enable a service application will be to deploy an AstroGrid CEA server in front of it rather than offering the CEA interface natively. The CEA is currently available in three flavours and the one to use depends on the type of legacy application.

Java Applications

If the service application has been written in Java then publishing it to the Euro-VO can be done using the astrogrid-cea-server.war available from http://www.astrogrid.org/maven/org.astrogrid/wars/?C=M&O=D&P=astrogrid-cea-*1.1-000*.war . This is a webapp that needs to be deployed to a container such as Apache Tomcat and configured to locate the target application. The astrogrid-cea-server webapp then uses Java-reflection to deduce the interface and create Registry entries. The application author should create an interface that just deals with simple Java types such as ints, doubles and strings. The astrogrid-cea-server will handle any conversion to and from URLs and MySpace/VOSpace references.

Deployment and configuration of the astrogrid-cea-server.war can be done in a number of ways but the simplest is to use the GUI installer, which is described below.

1. Prepare the target application
   • Create an "interface class" that exposes the interface to the application as public static methods. For example a service that offers arithmetic operations could have the class:

          public class MyApplicationInterface {
            public static int add(int a, int b) {
                return new ApplicationImpl().add(int a, int b);
            }
            public static int sub(int a, int b) {
                return new DifferentClassImpl().sub(int a, int b);
         }

• • Create a directory and place in it all the jar files for the application

1. If necessary download and install Tomcat 5.0+, ensuring that the Manager Application is enabled.
2. Download the latest installer from http://www.astrogrid.org/maven/org.astrogrid/apps/?C=M&O=D&P=astrogrid-cea-installer*.jar . The installer is an executable jar file and can be executed by double-clicking in Windows-like environments. Alternatively from the command line execute:

         java -jar astrogrid-cea-installer-VERSION.jar

or in a headless environment:

         java -jar astrogrid-cea-installer-VERSION.jar -h

1. Follow the instructions in the installer. You will be invited to choose whether to register the application. If you choose not to then you can register it later via webpages provided by the astrogrid-cea-server webapp. Amongst other things you will be prompted for basic metadata about the application including its IVORN, the type of the application (Java), the fully qualified name of the Java Class containing the interface (MyApplicationInterface.class in the above example) and the directory containing the application's jar files.

Note that Java-reflection is used to introspect the interface class file and generate the Registry entry for the application. Java-reflection is not able to deduce parameter names and therefore the Registry entry should be hand-edited to improve the user-orientated meta-data such as parameter names, units and description. AstroGrid Registries come with web pages to facilitate this - please consult the documentation for your specific Registry.

Command line Applications

Non-Java applications that can be compiled as command line applications can be given a CEA interface by deploying a astrogrid-cea-commandline.war, also downloadable from http://www.astrogrid.org/maven/org.astrogrid/wars/?C=M&O=D&P=astrogrid-cea-*1.1-000*.war

Deploying a CEA server for a command line application is similar to doing so for a Java application but slightly more complicated. Detailed instructions can be found on the AstroGrid website at http://www.astrogrid.org/maven/docs/HEAD/applications/multiproject/astrogrid-cea-commandline/ApplicationConfiguration.html but, briefly, the main differences are:

• The command line application must be placed in a directory where it can be executed by Tomcat.
• Since Java-reflection is not available an XML configuration file must be created describing the parameters that the application takes. This configuration file is used by the astrogrid-cea-commandline server to execute the application and also to generate the application's Registry entries.

An example of the configuration file can be found at http://www.astrogrid.org/viewcvs/*checkout*/astrogrid/applications/commandline/config/CeaApplicationConfig.xml?rev=HEAD&content-type=text/xml

The same installer used for Java applications can deploy an astrogrid-cea-commandline server. When prompted select "commandline" for the CEA flavour. Later in the installation process you will be prompted for the location of your configuration file.

Http Applications

A version of the astrogrid-cea-server is also available to act as a CEA proxy for existing web applications such as SIAP servers. This is beyond the scope of this document.

2.3 Client applications

In contrast to Service applications Client applications initiate any interactions with the infrastucture. Often these will be desktop applications and many visualisation tools such as Aladin, Topcat and VisIVO fall into this category. However some applications running on a remote server will also fit this description. One such example is the Remote Visualisation System (RVS) where the RVS client will send a message to the remote RVS Server instructing it to get access data via the Euro-VO.

Typically such applications will need to request files from VOSpace, authenticate a user in a Community and use a Registry to locate VO-resources. It is also likely that some applications will need to execute a CEA application or query a database through the DSA.

Interacting with Euro-VO components

As described above the Euro-VO infrastructure components created by AstroGrid have SOAP web service interfaces described by a WSDL document and are thus designed to not be tied to any particular platform or language.

An application writer who wants to use Euro-VO services has several options (in order of increasing ease of use):

Direct access to the web services

Direct access to Euro-VO components is available by sending and receiving SOAP messages via their webservice interfaces. Libraries for SOAP exist for all the major languages (see Appendix). Clients will require the Web Services Definition Language document for each service. These are available from the AstroGrid Software site at http://software.astrogrid.org/schemata/index.html.

Direct use of the web services requires detailed knowledge of the semantics of the components' interfaces which is beyond the scope of this document.

Access via Java Delegates

For the convenience of applications written in Java AstroGrid provides a set of delegate libraries which offer a Java API to the components and handle the conversion to and from SOAP messages transparently.

Some of the delegates are quite sophisticated and encapsulate some of the semantics of interacting with the Euro-VO infrastructure. For example a single method in a class might take a user's credentials (username, password, Community), look up the location of the Community in the Registry, contact that Community, log the user in and receive back another delegate to that user's home FileManager.

A tar or zip file of the AstroGrid delegate libraries and associated jar files is available as the AstroGrid Client Developer Kit, downloadable from the software site (see Appendix). The Appendix also has links to API documentation for the various delegates. This approach has been used successfully by both Aladin and Topcat to access VOSpace.

Access via the Astro Client Runtime

The preferred method for accessing AstroGrid components, and one that is appropriate for clients written in any language, is to use the Astro Client Runtime (ACR). The ACR is a Java application that will usually be co-located with the client application. It can be embedded in a Java application if desired but usually it will run in its own process space allowing a single instance to be shared amongst multiple client applications.

The ACR is downloaded, installed and executed using Java Web Start. The user is required to have Java installed on his machine but since this is necessary for running many applications such as Topcat, Aladin and the AstroGrid Desktop this is not unreasonable.

The ACR is automatically started when the AstroGrid Desktop is launched but can also be run on its own as a background process. It offers clients a unified interface to the Euro-VO infrastructure and external IVOA services via a number of protocols including Java-RMI and the language-neutral xml-rpc. The ACR then handles all the communication with the remote components.

This has the following benefits:

• The ACR offers a simpler interface than the components' own SOAP interfaces since it encapsulates a lot of the semantics of dealing with the infrastructure.
• The ACR can cache Registry and VOSpace data, improving the user experience when several different client applications need to access the same data.
• The ACR can deliver single sign-on � once you have authenticated with one client using the ACR all other clients will be automatically signed in.
• The ACR comes bundled with some commonly-needed dialogs such as VOSpace explorers and authentication dialogs. Thus a text-based client such as IDL could request a file from VOSpace and a graphical dialog will open to allow the user to select the file.

There are two chief disadvantages to using the ACR:

• It requires Java to be installed on the client machine,
• It must be started before an application can contact the Euro-VO infrastructure.

The first of these is unlikely to be a concern since Java will be required to run many VO applications and will be available on most desktops. The second concern is easily addressed if the client application has a means to control the user's web browser. The client application simply instructs the browser to fetch the ACR's .jnlp file from the web. This is will download the ACR (if necessary) and start it (if not already running). Note that the ACR is written as a singleton so multiple requests to start it will result in only a single instance. All applications that are delivered via Java Web Start have the means to control the user's browser and the ACR libraries provide a convenient method to start and locate the ACR instance. This is illustrated in the xmdv case study in the appendix. Non-Java and non-Java Web Start applications will need an alternative way of starting the ACR, including simply instructing the user to do it.

Please refer to the appendix for case studies illustrating the use of the ACR.

Locating, installing and executing Client Applications

Locating

Unlike Service applications, which by definition need to be automatically located to be of use, there currently is no standard way to locate Client applications. However several groups have begun the task of assembling lists of useful applications. These include the DS6 report on visualisation tools http://wiki.eurovotech.org/bin/view/VOTech/DS6SoftwareSurvey , the DS4 study on existing tools http://wiki.eurovotech.org/bin/view/VOTech/StudyOnExistingTools and the Astronomical Software and Documentation Service http://asds.stsci.edu/ .

The IVOA meeting in October had a session devoted to storing application metadata in Registries but did not reach a conclusion. Discussions are on-going, and will include the question of how to register client-side applications.

Installing and Executing

Desktop applications are written in many different languages, for a variety of platforms, and so there is no single way that such applications are installed on the user's machine and executed. However it is worth mentioning one particular technology that has proved to be a powerful tool for distributing and maintaining desktop applications.

Java Web Start is a technology for deploying Java-based applications to the user's desktop. The user simply clicks on a link on a web page and the web browser downloads, installs and executes the application, after first asking the user's permission. Subsequently the application can be executed offline. When started on-line Java Web Start first checks to see that the application is up to date and, if necessary, downloads and installs a new version of the application. The only requirement is that a recent version of Java has been pre-installed on the desktop. Aladin, Topcat and the ACR are all available via Java Web Start.

An interesting aspect is that Java Web Start is not limited to pure Java applications. If the application requires "native" libraries (e.g. DLLs on Windows or .so shared libraries on Linux) then these can be packaged for deployment along with the Java code. Java Web Start automatically only requests the libraries that are appropriate for the user's operating system. Java Web Start should therefore be investigated to see if it is a suitable deployment medium for non-Java applications.

Visualisation Tool Interoperability

Recently several visualisation tool teams have made progress on tool interoperability. Notable achievements include VisIVO's ability to control Aladin through its scripting interface and Aladin's inclusion of several other tools via its plugin interface. The benefits are obvious: each tool can concentrate on what it does best, and the effort involved in making tools interoperate will be significantly less than reimplementing existing functionality in each tool.

To build on this, the DS6 group has formed a team to examine ways to allow tools to collaborate in a more general fashion. This project, christened "PLASTIC" is still its early stages, but the interested reader is referred to the draft PLASTIC specification available from http://plastic.sourceforge.net . An implementation of PLASTIC will be available as a module in the ACR before Christmas 2005.

Appendix

ACR Case Studies

Case study: xmdv-lite

Xmdv-lite is a small visualization tool produced by Matthew Ward's group at WPI. The tool is written in Java and uses native OpenGL libraries (GL4Java ).

Java Web Starting

The first step in VO-enabling xmdv-lite was to make it available via Java Web Start. Not only does this make it easier for users to run but, as mentioned above, all Java Web Start applications can control the user's browser and thus we are able to exploit this to automatically start the ACR. Packaging the application for Java Web Start is straightforward. The first step is to create a JNLP file describing the application and including the URLs of all the resources. This JNLP file is reproduced below for reference

<jnlp spec="1.0+" codebase="http://software.astrogrid.org/votech/ds6/xmdv/" href="http://software.astrogrid.org/votech/ds6/xmdv/xmdv.jnlp">
  <information>
    <title>XMDV-Lite Version vo-0.1</title>
    <vendor>Matthew. O. Ward et al</vendor>
    <homepage href="http://davis.wpi.edu/~xmdv/"/>
    <description>This is a VO-enabled version of XmdvTool.  XmdvTool Light Java Version is a light-weight version of XmdvTool. It contains the four major multi-dimensional displays of XmdvTool: parallel coordinates, star glyphs, scatterplots, and dimensional stacking. It also supports the basic brushing: users can manipulate a brush to highlight a subset of data items. </description>
    <description kind="short">XmdvTool Light Java Version is a light-weight version of XmdvTool</description>
    <description kind="tooltip">XmdvTool</description>
    <icon href="http://software.astrogrid.org/votech/ds6/xmdv/04.gif" kind="default"/>
    <icon href="http://software.astrogrid.org/votech/ds6/xmdv/xmdvlogo.gif" kind="splash"/>
    <related-content href="http://davis.wpi.edu/~xmdv/">
      <title>Xmdv Home page</title>
    </related-content>
    <related-content href="http://www.astrogrid.org">
      <title>AstroGrid</title>
      <description>The main AstroGrid site</description>
      <icon href="http://software.astrogrid.org/votech/ds6/xmdv/aglogo.gif"/>
    </related-content>
    <related-content href="http://software.astrogrid.org/votech/ds6/xmdv/launchacr.html">
      <title>Launch ACR</title>
      <description>Launch the AstroGrid Client Runtime to enable MySpace access</description>
      <icon href="http://software.astrogrid.org/votech/ds6/xmdv/package.gif"/>
    </related-content>
    <offline-allowed/>
  </information>
  <security>
    <all-permissions/>
  </security>
  <resources>
    <j2se initial-heap-size="67108864" max-heap-size="268435456" version="1.3+"/>
    <jar href="http://software.astrogrid.org/votech/ds6/xmdv/xmdv-lite-vo-0.1.jar" download="eager" main="true"/>
    <jar href="http://software.astrogrid.org/votech/ds6/xmdv/gl4java.jar" download="eager" main="false"/>
    <jar href="http://software.astrogrid.org/votech/ds6/xmdv/acr-interface-1.1.jar" download="lazy" main="false"/>
    <jar href="http://software.astrogrid.org/votech/ds6/xmdv/commons-logging-1.0.3.jar" download="lazy" main="false"/>
    <jar href="http://software.astrogrid.org/votech/ds6/xmdv/rmi-lite-1.0.jar" download="lazy" main="false"/>
    <jar href="http://software.astrogrid.org/votech/ds6/xmdv/images.jar" download="eager" main="false"/>
  </resources>
    <resources os="Windows">
      <nativelib href = "natives-win32.jar" />
    </resources>
    <resources os="Linux">
      <nativelib href = "natives-linux.jar" />
    </resources>
  <application-desc main-class="xmdv.xmdv_interface.mainFrame"/>
</jnlp>

All of the application's resources such as binary code and images must be packaged as jar files and placed on the web server. If the application requires access to the local disk then the jars must be signed. In the case of xmdv-lite some slight modification to the code was required to ensure that images etc were loaded from the jar files. For instance

jRadioButton1.setIcon(new javax.swing.ImageIcon("images" + java.io.File.separator + "imgpa.gif"));

was changed to

jRadioButton1.setIcon(new javax.swing.ImageIcon(this.getClass().getResource("/imgpa.gif")));

and imgpa.gif placed in the top level of a jar file on the classpath.

Note that separate jar files are provided with the native GL libraries for each platform. For Windows the jar file contains:

GL4JavaJauGljJNI14.dll

while on Linux

libGL4JavaJauGljJNI14.so

It is important to respect the naming convention NAME.dll on Windows and libNAME.so on Linux.

An Ant script was written to perform the reasonably complicated task of compiling, packaging, signing and deploying the webstart code. For reference this is available here: VOTechDsSixXmdvAntScript .

VO-enabling

The second step was to give xmdv-lite the capability of loading its data files from VOSpace. Using the ACR this is almost trivial. Since xmdv-lite is a Java application it can use the ACR's Java-RMI interface and use the very convenient Finder class to locate and start the ACR. A new menu item "File/Open From MySpace" was added which executes the following code:

//Start an ACR if we don't already have one
Finder f = new Finder();
ACR acr = f.find(); 

//Open a VOSpace browser and wait until the user selects a file
ResourceChooser browser = (ResourceChooser) acr.getService (org.astrogrid.acr.dialogs.ResourceChooser.class);
URI uri = browser.fullChooseResource("Select .okc file",true,false,false);
//Using the URI of the selected file, obtain an InputStream on the remote file           
Myspace myspace = (Myspace) acr.getService(Myspace.class);
InputStream is = myspace.getReadContentURL(uri).openStream();

This InputStream was then passed to the existing code and the data loaded. The VO-enabled version of xmdv-lite is available from here.

Case study: Python scripting

Communicating with the ACR via RMI is easy but is only possible for Java applications. Fortunately the ACR offers several other protocols, the simplest of which is xml-rpc. Libraries for xml-rpc exist for all major languages (see Appendix). For example the Python equivalent of the above code is:

# Assume that the ACR is already running...
# For illustration purposes this code includes the lines required to make the initial
# connection to the ACR.  Pythonners should also check out the astrogrid.py module
# bundled with the ACR that handles starting the ACR and making the connection for you.
import xmlrpclib as x
import sys
import os

#Obtain URL of the ACR's locally running webserver which listens
#for xml-rpc requests.
#The file ~/.astrogrid.desktop is generated by the ACR on startup

prefix = file(os.path.expanduser("~/.astrogrid-desktop")).next().rstrip()
print "ACR Path:" + prefix
s = x.Server(prefix + "xmlrpc")
#Open a VOBrowser dialog and get the IVORN of the chosen file
ivorn = s.dialogs.resourceChooser.fullChooseResource("Select .okc file", True, False, False)
print "IVORN of selected resource is: "+ivorn
#Get the URL of the selected file
url = s.astrogrid.myspace.getReadContentURL(ivorn)
print "URL of file is " + url

Example python scripts demonstrating all the major ACR features come bundled with the ACR.

Case study: Aladin

Thomas Boch has recently published a report describing his experience integrating Aladin with the ACR. It is available from http://wiki.eurovotech.org/bin/view/VOTech/UsageOfAcrApiInAladin .

Further Information

The ACR is a very convenient way of accessing all VO services, especially Euro-VO components. Thorough documentation on the ACR and its uses, including code samples, is available from the main ACR page at http://www.astrogrid.org/maven/docs/HEAD/desktop/multiproject/acr-interface/apidocs/index.html

Links

Main AstroGrid Site

• http://www.astrogrid.org

The AstroGrid Software Site

• http://software.astrogrid.org
• Download Euro-VO infrastructure software from http://software.astrogrid.org/installdocs/index.html
• WSDL documents and other schemata http://software.astrogrid.org/schemata/index.html
• The Client Development Kit http://software.astrogrid.org/developerdocs/cdk.html
• The AstroGrid Client Runtime http://software.astrogrid.org/userdocs/workbench.html
• API Documentation for the latest release http://www.astrogrid.org/maven/docs/HEAD/

Visualization Tools mentioned in the text

• Aladin http://aladin.u-strasbg.fr/aladin.gml
• Topcat http://www.star.bris.ac.uk/~mbt/topcat/
• xmdv-lite http://davis.wpi.edu/~xmdv/
• RVS http://www.atnf.csiro.au/people/Anil.Chandra/rvs.htm

Libraries

• xml-rpc libraries http://xmlrpc.scripting.com/directory/1568/implementations
• SOAP libraries http://dmoz.org/Computers/Programming/Internet/Service-Oriented_Architecture/Web_Services/SOAP/Implementations/

Glossary

See the Main.AstroGrid glossary

-- John Taylor - Nov 2005

-- MartinHill - Nov 2005

DsSixToolIntReportNotesAndErrata