Main Page

From VxOware

Jump to: navigation, search

Documentation for VxOware


  1. News
  2. Abstract
  3. The Section
  4. Search: Internal
    1. Free Search
    2. Attribute Search
    3. Technical Details
  5. Search: External
    1. Configuration
      1. With text editor
    2. Web interface
    3. REST
    4. Query and Result
      1. Query
        1. Capabilities query
        2. Basic Query
        3. Query Option: noOutput
        4. Query Option: document
        5. Query Option: shareExternalSources
        6. Query Option: idsList
        7. Query Option: pointSearch
        8. Query Option: dateStart and dateEnd
      2. Result
      3. Web interface
    5. XSL Result Parser
      1. REST Request
      2. Web interface
      3. Example: KML-XML-SPIDR
      4. Java Result Parser
  6. Search: eXist client
  7. Adding Metadata
    1. Create Record
    2. Upload Records
    3. API
  8. Creating a Section
    1. Clone
    2. Manual
      1. The section schema
      2. The section template
      3. The section tree
      4. The section XSL
      5. Edit vo_description.xml
      6. Create Index
  9. Linking and Tagging
  10. Viewers
  11. Plug-ins
  12. Admin Tasks
    1. Batch modification of records
    2. Command line eXist
  13. The forum file
  14. Configuration
    1. vo_description.xml
    2. Config files
      1. mail.conf
    3. vo_store.conf
    4. vo_meta.conf
    5. textindex.conf
    6. users.conf
    7. Logging
  15. Installation
    1. General Instructions
      1. Pre-requisites
      2. Install and configure Java
      3. Install and configure Tomcat
      4. Install MySQL
      5. eXist
        1. eXist 1.4
        2. eXist 1.2
        3. Common to 1.2/1.4
      6. Install
      7. Set default databases
      8. Move config files
      9. Modify
      10. Modify web.xml
      11. Apache config
    2. Eclipse
      1. eXist Plugin
    3. EC2 Instructions

1. News

2. Abstract

The recent Heliophysics Virtual Observatory (VxO) effort involves the development of separate observatories with a low overlap in physical domain or area of scientific specialization and a high degree of overlap in metadata management needs. VxOware is a content and metadata management system. While it is intended for use by a VxO specifically, it can also be used by any entity that manages structured metadata. VxOware has many features of a content management system and extensively uses the W3C recommendations for XML (Extensible Markup Language), XQuery (XML Query), and XSLT (Extensible Style Sheet Language Transformations). VxOware has features such as system and user administration, search, user-editable content, version tracking, and a wiki. Besides virtual observatories, the intended user-base of VxOware includes a group or an instrument team that has developed a directory structure of data files and would like to make this data, and its associated metadata, available in the virtual observatory network. One of the most powerful features of VxOware is the ability to link any type of object in the observatory to other objects and the ability for every object to be tagged.

Paper in Earth Science Informatics: [1] | cached copy of journal article

3. The Section

From [2]

The section is the most basic grouping of metadata records in VxOware. A virtual observatory using an implementation of VxOware may contain any number of sections, each of them having different types of metadata records. Each section has an associated XML schema and may have a render XSLT file. For example, one section may contain metadata records that follow the RSS (Really Simple Syndication) document model. When a user uses a web browser selectsto view a metadata record from this section, an XSLT transform is applied to the raw XML to render it in HTML. Besides a render XSLT file, the section may have an edit XSLT file that creates an editable HTML form. Figure 12a showis a screenshot in which athat shows a previously stored metadata record that has been rendered in HTML using a render XSLT file that is associated with that section. Figure 12b shows the rendering of the same metadata record using an edit XSLT file. The XSLT file uses the raw metadata to create links to other services. For example, the Google and Yahoo links do a search on the person’s name, the NASA/ADS link does a search for journal and conference abstracts in NASA’s Abstract Data Service, and the address link does a search on the address in Google Maps.
Any XML metadata schema can be used to form a section. The predefined VxOware sections include News, which is used for notifications and automatically generated messages and is also the source of the VxOware RSS feed; Blogs, which contains records for user notes;, Message Boards, which contains discussions between the users; and Documentation, which has an editor that accepts wiki-formatted text. Other schemas have been used in VxOware and have render and edit XSLT files that are included in the application installation package available. These schemas include SPASE (Space Physics Archive Search and Extract) [3], FGDC (Federal Geographic Data Committee) [4], WISER (Wireless Information System for Emergency Responders) [5], and OE (Ordering Extensions) [6].

4. Search: Internal

4.1. Free Search

Free text search (left box on main page and text box in a given section) uses eXist full-text search on all sections having an associated special file in eXist settings folder /system/config/db/path/to/section/collection.xconf. By default, all sections have this applied because the default database you install in eXist when VxOware is installed has the file /system/config/db/myVxO/collection.xconf in the root of the VxO tree. (You may want to change this if size of db gets large and memory is an issue). For more about full-text search configuration, see eXist documentation [7].

Pattern matching is allowed (also in free-text and advanced search, but not "outer search"). For example, searching on a time as in 2008-01* will result in all documents with that year and month as the value for any element in the record (and the forum record if SQL search is used).

Both of the operators &= and |= accept simple wildcards in the keyword string. A ? matches zero or one character, * matches zero or more characters. You may use a backslash to escape wildcard characters.

Example: To find "havana" you can use:

  • havana
  • ha*na
  • ha?a?a
  • ha*

When you do a section-specific search, either eXist or SQL is used depending on if IS_FAST_INDEX is set in vo_description.xml (See Configuration). When a section is created with IS_FAST_INDEX set, any documents that are created are stored in eXist are also copied into a SQL database and the search is performed on the SQL database contents. Note that when IS_FAST_INDEX is set in vo_description.xml for a section, the admin must also create the SQL database in the SQL Index section of the Admin management page.

There are a few small differences between SQL and eXist search (see also #Technical_Details):

  • In SQL mode, the data that are indexed include information from the forum file (such as upload date, etc.; See the SQL browser in the admin section for more information). To see the difference, do a search on a date when a file was uploaded with and without IS_FAST_INDEX set to true.
  • In SQL mode, the search has a slightly different notation, for example, the + and - characters at the beginning and end are interpreted differently [8]. These characters may be escaped by double quoting the entire search string.
  • In SQL mode, space-separated words are connected by OR; in XML they are connected by AND.

4.2. Attribute Search

A IS_FAST_INDEX section can have special attributes associated with each record. A connection can be made between an element in the schema that can be used to mean FROM and TO coverage dates. This is indicated by placing


in the appropriate field in the schema file for that section. See the section "CoverageDemo". Other options include:


When searching a section with any of these special attributes, an entry box will appear for search on either time or space. The spatial search uses

SELECT count(*) FROM `test_f` INNER JOIN `test_p` ON `test_f`.pathId = `test_p`.id AND `documentPath` LIKE '/Test/%' AND (contains(`spcover`, GeomFromText('POINT(0 0)')))

If one document does not have temporal coverage information, the value of W=-180,S=-90,E=180,N=90 is assumed. If no start time is given, only the stop time is used for a constraint. If no stop time is given, only the start time is used for a constraint. All documents with no start or stop time will match any constraint.

4.3. Technical Details

Example: Debug output from SQL search on 2010 (using Person section):

SELECT count(*) FROM `spaseperson_f` INNER JOIN `spaseperson_p` ON `spaseperson_f`.pathId = `spaseperson_p`.id AND `documentPath` LIKE '/SPASEPerson/%' AND (MATCH (`viewCount`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`replyCount`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`publicationDate`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`pathId`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`periodStart`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`periodEnd`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`keyVersion`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`keyPersonName`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`keyResourceID`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`keyReleaseDate`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`keyOrganizationName`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`keyAddress`) AGAINST ('2010' IN BOOLEAN MODE) OR MATCH (`keyEmail`) AGAINST ('2010' IN BOOLEAN MODE))

Example: Debug output from XML search on 2010 (using Wiki section):

In terms of eXist notation, the search syntax is node-set &= 'string of keywords'. This operator selects context nodes containing ALL of the keywords in the right-hand argument in any order. The default tokenizer is used to split the right-hand argument into single tokens, i.e. any punctuation or white spaces are used to separate the keywords and, after which, are omitted. Note also that wildcards are allowed, and keyword comparison is NOT case-sensitive. (There is also |= operator available, for searching ANY of the keywords. This is not used, but could be in principle.)

5. Search: External

An external search may be made on VxO sections that are available using a framework of web-services. Thus, a p2p VxO federation may be created which allows for the interaction and data exchange across different application domains. To make a section available for external search, an administrator must specify a list of sections that are web-service accessible along with a list of metadata fields that are available for searching and displaying.

The external search web-service is available via REST protocol. The web-service returns a configuration-based self-description of VxO (information about the available capabilities) with a list of sections open for search. On requesting the information for a certain section, the web service returns a list of meta-record elements (and possibly a set of valid values), with values that may be used to find objects in this section. With this self-description, a context search query can be created which results in a list of objects. An important feature of the external search web-service is its dynamic result format. You may choose which of the available meta-record fields you need to see in the query result. These properties of the web-service make it suitable for case-based reasoning applications.

Searching in a time field with, for example, 2008-01, will result in all documents with that year and month.

5.1. Configuration

5.1.1. With text editor

External search capabilities are set in the VxO configuration files.

Observatory sections can be opened for the external search by the settings of the file /conf/vo_description.xml

<?xml version="1.0" encoding="UTF-8"?>
            <NAME>Documents</NAME> <!-- sectionId -->

The list of sections available for the external search is reported in the VO capabilities document.

A list of elements exposed for the external search is set in the observatory section configuration file /conf/[sectionId]/[sectionId]Schema.xml

<?xml version="1.0" encoding="UTF-8"?>
        <KEY>keyTitle</KEY> <!-- parameterId -->
        <IS_CAPABILITIES use="identification"/>

The list of searchable elements in a section is reported in the VO section capabilities document.

5.2. Web interface

Configuration is available on admin management page.

  1. External search is available only via link 1
  2. The list of known external sources is given in list 2
  3. You can add a new source in field 3
  4. The sources that should be available via the search pages (current VxO interface) are marked with checkboxes (4)
  5. The changes to the external sources list are committed by pressing button 5
  6. Press button 6 to automatically search for sources. When this button is clicked the system walks through all the known sources (present in the list) and makes a “shareExternalSources” request as shown below. All found sources are added to the common list.


  1. After clicking the “search source” button the known source are queried (group 1)
  2. The sources return the addresses of their known sources. The new addresses are added to the common list (group 2)
  3. VO administrator may set checkboxes for some of the new entries to make them available through the current VO interface

5.3. REST

When an xml file is uploaded into a subdirectory of an observatory section, the filename is stored internally and the xml file is given a document name that is a 36-digit hash (such as BE41D3AB-FE78-DC54-6379-2C2AB6668B96). For example, if you upload the file Robert.S.Weigel.xml, it will appear in the Exist db as virbo/SPASEPerson/BE41D3AB-FE78-DC54-6379-2C2AB6668B96.xml. You may access the xml file by filename and directory path or by its hash name. The raw xml is accessible via

This will also work:

Associated with each uploaded object is a forum object which includes information such as the name of the person who uploaded the object, the number times it was viewed, revision history, tags, cross-references, and discussion. To access the uploaded object in the observatory with a browser and see the HTML rendering of the uploaded metadata object and its associated forum object, use

The raw xml for the form object is also accessible:

5.4. Query and Result

Each parameter in self-description may be used for searching by element. Each parameter may be returned in search results

5.4.1. Query Capabilities query

This document describes the observatory structure, its sections available for outer search

VO self-description format (http://vo_app/

<?xml version="1.0" encoding="UTF-8"?>
   <NAME>VO name</NAME>
      <NAME>section name</NAME>
          <DESCRIPTION>section description</DESCRIPTION>

This document describes the records contained in sections. Each parameter describes the document’s XML element available for outer search

Section self-description format (http://vo_app/[sectionId])

<?xml version="1.0" encoding="UTF-8"?>
    <NAME>search parameter name</NAME>
    <DESCRIPTION>parameter description</DESCRIPTION>
    <TYPE use=“[notes]">parameterType</TYPE>
    <VALUE>parameter values (optional)</VALUE>
 </CAPABILITIES> Basic Query

http://vo_app/outersearch?section=[sectionId]&[parameterId]=[searchString]&[parameterId(2)]=[searchString(2)]&output=[parameterId(3)]&output=[parameterId(4)] Query Option: noOutput

If «output» parameters are missing, the result will contain only the number of found records

<?xml version="1.0" encoding="UTF-8"?>
 <result found_items=“[int]>

This parameter can be used to obtain only the list of found documents, for example:

<?xml version="1.0" encoding="UTF-8"?>
 <result found_items="508">
	<item doc_name="GeomStationsNPL">noDetails</item>
 	<item doc_name="GeomStationsEPN">noDetails</item>
 	<item doc_name="GeomStationsWES">noDetails</item>
 </result> Query Option: document

Returns a full XML document stored in VO by its ID (applies only to records stored in observatory sections)

http://vo-app/outersearch?specialRequest=document&docId=[documentId] Query Option: shareExternalSources




<?xml version="1.0" encoding="UTF-8"?>
        <!-- the underscore before the S is only there to avoid a mediawiki rendering problem--> 
 	<_SOURCE capability="true">http://vo-app1/</_SOURCE> 
 	<_SOURCE capability="true">http://vo-app2/</_SOURCE> 

A list of external observatories (federation) is returned. These observatories are known by the requested one. The attribute “capability” indicates whether the external observatory can be searched by the outersearch web service at the requested one. Query Option: idsList


The “idsList” value can specify directly a list of documents IDs to be exported, rather than the document search.

This document ID filter can be used in addition to the document search filter, to search only for the documents from the idsList. Query Option: pointSearch


Search for the geographic location (latitude - longitude pair, space separated). The result will be all geolocated records, which include the point into the geolocation rectangle.

This geolocation filter can be used in addition to other document search filters. Query Option: dateStart and dateEnd


Is used to search for the date range. Only documents with time stamp within the date range will be returned.

  • This date range filter can be used in addition to other document search filters.

5.4.2. Result

Search results are returned as a file, where “item” corresponds to a found document and “element” corresponds to a parameter declared as “output”

Search result format

<?xml version="1.0" encoding="UTF-8"?>
<result found_items=“[int]">
 <item doc_name=“recordId">
   <element_title>parameter name</element_title>
    <value>parameter value</value>

5.4.3. Web interface

Searchable Section List
Searchable Section List
Query Page
Query Page
For each document only requested elements are returned
For each document only requested elements are returned

5.5. XSL Result Parser

5.5.1. REST Request


“resType” – optional parameter, if it is absent – return XML

5.5.2. Web interface

5.5.3. Example: KML-XML-SPIDR

This document describes how to transform XML stream into KML with external search results from the VxOware by applying an XSL file (referred by URL).

Real world example of such XSL can be found at

Example of the VxOware “outersearch” request modified with the above XSL:

We have a special web form in the VxOware to experiment with the XSL transformations of the outersearch results; when the XSL transformation will give a satisfactory result, one can copy the search request URL from the search results address line (in web browser) and use it in another application.

5.5.4. Java Result Parser

Java class to parse the search results document. (For more details see VxOware sources.)

String searchRequestUrl = “[request]”;
ResultStructure searchResult= new ResultStructure(searchRequestUrl);

Compare Java method call with the XML(xPath) query:


6. Search: eXist client

Launch eXist webstart, select binoculars and enter:

for $doc in /* [text:match-all(.,'Filename')] return
concat(util:collection-name($doc), '/', util:document-name($doc))

to search for word Filename in all parts of all documents starting at the root of the eXist db. The documentation for eXist has many other examples.

7. Adding Metadata

Metadata may be added using

  1. Create record
  2. Upload record
    • URL Collection
    • Single XML File or ZIPed collection
  3. API

In all cases, the record is stored in a specified section and the docname is given a hash value. A human-readable URL may be specified in which case the filepath URL will show a regular filename for the record (given in the "Alternative URL" link").

7.1. Create Record

Using the "Create Record" button, you may create and/or edit metadata in a given section by entering information in a web-based form (login required). If the section has


set in vo_description.xml, then there is an option to give a name to the record that will appear as the filepath URL.

7.2. Upload Records

Using the "Upload records" button, you may upload a single XML file or a zip file with XML metadata files. See the URL Collections section for examples of URL Collection XML files. Name used in the filepath URL corresponds to the filename specified in the URL of the ingested record.

7.3. API

First, download the command-line tool from SVN: [9]

Example: modify an existing record using the VxOware API and command line tools.

Then, view existing metadata at

Download XML file

curl -o ./xml/SPASEPerson/PersonTest.xml

Make manual edit to file at ./xml/SPASEPerson/PersonTest.xml

Upload modified file

./ ./xml

View revised file using path ("Alternative URL")

View revised file using the document ID.

8. Creating a Section

8.1. Clone

You can clone a section by using the Admin management page. If the section is SQL-indexed, you must execute "Create index" in the Admin management page. See the following for a description of the section configuration files.

8.2. Manual

Start the eXist webstart client by going starting the web start program

Create a directory named conf/Help

8.2.1. The section schema

Create a document called helpSchema.xml. This file is used for generating the edit page. Note that attributes in an element are ignored. It is not possible to set up a section where attributes can be specified as an editable parameter in the edit form. However, attributes may be used in the render xsl file.

   <SCHEMA_TITLE>News addition form</SCHEMA_TITLE>
   <SCHEMA_DESCRIPTION>Please fill in the following fields according to the descriptions:</SCHEMA_DESCRIPTION>
       <DESCRIPTION>The name of the search result provider.</DESCRIPTION>
       <NOTE>Notes: Should contain the search terms themselves. Restrictions: Can not contain HTML markup. Required</NOTE>
       <IS_CAPABILITIES use="identification"/><!-- List this field in capabilities file for external search.-->
       <DESCRIPTION>A URL to the website that providers the search results.</DESCRIPTION>
       <NOTE>Restrictions: Can not contain HTML markup. Optional.</NOTE>
       <NOTE>Restrictions: Can not contain HTML markup. Required</NOTE>
       <IS_CAPABILITIES use="identification"/>

8.2.2. The section template

The template is used by the section XSL file to address information. Create a document named helpTemplate.xml with the following contents:


A more advanced template showing all options

    <KEY>keyElementName</KEY><!-- This key is used to connect this element to the times in the section template file -->
    <LONGNAME>Long name of element</LONGNAME>
    <DESCRIPTION>A short description of the element that will appear in the editor</DESCRIPTION>
    <NOTE>A short note about valid entries for the element</NOTE>
    <TYPE>text</TYPE><!-- The type of form to be used in the element -->
    <TYPE>select</TYPE><!-- A select element may be used -->
     <OPTION value="Asteroid"/>
     <OPTION value="Comet"/>
     <OPTION value="Earth"/>
    <TYPE>multipleselect</TYPE><!-- Checkbox -->
     <OPTION value="Asteroid"/>
     <OPTION value="Comet"/>
    <DATATEMPLATE/><!-- Default value to place in text or textarea element  -->
    <IS_CAPABILITIES><!-- Show as searchable field in outer search -->
    <IS_CAPABILITIES use="identification"/><!--? -->
    <IS_FAST_INDEX/> <!-- If set to true, uses SQL as indexing engine for search on this element.  If false, and SQL indexing is used, this element is ignored.  Also, this setting is ignored if IS_FAST_INDEX is false in vo_description.xml-->
    <IS_DISPLAY_TITLE/><!-- Set this to true to have the value of the field be represented as the title -->
    <IS_DISPLAY_DESCRIPTION/><!-- Set this to true to have the value of the field be represented as the title -->
    <IS_DISPLAY_LINK/><!-- The link to show in the listing of each records in the section -->
    <IS_DISPLAY_PREVIEW_IMG/><!-- Set this to a URL to any local or remote image (not cached locally!) -->
    <IS_DISPLAY_N_COVERAGE default="90"/><!-- If this field represents a northern boundary -->
    <IS_DISPLAY_E_COVERAGE default="180"/><!-- If this field represents a northern boundary -->
    <IS_DISPLAY_S_COVERAGE default="-90"/>
    <IS_DISPLAY_W_COVERAGE default="180"/>
    <IS_DISPLAY_DATE_FROM/><!-- If this field represents a coverage start time.  Format is any that MySQL DAYTIME supports -->
    <IS_DISPLAY_DATE_TO/><!-- If this field represents a coverage stop time.  Format is is any that MySQL DAYTIME supports -->

8.2.3. The section tree

Create a document called helpTree.xml:

   <ID>Help</ID> <!--Path of this section in eXist and top-level tree folder name -->
   <NAME>All Help Documents</NAME><!-- Name that will appear in the left menu..-->
       <ID>AddingMetadata</ID> <!--Path of this subdir in eXist to search when this link is selected-->
       <URI>AddMeta</URI> <!--Path of this section when filepath= is used -->
       <NAME>Info about adding metadata</NAME>
       <NAME>General help documents</NAME>
        <LABEL>Queries</LABEL> <!-- Top-level tree folder name -->
                <LABEL>All magnetosphere satellites</LABEL>

8.2.4. The section XSL

<xsl:stylesheet xmlns:xsl="" version="1.0">
    <xsl:output indent="yes" method="html"/>
    <xsl:template match="/">
                <xsl:apply-templates select="NewsDescription/Description/text()"/>
        <a href="{NewsDescription/Link}" target="_blank">
            <xsl:value-of select="NewsDescription/Link"/>
    <xsl:template match="text()">
        <xsl:call-template name="break"/>
    <xsl:template name="break">
        <xsl:param name="text" select="."/>
            <xsl:when test="contains($text, ' ')">
                <xsl:value-of select="substring-before($text, ' ')"/>
                <xsl:call-template name="break">
                    <xsl:with-param name="text" select="substring-after($text,' ')"/>
                <xsl:value-of select="$text"/>

8.2.5. Edit vo_description.xml

            <NAME>News</NAME><!-- Must match value of SCHEMA_NAME in schema file. This will be the name used in the URL.-->
            <LONG_NAME>News</LONG_NAME><!--This name appears in the section listing in the middle column when entering the observatory.  Usually should be chosen to match value in tree file.-->
            <DESCRIPTION>Any registered user may add news items.</DESCRIPTION>
			<!-- ADDITION_RESTRICTION :true|false|moderate -->
			<!-- DOC_NAME_TYPE : free|default -->

8.2.6. Create Index

If the section is SQL-indexed (<IS_FAST_INDEX/>) is in the section template, you must execute "Create index" in the Admin management page.

9. Linking and Tagging

10. Viewers

When an object is placed into the observatory, it may have a viewer associated with it.

  • Image section template should have. When this file is uploaded, the src/vops/storeAction/ places file in imageStoreDir as specified below. It also sets the OBJECT_TYPE to image.
   <TYPE action="imageAction">file</TYPE>
  • conf/vo_store.conf
# imagestore (ImageStoreAction)
  • conf/vo_store.conf
  • web/dataviewer.jsp
   String viewerPage = Settings.get("vo_store.dataType."+outputObjectType+".viewer");

11. Plug-ins

Plugins are used to place objects in the Observatory, either in the XML or Disk storage [10].

  • conf/vo_store.conf
upload.urlColl.storeClass=vobs.plugins.UrlCollectionPlugin (not yet written)

Correspond to


All of these call src/webapp/ to put an xml file in database.

To integrate a new plug-in into observatory, you can create a jar and put in lib folder and edit vo_store.conf and will then be able to access a new upload plug-in without re-creating the war file. The upload plug-ins are specified in

  • web/uploadMetadataForm2.jsp
<input type="radio" name="uploadFileType" value="data" />
<input type="radio" name="uploadFileType" value="dataZip" />
<input type="radio" name="uploadFileType" value="urlColl" />
<input type="radio" name="uploadFileType" value="dataCsv" />

12. Admin Tasks

12.1. Batch modification of records

(Using eXist Web Start interface query dialog that opens when clicking on binoculars.)

Delete note named Url on all records in SECTIONNAME.

for $url in collection("/db/virbo/SECTIONNAME")//Url return update delete $url

Replace old_string with new_string in all records:

for $doc in collection("/db/collection/")
return update replace $doc//text()["old_string"] with "new_string"

Remove ".xml" extension from OBJECT_NAME elements in forum file:

for $objname in collection("/db/virbo/forum")//DISCUSSION[ends-with(OBJECT_NAME,".xml")]//OBJECT_NAME//text()
return update replace $objname with substring-before($objname, '.xml')

To search for word Filename in all parts of all documents starting at the root of the eXist db. The documentation for eXist has many other examples.

for $doc in /* [text:match-all(.,'Filename')] return
concat(util:collection-name($doc), '/', util:document-name($doc))

Copy all records from a dir to another dir. Change record name based on element in record

let $dummy1 := xmldb:authenticate('/db/system/config', 'admin', '****EDIT THIS****')
for $obj in collection('/db/virbo/PersonUnderReview')
 let $fname  := util:document-name($obj)
 let $resid  := $obj//ResourceID/text()
 let $fname2 := replace($resid,"spase://SMWG/Person/","")
 let $tmp    := xmldb:copy("/db/virbo/PersonUnderReview","/tmp",$fname)
 let $tmp2   := xmldb:rename("/tmp", $fname, concat($fname2,".xml"))
 return <result>{$fname}, {$fname}, {$resid},  {concat($fname2,".xml")}</result>

Same as above but with subdirs for InstrumentUnderReview

let $dummy1 := xmldb:authenticate('/db/system/config', 'admin', '****EDIT THIS****')
for $obj in collection('/db/virbo/InstrumentUnderReview')
 let $fname-uuid  := util:document-name($obj)
 let $resid       := $obj//ResourceID/text()
 let $fname-spase := concat(replace($resid,"spase://SMWG/Instrument/",""),'.xml')
 let $coll-exist  := xmldb:collection-available(concat("/db","tmp"))
 let $tmp1        := xmldb:create-collection("/db","tmp")
 let $tmp2        := xmldb:create-collection("/tmp",tokenize($fname-spase,'/')[1])
 let $fname-tmp   := concat("/tmp/",$fname-uuid)
 let $dest-path   := concat("/tmp/",tokenize($fname-spase,'/')[1])
 let $tmp3        := xmldb:copy("/db/virbo/InstrumentUnderReview",$dest-path,$fname-uuid)
 let $tmp4        := xmldb:rename($dest-path, $fname-uuid, tokenize($fname-spase,"/")[2])
 return <result>{$coll-exist}, {$fname-uuid}, {$fname-tmp}, {$fname-spase}</result>

for PersonUnderReview section

let $dummy1 := xmldb:authenticate('/db/system/config', 'admin', '****EDIT THIS****')
for $obj in collection('/db/virbo/PersonUnderReview')
let $fname-uuid  := util:document-name($obj)
let $resid       := $obj//ResourceID/text()
let $fname-spase := concat(replace($resid,"spase://SMWG/Person/",""),'.xml')
let $fname-tmp   := concat("/tmp/",$fname-uuid)
let $coll-exist  := xmldb:collection-available(concat("/db","tmp"))
let $tmp1        := xmldb:create-collection("/db","tmp")
let $tmp2        := xmldb:create-collection("/tmp",tokenize($fname-spase,'/')[0])
let $dest-path   := concat("/tmp/",tokenize($fname-spase,'/')[1])
let $tmp3        := xmldb:copy("/db/virbo/PersonUnderReview",$dest-path,$fname-uuid)
let $tmp4        := xmldb:rename($dest-path, $fname-uuid, $fname-spase)
return <result>{$coll-exist}, {$fname-uuid}, {$fname-tmp}, {$fname-spase}</result>

Copy all records with tag "toreview" to a temp directory with filename corresponding to OBJECT_NAME

let $dummy1 := xmldb:authenticate('/db/system/config', 'admin', '****EDIT THIS****')
for $objname in collection("/db/virbo/forum")//TAG [text:match-all(.,'toreview')]
 let $store-return := xmldb:copy("/db/virbo/forum", "/db/tmp", util:document-name($objname) )
 let $object-path := $objname/ancestor::*//OBJECT_PATH
 let $coll-exist := xmldb:collection-available(concat("/db/tmp",$object-path))
 let $tmp := if ($coll-exist) then $coll-exist else xmldb:create-collection("/db/tmp",$object-path)
 let $filename := concat($objname/ancestor::*//OBJECT_NAME,".xml")
 let $source-path := concat("/db/virbo",$object-path)
 let $dest-path := concat("/db/tmp",$object-path)
 let $obj-id := concat($objname/ancestor::*//OBJECT_ID,".xml")
 let $tmp := xmldb:copy($source-path,$dest-path,$obj-id)
 let $tmp2 := xmldb:rename($dest-path,$obj-id,$filename)
 return <file>{concat($source-path, " ", $dest-path, " ", $objname/ancestor::*//OBJECT_PATH, $objname/ancestor::*//OBJECT_NAME,".xml", " ", $coll-exist, " ", util:document-name($objname))}</file>

12.2. Command line eXist

sudo ./ -s -u admin --password=PASSWORD

13. The forum file

Each record has an associated "forum" file that contains metadata about the record.

    <REPLIES discussRestriction="true"/>
    <VOTING averaged="0" votingRestriction="false"/>

14. Configuration

14.1. vo_description.xml

    <LONG_NAME>My Virtual Observatory</LONG_NAME>
    <DESCRIPTION>Description of my Virtual Observatory</DESCRIPTION>
    <OBSERVATORY_NEWS value=""/>
            <NAME>SPASEPerson</NAME> <!-- Where records are stored in eXist db /db/myVxO/SPASEPerson -->
            <LONG_NAME>Person</LONG_NAME><!-- Name that appears on left menu -->
            <DESCRIPTION>SPASE Records for persons from SMWG repository []</DESCRIPTION>
            <TREE>SPASEPerson/spaseTree_5.xml</TREE> <!-- Location of config files (/db/myVxO/conf/SPASEPerson) -->
            <DATAXSL>SPASEPerson/spase3.xsl</DATAXSL> <!-- Leave empty to display raw XML -->
            <PRE_DATA_URL>SPASEPerson/preprocess.xsl</PRE_DATA_URL> <!- Allow user to specify the URL to a record and apply this xsl transformation to record before putting in data base.  See Ingest Record Demo section -->
           <EDITOR_MODE>form</EDITOR_MODE><!-- Create Record mode form|text|upload.  If text, a textarea is given for entering XML.  If upload, an upload form is shown.  If form (default), then a text entry form is shown. --> 
           <ADDITION_RESTRICTION>true</ADDITION_RESTRICTION><!-- true|false|moderate (true = admin only, false = all  users, moderate = admin must approve using catalogue.jsp?searchSection=/waitingForApproval -->
            <IS_FAST_INDEX>true</IS_FAST_INDEX><!-- Set to true to use SQL indexing on this section.  If this is changed to true after documents exist in this section, you must initiate indexing from the admin page -->
            <DOC_NAME_TYPE>default</DOC_NAME_TYPE><!-- DOC_NAME_TYPE : free|default if free, then allow user to specify a document name (as in the Images section) -->
            <PARENT>SPASE Metadata</PARENT>

14.2. Config files

In WEB-INF/web.xml, there is a variable that points to init.conf. This is the main configuration file. Note that Windows-style paths must be the old eight-character style (C:\PROGRA~1\etc\). To figure out the correct path, use dir \x at a MS-DOS prompt.

14.2.1. mail.conf

Configure the sender of emails from the system. When an i-mail is sent to a user, it will also be forwarded to the user's external email address. The sender of the imail will be the account specified in smtpUser.

14.3. vo_store.conf

Set the directories on the local disk where various files are stored, including images and files.

14.4. vo_meta.conf

Set information about the location of the eXist database and the path to the observatory information within the database.

14.5. textindex.conf

SQL database configuration (for sections indexed with SQL)

14.6. users.conf

SQL database for user configuration information (username, password, email, etc.)

14.7. Logging

Catalina: Edit WEB-INF/classes/ parameter


The parameter


in init.conf is where looks for these log files.

Other: Edit init.conf parameter (jdbc:mysql connection log)


15. Installation

15.1. General Instructions

15.1.1. Pre-requisites

15.1.2. Install and configure Java

Please refer to your Linux distribution manual about installing the Java package. The Sun JDK version is preferrable. You can obtain it from

15.1.3. Install and configure Tomcat

The Tomcat installation directory (e.g., /usr/local/tomcat) will be referred to as CATALINA_HOME.

Enable Manager Servlet (see also

  • add "manager" role to tomcat's conf/tomcat-users.xml
  • add user with "manager" role
<role rolename="manager"/>
 <user username="tomcat" password="secret" roles="manager"/>

set privileged="true" in conf/context.xml

 <Context privileged="true"/>

Change session length from default of 30 minutes to 180 minutes


Retart Tomcat:


Test the Tomcat manager by pointing your web browser to:


15.1.4. Install MySQL

Please refer to your Linux distribution manual about installing MySQL. A useful browser interface for mySQL is

15.1.5. eXist

Download eXist, configure it, and install into Tomcat eXist 1.4

mkdir /usr/local/eXist
jar -xvf -c /usr/local/eXist eXist-setup-1.4.0-rev10440.jar

Change settings to use port 8088 in file /usr/local/bin/eXist/tools/jetty/etc/jetty.xml

<SystemProperty name="jetty.port" default="8088"/>

OR add a line to file /usr/local/eXist/tools/wrapper/conf/wrapper.conf

and start Jetty with


OR to run as a service make link:

sudo ln -s /usr/local/eXist/tools/wrapper/bin/ /etc/init.d/exist

and start as:

sudo service exist start

To install eXist in x64 Ubuntu as a service, use the x64 wrapper from

To restore an eXist backup:

/usr/local/eXist# ./bin/ -u admin -p PASSWORD -r /usr/local/eXist/webapp/WEB-INF/data/export/ --option uri=xmldb:exist://localhost:8086/exist/xmlrpc eXist 1.2

(VxOware past version 20100801 requires eXist 1.4. Previous versions may use eXist 1.2 or 1.4.)

Download the eXist war file from


Copy the eXist war file into Tomcat's webapps directory as "exist.war"

cp eXist-1.2.2-rev7828.war $CATALINA_HOME/webapps/exist.war

Test the eXist install by opening http://localhost:8080/exist/ Common to 1.2/1.4

Run the eXist Webstart client via the "Launch" button near the lower left of your eXist web page. Login as admin. You do not need a password the first time you login.

To set the admin password, select

  • Tools/Edit Users
  • Select "admin" in the list.
  • Enter a password.

Click "Modify User" (which provides very little apparent confirmation as a result)

Add the following to the <scheduler> section of conf.xml (which is located in the eXist installation directory). The backups will be located in

<job type="system" name="check1" 
        	cron-trigger="0 0 * * * ?">
		<parameter name="output" value="export"/>
    		<parameter name="backup" value="yes"/>
		    <parameter name="incremental" value="no"/>
			<parameter name="incremental-check" value="no"/>
			<parameter name="max" value="2"/>
         <job type="system" name="check2" 
		<parameter name="output" value="export"/>
    		<parameter name="backup" value="yes"/>
		<parameter name="incremental" value="yes"/>
		<parameter name="incremental-check" value="no"/>

15.1.6. Install

You can install VxOware from source by checking it out and building from SVN or by downloading a zip file that contains the war archive along with default databases and config files.

SVN-specific instructions are shown in red, zip file instructions in yellow.

 * Download the zip file using the "Download Now!" link at
 * Unzip the file and rename the directory vxoware
 * move the war file to the tomcat webapps directory ($CATALINA_HOME/webapps)
Get the application from svn
 svn co vxoware

* set catalina.home to $CATALINA_HOME
* set username and password to that of tomcat manager
* set the property to myVxO
* set manager.username property to define the name of a user having the manager role

Create a link to catalina-ant task in your home directory.
 mkdir -p .ant/lib
 ln -s $CATALINA_HOME/lib/catalina-ant.jar ~/.ant/lib
Download Ant from You may also need to add the ant bin folder to the path.

Build and deploy the web application
 ant install
If the webapp deployment fails because it already exists, try:
 ant try

Open http://localhost:8080/myVxO The default users and their passwords already existing in installation are


15.1.7. Set default databases

SVN-specific instructions are shown in red, zip file instructions in yellow.

Load vxoware/database/virtobs.sql into MySQL

mysql -u login -p -h localhost < virtobs.sql

Unzip voData directory

unzip vxoware/databse/

Move to appropriate directory

mv vxoware/database/voData $SOMEDIR

and edit configuration file accordingly


Unzip vxoware/database/

cd vxoware/database; unzip

and insert into eXist:

* Open http://localhost:8080/exist
* Press orange Launch button near the bottom of the left column
* Enter the password you entered installing eXist
* Press OK
* Select tools -> Restore
* Files of type: __contents__.xml files

Browse to the vxoware/myVxO/db and press 'Select backup file for restore'

15.1.8. Move config files

Set up a configuration directory, e.g.,

mkdir /etc/myVxO ; cp -r vxoware/conf /etc/myVxO/conf

Edit the file /etc/myVxO/conf/init.conf

set directory.conf=/etc/myVxO/conf

Edit other locations in settings file

In vo_meta.conf (e.g., /etc/myVxO/conf/vo_meta.conf), set the eXist database admin password.

You may also set the language and time format here.

15.1.9. Modify

Edit this line in vxoware/web/WEB-INF/classes


it must be a directory path and not a file path, e.g.,


15.1.10. Modify web.xml

In $CATILINA_HOME/webapps/vxoware/WEB-INF/web.xml, change the SettingsServlet configFile element to define the path of the permanent configuration file init.conf, e.g.,


15.1.11. Apache config

 Options Indexes FollowSymLinks MultiViews                                                                                                                                                                                                  
 <Proxy *>                                                                                                                                                                                                                                  
   Order Allow,Deny                                                                                                                                                                                                                         
   Allow from all                                                                                                                                                                                                                           
 ProxyPass /myVxO ajp://localhost:8009/myVxO                                                                                                                                                                                              
 ProxyPassReverse /myVxO ajp://localhost:8009/myVxO                                                                                                                                                                                        

To change name, place this before above

 RewriteEngine On                                                                                                                                                                                                                           
 RewriteRule ^/NewName/(.*)$ /myVxO/$1 [PT,L,QSA]

15.2. Eclipse

Note that we have successfully built and deployed VxOware in Tomcat using Eclipse Java EE IDE for Web Developers (Eclipse build id: 20100218-1602). (eXist and SQL were not running in Eclipse, however; they were run on a remote server.) We used an external eXist and SQL database.

Tested on Eclipse-jee-Europa-Winter and others

  • Check-out from SVN
  • Install Tomcat 6
  • Edit location of Tomcat in
  • Edit location of init.conf in web/WEB-INF/web.xml
  • Project->Properties->Java Build Path->Libraries tab should be the same JDK (1.5+, and not a JRE) as Window->Preferences->Java->Installed JREs
  • "Windows -> Preferences -> Ant -> Runtime, click on the Ant Home Entries (Default), and add your required JARs by clicking on the Add External JARs. The JAR file that contains Tomcat's Ant tasks is located %TOMCAT_HOME%\server\lib and is named catalina-ant.jar. By adding it to the Ant's runtime, you can now define Tomcat tasks in the Ant editor and invoke them if needed." [11]
  • Edit conf/*.conf paths
  • Select Tomcat under Servers tab -> Right-click -> Add and Remove -> Remove all is sometimes needed to get to run in Eclipse Galileo if 404 appears (some files are not being copied to local Tomcat webapp dir under Eclipse?).

To connect to ec2 user db, use (USERNAME and PASSWORD are in users.conf)

mysql -u USERNAME -pPASSWORD virtobs
grant all on virtobs.* to 'USERNAME'@'IP' identified by 'PASSWORD';
flush privileges;

To connect to ec2 user SQL index (used by some sections), use (USERNAME and PASSWORD are in textindex.conf)

mysql -u USERNAME -pPASSWORD textindex
grant all on textindex.* to 'USERNAME'@'IP' identified by 'PASSWORD';
flush privileges;

15.2.1. eXist Plugin


15.3. EC2 Instructions

First, obtain an EC2 account and install the ec2 tools on your local machine by following

From command line, execute

ec2-run-instances ami-bb709dd2 -k ec2-keypair

Wait a minute or two until


returns with a URL to your instance such as

ec2-authorize default -p 22   # Authorize SSH
ec2-authorize default -p 8080 # Authorize Tomcat
ec2-authorize default -p 3306 # Authorize mySQL
ec2-authorize default -p 80   # Authorize mySQL
ssh -i ~/.ec2/ec2-keypair.pem

remove * in URL http://us*.archive in sources.list

sudo nano /etc/apt/sources.list 
sudo apt-get update
sudo tasksel --section server # Select Tomcat and LAMP.  Write down mySQL password that you entered
wget http://localhost:8080/ # Should work
sudo apt-get install ant
sudo apt-get install subversion

get jdk-6u18-linux-i586.bin from

sh jdk-6u18-linux-i586.bin
ln -s jdk1.6.0_18 jdk
JAVA_HOME=/home/ubuntu/jdk;export JAVA_HOME
JRE_HOME=/home/ubuntu/jdk;export JRE_HOME
CATALINA_HOME=/usr/share/tomcat6;export CATALINA_HOME

Change line

<user username="tomcat" password="tomcat" roles="tomcat"/>


<user username="tomcat" password="tomcat" roles="tomcat,manager"/>
sudo nano /etc/tomcat6/tomcat-users.xml 
sudo nano /etc/tomcat6/context.xml # Add <Context privileged="true"/> inside of existing <Context> block.

Next three steps are required if using Sun's Java, which is recommended

sudo ln -s /etc/tomcat6 /usr/share/tomcat6/conf
sudo ln -s /usr/share/tomcat6/webapps/default_root /usr/share/tomcat6/webapps/ROOT
sudo mkdir /usr/share/tomcat6/logs

Install eXist:

sudo cp eXist-1.2.6-rev9165.war $CATALINA_HOME/webapps/exist.war

Re-start Tomcat:

sudo $CATALINA_HOME/bin/

Check to make sure everything is running (username and password default is tomcat)

svn co vxoware
sudo less /etc/tomcat6/tomcat-users.xml
nano vxoware/ # Set catalina.home=/usr/share/tomcat6 and change username and password to match what is in tomcat-users.xml
nano vxoware/build.xml # Replace virbo with myVxO
nano vxoware/web/WEB-INF/web.xml # Set SettingsServlet to /etc/myVxO/conf/init.conf
nano vxoware/conf/init.conf # set directory.conf=/etc/myVxO/conf
sudo mkdir /etc/myVxO
sudo cp -r vxoware/conf /etc/myVxO
sudo mysql -u root -p -h localhost # Enter create database myVxO;quit;
sudo mysql -u root -p -h localhost myVxO < vxoware/user_sql_database/virtobs.sql
sudo apt-get install unzip
cd /var/www; sudo unzip ~/vxoware/voData/
scp vxoware/exist_database/ yourlocalmachine:
Run eXist webstart client from
Select tools -> Restore
Files of type: __contents__.xml files

Browse to the vxoware/exist_database/db and press 'Select backup file for restore' and select __contents__.xml file

sudo ln -s /usr/share/tomcat6 /usr/local/tomcat
mkdir -p ~/.ant/lib
ln -s $CATALINA_HOME/lib/catalina-ant.jar ~/.ant/lib

Note that sudo ant try does not pass user's environment vars for JAVA_HOME and CATALINA_HOME, so sudo cp is needed.

cd vxoware ; ant try 
sudo cp vxoware/deploy/myVxO.war /usr/share/tomcat6/webapps
Personal tools