From Vxoware

[edit] 1 September 12th telecon

1. Upload new SPASE directories into eXist
2. REST search documentation
3. Fix space and time format bug
4. Link in data ordering page
5. Joint ViRBO/SPIDR user case (NGDC)
6. Report on integration of Doug and Anne's install procedure.

[edit] 2 June 13th telecon

• Make file upload optional in software section
• XSLT to transform spase://SMWG/Person/Robert.S.Weigel to hyperlink to document.do?Robert.S.Weigel
• Can’t upload single XML file?
• Fix css to look more like VMO
• Need error when no parameter selected

• Javascript collapsible tree menu
• Embed thumbnail under metadata? Connected to images section?
• Path for fixing New and Total list?
• Remove Section Filters button

[edit] 3 June 6th telecon (11 am Eastern)

• Problem uploading Person XML
• What determines what you see on metadata upload pull-down
• Progress on "upload metadata" button
• Dima to create default mysql db with admin and guest as only users.
• Dima: Re-name "Fuzzy" check-box to "Exact" (and reverse logic)
• Dima - free text search on exist-db.
• Eric - Send Bob FGDC Metadata for DMSP
• Dima add option for showing "Observatory Digest" as admin option (is it possible to not show it if it is empty?)
• Bob - Privacy policy link broken add document with same name to wiki
• Doug - continue efforts with simplifying install, user db, etc.
• Bob - continue work on interface with TimeSeriesPlugin. Continue work with interfacing to git repository.
• What happened to my username? don't know
• How to give admin privs? added to documentation
• Explain difference between Advanced external search and Federation search.done
• Can we do "advanced section search" on current vxo?yes IS_CAPABILITIES in vo_description.xml sets what sections can be searched.
• Recover "SPASE inside" logo and "VxOWare" logo on left. done
• Is ADMIN a real user? How do I set its icon? I don't see ADMIN or admin in mysql db.will be implemented
• Dima and Dima - Add example metadata for OE for SPIDR services (any any other services that VxOware can now handle)
• <DataSources> still exist in XML files for OE
• Can't add to federation http://aurora.gmu.edu:8080/virbo2 Fixed
• Dima and Dima - give a demo of a federation search (during telecon). Bob will document demo and use it at VxO meeting done

[edit] 4 May 30th telecon (11 am Eastern)

• Bob try creating observatory digest. Should the default install should be the more traditional view (sections on left, etc). This is what documentation says.
• Dima K. Change font color of "Advanced Search" on front page (I can't see it with a blue background).
• Eric what is the renewal date?
• Add data to allow upload or adding through editor? Where did we leave that discussion?
• Bob, Dima, and Dima. What am I doing wrong when renaming sections? I get document does not exist error. (Try View XML on LICA. SpaseInstruments does not show up in exist DB search).
• Go through how to correct typos (finding text in db). Go through creating an account
• Wrong numbering of number of documents
• How to get rid of News section
• Dima - free text search on exist-db.
• Discuss creating configuration section what has vo_description.xml, settings.xml, etc. in it.
• Recover "SPASE inside" logo and "VxOWare" logo on left.
• Based on email from Dima and Dima, Bob has added section http://vxoware.org/vxoware/index.php/Main_Page#Exist_DB_Layout. Please proofread and discuss.
• Dima K. update on TimeStep and allowing YMD to only be shown in drop-down.
• Dima M. update on allowing xmlns to be allowed (i.e., in <Spase xmlns="http://...">)
• Bob will create a default exist db for default vxo
• Bob continue work on TimeSeries plugin
• Bob continue work on command-line update of exist db
• Doug update on build6.xml
• Dima K. and Dima M. find out why avatar upload no longer works
• Dima M. commit new spidr.jar to CVS with log4j.properties removed

[edit] 5 May 20th telecon (11 am Eastern)

[edit] 5.1 Discuss

1. How do I add cadence selector (to OE) once I have modified TimeseriesPlugin.java? One of my XML files already has a "Step" element, but I don't see it rendered/
2. mLog.debug("Reading file: " + curPath); Where is this log file?
3. How do I suppress hour and minute pull-down menus? (Let's walk through xslt file on phone)
4. Doug discuss install and propose config simplifications. Also MySQL dependency discussion? HSQL, Derby?
5. Bob will try changing vo_store.conf and /virtualObservatory/src/vobs/plugins/TimeseriesPlugin.java to include ppd parameter and remove pull-down for hour and minute. Also need to change OE file to allow for dt parameter.
6. Bob will modify TimeseriesPlugin.java so that the files voData/files/HASH-HASH.xml have contentType that is a mime type, and fileName that is based on the timeseries.org query string.

<metadata>
  <OBJECT_ID>C05932DD-6BBA-F466-D109-8A8A3DF91489</OBJECT_ID>
  <DESCRIPTION>uploaded/form</DESCRIPTION>
  <ACTION_ID>fileAction</ACTION_ID>
  <fileName>C05932DD-6BBA-F466-D109-8A8A3DF91489.cdf</fileName>
  <contentType>SAMPEX cdf data</contentType>
  <fileSize>4468</fileSize>
</metadata>

1. Discuss Parameter groups
2. Command-line image upload?
3. Change "DataBasket" on left to "Data Cache"?
4. Continue external search discussion

[edit] 5.2 Bugs

1. Multiple select=false, but still allows multiple boxes to be checked?
2. Need Javascript pop-up error when no parameter is selected by user selects "Go"
3. Bug when trying to upload new avatar
4. Is <SourceShortName>Get Data</SourceShortName> ignored? No, but SourceName is repeated.

[edit] 6 May 13th telecon (10 am Eastern)

• Dima will send Bob instructions for OE to bypass the data basket.
• Bob will have timeseries.org return a jnlp.
• When I delete a record (via exist or the broswer) and then re-upload that record (with the same name), all of the history information from the original document is still there. Should it be? I can see where uploading a file with the same name replaces the old version, but if I delete the old version first it seems that I should also delete the meta-meta data associated with it.

[edit] 7 May 6th telecon (11 am Eastern)

May 6th is a Tuesday

•  ! Dima K. See why the command line stuff is not working.
•  ! Dima K.: Update the ViRBO Exist (done) and war (not done) file on aurora.gmu.edu. Bob will update the war file
• Default install status. Can we create a continuously updated version at http://aurora.gmu.edu:8080/myVxO for documentation? Bob will do this
• (done) Bob: Merge V-catalog.ppt into documentation
• (done - see below) Bob: Write up statement of work for new developer (Doug) and write some text that describes the problems that were discussed on the telecon
• Discuss license dust-up as a result of teamvxo.org announcement (Bob). Sourceforge indicates vxoware is BSD, but I don't see a file named LICENSE.Will be done by Dima
• Discuss Tomcat session time. At least we should document how to modify this session time. To change default session time in tomcat, you need to edit tomcat/conf/web.xml file. The section "default session configuration", parameter session-config/session-timeout. It will change session time for all the contexts loaded in tomcat.
• Is it possible to get rid of the MySQL dependency for user management? This will simplify installation. Also, it seems that there must be a native java approach. Something like this: http://java.sun.com/j2se/1.4.2/docs/guide/security/jaas/JAASRefGuide.html. Doug will suggest some ideas, Dima and Dima will look into
• Walk through creating virtual section (so Bob can document it) Wait until new installation of myVxO is created
• Misha and Dima K.: Think about how to make the search interface for External and Internal look more similar. Most search engines have a default of "search everything" and then you can restrict your search with an advanced search. Do we want to go this route, or do we think that most people will want to have a restricted search by default?Dima will create a "simple federation search" option for the External Search Service. The Search box on the front page will changed to be similar to that suggested by Bob in the PDF scan
• Misha, Dima K., and Bob: Think about how to resolve this issue: The new V-catalog tree on the home page resolves Bob's feature request. Where do we put the old "full" section list? We could just get rid of it because all of the entries are already available on the advanced search page. However, it looks nice, and one can anticipate putting in icons and stylizing the list, which would look much better than the V-catalog list.Discussion tabled until Bob has a chance to look at new code base

[edit] 7.1 Project for Doug

1. Set up your own installation of vxoware on a lasp server, report any problems that you run into, and make suggestions on how to simplify the installation process.
2. Try the "Creating a Section" instructions in section 6.2 of http://vxoware.org/vxoware/index.php/Main_Page. If you can think of a way to simplify these instructions, feel free to post them to the wiki.
3. I am curious about using amazon's s3 for the vxoware "data store". If you study up on amazon's s3, we can tell you how the vxoware "data store" works and decide if this is something that would be worthwhile.
4. Using the web service at http://virbo.org/wiki/index.php/Main_Page#Creating_a_SPASE_Person_Record onvert links in SPASE records like this spase://nssdc/person/3637 to a hyperlink before they are rendered on the page.
5. Modify XSLT to right-justify bold-face text in first column on observatory pages such as http://virbo.org/virbo/viewdata.do?docname=AMPTE-Observatory

Large project (1): At present, the plan is to store high-level metadata (such as observatory or satellite metadata) at a NASA site. The problem with this is that it will be (and has been) very difficult to update the metadata. And high-level metadata should be a community resource. There are many problems of having multiple repositories holding slightly different versions of metadata; "high-level" metadata should live in a common community repository in the same way that wikipedia holds common encyclopedic entries. Having near-identical copies of metadata for satellites in multiple locations makes as little sense as creating a clone of wikipedia that has only slight differences. There are cases where some metadata is so domain- or infrastructure-specific that is does not make sense to store it is a community repository, but this is more often the exception than the rule.) The individual observatories should not be the master node for metadata that is of interest to other VxOs. Also, think about how and if VxOWare can resolve the fact that SPIDR has FGDC metadata for WDC magnetometers while VMO/U has created SPASE records for each.

The goal is to show that VxOware can be used for high-level metadata management. This is the easy part. The hard part is getting people to see that the old method of maintaining metadata can be improved: (1) receive an email request for change (2) ignore email (3) receive an email asking about original request (4) forward request to secretary who is on leave (5) secretary sends email after a few weeks with question (6) original requester responds … 1 month passes … (7) change is made … one month passes … (8) original requester wonders what happens. Notices that metadata has been updated. Original requester wonders why nobody told him when the update was made.

One “social” issue that we need to deal with is that when a “wiki”-style management system is proposed, people tend to say “but what if someone makes a bad edit”. I have suggested a number of ways of dealing with this.

Here is the plan. Create a vxo instance for this.

• User manually uploads satellite-level metadata. Say one of these: http://vho.nasa.gov/component/option,com_wrapper/Itemid,57/

• On upload, validation is performed based on call to http://www.spase-group.org/tools/validate
  • If error, put content into simple HTML form that allows user to edit (temporary solution until the editor at spase-group.org is completed.
  • User has option of saving invalid SPASE

• User should have option to add “Email on change” addresses (this information should be stored in the SPASE document in the <Extension> section). External emails must be allowed (and it must be easy to opt-out; opt-out information should be given in email)

<Extension>
  <Email-on-change>
      email1@abc.com
  </Email-on-change>
  <Email-on-change>
      email2@abc.com
  </Email-on-change>
</Extension>

Large project (2): Integrate an orbit viewer into VxOWare. Google Earth or Orbit Viewer. Details to follow.

[edit] 8 April 27th telecon (11am Eastern)

• http://magnet.gmu.edu/wiki-private/index.php/VxOWareDocs#Command-line_modification_of_DB

(for some reason, I can't get this to work. I noticed a typo in the instructions, but even with it fixed, I get errors.)

• Discus ideas VxOware tasks for new developer from University of Colorado.
• Go over the PDF file I sent last week.

[edit] 9 April 11th telecon (11am Eastern)

[edit] 9.1 Bob

• Cron to update CVS daily on Sourceforge
• How can documentation process be sped up?
• Look into Visa for Russia. Possible visit in June. Misha will be in Boulder at start of August. Eric will be in Kiev in start of October.

[edit] 9.2 Misha and Dima K.

• Document how to do outer search
• Implement virtual section and try workflow for it. Then document it.
• Create web editor for vo_description.xml (not a priority).

[edit] 9.3 Dima

• Test command-line equip instructions.

[edit] 9.4 Justin

• Look into converting png images to SWF movie for easy browsing

[edit] 9.5 Misha

[edit] 10 April 4th telecon (11am Eastern)

• http://groups.google.com/group/vxo-developer
• Review progress/status of past assignments
• Need to either make russian cvs accessible to public for browsing (is it?) or sourceforge CVS daily
• Discuss: http://magnet.gmu.edu/wiki-private/index.php/VxOWareDocs
• Need a compelling use-case demo for VxOware. Would like our new developer from UC Boulder to see if we can "fix" the SPASE person issue (I will describe during the telecon)
• Go through equip example
• Regular telecons to work through use cases like above

[edit] 11 February 15th telecon (11am Eastern)

• Have the a new version of VxOware that uses the new OE with at least the new time options and the "select all" option. Having Groups implemented is not expected.
• Try a build of vxoware + autoplot from the sourceforge SVN during the next telecon.
• Misha's word doc on the new OE is here http://docs.google.com/Doc?id=dhdk5g33_38gcc73mc6

[edit] 12 February 1st telecon (11am Eastern)

[edit] 12.1 Post-meeting Email

All,

I have entered assignments in sourceforge as either bugs or feature requests. A few tasks that were not included are:

• Jeremy - Put README in http://vxoware.svn.sourceforge.net/viewvc/vxoware/autoplot/trunk/ with NB build instructions. Put this information in the README too http://autoplot.org/autoplot/index.php/Autoplot_from_source.
• Dima - Send Bob instructions for how he backed up the eXist DB and instructions for adding or replacing an XML file in the eXist database from the unix command line (for doing bulk updates of XML files).
• Misha - Review Bob's instructions for adding a section: http://mag.scs.gmu.edu/wiki-private/index.php/2007Meetings#Request_for_Review

I hope that by the next telecon we will have the a new version of VxOware that uses the new OE with at least the new time options and the "select all" option. Having Groups implemented is not expected. I also hope that I can try a build of vxoware + autoplot from the sourceforge SVN during the next telecon.

Misha, here is my write-up on the features wanted for Groups in OE

Given an OE file with the parameter list

• Parameter 1
• Parameter 2
• Parameter 3
• Parameter 4
• Parameter 5
• Parameter 6

I would also like to specify groups of parameters. The appearance on the web form will be

_________________________

0 Select all groups

0 Group 1

0 Group 2

0 Group 3

_________________________

0 Select all parameters

0 Parameter 1

0 Parameter 2

0 Parameter 3

0 Parameter 4

0 Parameter 5

0 Parameter 6

[edit] 12.2 Bob

• Bob report on SRB and Amazon
• Report on progress with SPASE with scientists
• Discuss draft of paper for discussion (which will partially serve as the user guide)
• Bob finish code for calling Autoplot from command line
• Discuss Matlab code that generates SPASE metadata + OE for a dataset http://virbo.org/wiki/index.php/Main_Page#Creating_a_SPASE_Numerical_Data_Record

[edit] 12.3 Jeremy and Dima

• Provide Autoplot and VxOware SVN and build instructions (I'll need these sent out a day or two in order to be able to test the instructions). Add SVN link on Autoplot page. Jeremy will have done very soon. Misha will start to work on exporting tagged CVS versions to SVN and uploading to Sourceforge with each release.
• Address (but not necessarily fix or implement) outstanding Sourceforge bugs and requests

[edit] 12.4 Misha

• Provide any updates regarding allowing multiple schema for internal DB (so multiple sections are allowed). Can be done. In process of implementing.

[edit] 12.5 Eric and Misha

[edit] 12.5.1 OE

General

• Description is spelled "Discription" in OE document model.
• Convert autoplot to hyperlink in OE. Is this allowable?
• I would also like the ability to associate each parameter name to a link of a parameter-level SPASE document in the observatory. Allow the <Info> to be linked to Parameter level metadata?
• Need to be able to set "Select All" by default or to specify the default parameter selection list.
• For discussion: can we have a pop-up if number of points exceeds a given value or if request matches a certain criteria? Or can I specify Javascript function in the OE that parses the order before it is sent and then decides if a pop-up message should be given (Such as "Request is too large. Access merged file instead.").
• How can I specify a "group" of parameters. For example, suppose there are 96 time series associated a data set (e.g., the O'Brien reanalysis data set). Can OE handle the case where I want to call allow either all 96 parameters to be selectable _or_ 4 groups of 24 parameters.

Time options

• Add dt options (e.g., 1-second, 2-sec = original cadence, 1-minute, 1-hour, 1-day)
• Other Time options - uniform time grid, remove fill, etc.
• Add averaging options (return # of points used in average)
• Add caching options? Name, Expiration, public, etc.
• Need to be able to set default dates and default dt.

[edit] 12.5.2 Request for Review

Here are my instructions for how to manually create an Help section in the observatory. This will be one of the documents in the Help section in the observatory at vxoware.org once the wiki syntax parser is working correctly. I have put ??? in areas where I would like you to add a few lines of explanations.

The following instructions show how to manually create a new section in the observatory.

1. Start the eXist webstart client by going starting the web start program http://vxoware.org/exist/exist.jnlp
2. Create a directory named conf/Help
3. Create a document called helpTree.xml. The <ID/> tag indicates the name of the section that will appear on the left-hand menu of the VxO. The <NAME/> tag indicates what will appear as the description of all subsections. Each subsection has a node with an ID and a NAME. The <FILTER_DEFAULT> indicates ???.

<SECTION>
  <ID>Help</ID>
  <NAME>All Help Documents</NAME>
  <NODE>
      <ID>AddingMetadata</ID>
      <NAME>Adding Metadata</NAME>
      <FILTER_DEFAULT>false</FILTER_DEFAULT>
  </NODE>
  <NODE>
      <ID>General</ID>
      <NAME>General Help</NAME>
      <FILTER_DEFAULT>false</FILTER_DEFAULT>
  </NODE>
</SECTION>

• Next, create a document called helpSchema.xml. This file is used for ???. The possible fields are ???.

<SCHEMA>
  <SCHEMA_NAME>News</SCHEMA_NAME>
  <SCHEMA_TITLE>News addition form</SCHEMA_TITLE>
  <SCHEMA_DESCRIPTION>Please fill in the following fields according to the descriptions:</SCHEMA_DESCRIPTION>
  <FIELD>
      <KEY>keyTitle</KEY>
      <LONGNAME>Title</LONGNAME>
      <DESCRIPTION>The name of the search result provider.</DESCRIPTION>
      <NOTE>Notes: Should contain the search terms themselves. Restrictions: Can not contain HTML markup. Required</NOTE>
      <TYPE>text</TYPE>
      <IS_DISPLAY_TITLE/>
      <IS_CAPABILITIES use="identification"/>
      <IS_FAST_INDEX/>
  </FIELD>
  <FIELD>
      <KEY>keyLink</KEY>
      <LONGNAME>Link</LONGNAME>
      <DESCRIPTION>A URL to the website that providers the search results.</DESCRIPTION>
      <NOTE>Restrictions: Can not contain HTML markup. Optional.</NOTE>
      <TYPE>text</TYPE>
      <IS_DISPLAY_LINK/>
  </FIELD>
  <FIELD>
      <KEY>keyDescription</KEY>
      <LONGNAME>Description</LONGNAME>
      <DESCRIPTION>A text.</DESCRIPTION>
      <NOTE>Restrictions: Can not contain HTML markup. Required</NOTE>
      <TYPE>textarea</TYPE>
      <IS_DISPLAY_DESCRIPTION/>
      <IS_CAPABILITIES use="identification"/>
      <IS_FAST_INDEX/>
  </FIELD>
</SCHEMA>

[edit] 13 January 14th telecon (11am Eastern)

Post AGU wrap-up

All,

As you may have heard, the AGU meeting went very well. I want to thank everyone for their efforts and help in pulling this off. There was a lot of interest in Autoplot, and based on my extensive use of VxOware prior to and during AGU week, I am confident that it will generate as much interest or more when we are ready for a formal software release. Here is were I think we are on the various projects:

(1) Autoplot - We are in the refinement phase right now, and I think there will be a planning and then expansion that takes place this spring in collaboration with at least one other VxO.

(2) VxOware - I have registered vxoware.org. I think we should create a new observatory instance and start using it ourselves for documentation and demonstration of various features (such as the weather forecast examples and the image server connections that we used to have in virbo.) Although there are many, many CMS systems out there, I think vxoware has the advantage that (1) it is (almost) pure java, (2) it has a clean UI, and most important, (3) it (will) have well-documented examples of how to hook into other data sources.

(3) timeseries.org - now serves das2binary/(compressed and uncompressed), das2text, hdf5, cdf, netCDF, text in various forms, matlab binary, flat binary. The only thing left is for me to make the modifications necessary to combine multiple parameters. Misha and Dima decided that they did not like my proposed approach (wildcards and range specifications in the URL).

(4) Data - The entirety of the SAMPEX data center will soon be available through timeseries.org and hence ViRBO. Josh Rigler is finishing off the metadata using some matlab codes I put together to convert CDAWeb metadata to SPASE (and Reiner's LANL/GEO data to SPASE). Reiner just gave me a bunch of Polar data which I need to look at soon. I find the bottleneck (or at least the most time consuming part) in making these data available through the timeseries.org server is dealing with the metadata.

(5) Metadata - I put together a bunch of Matlab scripts that output SPASE records for certain data types. I need to document and distribute this software so that the data people can help me with the metadata bottleneck.

(6) OE - The space physics community has been discussing this extensively. I think what you guys have is fairly well developed and you are probably ahead of everyone else that is thinking about it. I have lots of suggestions for it based on my recent use associated with timeseries.org.

(7) SPASE editor - I was told that they received money to work on this more and will be willing to work with us more than they have in the past. Also, a few people commented that they liked the editor form that was currently implemented in virbo/vxoware.

(8) Search - There were a number groups that showed off search functionality at the AGU. I am certain that you guys have something that already has the same or better functionality, and we should discuss how much time we should put into this. As I have mentioned previously, until ViRBO gets more data, it does not make sense to work on a huge search engine, but for vxoware, it may make sense to tailor the existing search capabilities to SPASE more.

I'll be ready for a telecon starting on and after Wednesday of this week. Please let me know when you can meet next. My working list of items that we could discuss is here: http://mag.scs.gmu.edu/wiki-private/index.php/2007Meetings#Items_for_next_telecon. I will clean up and clarify these items before the telecon.

[edit] 13.1 Autoplot

• Tab that lists log notes from autoplot (getting completions, etc.)?
• Improve style page (right-justify and remove fill to reference checkbox (use fill color "none" instead)?
• Allow line thickness of zero?
• Why 8/10 instead of 80%
• Decrease delay from first feedback after first hit of enter
• This failed: http://cdaweb.gsfc.nasa.gov/istp_public/data/polar/hyd_h0/2000/po_h0_hyd_%Y%m%d_v01.cdf?ELECTRON_DIFFERENTIAL_ENERGY_FLUX&timerange=20000109-20000111
• This gave exception but the plot still appeared: http://cdaweb.gsfc.nasa.gov/istp_public/data/polar/hyd_h0/2000/po_h0_hyd_%Y%m%d_v01.cdf?ELECTRON_DIFFERENTIAL_ENERGY_FLUX&timerange=20000109
• We should unify our usage of time and ppd in urls with timeseries.org. What about using to=2000 timeseries.org/ssn gives error now.
• Autoplot as a applet? Put note about answer on web page.
• Change "url rejected, getting completions" to "url ambiguous, inspecting file for plotting parameters"
• Problems when no epoch variable in cdf file?
• spacecraft_data/omni/omni2_1963.dat
  • Say "loading file" instead of "loading data".
  • or say "loading file\n omni2_1963.dat"
  • After download has started, spawn another process that reads the directory to find the file size and then provide a progress meter?
  • Also list an estimate of bits/sec?
• After file has been loaded for this example, http://autoplot.org/data/autoplot.cdf?BGSM you could leave some info in the bottom bar. For example, file size and bits/second download time. Or say "cached file used". This example should indicate the YMD somewhere too.
• I still don't know about that file folder icon. I think the mouse-over should say "list parameters in file" and the icon should be that of a bulleted list. Perhaps this icon should only appear when looking at a file where distinct variables were found. Then again, on a columnar file it could just say "column 1", "column 2", ..., "column 3".

[edit] 13.2 Discussion

• Unique ID should be off by default. A warning should be given if there is a name collision. We need to discuss the ViRBO namespace for XML documents. I don't think having a string of numbers associated with each file is good. Should the default behavior be for the document to have a name that is extracted from the file? And have the long string of numbers be for internal use only. (Pending new release)
• Paper. Discussed.
• Why are we serving netCDF?
• Justin - I am not sure what his support needs are, but what I really need is someone to help with server or metadata stuff (does he have matlab experience? If so, he could he work with Janet)
• Would it be possible to pull the metadata from certain sections out of a repository? And push changes back to a repository? I know that we already have revision control, but this approach would help two types of users (1) the ones that make bulk changes and (2) the ones that only want to make a tweak and are not interested in installing svn, cvs, git, etc. Discussed. Misha is thinking about P2P solutions to which this would apply
• The "Related" section problem that I created. Submitted to feature request on SF

[edit] 13.3 OE

Emailed to Misha on January 14th, 2008

[edit] 13.3.1 General

• Description is spelled "Discription" in OE document model.
• Convert autoplot to hyperlink in OE. Is this allowable?
• I would also like the ability to associate each parameter name to a link of a parameter-level SPASE document in the observatory. Allow the <Info> to be linked to Parameter level metadata?
• Need to be able to set "Select All" by default or to specify the default parameter selection list.
• For discussion: can we have a pop-up if number of points exceeds a given value or if request matches a certain criteria? Or can I specify Javascript function in the OE that parses the order before it is sent and then decides if a pop-up message should be given (Such as "Request is too large. Access merged file instead.").
• How can I specify a "group" of parameters. For example, suppose there are 96 time series associated a data set (e.g., the O'Brien reanalysis data set). Can OE handle the case where I want to call allow either all 96 parameters to be selectable _or_ 4 groups of 24 parameters to be

[edit] 13.3.2 Time options

• Add dt options (e.g., 1-second, 2-sec = original cadence, 1-minute, 1-hour, 1-day)
• Other Time options - uniform time grid, remove fill, etc.
• Add averaging options (return # of points used in average)
• Add caching options? Name, Expiration, public, etc.
• Need to be able to set default dates and default dt.

[edit] 13.4 UI

• "List all sources" "List Satellites" "List Datasets"

[edit] 13.5 Test procedure for ViRBO

• Create a new account named test_user-YYYY-MM-DD
• Visit the News section and add an item
• Make sure the "New" column reflects his addition after a refresh
• Go to the Data. Select a random parameter and request one month of

data at 1-hour resolution. Do all of the possible actions for that parameter.

• Go to the Data Services and test the SPIDR service by requesting one

month of Dst or Kp data.

• Test one other Data Service.
• Do a search in "friendly external resouces".
• Log off and re-logon in as himself and delete the News item. Remove the

test_user-YYYY-MM-DD account.

[edit] 13.6 Test procedure for Autoplot

• Try all of the default bookmarks (In a random order)
• Try an invalid URL
• Export data to ASCII
• Change the plot style in some way
• Save the current plot as a .vap file
• Look at a different data set from the bookmarks list
• Load the .vap file that was just saved.
• Try a number of zoom features
• Change the line style and thickness
• Change the background color to a custom color
• Change the font size or style

[edit] 14 November 1 Telecon (11 am Eastern) Results

[edit] 14.1 Background

First, I think that the tasks outlined in the "Data Section of Observatory" part of this email should be given highest priority. Don't worry about the data or the aggregation. I will take responsibility for that. I don't know why I let the SPASE issue slip, but we really need to get the following done so that we are serving data and I can continue to generate SPASE records for the rest of the satellites.

I have attached a single SPASE XML file (NGDC_POES15_SEM2.xml) that refers to the NOAA 15 SEM-2 instrument (for now, ignore the <Instrument> section. At this point we only need to discuss the NumericalData and Service sections).

For clarity, the SPASE record only contains two parameters of 55 that will be there in the final version. I expect there to be one SPASE record like this per Satellite/Instrument pair. At present, we have about 23 Satellite/Instrument pair descriptions in the observatory. I have also attached an OE file (SPASE2OE.xml) with information about mapping from this SPASE record to OE. Before describing how I think a data request page should be rendered, I should make a few comments about some things I realized while reading your OE description, preparing this email and SPASE record, and reflecting on why my previous idea of just giving you a list of SPASE files did not work out (and was perhaps ill-concieved).

The key thing to remember is that we are only dealing with a few "stations", each of which have _very_ different parameters that are being measured. This is in contrast to some of the CLASS pages you showed, which deal with a number of "stations", all with the same instruments (or at least that measure the same physical parameter). I'll call this the Parameter/Position view. POES is one exception - there are a number of these satellites with essentially the same instruments measuring the same physical parameters. However, even in the case of POES, we think first which satellite (POES 15, for example) and then which instrument (SEM-2) and then which parameter (proton-1 field aligned .03-.08 MeV). Cluster and THEMIS are other exceptions to the Station/Instrument/Parameter view.

Radiation belt people think in terms of data in this order: (1) satellite, (2) instrument, (3) available parameters. I'll call this Station/Instrument/Parameter view. My description on how the data page should be rendered is based on the Station/Instrument/Parameter way of thinking. For now we will present the Station/Instrument/Parameter view and will rest assured that when we really need the Parameter/Position or Parameter/Satellite view, we will have the infrastructure (OE+CLASS) to deal with it.

[edit] 14.2 Data Section of Observatory

Selecting the "Data" link on the left column of the main page should take the user to the Data Section of the virtual observatory (as it currently does). Right now the user sees a large tree and then 23 records. This tree should be reduced to

All ViRBO Physical Parameters
 - Charged Particle Flux
 - Ephemeris
 - Energetic Particles
 - Differential Flux
All ViRBO Satellites
 - POES 15
 - POES 14
 - SAMPEX
 - etc.

The Physical Parameter list will be generated from the <MeasurementType> elements in the SPASE record found by searching all of the 23 Satellite/Instrument pair SPASE records. Eventually we will want this list to take on many of the features that SPASE allows, but for now we will need to leave it in this form. After the tree, there should be the same list of 23 records as they are currently rendered.

All of the 23 records will stay essentially same except the POES (SEM-2) description will be adjusted to refer to POES 15 SEM-2 (It currently just says POES which reflects a Satellite/Parameter view not a Satellite/Instrument/Parameter view as I want. The attached XML file has the correct wording.).

When a record is selected for a Satellite/Instrument a page should appear that is generated from a single SPASE record. (The page layout currently shows the FGDC record and a discussion element). The following is how I would like the page to be rendered.

• [Long description in NumericalData-->ResourceHeader-->Description part of SPASE record]
• [Link to SPASE record used to generate page (don't worry about rendering the XML in HTML for now)]
• [Time Range Selector with valid ranges determined from NumericalData-->TemporalDescription-->TimeSpan-->StartDate/EndDate]
• [Cadence Selector with vaild values determined from NumericalData-->TemporalDescription-->TimeSpan-->Cadence]
• [File Format Preference with valid values determined from NumericalData-->AccessInformation-->Format]

• [multiple select check-box list with parameters given by all listed in NumericalData-->PhysicalParameter-->Name] For the example spase record shown it would be
  • noaa15 time [get data] [view granules] [view in autoplot] [view previews]
  • noaa15 proton-1 field aligned .03-.08 MeV [get data] [view granules] [view in autoplot] [view previews]
  • 53 more parameters

After each parameter name there should be four action links/buttons:

1. [get data] will return data in the specified time range. We will either get the aggregated data from timeseries.org or your aggregation server. If the data are available from timeseries.org, there will be a Service section for timeseries.org in the xml document. To get the data from timeseries.org, you will need to form a URL based on the time, cadence, and data formats selected http://www.timeseries.org/index.php/Main_Page#Web_Service.
2. [view granules] will return a list of associated granules. This list will be returned from your server.
3. [view previews] will make a request for preview images at your server.
4. [view in autoplot] will launch autoplot by making a request to timeseries.org (with EXT in the URL set to jnlp), which will return a .jnlp file.

Now, if a user selects multiple parameters, then they must select an action button that is at the bottom of the page (as it is now). The possible actions can be [get data] [view granules] [view previews] (no autoplot option for now!).

[edit] 15 August 30 Telecon (11 am Eastern) Agenda

[edit] 15.1 All

• Fall AGU plans?

[edit] 15.2 Misha

• Progress update on SPASE
• VxOware update ppt
• Data processor description doc
• Amazon Simple Storage Service (S3) http://www.amazon.com/gp/browse.html?node=16427261
• Amazon Elastic Compute Cloud (EC2) http://www.amazon.com/gp/browse.html?node=201590011
• Common Data Model with some use cases.
• We have added a read-only user to our CVS repository on clust1.wdcb.ru, user name cvsr, password (see email)

[edit] 15.3 Bob

• Progress update on VMWare

[edit] 15.4 Jeremy

• Update on Autoplot

[edit] 15.5 Eric

• Update on NOAA data

[edit] 16 August 10 Telecon (11 am Eastern) Agenda

In red, hopefully done _before_ the telecon ...

[edit] 16.1 Bob

• Discuss paper
• Hyrax installation notes are up Hyrax installation
• Discuss simple CDF reader email discussion with McGuire, Faden, and Han
• Discuss XSLT file given to me by Merka
• Discuss VMWare work
  • Install VxOWare + Hyrax on FC 7 VMWare image
  • CVS checkout of VxOWare on FC 7 VMWare image
  • Discuss phyiscal2virtual
• Discuss how the VxOWare developers can best communicate with users. What makes it easiest on he users and makes it most likely that they will report a bug?
• Discuss how the ViRBO developers can best communicate with users
• Shri about RBC index data 303-492-9422 [done. have data, but need to apply method in paper to get RBC].
• Upload Baker's SPA tutorial when spidrd is working. What happened to my upload?
• Ask Curtis about converting PDF to embeddable slideshow [done]
• Contact Sibek about Science Working Group Meeting for RBSP
• Contact Terry about GOES data 303-497-5713 [done. have approval]

[edit] 16.2 Jeremy

• Summarize progress on Hyrax and Autoplot
  • New autoplot at: http://www.cottagesystems.com/virbo/apps/autoplot/
    • example .vap file: vap
  • Hyrax
    • server at: http://www.papco.org:8080/opendap
    • cdf_handler: http://www.papco.org:8080/opendap/cdf/sampex/30f/1994/
• Plan for configuring a Hyrax server: get it working on a reference machine and hand it off to Bob and Misha [done. bob has this working on a reference machine]
• Plan for simple Java CDF reader: Bob will talk to GSFC folks about this (McGuire noted that someone on vacation may have already worked on a similar problem). [Bob and Jeremy's work of convincing the GSFC folks that it can be done without too much work is done. Bob will meet with McGuire's group at GMU on August 31st at 11am. We can do a telecon if desired.]

[edit] 16.3 Eric

• Send Bob info he needs to map ftp://ngdc.virbo.org to his server [done. next bullet]
• Have Justin Mirror ftp://virbo.org hourly with rsync. Put his notes on command for this in wiki. Actually, I think this information should in an "admin" section of virbo.

[edit] 16.4 Misha

• Send Bob info he needs to map ftp://wdcb.virbo.org to his server
• provide details about software packages (section 4 of vxoware paper).

[edit] 17 July 23 Telecon (11 am Eastern) Assignments

[edit] 17.1 Jeremy

• Send Misha SF username.
• Write up paragraph about CDF/netCDF/THREDDS/OpenDAP issues/complications and a proposed approach to dealing with them [Jeremy, please paste your notes here. I lost the link you sent me to your notes on your wiki].
• Find out about compression mode for netCDF.

[edit] 17.2 Bob

• Send Misha SF username [done].
• Send Misha use case scenario for bug reports for both ViRBO and VxOWare [done]
• Send Eric info on Sebastien's work [done - Contributions from Sebastien are listed on virbo.org/wiki].
• Send Misha text for explaining status of VxOWare to prospective users.
• Given Jeremy's paragraph, communicate with the OpenDAP developers what we plan to do and see if they have any comments [done].

[edit] 17.3 Eric

• Send Bob info he needs to map ftp://ngdc.virbo.org to his server

[edit] 17.4 Misha

• Recent changes: ppt
• Send Bob info he needs to map ftp://wdcb.virbo.org to his server
• Fix misspellings of "Sebastien Bourdarie" on ViRBO portal
• Send some links to Bob about Amazon "data cloud" concept

[edit] 18 June 29 Telecon (11 am Eastern) Assignments

[edit] 18.1 All

Revise, contribute, and comment on this list of milestones

• Year 1 (Fall AGU, 2007)
  • "Most" data available behind DODS server and viewable with DAS2
  • Metadata goals?
  • One concatanated data set
  • A DODS server that can read CDF files
  • All data in CDF or netCDF format
  • A paragraphs for each of five data sets or events that describe it and link to pdfs and link to das2 plots to view it.
  • 2 demo screencasts. One to use wiki, one to search for data.
  • A demo for preparing and formatting a data set for contribution into ViRBO
  • "Most" metadata complete and SPASE compliant
  • A clearly-defined goal and plan for "level 2" plotting
  • ViRBO Live CD [we need to discuss this in the telecon]
  • Community buy-in and use [am using wiki and meetings to effect this]
• Year 2 (Fall AGU, 2008)
  • Add data from simulations
  • Finish level 2 time series plot
  • Possibly work on "level 2" 2D and 3D plotter (after defining what it is).
  • Work on "shopping cart" capability.
  • Public distribution of code to build ViRBO Live CD
  • Seamless "User Contribution" process

[edit] 18.2 Jeremy

• Help Bob make sense out of what is in the AMPTE/CHEM data files and converting them to CDF. (Jon Vandegriff responded and said he will be back in a week to answer my questions. Update: I sent him a second email on June 1st. Update: I met with him at the VOiG meeting and asked him to provide more metadata information. Bob. I need to do more to check the usability of the data before we work on this more. Bob.)
• Report on program to concatanate CDF files (get info on what SPDF uses because they are a CDF shop but also have a DODS server). serving and aggregating CDF with DODS
• Most important have NPOES data viewable with das2 from DODS server on either spidrd or virbo.org! (Notes on installing POES data onto virbo.org)

[edit] 18.3 Misha

Information on recent changes: ppt.

• Give shopping cart demo
• Take Bob's wiki list below and think about work required to implement features
• Move project to Sourceforge
• Have email2news testing complete by June 6th

[edit] 18.4 Eric

• Get ViRBO on http://vmo.igpp.ucla.edu/search/registry.jsp (see last Telecon notes for more info)
• Add details about SW reanalysis to main data matrix on http://virbo.org/wiki
• Most important have NPOES data viewable with das2 from DODS server on either spidrd or virbo.org!
• virbo ftp backup?

[edit] 18.5 Bob

• Check http://www.unidata.ucar.edu/projects/THREDDS/DataPublications/SampleDataPublicationFor2005AMS.htm
• Write up list of most useful features of mediawiki for Misha. [Now that I think about it, it looks like virbo already has most of the features I want. So I need to start using it and provide feedback to Misha.]
• Look into hosting RB workshop in Cook Islands [not sure if I will go, will contact organizers soon]
• Finish AMPTE/CHEM processing. ["Done". See note about this under Jeremy]
• Agenda for July meeting [Done. See http://virbo.org/wiki]
• Work with Eric on SPASE records
• I think a good next step is for me to create a cdf of a full concatanated data set (say, one for Dst at 1-hour averages from 1957-2005, and one for SAMPEX daily-averages 1993-2004), create the SPASE metadata record, add metadata to the cdf file, and then let DAS2 have at it. See html and html [Done. Report on this.]

[edit] 19 June 1 Telecon (11 am Eastern) Assignments

[edit] 19.1 For Discussion

Hi fellow VxOers -
  
I've populated a registry server with a list of descriptions for each of the
VxO.  This registry of registries uses the SPASE registry server to provide
a search and retrieval service for resource descriptions. You can take a
look at the search capability at:

  http://vmo.igpp.ucla.edu/search/registry.jsp

If you like to see XML you can tap the registry server at: 

   http://vmo.igpp.ucla.edu/registry/registrySearch.jsp 

which will return all of the registry description profiles or use:

  http://vmo.igpp.ucla.edu/registry/registrySearch.jsp?words=VITMO

to see just the one for VITMO.

You can view the associated registry description (in XML) with:
 

http://vmo.igpp.ucla.edu/registry/registrySearch.jsp?resourceID=spase://Regi
stry/VITMO

If this makes you curious how the search capability will work with other
resources you can check out the very nascent VMO search capability at UCLA.
This is located at:

  http://vmo.igpp.ucla.edu/search/

Try entering "*" in the box and hit enter, then try "isee" and enter; then
try "isee plasma" and enter. This will give you a sense of what the SPASE
registry server is capable of achieving.

Please let me know what you think. We would like to hear from you.

-Todd-

[edit] 19.2 Misha

• Think about how/if we should make the (mediawiki) wiki page and the ftp site both a part of a single entitiy as viewed from the outside. That is, if a user registers for an account, a ftp directory and a wiki account is also created. Are we going in the wrong direction? For perspective, take a look at the plans of some of the other VxO's (on the wiki page). [Bob to provide desired feature list]

[edit] 19.3 Bob

• MySQL comment to Jeremy [ok]
• Meeting Dates in July/August? [Looks like July 23 will be best.]
• Test new features that were completed by Misha and co. for the last telecon. [ok except need forwarding address for mail that hits news@virbo.org]
• Get metadata for HEO data from O'Brien. [ok]. Now need to get his cdf file generation code working.
• Look at AGU goals and revise them based on current knowledge [here they are:]
  • "Most" data available behind DODS server and viewable with DAS2
  • A DODS server that can read CDF files
  • All data in CDF or netCDF format
  • A paragraphs for each of five data sets or events that describe it and link to pdfs and link to das2 plots to view it.
  • 2 demo screencasts. One to use wiki, one to search for data.
  • A demo for preparing and formatting a data set for contribution into ViRBO
  • "Most" metadata complete and SPASE compliant
  • A clearly-defined goal and plan for "level 2" plotting
  • ViRBO Live CD [we need to discuss this in the telecon]
  • Community buy-in and use [am using wiki and meetings to effect this]
• List of year 2 goals
  • Add data from simulations
  • Finish level 2 time series plot
  • Possibly work on "level 2" 2D and 3D plotter (after defining what it is).
  • Work on "shopping cart" capability.
  • Public distribution of code to build ViRBO Live CD
• Continue to work with Jeremy on trying to define what "level 2" plotting functionality is. [ok]
• Move the Data Matrix to the public Wiki [done]
• Add security measures to virbo.org. [not done, will have Brian Curtis look into this]
• Is there an open source program that can to a ppt2pdf2png2wiki_img_gallery? [nevermind]
• Report on virbo.org wiki use at GEM [O'Brien is enthusiastic, I started page]
• Report on metadata efforts [what I am thinking about]

[edit] 19.4 Eric

• Finish NPOES data [needs to run again ...]
• Specify what he wants for Denver meeting [Bob to present ViRBO on Wed., Eric to present stewardship, Bob to present ViRBO status at Monday meeting]
• Try and find out when we will know more about the VOiG agenda (more than just session names). [ok. it is posted now]

[edit] 19.5 Jeremy

• Finish SAMPEX plotter [done. will send demo to Bob next week]
• Install VxO stuff on virbo.org. Put notes and instructions on Wiki. [almost done]
• Report on program to concatanate CDF files [he has developed limited code to do this]

[edit] 20 May 11th (11am EDT) Telecon Assignments

[edit] 20.1 Jeremy

• an applet version of the autoplotter, which allows for the initial URL to be specified from the launcher. We needed this functionality for a project for the plasma wave group, and it translates easily to our plotter.
• SAMPEX( DOY, L Shell ) make_time_L_spectrograms
• THREDDS meta data discovery problem definition
• Ask Eric about the POES data, verify POES data verifyPoesData

[edit] 20.2 Misha

• Set up and test email2news
• Change "Vote" to "
• Guest users should see all tabs ([edit, view, move, etc]) but should get notice that they need to login in order for the feature to work.

[edit] 20.3 Bob

• Convert Server logs to CDF?
• How to cross link data and documents. We don't want to enter metadata for data, get permalink, enter metadata for document add permalink and get doc permalink, then return to data metadata to add newly-generated permalink. Want bi-directional ability. Can we have "objects" tab bring up (ajax) list of recently added permalinks?
• Have name and address fields filled out using the user's information.
• modify the metadata list for "Data" section.
• All documents list should include items in "New" category

[edit] 21 April 6th (11am) Telecon Assignments

[edit] 21.1 Bob

• Need a developer mailing list [Bob look into Google Groups, etc.]
• Discuss meeting wiki with O'Brien [ok]
• Review Wiki syntax [ok]
• Review SPASE Maillist [ok]
• Figure out how to do a telecon with my phone (or not if it is still hanging up randomly) [let Eric connect since he knows better how to connect to Misha]
• Install Matlab on SPIDRD? [Eric to look into install of a permanent version (no license manager)]
• Discuss when to freeze content.
• Look at swivel.com
• Write a single sentence that describes THREDDS, DODS, OpenDAP and their differences. DAP = data access protocol (highracks is an implementation of dap protocol, DODS is another implementation?). THREDDS Is used to for creating inventory of metadata. [done. see the following three bullets]
  • DODS: Appears to be the predecessor to OPeNDAP. Much of the documentation uses OPeNDAP and DODS interchangeably.
  • http://www.opendap.org : "OPeNDAP provides software which makes local data accessible to remote locations regardless of local storage format. OPeNDAP also provides tools for transforming existing applications into OPeNDAP clients (i.e., enabling them to remotely access OPeNDAP served data)."
  • http://www.unidata.ucar.edu/projects/THREDDS "... middleware to bridge the gap between data providers and data users. The goal is to simplify the discovery and use of scientific data and to allow scientific publications and educational materials to reference scientific data."
  • Also note: dap_server="General Purpose Handler", "DAP++ SDK = libdap", and Hyrax=OLFS+BES.
• Look into VMWare stuff some more. Review Jeremy's ViRBO setup notes. [The students that showed interest in this never appeared. I'll continue to 'look into this'.]
• Discuss alternatives to "Vote"? [done. see Misha's assignments for next week]
• Use Javascript to detect browser width. If wide enough, use larger font size? [Bob look into canned solutions. Don't want to research, develop and test this ourself.]
• Ask Misha if is possible to auto-login if visited previously.

[edit] 21.2 Misha

• Set up and test email2news
• Change "Vote"
• Change expiration time to 3 hours.
• Guest users should see all tabs ([edit, view, move, etc]) but should get notice that they need to login in order for the feature to work.

[edit] 21.3 Eric

• Have server populated with .nc files

[edit] 21.4 Jeremy

• Set up virbo in Iowa City for testing development.
• Have mods sent by scan complete (Bob's app) (autoplot application)
• Identify dods server that serves netCDF files.
• Get metadata from netCDF dods server. (Note Thredds API is not needed--dods should have all the needed metadata.)

[edit] 22 March Telecon (January Post-meeting Assignments)

Progress reports on telecon in Mid-Feb. Test build for week of March 11-18 (Bob's spring break). Misha is in US starting in July-Aug, extending for 3 months.

[edit] 22.1 Bob

• http://java-source.net/open-source/wiki-engines
• Think about indexing virbo content so web crawlers can index [done. no resolution]
• Think about defining supported data pile formats [done. need to write them down.]
• Screenshot of wanted virbo changes [done]
• Meet with SPASE person and learn about FGDC to SPASE [not done. meet with someone at Spring AGU?]
• Set up meeting for March 11-18 [done]
• Misha will be here 2-3 months starting in July
• POES2NC to Eric [done]
• Post something to virbo [done] Try linking to something in ViRBO Is it possible to link to the presentation page to see any discussion? java web file system and webdav+web file system
• Schedule Telecon dates [done]
• Investigate spidrd2vmwareimage possibilities [done. meeting with student this week]
• Understand DODS, THREDDS, and DQC [done]
• Email2ViRBO News - set up news@virbo.org [done. goto mail.virbo.org user=virbo, pass=]
• Set-up procmail that writes the following to a web-acessible location on virbo.org:

<NewsDescription>
  <Title>ViRBO site</Title>
  <Link>http://spidrd.ngdc.noaa.gov/virbo</Link>
  <Description>ViRBO site prototype is open for reviewing</Description>
</NewsDescription>

Procmail does not seem like a robust solution. It requires pop3, sendmail, procmail, and perl. One could probably do it with only procmail (which can server as a mail transfer agent and allows regular expression parsing). However, this is not likely to be very maintainable. A better solution is to pop the mail off of mail.virbo.org with Java. Here is an outline of how it would be done:

Test procmail

procmail DEFAULT=/dev/null VERBOSE=yes /dev/null < /dev/null 

Send test mail from command line

mail -s "Hello 4" -r rweigel@gmu.edu rweigel@gmu.edu < bin/perl/procmail.pl 

.procmailrc

ProcmLOGFILE=/home/weigel/procmail.log
MAILDIR=/home/weigel/Mail

:0
* ^From.*rweigel
| /usr/bin/perl /home/weigel/bin/perl/procmail.pl > /home/weigel/procmail.txt

procmail.pl

#!/usr/bin/perl

while (<>) {
    if ($_ =~ /^Subject/) {
	print "$_";
    }
    if ($_ =~ /^From/) {
	print "$_"; 
    }
    # Grab mail body and format as XML here.
}

[edit] 22.2 Jeremy

• DAS2 read DODS data stream das2 reads dods notes
• Post something to virbo
• Think about indexing virbo content so web crawlers can index
• Set up model ViRBO for development, documentation. ViRBO setup

[edit] 22.3 Misha

• Think about indexing virbo content so web crawlers can index
• Install CDF module on SPIDRD and have it working with DODS.
• Find out if DODS can do equivalent of netcdfmerge with cdf files
• Find out if DODS can deal with case where data provider does not want to run DODS server, but has http directories exposed.
• Post something to virbo
• Ability to associate powerpoint to data set.

[edit] 22.4 Eric

• POES.nc from Bob to populate THREDDS server by Feb.
• Post something to virbo

[edit] 23 January 2007 Developers Meeting

[edit] 23.1 Pre-Meeting Assignments

• All: Look through the meeting agenda and ask for clarifications or make additions.
• All: If you see something that could use a link, add one.
• All: If you see something that could use a file, upload it.
• All: Browse the sites linked to in sections 1.4 and 1.5 on the main page. Add any links that you know of.
• All: Scan the ViRBO proposal
• All: Prepare slides or handouts for the sessions your name is listed in
• Misha: Be prepared to explain the details (at the level of directories, source code, servers, etc.) of the software infrastructure used for ViRBO.
• Jeremy: Take a few minutes to learn about what technology is used by ViSBARD and the SSC 3-D orbit viewer
• Jeremy: Become familiar with SPIDR.

[edit] 23.2 Meeting Time line

• Jeremy arrives 9:00 pm Tuesday
• Bob arrives Sunday and will work at NCAR with Mike Wiltberger on Tuesday.
• Wednesday will be primarily for presentations. It is quite likely that we will need to schedule additional presentations based on what comes up on Wednesday.
• Thursday and Friday will be work days

[edit] 23.3 Meeting Agenda (all meetings at NGDC in Boulder, CO)

[edit] 23.3.1 9:00 am - 10:00 am Introduction

• Bob - Purpose and goals of this meeting
  • A clear time line for software-related work that needs to be completed by 2007 AGU
  • A rough time line of software-related work that needs to be done in years 2 and 3
  • A list of what will be in the ViRBO software repository and on the ViRBO server
  • Familiarity of everyone with accessing the ViRBO software repository and server
• Bob
  • Discussion of budget.
  • Discussion and summary of VxO news and trends over the past six months. VSPO VSTO VHO VSO VGMO
  • Discussion of radiation belt data news and trends over the last year, including the Radiation Belt Storms Probe Mission (NASA), and the new round of radiation belt data assimilation proposals (MURI).
• Eric - Discussion and summary of VxO news and trends over the past six months. Discuss results from the ([1]) poster at the Fall 2006 AGU (Abstract). Discussion of SPASE progress. Anything else?
  • Need to say up front is what is in VO and what is elsewhere. "Domain Representation" issue of VMO.NET
  • VO Definition
  • Not a VO definition Media:Virbo meeting jan07.ppt
  • Data provider definition
  • Discussion of CEDAR

[edit] 23.3.2 10:00 am - 11:00 am What is a VxO?

Many people have an idea of what a VxO is. Lets discuss the following:

• Definition of a VxO according to http://lwsde.gsfc.nasa.gov
• Definition of a VO according to http://www.virtualobservatory.org and etc
• What we proposed (ViRBO proposal) that is different from this
• What others in our VxO class proposed
• What we want to do that strays too far from this
• What we want done that someone else is working on (Eric and Bob can both comment on what we know about the VxO proposals being submitted this year - here is the NASA AO).

[edit] 23.3.3 11:00 am - 12:00 What capabilities should ViRBO have?

To guide our answer to this, ask yourself what we need to do in order to make ViRBO a central place for radiation belt information and data? Building a community is difficult, and the methods for radiation belt data access by scientists are fairly entrenched. How will we provide something that is so compelling that people are motivated to use it? I worry that creating only a VxO (according to its traditional definion) will not result in a long term change and hence long term success of ViRBO. Here are some ideas:

• User Comments about VxOs
• Storage for data? What data should be local and what should be remote?
• Data set storage tools and data managment tools? Do we know of any? Is there an analog of CVS/SVN et al for data?
• ViRBO Users Guide?
• IPSAT (ONERA Radiation Belt VxO)
• Models at SPDF's ModelWeb
• Conference Organizer and Archiver
• Lots of Links. Create outline and ask Shri to fill in. Should we have introductory material on the radiation belt?
• Publication watcher (auto search key words in NASA ADS and JGR)
• Ability to click on link that says LEO Orbit and have an applet pop up that shows it exactly in say ViSBARD or the other NASA orbit viewer Java tool?
• Web watcher (auto search key words from Google searches)?
• Repository for people like O'Brien to simplify his distribution of code and bug fixes?

[edit] 23.3.4 1:00 - 4:00 pm Review of Tools

Over the past 6 months I have asked for and performed many "experiments" and done research into the different VxO related tools that are available. As you know, many of these tools fall outside of the scope of a traditional VxO. However, we proposed to develop a VxO infrastructure as well as a VxO, so some work is really in our scope. I would like everyone to report on their experiences and opinions on these projects. One goal of this meeting is to have set a clear time line for the software-related work that needs to be completed for the 2007 AGU. I would like to make our decisions based on these discussions.

• Bob - TSDS, Data Matrix, and new data (Abstract).
• Eric - SPIDR, AGU 2006
• Misha - netCDF, THREDDS (powerpoint, Environmental Scenario Search Engine)
• Jeremy - DAS2, DAS2 Video, PaPCo, NetBeans IDL IDE, ViSBARD

[edit] 23.3.5 6:30 pm Welcome dinner at Sushi Zanmi

[edit] 23.4 Use Cases

• What rad belt data is here, what’s not here?
  • Need web page that addresses this. Faq->Bob
• What is going on in rad belt world?

1. Need news site updated
2. Cc: virbo-news@virbo.org -> into queue
3. Rss searches into queue

• I have software(.m or .pro) files I want handled by virbo

1. Test this Thursday or Friday
2. Test revision scenario too.

• I want to post a raw data set that will be ingested into virbo

1. Step 1: upload the file & edit the meta data
2. Step 2: admins see it and work with scientist to create meta data record and convert to VIRBO-readable

• Query (Consumer/User) (Eric emphasizes that a. and b. support them as the same - human view uses the same API as we publish)

a.	Human
b.	Computer
i.	getCategories()
ii.	getCapabilities(category)
iii.	autoSearch( matching key filtered metadata)

• Browsing
• Level2 Abstraction use case
  • SAMPEX( DOY,L Shell),HIST ( DOY,L Shell) rebinning to (2:7:0.1)
  • Conjunctions of 2 spacecraft
  • Stack of timeseries on common time axis.
• Terry Onsager--IDL C API

• Data Abstraction Levels
  • Level 0: granules and “big zip files”—no interpretation
  • Level 1: data stream interpretation, viewer for human consumption available. das2Stream, cdf file pile, dods, netCDF. Human applies science abstraction
  • Level 2: DQL, data processing, can be plotted together, correlative studies. “Science Abstraction Level”

0. all 22 datasets
1. 10 available
2. 5 available