SkyView in a Jar

 

Using SkyView as a Standalone Java Application

 

 

Thomas McGlynn and Laura McDonald

High Energy Astrophysics Science Archive Research Center

Goddard Space Flight Center

Greenbelt, MD 20771

 

Version 2.5

August 1, 2008

Three color all-sky image in using IRIS 100, 60, and 25 micron data


Table of contents

 

1        Introduction. 3

1.1        Setup. 3

1.2        Running SkyView in a Jar 4

1.3        Simple Examples. 5

1.4        Incorporating ImageJ. 6

1.5        Other included software. 6

1.6        Dependencies on Remote Systems. 7

1.7        Batch processing. 7

2        Settings and Program Arguments. 8

2.1        Settings Files. 9

2.2        Settings Summary. 10

2.3        Detailed Settings. 13

2.4        Batch compatibility keywords. 52

3        Program internals. 53

3.1        Order of processing. 53

3.2        Software organization. 59

4        CGI Processing. 61

4.1        CGI Image requests. 62

4.2        Displaying survey descriptions. 63

4.3        Building a simple image access service. 63

5        Survey configuration. 64

5.1        Contents of the description file. 64

5.2        A Simple Survey Definition File. 70

6        Organization of the jar file. 70

 


1      Introduction

SkyView-in-a-Jar provides users with a local SkyView system on their own machines.  Users can generate FITS, GIF, JPEG, … images from major surveys in any requested geometry, resample and mosaic their own data, overlay grids and catalog positions, and create their own surveys.  SkyView-in-a-Jar comes completely ready to use as a single file. 

In this default configuration all surveys accessible through the SkyView web pages are available.  It is easy to add new surveys using local files or the Virtual Observatory Simple Image Access protocol.   Images can be converted to standard data formats and overlaid with coordinate grids or catalogs object locations.   Any HEASARC, Vizier or  VO-Cone-Search compliant catalog can be queried.

The code itself is itself extensible.  Users can add new samplers, projections, and coordinate systems.  Even the classes which define the basic algorithms for finding surveys, selecting images within the survey, and mosaicking the images can be overridden.  Users can insert their own pre- and post- processing functions to modify the input data or the resulting image.

Although there are many options available to the user, there are very few required inputs.  Users need only specify the position around which they would like an image, and the survey from which the image is to be extracted.  SkyView will take care of the rest.

This document provides a comprehensive user’s guide to SkyView-in-a-Jar.

This software is used as the basis for the SkyView web site http://skyview.gsfc.nasa.gov.  Many, though not all, of the options can be used through the Web site and web users should use this document to understand the server side processing that the Web site perform.

1.1    Setup

SkyView-in-a-Jar is self contained within a single executable jar (short for Java ARchive) file.   Jar files are a modified version of zip files used to store Java code.  Jars can include data and program control files as well as the Java code.  SkyView-in-a-Jar contains all of the control files needed to access the surveys available from SkyView’s standard Web interface.  For a few surveys the survey data itself is included.  For these surveys not even network connections are required.  The input data from other surveys is accessed through HTTP and FTP protocols.  Even when the underlying survey data is accessed over the Web, all processing and resampling is done on the local machine.

Users can download the jar file from

http://skyview.gsfc.nasa.gov/jar/skyview.jar

The same jar file is used for all operating systems.  The file can be placed in any convenient directory.  

SkyView requires at least a Java 1.5 (sometimes called Java 5.0) environment to run.  Java distributions for most major operating systems are available at

http://www.java.sun.com

Java should be pre-installed on MacIntosh machines.

 

 

1.2    Running SkyView in a Jar

To run the SkyView program a user simply enters

     java -jar path/skyview.jar   command-line-arguments

Here path is the directory in which you have placed the SkyView jar file.  The next chapter will describe the command line arguments (settings) in detail, but we will give a few examples below.  You can use alias or macro capabilities in your native operating system to make the command more convenient.  E.g., for in the sh or bash Unix shells you can define an alias with

alias skyview='java -jar mypath/skyview.jar'

and later just enter

Text Box: java -jar skyview.jar survey=DSS,FIRST position=3c273 quicklook=jpg
Processing survey:dss
  Number of candidate source images is 7.
  Number of source images to be resampled is 1.
  Processing image #3
  Creating quicklook image: output1.jpg
  Opening output file output1.fits.

Processing survey:first
  Number of candidate source images is 5.
  Number of source images to be resampled is 1.
  Processing image #3
  Creating quicklook image: output2.jpg
  Opening output file output2.fits.

                          
DSS                                                                      First
skyview command-line-arguments

to run a query.

 Suppose you would like to get a image of the quasar 3c273 in the optical and radio.  The Digitized Sky Survey (DSS) is a popular all-sky optical survey.  The FIRST (Faint Images of the Radio Sky at Twenty-one centimeters) radio survey also has coverage in this region.  The first line (in bold) in the box above is all you need to enter to get FITS and JPEG images from both of these surveys.

The output shows that the program looks at 7 candidate source files for the DSS but uses only one of them in generating the output.  It uses 1 of 5 for the FIRST survey. If the quicklook argument had been omitted, no JPEGs would have been produced, just FITS files. 

The resulting images are also shown in the box.  The jet is barely visible in the optical, but in the radio the source is double: the central AGN and the jet are both bright radio sources.  [Note that the scale of the images is not the same.  We did not specify either the size of the image or of the pixels, so the default pixel resolution for each survey was used, 1.7" and 1" respectively.]

The SkyView Java classes in the jar file can be used in other applications as well.  Simply include the JAR file in your CLASSPATH.   The only difference between a regular and executable jar file is that the later contains a manifest file that indicates how to start up when the -jar option is used in the java command. The SkyView Java application could also be started up with

java -cp pathto/skyview.jar skyview.executive.Imager key1=arg1...

If you want to use the Imager class or any of the other software packaged in the jar in other Java applications go right ahead.   There are no limitations on the use of this software.

1.3    Simple Examples

A few examples of how to generate images using SkyView are given here.  A detailed discussion of the meaning of the arguments is given in the next chapter.

 

java -jar skyview.jar position=3c273 survey=2mass-j,2mass-k,2mass-h

This returns three 300x300 [default size] pixel images named output1.fits, output2.fits, and output3.fits with the survey default pixel size for the three surveys selected.

 

java -jar skyview.jar position=3c273 survey=2mass-j,2mass-k,2mass-h   \

    quicklook=jpg nofits

 

Get the same images but only as JPEGs

 

java -jar skyview.jar position=3c273 survey=2mass-j,2mass-k,2mass-h \ 

    quicklook=jpg scaling=sqrt

 

Get the same images as both FITS and JPEGs.  Scale the JPEG brightness according to the square root of the intensity of the pixel.

 

java -jar skyview.jar position=3c273 survey=2mass-j,2mass-k,2mass-h rgb

Get the FITS images along with a 3-color JPEG where the rgb planes are from the three surveys.

 

java -jar skyview.jar position=3c273 survey=2mass-j \

scale=.0005 pixels=1200

 

This returns a single file 1200x1200 pixel file named output.fits with pixels of 1.8”.

 

 

java -jar skyview.jar position=3c273 survey=user \

   userfile=ff1.fits,ff2.fits,ff3.fits   scale=.003 \

   sampler=Lanczos4 deedger=skyview.process.Deedger

 

This request looks at the three local files ff[1-3].fits and tries to create a mosaic from them using a fourth order Lanczos resampler. Boundaries between the input images are to be masked using the standard deedger class. 

 

java -jar skyview.jar position=’horsehead nebula’ survey=dss \

postprocessor=null output=horsehead \

float coordinates=B1950

 

This asks for a DSS image of the Horsehead nebula but turns de-edging off.  The output image should use 4-byte reals and be in B1950 coordinates.  The output file will be named ‘horsehead.fits’

1.4    Incorporating ImageJ

Starting with version 1.9, the SkyView jar also contains a complete distribution of the ImageJ image display and analysis tool.  ImageJ is a very powerful public domain software package developed by Wayne Rasband (wsr@nih.gov) at the National Institutes of Health.  SkyView currently uses ImageJ functions for creating quicklook images in GIF, JPEG and other formats, including various scaling and color table operations.  ImageJ has many other capabilities for users may to analyze images created by SkyView.   Hundreds of plug-in modules are also available that provide powerful functionality in diverse areas.

Users should be able to start ImageJ itself with the SkyView jar using the command

    java -cp skyview.jar ij.ImageJ [other arguments]

or they can download the ImageJ jar file at http://rsb.info.nih.info/ij.  Much more information on ImageJ is available at that site.   Future versions of the SkyView are likely to use additional ImageJ functionality.   The SkyView jar includes both the ImageJ class and source code.  Version 2.3 of SkyView-in-a-Jar includes version 1.38k of ImageJ.  A few modifications to this version were made primarily to support drawing to text at angles and addition of specified colors to the current look up table.

You can start an ImageJ session to analyze SkyView images generated by this application by specifying the ImageJ setting.

1.5    Other included software.

Spline sampler.

The Spline sampler included here is a translation from C of the spline sampler of Philippe Thévanez (EPFL).  Later versions of the C code are available at http://bigwww.epfl.ch/thevenaz/interpolation/.

 

DSS Decompression

The DSS Decompression (skyview.survey.HDecomp) class is based upon the Java code of Pierre Fernique (CDS) which is in turn based upon the C routines of R. White (ST ScI).  Significant changes were made to the code before inclusion in SkyView.

 

FITS Library

SkyView uses the nom.tam.fits FITS library of Tom McGlynn (NASA/HEASARC). The complete library is included although no use is made of the methods to read or write tables.

1.6    Dependencies on Remote Systems

While all data processing occurs in the local system, the SkyView-in-a-Jar application may fail if remote resources are not available.  The underlying data for a few low resolution surveys are included inside the jar file (e.g., 408 MHz and EGRET.  The application will fail if it is unable to retrieve data for other surveys.  For many surveys that data is fetched from the SkyView server at skyview.gsfc.nasa.gov. However the FIRST survey links to archives at the Space Telescope Science Institute.  The 2MASS and NEAT surveys use archives at JPL.  A number of other surveys, e.g., the SDSS, now use the VO Simple Image Access protocol, and link to URLs described in the survey descriptions.  The <Images> area of the survey description will generally indicate the dependencies.

Catalog queries normally involve links to remote catalog sites and will fail if those sites are unavailable. 

The other remote dependency for the application is in the resolution of target names.  If the user specifies a target rather than a pair of coordinates, the coordinate converter service at the HEASARC is used to resolve this name into coordinates.  The HEASARC service in turn calls the SIMBAD and NED name resolvers.  The HEASARC service may also be used if the user enters the coordinates using characters other than numerics, signs, colons, commas and decimal points (e.g, 10h30m10.3s).

1.7    Batch processing.

In version 2.3 of the jar a beta-version of a new class skyview.executive.Batch was added to allow users to run a series of image requests in a single Java session.  If this class is invoked the first argument is a special requests file and while subsequent arguments are standard settings.  The first argument may be '-' to use the standard input for the requests file.

Each line of the requests file may contain one or more settings which will be concatenated to the settings given on the command line.  Any number of requests can be made.  E.g., suppose we have a requests file. myrequests.fil, containing

survey=dss,dss2r position=3c273

position=3c279

position=3c279 quicklook=gif pixels=600

position=Abell1656 output=special

 

 

If we try

     java –cp .../skyview.jar skyview.executive.Batch myrequests.fil survey=dss2b quicklook=jpg

we will run four Imager requests.  The first will create two output images output1_1.fits and output1_2.fits (and .jpg).  The second will create a single output image.  The thrird a larger output image and use the GIF format for the quicklook image.  The last will create files names special4.fits and special4.jpg since the output stem was specified explicitly.  The row number is concatenated to the stem to distinguish the outputs.

 Note that settings in the request file override those specified on the command line.

2      Settings and Program Arguments

The operation of the program is controlled by settings.  Settings are simply key-value string pairs, similar to Java Properties or Preferences, specified as an equation, e.g., survey=dss or position=3c273.   When a value contains a comma, it is treated as an array of values, e.g., survey=dss,heao1 indicates that there are two values for the survey, dss and heao1  For some settings the value is not used, only that the setting is present.  In this case the value need not be specified, e.g., the user can specify NoFITS=1 or just NoFITS to suppress FITS output.   The keywords for settings are case-insensitive, but the values may not be, especially when they refer to files or class names.  E.g., output=test is equivalent to OUTPUT=test but not to output=TEST [unless the underlying file system is case-insensitive].   In this document we tend to use mixed capitalization of the keywords that helps make their meaning clearer but we have not attempted to be rigorously consistent.

The special string "null" can be used as the value for a setting.  This unsets a previously set setting.  E.g., Deedger=null may turn off de-edging for a survey which enables it by default.   A system setting of _NullSettings is used to track settings that have been explicitly nulled so that they are not inadvertently turned back on.

The user can set settings on the command line, but they are also set in several other ways.  Settings are processed in the following order:

 

  1. At startup the application attempts to find a system settings file.  This establishes base defaults for the system.  The application also ensures that values are set for critical system settings (e.g., the class names for key processing objects). These default settings are normally read in through the system resource skyview.settings.  If users wish to provide their own default settings, they can override the settings provided in the jar file by creating a skyview.settings file in the current directory, or by pointing to a file in any location (and of any name) using the environment variable SKYVIEW_SETTINGS.
  2. Users can specify settings on the command line.  Any number of settings can be given on the command line.  If the value for a setting includes a blank or character that will be treated specially by the shell processing the command, then setting may need to be enclosed in quotes or the special characters may need to be escaped in some fashion.  E.g., typically you will need position=’3C 273.
  3. The user can specify one or more files of settings using the eponymously named setting 'Settings'.  These are parsed immediately after the command line arguments.  Note that if you want to process two settings files use "settings=file1,file2" not "settings=file1 settings=file2".  The latter will only read  file2.
  4. After the user specified settings are retrieved, one or more SettingsUpdater classes may be called depending upon the value of the SettingsUpdaters setting.  These classes may look at the current settings and modify them.  This can be used, e.g.,  to support obsolete settings or to fill in defaults for new featus.
  5. As each survey is processed, the <Settings> element in the survey definition file is parsed for survey specific settings.  These settings will not override any settings already in place.  These are used for survey defaults like image scaling, or de-edging.
  6. Later in survey processing, settings may also be set in the <Images> element.  These will override any user specified settings.  Any XML element found other than the <Image> elements inside the <Images> element specifies a setting.  These are intended to define and communicate with the classes that are used to build survey images but users are free to exploit this behavior to set settings that the user cannot override (at least without editing the survey description).
  7. Settings are also specified and consumed internally by various classes but normally this is invisible to the user.  The keys for such internal settings are usually prefixed with a “_”.

 

When a setting is set multiple times, only the last value is effective.   Some settings, especially the internal settings, may change as the program continues processing. E.g., if the user does not define the size or pixel scale of the image to be output, then the pixel scale will normally be taken from a default established for each survey.  If the user asks for multiple surveys, the pixel scale (and hence the total size of the image) could be different for each.

2.1    Settings Files

Both system and user settings files are simple ASCII files.  Each line can be a blank, a comment line (first character of the line is ‘#’) or specify a single setting.  This example might be used as the systems settings:

 

# An example skyview.settings file.

 

Pixels=500

Projection=Sin

Coordinates=J2000

Sampler=Lanczos

Equinox=2000

 

Resolver=NED-SIMBAD

XMLroot=$SKYVIEW_XMLROOT

 

Mosaicker=skyview.process.Mosaicker

SurveyFinder=skyview.survey.XMLSurveyFinder

ImageFinder=skyview.process.imagefinder.RecursiveImageFinder

SettingsUpdaters=skyview.executive.BatchCompatibility,skyview.ij.IJProcessor

 

System environment variables can be used within settings.   In this example an environment variable is used to set the XMLRoot (the directory where survey XML files may be found). When a value begins with a $ the value of the corresponding environment variable will be used but the line will be ignored if the environment variable is not defined.  This allows users to override the default settings using environment variables using two lines in the settings file, e.g.,

Projection=Sin

Projection=$PROJ

will use a Sine projection unless the user overrides it by specifying the PROJ environment variable or on the command line.

While a user can specify any number of settings on the command line, only one setting may be given on each line in a Settings file.

2.2    Settings Summary

The following table gives a complete listing of all settings currently used in SkyView and where they are discussed in this document.  A few settings that most users will want to be sure to understand are highlighted in bold type.  Settings beginning with an underscore are used internally by the program but are not normally visible to users.

 

`

Keyword

Discussed

Short  Description

 

Cache

2.3.4.5

Directory[ies] to save remote files for later reuse.

_CachedFile

4

List of files downloaded for current survey

Catalog

2.3.9.1

Catalog[s] queried for objects in output image

CatalogFile

2.3.9.2

File to save catalog data in

CatalogFilter

2.3.9.4

Constraint to be applied to catalog data

CatalogIDs

2.3.9.6

Plot catalog identifier numbers

CatalogRadius

2.3.9.3

Set the maximum radius for searching for catalog data.

COLTAB

2.4

Batch compatibility synonym for LUT

ComboSamplers

2.3.3.2

Paired set of samplers to do sampling

Compress

2.3.4.2

Compress FITS output (gzip)

Contour

2.3.6.1

Survey for which a contour map is to be generated

ContourSmooth

2.3.6.2

Smoothing  applied to contour images

Coordinates

2.3.2.7

Coordinate system used in output image

_CoordinateSystem

4

Coordinate system name suitable for HTML output

CopyWCS

2.3.2.5

Use frame of this file for output image

CornersOnly

2.3.10.7

Check only corners when selecting images

_CurrentSurvey

4

Name of survey currently being processed

Deedger

2.3.10.14

Deedging class to match backgrounds of images

DefaultSIASurveys

4.3

Surveys to be searched by default in SIA request

DescriptionXSLT

2.3.11.7

XSLT for transforming survey descriptions to HTML

Draw

2.3.8.1

One or more drawing commands

DrawAngle

2.3.8.2

Initial rotation angle for drawing

DrawFile

2.3.8.3

Draw file[s] to be read and executed.

DSS2Prefix

5.1.4

Prepended to cached files from DSS2 style surveys

Ebins

2.3.3.3

Specify the energy binning of 3-d images.

Equinox

2.3.2.9

Equinox of the coordinate system

EQUINX

2.4

Batch compatibility synonym for Equinox

ErrorMsg

4

The last error encountered by the program

ExposureFileGen

2.3.10.13

Replace string used to generate exposure file names

ExposureFileMatch

2.3.10.12

Matching string used in generating exposure files

ExposureFinder

2.3.10.11

Class to find exposure for input images

ExposureKeyword

2.3.10.8

FITS keyword containing exposure value

File

2.4

Batch compatibility synonym for Output

FilePrefix

5.1.8.1

String prepended to file names in Image elements

FindRetry

2.3.10.6

Retry with all candidates when no source pixel found

FITS

5.1.7

Content to be included in FITS files

Float

2.3.4.3

Use 4-byte rather than 8-byte floating point FITS

FooterTemplate

2.3.11.5

Template file for bottom of HTML pages

GeometryTwin

5.1.4

Surveys with the same geometry as current survey

Grid

2.3.7.1

Include coordinate grid on output image

GridLabels

2.3.7.2

Label coordinate grid with coordinate values

HeaderTemplate

2.3.11.3

Template file for top of HTML pages

HTMLWriter

2.3.11.1

Class that writes HTML wrapper

ImageFactory

5.1.8

Class that creates images from a given survey

ImageFinder

2.3.10.3

Class to find best candidate image for each pixel

ImageGenerator

5.1.8.2

Class that finds candidate images from a given survey

ImageJ

2.3.5.12

Keep the ImageJ user interface running

_ImageMax

4

Maximum value in output image

_ImageMin

4

Mininum value in output image

ImageSize

5.1.8

A typical size for images in the survey

_ImageXPixel

4

Width of output image in pixels

_ImageXSize

4

Width of output image in degrees

_ImageYPixel

4

Height of output image in pixels

_ImageYSize

4

Height of output image in degrees

Invert

2.3.5.5

Invert the specified lookup table.

ISCALE

2.4

Batch compatibility synonym for Scaling

Lat

2.3.2.4

Dec/Latitude coordinate of center of image

Level

2.3.2.8.1

TOAST Grid Parameter

LocalURL

2.3.4.8

Translate a URL to a local file

Lon

2.3.2.3

RA/Longitude coordinate of center of image

LUT

2.4

Load the specified lookup table.

MAPROJ

2.4

Batch compatibility synonym for Projection

Max

2.3.5.7

Maximum value to use for output quicklook images

_Meta_xxxx

5.1.5

Standard metadata regarding the survey.

Min

2.3.5.6

Minimum value to use for output quicklook images

Mosaicker

2.3.10.10

Class to be used to resample and combine images

Name

5.1.2

The full name of the survey

NoExit

2.3.10.1

Do not exit after completing image generation

NoFITS

2.3.4.4

Suppress output FITS image

_NullSettings

2

Settings explicitly deleted by user

Offset

2.3.2.14

Offset of the image center from specified position

OnlineText

5.1.6

Text to include in HTML output

Output

2.3.4.1

Stem for output files

_Output1 (2,3)

4

The output FITS name for the n’th survey

_Output_FITS

4

Name of output FITS file

_Output_QL

4

Name of output quicklook image file

_Output_RGB

4

Name of output RGB image

OutputRoot

2.3.11.6

Root for output images in CGI request handling

Pixels

2.3.2.11

Number of pixels in the output image

PIXELX,PIXELY

2.4

Batch compatibility synonym for Pixels

PlotColor

2.3.5.10

Color of overlays in plots

PlotFontSize

2.3.5.11

Size of the font for plot overlays

PlotScale

2.3.5.9