The UDF Archive is built around two coupled applications, aTool which is the archive manager and the archive Server which is the primary interface between the archive and the outside world. It is intent of this section is to describe the fundamental concepts, structure, and workings of the UDF archive. This will include discussion of the archive directory structure, archive types, the various archive databases, Server functions, and the file formats required for data to be recognized for inclusion in an archive.
All archives have a common directory structure. Each archive is associated with a top level Home directory which has beneath it the subdirectories Incoming and Database. The Incoming directory is where any new content which is to be included in the archive is initially placed. The Database directory holds the archive Database(s). With only one exception all archives make use of one or more databases to keep track of their contents. The Home directory also contains the ArchiveConfig file which is an ASCII file holding the archive configuration parameters.
There are two top level archive classes: software and UDF data. UDF Data archives are restricted to contain only one compoent of UDF data. They may hold UDF header/data file pairs, UDF VIDF files, or UDF PIDF files but not a mixture. Archiving the entire data set from a single satellite would then require three separate archives. Software archives can hold any currently defined software package.
An archive is identified to outside programs by its Archive ID. This tells any application interacting with an archive its class and should it exist, its subclass. In the case of data archives it also conveys what datasets may be held within it.
An archive ID is a an ASCII string containing from one to three "." separated fields. Data archives use the first two fields to give the UDF project and mission acronyms of the data sets being stored. The last field is either empty, a P or a V depending on whether the archive holds UDF header/data files, PIDF files or VIDF files respectively. As an exmaple, to archive the UDF data for the entire IMAGE satellite mission would require three archives whose IDs would be:
| IMAGE.IMAGE1 |
| IMAGE.IMAGE1.P |
| IMAGE.IMAGE1.V |
The databases maintained by the different classes of archives to keep track of their contents have different structures and formats. Not all archives use databases. The PIDF subclass of data archives does not make use of a database. The archive is generally small in size, the archived files have a unique naming convention, and they are not time dependent. Without the benefit of an database PIDF archives are not allowed to be split across directories. All of the ingested files are archived in the single directory set up in the intial configuration of the archive. As mentioned this is not a burden since PIDF archives are generally under an megabyte in total size.
Archives which hold UDF header/data files make use of multiple databases. There is one database established per defined UDF experiment under the UDF project/mission data set being archived. The database entries are fully derived from the contents of the data files being ingested together with the current archive configuration parameters. There is no need for any information pertinent to the database entry to be included with the ingested files. The database entries which are identical to those built for the local online UDF databases provide the UDF virtual name as well as the start and stop time of the data within the file, the archive code, (which points to the directory the data files are stored in) and the compression code (which provides any extension on the file names). The latter two fields while present in the local UDF databases are not used.Archives of UDF VIDF files use only a single database for the entire project/mission. Their database entry format is identical to that used in the header/data archive database and is also fully derived from the contents of the VIDF file and the current archive configuration parameters.
Software archives make use of multiple databases. Software packages currently are divided into 5 catagories and there is one database per catagory. The current categories and what packages occur under then are:
| udf | All of the kernel UDF C-code software packages and any other packages which use $UDF_HOME as their base directory. |
|---|---|
| gph | All of the kernel graphics C-code software packages and any other packages which use $GPH_HOME as their base directory. |
| uTools | All higher level analysis and display packages which use $UDFTOOL_HOME as their base directory. |
| tTools | All of the Tcl/Tk based analysis packages and any other packages which use $TCLTOOLS_HOME as their base directory |
| iTools | All of the IDL based analysis packages and any other packages which use $IDLTOOLS_HOME as their base directory |
The software databases are ASCII files and individual entries are a set lines of keywords and variables. The keywords supported by each database are identical. They are used primarily to identify to the Server where the package is stored and to the Client what steps need to be taken in installing the package. Part of the database entry is submitted with the software package. The contents of this submission are discussed in the section on package format below. The Archive Code assigned to the package is obtained on ingest from the archive configuration.
One of the common input formats used to get data into a data arhive is the UDF data bundle. The UDF data bundle holds one or more UDF data files of a single class; either PIDF, VIDF, or header/data files and all from a single UDF project/mission definition. The files are placed in a single tar file which is then gzipped for compression. The files within the tar file are all uncompressed. If data bundle contains header/data files then it must also contain a Manifest file named UDFXFER. This is an ASCII file which lists the bundle contents. Each line in the file lists on of bundle files followed by the UDF Experiment and Instument acronyms associated with that file as EXPERIMENT/IMSTRUMENT. The input lines for the files IMOORBIT20001810000D and IMOORBIT20001810000H which are found under the IMAGE OA and ORBIT Experiment and Instrument acronyms would be
IMOORBIT20001810000D OA/ORBIT IMOORBIT20001810000H OA/ORBIT
A UDF data bundle is identified to the Server by its file name. The file name consists of 5 "." separated fields. It's form is:
The Server expects files it finds in the Incoming directory of an Archive to be of a known format which depends on the archive type. All data archives will accept the udf data bundle as an input format. This is the only format which can be used of inputing UDF header/data files. PIDF and VIDF files can also be placed individually in the Incoming directory.
Software files are tarred files each containing a tarred-gzipped software package file and an ASCII file containing a subset of the package database entry fields. The Incoming file, as well as the two files it contains all use the same naming convention with the only difference being the file extensions. The file name format is
GRASTER.3.0r0.tar
tar -vtf GRASTER.3.0r0.tar
4386 2004-02-29 20:23:49 GRASTER.3.0r0.tgz
78 2004-02-29 20:23:49 GRASTER.3.0r0.dbe
The .dbe file is an ASCII file. Each line in the file consists of a database keyword followed by the keyword value. An example database entry file is shown below.
PKG UDFPLOT REV 3.0r0 DIR UDFTOOL_HOME+Plots OPT 1|UDFplot|1||0|1|0 ENV UDFTOOL_HOME+UDF_HOME+GPH_HOME+XLIB UDEP udf GDEP gph
Only defined keywords need to be included in the file. Keywords which have no definitions are omitted. The full list of keywords and their definitions are given below
The package name sans revision number an extension.
The package revision as it appears in the package file name (eg 3.0r0)
The directory in which the package contents are to be placed. This is the full path name with the individual directories which make it up separated by the + sign. The first directory is always the environment variable which points to the package Base directory (currently UDF_HOME, GPH_HOME, UDFTOOL_HOME, TCLTOOLS_HOME, or IDLTOOLS_HOME). The database that the package is added to also is determined from the environment variable. Directories under the package directory which may have content added to them and which need to be removed prior to the installation can be added to the line by adding a ^ at the end of the package directory followed by a set of + separated subdirectory names as:
DIR UDFTOOL_HOME+UDFToIDL^DIR1+DIR2In the installation process both directories $UDFTOOL_HOME+UDFToIDL+DIR1 and $UDFTOOL_HOME+UDFToIDL+DIR2 will be deleted prior to the software package being unpacked.
The installation options. These are listed as a string or | separated values. The option definitions are:
| Option | DESCRIPTION |
|---|---|
| 1 | 0 - no make required 1 - non-shared object make required 2 - shared object make required |
| 2 | If an executable is built using a call to make, the name passed into the makefile |
| 3 | 0 - no "make release" required 1 - requires a "make release" be issued |
| 4 | A + separated list of any files that need to be moved to the bin directory under the package HOME environment setting. |
| 5 | 0 - does not require pkg_mkIndex command 1 - requires a pkg_mkIndex command |
| 6 | 0 - do not remove the package directory before
installing 1 - remove the packge directory before installing |
| 7 | 0 - install under all conditions 1 - don't install if previous installation exists |
Option 7 should always be 0. The option is used in several full kernel installs where partial packages outside the kernel are installed. In this case these packages are not installed if the full package has been previously installed.
A list of + separated environment variables that must be set for installation to preceed.
A + separated list of required installed packages from the UDF software kernel. Any package can have a minimum revision attached to it by adding ,REV such as LIBBASEUDF,3.0r0. The entire UDF kernel can be specified by the entry udf.
A + separated list of required installed packages from the GPH software kernel. Any package can have a minimum revision attached to it by adding ,REV such as GPLOTTING,3.0r0. The entire GPH kernel can be specified by the entry gph.
A + separated list of required installed packages from the GPH software kernel. Any package can have a minimum revision attached to it by adding ,REV such as GPLOTTING,3.0r0. The entire GPH kernel can be specified by the entry gph.
A + separated list of required installed packages from the TCL database. Any package can have a minimum revision attached to it by adding ,REV such as TCLUDF,3.0r0.
The procedure name to use to install the package if the generic AUpkgInstall procedure will not work.
| &larr Previous | TOC | HOME | |
| Server Options |