Document title: UCLA IGPP Flat File - Header Format Project: ISEE1 and ISEE2 NDADS Datatype: MAG_4S_FF and MAG_1M_FF Super-EID: DOCUMENT There may be other documents also identified by this super-EID. NDADS filename: FFHEADER.SFD TRF entry: b46617.txt in NSSDC's controlled digital document library, Mar. 1998. Document text follows: ---------------------- CCSD3ZF0000100000001CCSD3FF0000500000001CCSD3CS00004markeraa ADIDNAME = NSSD0210; CCSD$$MARKERmarkeraaNSSD3KS00020markerbb ADIDNAME = "NSSD0210"; REVISION_NUMBER = 0; SUBMISSION_DATE = 1994-09-21; REGISTRATION_DATE = 1995-03-17; REVISABLE = Y; RELEASABLE = Y; EXPECTED_RELEASE_DATE = 1995-03-17; BRIEF_TITLE = "UCLA IGPP Flat File - Header Format"; BRIEF_DESCRIPTION = "A flat file system is a two-dimensional data base system developed at UCLA/IGPP. Flat files are used to store data composed of multiple records consisting of one or more data fields. This describes the header portion."; REVISION_COMMENT = "Original"; GROUP = REGISTRANT; TITLE = Mr.; FIRST_NAME = Bryan; MIDDLE_NAME = ""; LAST_NAME = Littlefield; AFFILIATION = "UCLA"; GROUP = ADDRESS; ADDRESS_LINE = "University of California at Los Angeles"; ADDRESS_LINE = "Institute of Geophysics and Planetary Physics"; ADDRESS_LINE = "5833 Slichter Hall"; ADDRESS_LINE = "Los Angeles, CA 90024-1567 USA"; END_GROUP; GROUP = EMAIL_ADDRESS; NETWORK = NSI/DECnet; NET_ADDRESS = "BRUNET::BRYAN"; END_GROUP; GROUP = EMAIL_ADDRESS; NETWORK = NSI; NET_ADDRESS = "bryan@igpp.ucla.edu"; END_GROUP; PHONE = "+1-310-206-9955"; FAX = "+1-310-206-3051"; END_GROUP; GROUP = REVISOR; TITLE = Mr.; FIRST_NAME = Bryan; MIDDLE_NAME = ""; LAST_NAME = Littlefield; AFFILIATION = "UCLA"; GROUP = ADDRESS; ADDRESS_LINE = "University of California at Los Angeles"; ADDRESS_LINE = "Institute of Geophysics and Planetary Physics"; ADDRESS_LINE = "5833 Slichter Hall"; ADDRESS_LINE = "Los Angeles, CA 90024-1567 USA"; END_GROUP; GROUP = EMAIL_ADDRESS; NETWORK = NSI/DECnet; NET_ADDRESS = "BRUNET::BRYAN"; END_GROUP; GROUP = EMAIL_ADDRESS; NETWORK = NSI; NET_ADDRESS = "bryan@igpp.ucla.edu"; END_GROUP; PHONE = "+1-310-206-9955"; FAX = "+1-310-206-3051"; END_GROUP; GROUP = REVISOR; TITLE = Mr.; FIRST_NAME = Harry; MIDDLE_NAME = ""; LAST_NAME = Herbert; AFFILIATION = "Univ. of California"; GROUP = ADDRESS; ADDRESS_LINE = "University of California at Los Angeles"; ADDRESS_LINE = "Institute of Geophysics and Planetary Physics"; ADDRESS_LINE = "5833 Slichter Hall"; ADDRESS_LINE = "Los Angeles, CA 90024-1567 USA"; END_GROUP; GROUP = EMAIL_ADDRESS; NETWORK = NSI/DECnet; NET_ADDRESS = "BRUNET::HARRY"; END_GROUP; GROUP = EMAIL_ADDRESS; NETWORK = NSI; NET_ADDRESS = "hherbert@igpp.ucla.edu"; END_GROUP; PHONE = "+1-310-825-9030"; FAX = "+1-310-206-3051"; END_GROUP; REFERENCED_ADID = 0; PROJECT_NAME = ""; MISSION_NAME = ""; INSTRUMENT_NAME = ""; CCSD$$MARKERmarkerbbCCSD3SS00002markercc =============================================================================== UCLA IGPP FLAT FILE SYSTEM OVERVIEW =============================================================================== The flat file system is a two-dimensional data base system developed at UCLA/IGPP. Flat files are useful for storing data which exists as multiple records, called rows, each consisting of one or more data fields, called columns. Here it is most often used for time series lists of real numbers, but because the data fields can contain most common computer data types - floating point, integer, character, logical - the system is quite flexible. A flat file consists of two data objects, each typically in a separate file: 1. An ASCII character header object containing meta data, i.e., information that describes the values in the binary data object. This header object contains such information as: record length, number of rows, creation date, description of each data field or column, and room for a user-written abstract about the data. Please refer to the Data Description Record Object in the CCSDS registered description whose ADID=NSSD0210 for a complete description of the format and information in a header object. On the Sun/UNIX and VAX/VMS systems this object is typically found in a file with a type extension of '.ffh' or '.FFH'. Future mentions of this file will use the lower case style. 2. A binary data object containing the actual data values. The data object is handled as a fixed record length binary file. This structure provides for the following features: it allows the data to be access either randomly or sequentially, there is no need to interpret character strings before numerically manipulating the data, and the storage requirements are much lower than they would be for data stored as character strings. On the Sun/UNIX and VAX/VMS systems this object is typically found in a file with a type extension of '.ffd' or '.FFD'. Future mentions of this file will use the lower case style. UCLA has developed a number of software packages to manipulate flat files. Descriptions of those packages available at the time of registration of this description are found in another supplementary object in this description. This software matches the header object (file) and data object (file) by matching the file names whose type extensions are ".ffh" and ".ffd" respectively. CCSD$$MARKERmarkerccCCSD3DS00002markerdd ------------------------------------------------------------------------------- HEADER FILE: DETAILED DESCRIPTION ------------------------------------------------------------------------------- The header object portion of a flat file system is an ASCII character object that describes the contents of the data object. The header object contains information about the data object's physical characteristics (name, size, record length, etc). It also contains descriptions and locations of data columns, and a semi-free-form abstract, which may consist of start and stop times, owner, description of the data, etc., along with information about any changes to the data or other activities involving the data. The header object is made up of ASCII characters. Each line is 72 characters long including trailing blanks as needed. There are no line delimiters, the object may be thought of as a fixed record length file. This design was selected so that a header object has the identical format regardless of file system or operating system. Since there are no line delimiters, the header object is NOT a standard text file. On Sun/UNIX systems the header file may be converted to a text file using the program "ffhcat". On VAX/VMS systems, the header file may be read directly by the VMS editors and written out as a text file. Listed below is a sample header object, with line numbers added for reference on the left side. Following this listing is a detailed description of the components of the flat file system header object. LINE # HEADER RECORDS ----- ------------------------------------------------------------------------ 1 DATA = myfile.ffd 2 CDATE = 86 003 JAN 3 14:21:49 UPDATE = 86 336 DEC 2 14:21:05 3 RECL = 24 4 NCOLS = 5 5 NROWS = 146 6 OPSYS = SUN/UNIX 7 # NAME UNITS SOURCE TYPE LOC 8 001 UT SEC UNIVERSAL TIME T 0 9 002 BX ROT nT PVO OMAG R 8 10 003 BY ROT nT PVO OMAG R 12 11 004 BZ ROT nT PVO OMAG R 16 12 005 BT nT PVO OMAG R 20 13 ABSTRACT 14 FIRST TIME = 86 068 MAR 9 04:30:30.000 15 LAST TIME = 86 068 MAR 9 07:18:30.000 16 OWNER = C. T. RUSSELL 17 MISSING DATA FLAG = 1.0000000E+34 18 AVERAGE INTERVAL = 00:01:00.000 19 ORBIT NUMBER(S) = 2650 20 PVOFF: 86 003 JAN 3 14:21:49 21 Data request for Dr. Russell 22 FFROT: 86 336 DEC 2 14:21:05 23 BX ROT .9630 .2660 .0014 BX VSO 24 BY ROT = -.2656 .9625 .0544 X BY VSO 25 BZ ROT .0132 -.0528 .9985 BZ VSO 26 END The flat file system header object has three parts: 1. Physical description of the data object. Lines 1 to 6 above. 2. Data column descriptions. Lines 7 to 12 above. 3. Semi-free-form textual abstract. Lines 13 to 26 above. NOTE: The abstract always begins with a line containing only the keyword "ABSTRACT", line 13 above, and it always ends with a line containing only the keyword "END", line 26 above, which also indicates that the end of the flat file system header object has been reached. Also, while there are no requirements for the abstract, certain semi-standard formats have been developed which are described below. NOTE: The spacing of "KEYWORD =" information is significant in the physical description section and abstract section. The spacing is also significant in the data column description section. In general, each line of the physical description section begins with 8 character keywords (include blanks and "=") and each line of the abstract keyword information begins with 21 character keywords (again including blanks and "="). FORTRAN formatting information for writing and reading each of these lines is included below with the description of each keyword. NOTE: For all time references that include a Day-Of-Year (DOY) value, the DOY for January 1 is "001" and the range of values is "001" to "366". Please refer to the "CTIME quick reference guide" included in this document for a description of how time is represented and manipulated in UCLA/IGPP flat files. The following is a detailed description of the various components of the flat file system header object, using the example above as a reference. ------------------ Line 1: "DATA =" ------------------ The "DATA =" line of the flat file system header object contains the name of the binary data file associated with this header object. This item is included as reference information, it is NEVER used by the UCLA/IGPP software to determine the name of the data file that is associated with the header object. The UCLA/IGPP software always assumes that the names of the header file and data files are identical, except for the type extension, "ffh" for header files and "ffd" for data files. The FORTRAN formats used with this keyword are: WRITE:('DATA = ',a) READ:(8x,a) ------------------ Line 2: "CDATE =" ------------------ The "CDATE =" line gives the creation date of the flat file. There is an optional "UPDATE =" field permitted on the same line which indicates the LAST TIME the flat file was updated, if at all. The "CDATE = " and "UPDATE =" strings always begin at the same character positions in the line (characters #1 and #39). Since these times are only included as reference information for the user and are not utilized by the UCLA/IGPP software, there are no absolute format requirement for the time. However, in actual use, the time is always written as shown in the sample above, namely, " YY DOY MMM DD HH:MM:SS ". The FORTRAN formats used with this keyword are ("UPDATE =" is optional): WRITE:('CDATE = ',a24) READ:(8x,a24) --OR-- WRITE:('CDATE = ',a24,6x,'UPDATE =',a24) READ:(8x,a24,14x,a24) ------------------ Line 3: "RECL =" ------------------ The "RECL =" line gives an integer value that is the length of each fixed-length record in the flat file system data object, measured in bytes. This information is used by UCLA/IGPP software to access the data object. The FORTRAN formats used with this keyword are: WRITE:('RECL = ',i6) READ:(8x,i12) ------------------ Line 4: "NCOLS =" ------------------ The "NCOLS =" line gives an integer value that is the number of columns in each record in the flat file system data object. This information is used by UCLA/IGPP software to access the data object. The FORTRAN formats used with this keyword are: WRITE:('NCOLS = ',i6) READ:(8x,i12) ------------------ Line 5: "NROWS =" ------------------ The "NROWS =" line gives an integer value that is the number of rows, also referred to as records, in the flat file system data object. This information is used by UCLA/IGPP software to access the data object. The FORTRAN formats used with this keyword are: WRITE:('NROWS = ',i11) READ:(8x,i12) ------------------ Line 6: "OPSYS =" ------------------ The "OPSYS =" line contains a character string that identifies which operating system's data storage format the flat file system data object is written in. This information is used by UCLA/IGPP software to determine if the data requires conversion of floating point and integer values before it can be correctly accessed. The valid entries for this field are currently "SUN/UNIX" and "VAX/VMS". Please refer to the "DATA REPRESENTATIONS GUIDE" included in this document for a description of the integer and floating-point formats used in UCLA/IGPP flat files. The FORTRAN formats used with this keyword are: WRITE:('OPSYS = ',a) READ:(8x,a) -------------------------------------------- Line 7: Data column description header line -------------------------------------------- The next line contains a header line for the data column description lines. This line is included to help the user to read the data column description lines, it is not used by UCLA/IGPP software. The allowed format of the lines is as follows: " # NAME UNITS SOURCE TYPE LOC " ----------------------------------------------------- Lines 8 to "ABSTRACT": Data column description lines ----------------------------------------------------- These lines are used to describe the data columns in the flat file system data object. There is one line per data column so there should be as many data column descriptor lines as the value indicated in Line 4 for the NCOLS item. The following is a sample of the column description header line with a column descripter included. Above these are lines showing the character position of the items in the line. Following this is a description of each of the items along with their character position in the line: 1 2 3 4 5 6 7 123456789012345678901234567890123456789012345678901234567890123456789012 # NAME UNITS SOURCE TYPE LOC 001 UT SEC UNIVERSAL TIME T 0 The FORTRAN formats used with the column description lines are: WRITE:(i3.3,1x,a10,a10,a24,2x,a4,i5) READ:(i3.3,1x,a10,a10,a24,2x,a4,i5) CHARACTERS 1-3: # - The column number in the data object, with leading zeroes as needed. There are only three character positions for this field and in fact the UCLA/IGPP software only supports from 1 to 999 data columns. CHARACTER 4: blank - Character 4 is a field separator. It is always blank. CHARACTERS 5-13: NAME - The name of the data column. This is a short but meaningful title identifying the data column. Any ASCII characters, including blanks, are permitted. This field is often used by the UCLA/IGPP software to locate particular data items in the data object and for labelling of plots and listings. CHARACTER 14: blank - Character 14 is a field separator. It is always blank. CHARACTERS 15-23: UNITS - The units of measurement of the data column. Any ASCII characters, including blanks, are permitted. This field is often used by the UCLA/IGPP software for labelling plots and listings. CHARACTER 24: blank - Character 24 is a field separator. It is always blank. CHARACTERS 25-49: SOURCE - This field may be used to indicate the source of the data or to provide any additional useful information. Any ASCII characters, including blanks, are permitted. This field is not typically used by the UCLA/IGPP software and is instead provided as a reference for the user. CHARACTER 50: blank - Character 50 is a field separator. It is always blank. CHARACTERS 51-54: TYPE - This field is used to indicate the storage type of the data column, using FORTRAN-like references. There are a limited number of legal entries in this field: T - Time, a real*8 value that is the number of seconds since Jan 1, 1966 at 00:00:00.000 R - Real*4 floating point number R*4 - Real*4 floating point number R*8 - Real*8 floating point number I*2 - Integer*2 whole number I*4 - Integer*4 whole number A*## - ASCII character string. ## MUST be a multiple of two and cannot exceed 98, for example, 2,4,6,8,10,12,14...94,96,98 Please note that while the low level UCLA/IGPP software supports all these storage types, most high level software currently assumes that the first data column is type "T" and that the remaining data columns are type "R". This limitation is gradually being addressed. CHARACTER 55: blank - Character 55 is a field separator. It is always blank. CHARACTERS 56-59: LOC - This field is used to indicate the starting byte location of the data column in the data record where the first byte is zero. Please note that, as with the TYPE field, while the low level UCLA/IGPP software supports the full range of possible data locations, most high level software currently assumes that the first data column is type "T", starting in byte location zero, and that the remaining data columns are type "R", starting in byte locations 8, 12, 16, 20, etc. This limitation is gradually being addressed. CHARACTERS 60-72: blanks - Characters 60-72 are filler. They are always blank. --------------- "ABSTRACT" line --------------- The "ABSTRACT" line is required and is used by the UCLA/IGPP software to indicate that the end of the data column description lines has been reached. All lines after the line "ABSTRACT" are optional except the "END" line, which indicates that the end of the header object has been reached. However, two semi-standard documentation formats have been developed to summarize information that is often useful in tracking the characteristics, history and modifications of the data. The first is a fixed-form keyword section which includes general information about the data. The second is a less structured section that identifies any modifications that have been made to the data object by the various high level flat file system programs. Both these sections are described below. ------------------------------------- ABSTRACT: Fixed-form keyword section ------------------------------------- The following lines may be added to a flat file system header object abstract using the subroutine "fhputinfo" and they may be read using the subroutine "fhgetinfo". These subroutines were developed to make it easier to provide a standard set of information to the user and to many of the high level flat file system programs. The keyword items may appear in any order after the "ABSTRACT" line and before the "END" line. If there are multiple occurances of a keyword then the last occurance is used. If there is no occurance of a keyword then a default value is returned to the user. Thus, it is recommended that the following items be included in the abstract, preferably at the beginning for easy of viewing by an interactive user: "FIRST TIME =" - The time of the first data record in the data object. The time format is typically " YY DOY MMM DD HH:MM:SS:MMM " The FORTRAN formats used with this keyword are: WRITE:('FIRST TIME = ',a28) READ:(a18,3x,a28) DEFAULT:-1.0d34 "LAST TIME =" - The time of the last data record in the data object. The time format is typically " YY DOY MMM DD HH:MM:SS:MMM " The FORTRAN formats used with this keyword are: WRITE:('LAST TIME = ',a28) READ:(a18,3x,a28) DEFAULT:1.0d34 "OWNER =" - This is typically the name of the person who created or requested the flat file. The FORTRAN formats used with this keyword are: WRITE:('OWNER = ',a) READ:(a18,3x,a) DEFAULT:' ' "MISSING DATA FLAG =" - The value placed in a field when no valid value exists, the fill value. The FORTRAN formats used with this keyword are: WRITE:('MISSING DATA FLAG = ',g14.7) READ:(a18,3x,g14.7) DEFAULT:1.0e32 "AVERAGE INTERVAL =" - If the data is averaged, this field indicates over what time interval the average was taken. The text string "HIGH RESOLUTION" indicates that no averaging was performed, the data in this flat file is the same resolution as the source data. The text string "UNKNOWN" is used when no AVERAGE INTERVAL was available or if the status of the source data is unknown. The FORTRAN formats used with this keyword are: WRITE:('AVERAGE INTERVAL = ',i6.2,':',i2.2,':',i2.2,'.',i3.3) READ:(a18,3x,i6.2,a1,i2.2,a1,i2.2,a1,i3.3) i.e. 'AVERAGE INTERVAL = hour:mn:sc.msec' DEFAULT:-1.0d0 --OR-- WRITE:('AVERAGE INTERVAL = HIGH RESOLUTION') READ:(a18,3x,a) DEFAULT:-1.0d0 --OR-- WRITE:('AVERAGE INTERVAL = UNKNOWN') READ:(a18,3x,a) DEFAULT:-1.0d0 "ORBIT NUMBER(S) =" - This field provides the orbit number or numbers for data in the data object. A value of zero indicates that there is no orbit number. A range may be specified to indicate that multiple contiguous orbits are in the same flat file. There is no supported way to indicate that there are multiple non-contiguous orbits in the same flat file. The FORTRAN formats used with this keyword are: WRITE:('ORBIT NUMBERS(S) = ',i6) READ:(a18,3x,i6) DEFAULT:0 --OR-- WRITE:('ORBIT NUMBERS(S) = ',i6,1x,'-',1x,i6) READ:(a18,3x,i6,3x,i6) DEFAULT:0,0 --------------------------------------- ABSTRACT: Record of data manipulations --------------------------------------- Many high level flat file system programs append additional information to the flat file system header object indicating changes, additions or manipulations of the values contained in the flat file system data object. The preferred format for these lines is as follows: PROGRAM_NAME: DATE_OF_ACTIVITY COMMENTS_CONCERNING_ACTIVITY COMMENTS_CONCERNING_ACTIVITY . . . COMMENTS_CONCERNING_ACTIVITY The following example is from the sample flat file system header object listed above: PVOFF: 86 003 JAN 3 14:21:49 Data request for Dr. Russell FFROT: 86 336 DEC 2 14:21:05 BX ROT .9630 .2660 .0014 BX VSO BY ROT = -.2656 .9625 .0544 X BY VSO BZ ROT .0132 -.0528 .9985 BZ VSO In this example there are records from two different programs that operated on the data. The first entry is from the program "PVOFF" which created the on-line disk flat file by reading a magnetic tape containing the data. The only additional information added by "`pvoff" was to indicate who the flat file was created for. The second entry is from the program "FFROT", which rotated the data into a different coordinate system. In this case, the rotation matrix that was applied to the original data has been included for reference. ---------- "END" line ---------- The "END" line is required and is used by the UCLA/IGPP software to indicate that the end of the flat file system header object has been reached. CCSD$$MARKERmarkerddCCSD3SS00002markeree ------------------------------------------------------------------------------- FLAT FILE UTILITY PROGRAMS ------------------------------------------------------------------------------- FF2ASCII Write flat files to ASCII tape or disk file. Reads a flat file and writes it out as ASCII text. The ASCII text may be directed to a disk file or to magnetic tape. FFAVG Average flat file data. Reads one or more flat files, computes averages of specified data columns and writes these to another file. Can also compute maxima, minima, root-mean-square of the deviation, and quartiles (2nd quartile=median). FFCALC Perform math operations on flat file data. This program will create a new flat file whose columns are the results of operations on another file. You can do most mathematical and trigonometric operations on flat file data. FFCLEAN Removes 'glitches' in the data. Reads a flat file and writes a new flat file with spikes in the data converted to flags. FFCONVERT Converts a flat file from Vax to Sun or Sun to Vax. Program on the Vax converts Sun flat files to Vax flat files, the program on the Sun converts Vax flat files to Sun flat files. FFHCAT Convert flat file header file to or from ascii text. Reads a header object and writes an ascii file to standard output, or Reads an ascii file from standard input and writes a header file. FFHEDIT Edit header file. Copies a header file into a temporary standard ASCII file, runs vi on it for you, and then when you've exited vi, allows you to overwrite the original header file. FFMERGE Merge flat files. Merges, concatenates, and extracts flat file data. Files can be merged by linear or quadratic interpolation, or by nearest point. FFSORT Sort flat file data. Sorts the data by time, allows you to remove or keep duplicate times. FFSR Flat File Save and Restore. FFSR copies flat files to and from magnetic tape. Currently FFSR will read tapes written by FFSR on an HP/UNIX computer, a VAX/VMS computer or on an HP1000/A900. FL Flat file Lister and Plotter. Interactively displays flat files. Within FL you can: a. Do fast and sequential searches for a given time, b. Compute averages, minima, maxima and standard deviations. c. Apply a mask to the data columns - listing data fields which fall in a certain range. d. Create an ASCII file from your flat file. e. Plot selected columns versus time on a variety of graphics devices using the "FPLOT" program described below. f. Plot one column versus another. g. Flag selected data columns of selected rows. h. Interactively select data points to flag from a time series plot. FPLOT Flat file plotting program. Reads a parameter file and plots selected columns versus time or versus another column on a variety of graphics devices. PVROT Rotates PVO magnetic field data files This program can rotate PVO magnetic field data from spacecraft coordinates to VSO coordinates. The Ephemeris data is also needed to perform this operation. FFTRANS Rotates Geophysical data into various coordinates. ISEE_ROT Rotates Isee data into various coordinates. ------------------------------------------------------------------------------- FLAT FILE SUBROUTINES AND FUNCTIONS ------------------------------------------------------------------------------- The flat file routines are callable by user written programs. There are versions of the routines in C and in FORTRAN. The quick reference guides for these two libraries are included below. Also included below is a help file for the FORTRAN callable versions of the UCLA/IGPP time manipulation routines that are typically used when accessing flat files. C callable versions of these routines are also available. ----------------------- C quick reference guide ----------------------- #include char fname[]; FF_ID *id; FF_O_PARAM o_param; FF_COL_DESC col_desc; FF_H_INFO info; FILE *fp; int recl,ncols,nbufs,len; long nrows,row; char hrec[76]; struct flat { double time; float data[]; } drec; /* Choose from following status options */ o_param.status = FF_EXIST | FF_CREATE | FF_READ | FF_WRITE; o_param.recl = recl; o_param.ncols = ncols; o_param.nrows = nrows; o_param.nbufs = nbufs; if ((id=ff_open (fname,&o_param))==NULL) exit(1); /* Write column descriptor record */ col_desc.ncol = 2; strcpy(col_desc.name,"Bx"); strcpy(col_desc.units,"nT"); strcpy(col_desc.source,"ISEE1"); strcpy(col_desc.type,"R"); if (ff_hput(id,&col_desc) < 0) goto ff_error; /* Read column descriptor record */ col_desc.ncol = 7; if ( (n=ff_hget(id,&col_desc)) < 0) goto ff_error; if ( n==0 ) break; /* end of file */ /* List header file on standard output */ if (ff_hlist(id) < 0) goto ff_error; /* Write header file record */ if (ff_cput(id,"header line") < 1) goto ff_error; /* Get info records from header file */ if (ff_hgetinfo(id,&info) < 0) goto ff_error; /* Put info records to header file */ if (ff_hputinfo(id,&info) < 0) goto ff_error; /* Read header file record */ if ( (n=ff_cget(id,hrec)) < 0) goto ff_error; if ( n==0 ) break; /* end of file */ /* Efficient write of record(s) to data file */ fp = id->ffd->fp; if (fwrite(&drec,sizeof(drec),1,fp) == 0) goto ff_error; /* Efficient read of data file record(s) */ fp = id->ffd->fp; if (fread(&drec,sizeof(drec),1,fp) == 0) { if (ferror(fp)) goto ff_error; /* error */ break; /* end of file */ } /* Efficient way to position file at row */ if (fseek(fp,row*id->ffd->recl,0) != 0) goto ff_error; /* Position data file to before record 99, numbered from 0*/ if (ff_setrow(id->ffd,99) < 0) goto ff_error; /* Return next row number, numbered from 0 */ if ( (row=ff_getrow(id->ffd)) < 0) goto ff_error; /* Binary search for time, set row1 and row2 = 0 for whole file */ if ( (row = ff_bsearch(id,time,row1,row2)) < 0) go to ff_error; ff_close(id); exit(1); ff_error: ff_reporterror(id); Flat file structure members: FF_ID *id; FILE *(id->ffh->fp); char *(id->ffh->name); FILE *(id->ffd->fp); char *(id->ffd->name); int id->ffd->nrows; int id->ffd->recl; int id->ffd->ncols; FF_O_PARAM o_param; int o_param.status; int o_param.recl; int o_param.ncols; int o_param.nrows; int o_param.nbufs; FF_COL_DESC col_desc; int col_desc.ncol; char col_desc.name[FF_COL_NAMELEN]; char col_desc.units[FF_COL_NAMELEN]; char col_desc.source[FF_COL_SOURCELEN]; char col_desc.type[FF_COL_TYPELEN]; int col_desc.loc; FF_H_INFO info; double info.first_time; double info.last_time; double info.resolution; double info.flag; char info.owner[L_cuserid]; int info.first_orbit; int info.last_orbit; double info.first_time; ----------------------------- FORTRAN quick reference guide ----------------------------- All integers are default integers (INTEGER*4). FNAME is character. Other needed declarations are shown. The flat file system uses Fortran file unit numbers 23-34. Do not use these numbers for other files. New create/open/close routines. IRECL is in BYTES!! call FFCREATE ( FNAME, IRECL, NCOLS, NROWS, ID1, ID2, IERR, NBUF) call FFOPEN ( FNAME, IRECL, NCOLS, NROWS, ID1, ID2, IERR ,NBUF) call FFUPDATEOPEN ( same arguments as FFOPEN ) call FFCLOSE ( ID1, ID2, IERR) Data read/write. Note LEN is in BYTES!!! call FPUT ( ID, RECORD, LEN, IERR) ! write record (array of LEN bytes) call FGET ( ID, RECORD, LEN, IERR) ! read record ( " " " " ) Header read/write. character NAME*10,UNITS*10,SOURCE*24,TYPE*4,OWNER*16,STRING*72 real*8 T1,T2,DTAVG call FHPUT (ID1,IERR,NCOL,NAME,UNITS,SOURCE,TYPE,LOC) call FHGET (ID1,IERR,NCOL,NAME,UNITS,SOURCE,TYPE,LOC) call FHGETINFO (ID1,IERR,T1,T2,OWNER,FLAG,DTAVG,IORBIT1,IORBIT2) call FHPUTINFO (ID1,IERR,T1,T2,OWNER,FLAG,DTAVG,IORBIT1,IORBIT2) call FHUPDINFO (ID1,IERR,T1,T2,OWNER,FLAG,DTAVG,IORBIT1,IORBIT2) call FCPUT ( ID1, STRING, IERR) ! write character header record call FCGET ( ID1, STRING, IERR) ! read character header record call FHCOPYABSTRACT (IDIN,IDOUT,IERRIN,IERROUT) Positioning. Note that IERR is required with FFPOS. call FFPOS ( ID, IREC, IERR) ! set file position to IREC. IREC = JFPOS (ID) ! return current file position. NROWS = JNROWS (ID2) ! return length of data file. Fast time search. real*8 TIME call FBSEARCH ( ID, TIME, JREC, IERR , JREC1 , JREC2) ! search for first record whose ! first 8 bytes are >= TIME Miscellaneous routines. call FFPURGE (FNAME, IERR) ! purge flat file pair call FLATFILENAME (ID, IERR, FNAME) ! return full name. call FLATREPORTERROR (ID,IERR) ! Report error to output device. Old create/open/close routines kept for backward compatibility. call FCREA ( FNAME, IRECL, NCOLS, NROWS, ID1, ID2, IERR) ! IRECL=bytes call FOPEN ( FNAME, ID1, ID2, IERR) call FCLOS ( ID1, ID2) Old FPOS (note no error return - does nothing if ID passed incorrectly) call FPOS ( ID, IREC) ! set file position to IREC. Specify number of buffers to use. FFBUFFS is unneeded on the SUN/UNIX or VAX/VMS unless you want to change LUOUT. call FFBUFFS (NFILES, NBUFS, LUOUT) All the above routines are located in a default library which is searched automatically by the linker. ------------------------------------------------------------------------------- "CTIME" quick reference guide ------------------------------------------------------------------------------- Cline time "Cline time" (sometimes referred to as "ctime") is an 8-byte floating point number that is is the number of seconds since Jan 1, 1966 at 00:00:00.000. It is named for it's original designer, Neal Cline, of UCLA/IGPP. Cline time routines are used to manipulate and display time values in many programs. With them you can compare, compute and display time values. Since most UCLA/IGPP programs rely on data which is time dependent, these routines are handy for many applications. Additionally, they are quite easy to use. Cline time values can be stored in four formats: DOUBLE PRECISION (REAL*8) ('T' format) 8 bytes per value INTEGER(8) ('IT' format) 32 bytes per value CHARACTER ('CT' format) Up to 28 bytes per value INTEGER*2(14) or INTEGER*4(7) ('AT' format) Up to 28 bytes per value The Cline time routines can handle most any time thrown at them - currently, dates from 1600 MAR 01 to year 9999 seem to work OK. In the T format, a time value is stored in a double precision number, also known as REAL*8, representating the number of seconds since Jan 1, 1966 at 00:00:00.000. This format has three main advantages: it is very easy to compare several values to each other, it is the best format to manipulate time mathematically, and it takes up very little internal program space. Its major disadvantage is that it must be converted for display in human readable form. Two conversion types are available, to integer form (referred to as "IT format") and to character form (referred to as "CT format" or "AT format"). In the IT format, a time value is broken up into eight parts and stored in an eight element integer array. The values are stored as follows: IT(1) - Year IT(5) - Hours IT(2) - Day of year (1-366) IT(6) - Minutes IT(3) - Month (1-12) IT(7) - Seconds IT(4) - Day of month (1-31) IT(8) - Thousanths of seconds Years 0-99 designate 1900 to 1999, otherwise use four digits. The advantage of this format is that individual components of a time can be compared or viewed. It is also relatively easy to look at a time in IT format on the screen. The major disadvantage of IT format is that it is very difficult to compare IT times. In the CT format, a time value is stored as ASCII data in a character string, that is, in a fortran CHARACTER data element. The purpose of the CT format is for ASCII input or output, it must be converted to a Cline time (seconds since Jan. 1, 1966) before it can be manipulated mathematically. The standard CT output (from routines ICONC and TCONC) is 28 characters and comes in 1 of 2 formats: columns 1 5 10 15 20 25 ....+....+....+....+....+... format 1: 79 031 JAN 31 12:32:07.110 two digit year for 1900's - or - format 2: 1879 031 JAN 31 12:32:07.110 four digit years As input, the time string can be in several more or less standard formats: 79 031 123207 1979 31 123207 79 jan 31 123207.110 79 031 12:32:07.11 NOTES: 1. Four digit years are OK, two digit years represent the 1900's. 2. Month abbreviations are the first three letters of the month. Lower case is OK. The AT format is identical to the CT format except that the ASCII characters are stored in arrays of non-character type: INTEGER, REAL, LOGICAL - anything except CHARACTER (as in the days before FORTRAN 77). In these arrays each ASCII alpha-numeric character occupies one byte. Thus, an INTEGER*2 array of 14 elements contains 14*2=28 characters. An INTEGER*4 array of 7 elements also contains 28 characters. There are ten Cline time routines which convert times from one format to another. All these routines require that you specify both the input and output values as formal parameters. TCONI Usage: CALL TCONI(T,IT) Converts a DOUBLE PRECISION time, T, into an 8 element INTEGER array, IT. Year will be 0-99 for 1900's. ICONT Usage: CALL ICONT(IT,T) Converts an 8 element INTEGER array, IT, into a DOUBLE PRECISION T. ICONC Usage: CALL ICONC(IT,CT) Converts an 8 element array IT into a 28 character string, CT. The year, IT(1), is converted to ASCII as specified (1900 is not subtracted). CCONI Usage: CALL CCONI(CT,IT) Converts a character string, CT, into an 8 element INTEGER array, IT. TCONC Usage: CALL TCONC(T,CT) Converts the DOUBLE PRECISION time T into a 28 character string. Year will be 2 digits for 1900's. CCONT Usage: CALL CCONT(CT,T) Converts string CT, to the DOUBLE PRECISION T. ICONA Usage: CALL ICONA(IT,AT) Converts the 8 element array IT into a 28 character string in the array AT, which should be either INTEGER*4 AT(7) or INTEGER*2 AT(14). ACONI Usage: CALL ACONI(N,AT,IT) Converts N characters in array AT into an 8 element INTEGER array IT. TCONA Usage: CALL TCONA(T,AT) Converts the DOUBLE PRECISION time T into a 28 character string in the array AT. ACONT Usage: CALL ACONT(N,AT,T) Converts N characters in array AT into a DOUBLE PRECISION T. TNOW Usage: T = TNOW() Function which returns current time. Both T and TNOW must be declared DOUBLE PRECISION. The following is a sample program which uses the three time formats: PROGRAM TEST C C Program to round off two times to the nearest hour and then say C which is larger. C INTEGER ITA(8),ITB(8) DOUBLE PRECISION TA,TB CHARACTER CT*28 C C Get two times from user and store in TA and TB C WRITE(*,'(A)') 'Enter first time' READ(*,'(A)') CT ! Read character time into CT CALL CCONT(CT,TA) ! Convert to T format C WRITE(*,'(A)') 'Enter second time' READ(*,'(A)') CT ! Use same CT array for second time CALL CCONT(CT,TB) ! Convert second time to TB C C Now add 1800 seconds (half hour) to each time so that the C rounding off process will go to the nearest hour. C TA=TA+1800.0D0 ! Add to first time TB=TB+1800.0D0 ! Add to second time C C Now round off both times by converting to IT format and setting C the minutes, seconds, and milliseconds to zero. C CALL TCONI(TA,ITA) ! Convert first time CALL TCONI(TB,ITB) ! Convert second time C DO I= 6,8 ! Element 6 is minutes, 7 is seconds, and 8 is ms. ITA(I)=0 ITB(I)=0 ENDDO C C Now convert rounded off numbers back to T format and compare. C CALL ICONT(ITA,TA) CALL ICONT(ITB,TB) C IF(TA.GT.TB) THEN ! If first time is larger CALL TCONC(TA,CT) WRITE(*,'(A,A)') CT,' is the larger time' ELSE IF(TB.GT.TA) THEN ! If second time is larger CALL TCONC(TB,CT) WRITE(*,'(A,A)') CT,' is the larger time' ELSE ! Times are the same WRITE(*,'(A)') 'The times are the same' ENDIF C STOP 'Thank you' END ------------------------------------------------------------------------------- DATA REPRESENTATIONS GUIDE ------------------------------------------------------------------------------- ------------------------------- DEC VAX/VMS Data Representation ------------------------------- =============================================================================== The following is from the "Programming in VAX FORTRAN" manual from Digital Equipment Corporation, AA-D034D-TE, September 1984, Pages C-1 through C-3. =============================================================================== Appendix C - FORTRAN Data Representation ---------------------------------------- (NOTE: Section C.1, which describes LOGICAL*1 representation, is not included because this data type is not used by UCLA/IGPP flat files.) C.2 INTEGER*2 Representation ---------------------------- 15 14 0 --------------------------------- |S| | |I| BINARY NUMBER | :A |G| | |N| | --------------------------------- SIGN = 0(+), 1(-) Integers are stored in a twos complement representation. INTEGER*2 values are in the range -32768 to 32767, and are stored in two contiguous bytes aligned on an arbitrary byte boundry. For example: +22 = 0016(hex) -7 = FFF9(hex) C.3 INTEGER*4 Representation ---------------------------- 31 30 0 ------------------------------------------------------------------ |S| | |I| BINARY NUMBER | :A |G| | |N| | ------------------------------------------------------------------ SIGN = 0(+), 1(-) INTEGER*4 values are stored in twos complement representation and lie in the range -2147483648 to 2147483647. Each value is stored in four contiguous bytes, aligned on an arbitrary byte boundry. Note that if the value is in the range of an INTEGER*2 value, that is, -32768 to 32767, then the first word can be referenced as an INTEGER*2 value. C.4 Floating-Point Representations ---------------------------------- The exponent for the REAL*4 and REAL*8 (D__floating) formats is stored in binary excess 128 notation. Binary exponents from -127 to 128 are represented by the binary equivalents of 1 through 255. (NOTE: Text describing REAL*8 (G__floating) and REAL*16 are not included because these data types are not used by UCLA/IGPP flat files.) For each floating-point format, fractions are represented in sign-magnitude notation, with the binary radix point to the left of the most significant bit. Fractions are assumed to be normalized, and therefore the most significant bit is not stored (this is called "hidden bit normalization"). This bit is assumed to be 1 unless the exponent is 0. If the exponent equals 0, then the value represented is either zero (refer to the section entitled "Representation of 0.0" in VAX FORTRAN User's Guide) or is a reserved operand (refer to the section entitled "Reserved Operand Faults" in VAX FORTRAN User's Guide). C.4.1 REAL*4 (F__floating) -------------------------- REAL*4 (F__floating) data is four contiguous bytes starting on an arbitrary byte boundry. Bits are labeled from the right, 0 through 31. 15 14 7 6 0 --------------------------------- |S| | | |I| EXPONENT | FRACTION | :A |G| | | |N| | | --------------------------------- | | | FRACTION | :A+2 | | | | --------------------------------- 31 16 SIGN = 0(+), 1(-) The form of REAL*4 (F__floating) data is sign magnitude, with bit 15 the sign bit, bits 14:7 an excess binary exponent, and bits 6:0 and 31:16 a normalized 24-bit fraction with the redundant most significant bit not represented. The value of F__floating data is in the approximate range: 0.29*10**-38 through 1.7*10**38. The precision is approximately one part in 2**23, that is, typically seven decimal digits. C.4.2 REAL*8 (D__floating) -------------------------- REAL*8 (D__floating) data is eight contiguous bytes starting on an arbitrary byte boundry. Bits are labeled from the right, 0 through 63. 15 14 7 6 0 --------------------------------- |S| | | |I| EXPONENT | FRACTION | :A |G| | | |N| | | --------------------------------- | | | FRACTION | :A+2 | | | | --------------------------------- | | | FRACTION | :A+4 | | | | --------------------------------- | | | FRACTION | :A+6 | | | | --------------------------------- 63 48 SIGN = 0(+), 1(-) The form of REAL*8 (D__floating) data is identical to an F__floating real number, except for an additional 32 low-significance fraction bits. The exponent conventions and approximate range of values are the same as those for F__floating. The precision is approximately one part in 2**55, that is, typically 16 decimal digits. ---------------------------- SUN/UNIX Data Representation ---------------------------- =============================================================================== The following is from the "SPARCompiler FORTRAN 2.0.1 Reference Manual" from SunPro, a Sun Microsystems, Inc. Business, Part No: 800-6554-11, Revision A, October 1992, Pages 421 through 427. =============================================================================== Appendix C - Data Representations --------------------------------- This appendix is a brief introduction to data representation. For some detail and explanation, read the "FORTRAN User's Guide"; for even more, read the "Numerical Computation Guide". Whatever the size of the data element in question, the most significant bit of the data element is always stored in the lowest-numbered byte of the byte sequence required to represent that object. C.1 Real, Double, and Quadruple Precision ----------------------------------------- Real, double precision, and quadruple precision number data elements are represented according to the IEEE standard by the following form, where "f" is the bits in the fraction. (-1)**sign * 2**exponent-bias * 1.f Table C-1 Floating-Point Representation ----------------------------------------------------------------- | | Single | Double | Quadruple | |===============|===============|===============|===============| | Sign | Bit 31 | Bit 63 | Bit 127 | |---------------|---------------|---------------|---------------| | Exponent | Bits 30-23 | Bits 62-52 | Bits 126-112 | | | Bias 127 | Bias 1023 | Bias 16583 | |---------------|---------------|---------------|---------------| | Fraction | Bits 22-0 | Bits 51-0 | Bits 111-0 | |---------------|---------------|---------------|---------------| | Range approx. | 3.402823e+38 | 1.797693e+308 | 3.362e-4932 | | | 1.175494e-38 | 2.225074e-308 | 1.20e+4932 | ----------------------------------------------------------------- (NOTE: Sections C.2 through C.4, which discuss "Extreme Exponents", "IEEE Representation of Selected Numbers", and "Arithmetic Operations on Extreme Values", are not included because they are beyond the scope of a document describing UCLA/IGPP flat files. For a complete description of Sun/UNIX floating point arithmetic please refer to the "IEEE Standard 754 for Binary Floating-Point Arithmetic" or the compiler manuals for any computer vendor that implements this standard.) C.5 Bits and Bytes by Architecture ---------------------------------- (NOTE: While the following information does not directly define Sun/UNIX INTEGER*2 and INTEGER*4 representations, it does explain how these representations differ from those of DEC VAX/VMS, which are described above. By reviewing the DEC discussion and then implementing the changes below, one will be able to understand Sun/UNIX integer representations since the meaning of the bits are the same, the only difference being that the bytes are in a different order. The information in this section is also required to understand how to convert bot integer and floating-point values from one data representation system to the other.) The order in which the data--the bits and bytes--are arranged differs between VAX computers on the one hand and SPARC computers on the other. The bytes in a 32-bit integer, when read from address "n", end up in the register as shown below. Table C-8 Bits and Bytes for Intel and VAX computers ------------------------------------------------------------------------- | Byte n+3 | Byte n+2 | Byte n+1 | Byte n | |=================|=================|=================|=================| | 3 3 2 2 2 2 2 2 | 2 2 2 2 1 1 1 1 | 1 1 1 1 1 1 0 0 | 0 0 0 0 0 0 0 0 | | 1 0 9 8 7 6 5 4 | 3 2 1 0 9 8 7 6 | 5 4 3 2 1 0 9 8 | 7 6 5 4 3 2 1 0 | ------------------------------------------------------------------------- Most Significant Least Significant Table C-9 Bits and Bytes for 680x0 and SPARC computers ------------------------------------------------------------------------- | Byte n | Byte n+1 | Byte n+2 | Byte n+3 | |=================|=================|=================|=================| | 3 3 2 2 2 2 2 2 | 2 2 2 2 1 1 1 1 | 1 1 1 1 1 1 0 0 | 0 0 0 0 0 0 0 0 | | 1 0 9 8 7 6 5 4 | 3 2 1 0 9 8 7 6 | 5 4 3 2 1 0 9 8 | 7 6 5 4 3 2 1 0 | ------------------------------------------------------------------------- Most Significant Least Significant The bits are numbered the same on these systems, even though the bytes are numbered differently. (NOTE: Text describing the use of the Extended Data Representation (XDR) format is not included because this data type is not used by UCLA/IGPP flat files.) CCSD$$MARKERmarkeree