Figure 6: Unpopulated Parent Variable Definition Menu
Importing measurements into the UDFAnalysis framework is initiated from the Variable Definition GUI menu which is invoked from the Main Menu. This brings up a parent or top level menu. The complete set of accessible menus are described below.
The parent menu is used to select the type of data being imported which will be either UDF or ASCII. Each data type has its own GUI interface in which the data file, associated measurements and any ancillary information are defined. The are described separately. An unpopulated and populated example of the parent GUI are shown in figures 6 and 7 respectively.
The menu consists of a central text window which shows a summary of the defined variables and a small subset of selected options. The first column of information in the text window is an offset into the variable definition structure where the options for this definition are kept. It is not editable by the user. This is followed by a list of the variables being imported into the UDFAnalysis framework, the data source type, and whether gaps in the data are being automatically filled.
Sources are read in the order shown in the text window, from top to bottom. You can change this order by hilighting a source and then using the up and down arrows to move it up and down in the list.
The options associated with the parent menu are described below.
Close the menu window. The window can also be closed (and reopened) by clicking the VARIABLE DEFINITIONS button in the Main Menu.
Sets the source of the next variable definition to be added. This is either UDF or ASCII. UDF includes the IDFS data format. Variables which have a UDF/IDFS source will autopromote if they are not locally available when the menu is run. This entails locating a data source on the web and then retrieving the data and installing it locally. ASCII files must be physically present on the local system or manually retrieved.
Creates an entry for the selected SOURCE type in the menu text window similar to those seen in the populated menu above. If there is a hilighted entry in the text window the new definition is placed above that entry. If no entry is hilighted the new definition is placed at the bottom of the list.
The variable field in the new definition is initially blank but the other fields are populated with default values. The Variable field is filled when data source options are set up. The GUI through which the options are accessed and set is brought up through the EDIT button.
This brings up the menu associated with the currently hilighted source definition in the text window. No action occurs if no source has been identified. The source definition menus are described in detail later and contain all of the options needed to both define the data source and to link named variables in the UDFAnalysis framework to the measurements being acquired.
This updates the information on all of the lines in the text window reflecting any changes or additions to the variable name lists made when editing a definition.
This option unhilights the currently hilighted source definition in the text window.
Deletes the currently hilighted source definition in the text window. This removes the entry line and clears all variables associated with the definition. Not to be used lightly as there is no recovery. No action occurs if no source has been identified for deletion.
Editing an ASCII source entry in the parent variable definition GUI brings up the menu:
The menu consists of three sections, the definition of the internal variable names at the top, the variable storage options and extra variable definitions to the left, and the interface to the ASCII source at the right. You can have multiple versions of this window active at one time if you are editing multiple ASCII source definitions.
The options in the menu are described below.
Close the menu window. The window can reopened by re-editing the variable entry.
The list of variable names associated with the input measurements. These names are used as references to the input measurements in all subsequent menus. There must be one variable name in the list for each measurement being input from this source. The first name in the list will be associated with the first measurement being imported, the second with the second measurement, and so on. Variable names are space separated and must be unique. You can use the N.M,, vec, or ten, prefixes to assign sets of measurements to sets of linked variables.
NOTE: Not all variables read in an ASCII file need to be imported into the UDFAnalysis framework.
Set this option to YES if the measurements in the source are time based. Time based measurements have at least one time associated with them and possibly two (a start and a stop time).
This option is valid only for time based measurements. Set this option to YES if you want the measurements to be stored in the system-wide time grid. If set to NO then the measurements will be kept at the resolution they are read in. In general you will want to keep this option set to YES but if you do elect to keep the variables at their native time resolution then you will probably also want to define variables for the associated measurement time under Extra Variables.
The method used to store time based measurements in the time grid. This is a toggle option which switches between BAND and POINT. The BAND storage method spreads data across all of the cells in the grid which cover the duration of the measurement. Cells which are partially filled by a measurement are assigned a time weighted average of the value. Measurements which only have a single time associated with them use it as both the start and the stop time (zero time duration) in which case BAND storage is identical to POINT storage. With POINT storage, data is placed into the cell which contains the measurement starting time.
Data placed in a grid cell is added to any data which may already exist in it. Once the total data set has been accumulated each cell in the grid is normalized by the summed weighting factors of data stored in it.
NOTE: Non-time based data is always stored as an array of values beginning at 0 and ending at N-1 where N is the number of values read in.
Set this option to YES to use a linear interpolation to fill unfilled cells in a data grid. Unfilled cells can occur for several reasons. Time based data stored using POINT storage can have unfilled cells when the measurements are stored in a grid which has a higher temporal resolution than the data itself. Data exclusion based on the next option can also result in unfilled cells in the data grids.
This field establishes a data exclusion criteria. It consists of a conditional statement, either < or >, followed by a value. All measurement values which meet the condition are omitted from storage in the time grid. It is useful in many instances to remove fill data or data spikes. Any holes in the grid caused by data removal can be filled using the Fill option above.
This option is used only if the measurements are Time Based and allows the start and stop time associated with the measurements be stored as independent variables. The times will be stored as deltas from the beginning time given in the Time Definition Menu. The first listed variable will hold the start time and the second the stop time. If there are no stop times associated with measurements the second variable, if present, will be ignored. Note: This field should be left blank if no time variables need to be returned. The time variables are used if the data being input may need to be gridded or regridded.
The name of the file holding the measurement data to be read. If the file is not in the directory from which UDFAnalysis was invoked it must be prefixed with the full path name. It is always safest to use the full path name in the specification as this removes having to invoke UDFAnalysis from a specific location.
The type of ASCII file being accessed which can be either be Generic or MOM-UDF. Generic files have arbitrary format. The format as well as the file content is defined in the fields listed under the Generic File Information heading. MOM-UDF are ASCII files which contain plasma moments generated by the MOments application. For this type of file both the format and measurements are known so there is no need to describe how to parse the file or to specify its contents.
Figures 9and 10 show examples of a populated ASCII definition GUI for a generic and MOM-UDF file respectively.
The list of the measurements to be imported into the UDFAnalysis framework. This can be either all or a subset of the available measurements. There should be a one to one correspondence between the list of measurements being imported into the UDFAnalysis environment and the list if variable names given in the Variables field. The association between the imported measurements and the variable names follow the order in which they are specified.
With generic ASCII files the set of available measurements is listed in the FILE VARIABLE field under the Generic File Information heading in the menu. The measurements available to be imported into the UDFAnalysis environment have names beginning with an underscore (_). Time variables, which do not have the preceding underscore, are not available for import.
When the source file is a UDF-MOM the measurements which can be imported are given the table below. All of the vector and tensor moments are returned in GSE coordinates unless the measurement name is preceded by a b (such as bVx) in which case the measurement is returned in a magnetic field based coordinate with the x direction along the field. You can import measurements in both coordinate systems. Unlike generic variables, these do not start with and underscore.
| Measurement | Description | Units |
| N | Plasma Density | m−3 |
| Vx, Vy, Vz | Bulk Velocity Elements | m/s |
| Ex, Ey, Ez | Average Energy | eV |
| Vxx, Vxy, Vxz | Second Moment of Velocity | m2∕s2 |
| Vyx, Vyy, Vyz | ||
| Vzx, Vzy, Vzz | ||
| Tx, Ty, Tz | Plasma Temperature | eV |
| Pxx, Pxy, Pxz | Pressure Tensor Elements | Pascals |
| Pyx, Pyy, Pyz | ||
| Pzx, Pzy, Pzz | ||
| Vxxx, Vxxy, Vxxz | Third Moment of Velocity | m3∕s3 |
| Vxyy, Vxyz, Vxzz | ||
| Vyyy, Vyyz, Vyzz | ||
| Vzzz | ||
| Qxx, Qxy, Qxz | Heat Flux Tensor Elements | eV-m/s |
| Qyx, Qyy, Qyz | ||
| Qzx, Qzy, Qzz | ||
| Qx, Qy, Qz | Equivalent to Qxx, Qyy, Qzz | eV-m/s |
| F1 thru F10 | User Defined Additions | |
The measurements F1 through F10 are associated with additional data which may have been added to the moments file when it was created. These were added at the users discretion and have no standard definition. Their contents will be listed in the header of the moments file.
This option only pertains to UDF_MOM ASCII files. If any of the measurements are being imported in the magnetic field aligned coordinate system then this field contains the variable name of the magnetic field vector components in GSE which are used to rotate the coordinate system. This data must be read in prior to reading in the moment data.
This field applies only to GENERIC ASCII files. A generic ASCII file is assumed to consist of a block of header information followed by a block of measurements. The block of measurements is assumed to of a set of identically formatted lines each containing a mixture of timing information and measurements of which the timing information is only present for time based data. The measurements in each line are assumed to be unique, that is the same measurement is not repeated in the line. The File Format field is the C format specification used to read the data line. As an example, consider the following line from an ASCII file of magnetic field data.
2007 020 13:59:59.9000, +1.070e-01, +2.557e+00, -1.490e+00
The format to read this line is given by
%d %d %d:%d:%f, %e, %e, %e
This field links the elements associated with each format element specified in the File Format field to a variable name. The variable names are listed in the order that they are read on the line. The names associated with measurements (as opposed to time variables) are free form but must begin with an underscore (_). Time variables when present on the line must conform to the following naming conventions:
| Variable Name | Definition |
| bY | Beginning Year |
| bD | Beginning Day of Year or Day of Month |
| bMo | Beginning Month |
| bH | Beginning Hour |
| bM | Beginning Minute |
| bS | Beginning Second (may be fractional) |
| bMs | Beginning Millisecond |
| eY | Ending Year |
| eD | Ending Day of Year or Day of Month |
| eMo | Ending Month |
| eH | Ending Hour |
| eM | Ending Minute |
| eS | Ending Second (may be fractional) |
| eMs | Ending Millisecond |
Keeping to the example of magnetic field data given in the previous section, the variables associated with the line of data shown could be listed as:
bY bD bH bM bS _Bx _By _Bz
or similarly:
bY bD bH bM bS vec,_B
were the vec,_B expands to _Bx, _By, _Bz
This field identifies how to recognize the start of the data block or equivalently the last line in the file header information block. If there is no header information in the file and the data starts at the first line of the file then this field should be set to _FirstLine_. Leave this field blank if the data begins after the first blank line in the file otherwise provide the first word(s) on the line preceding the data. Note that these word(s) cannot appear as the first word(s) in any other line in the header block. As an example if the line BEGIN DATA precedes the data block then set the field to BEGIN or to BEGIN DATA.
NOTE: At times it might be necessary to add a line separating the header information from the data in the file if no unique separator exits. You also need to remove any trailing lines of information which may appear at the end of the file after the data block.
If this is a time based set of data which only has a single time tag associated with it and if the data is regularly spaced in time then setting this option to YES will direct the read routine to supply an ending time to each measurement. This is done using an algorithm which looks at the time separation between adjacent the lines of data and determines the most likely value of the measurement duration to use for all data values. It should be emphasized that the ending time of a line is not just set to the beginning time of the next line. The use of a common measurement time to set the end times for each line ensures that time gaps in the data will be recognizable and preserved. The option is ignored both for non-time based data and if there is a second time tag provided in the data.
Editing a variable entry associated with an UDF/IDFS source opens the menu:
The menu consists of three sections, the definition of the internal variable names at the top, the variable storage options to the left, and the interface to the UDF/IDFS source at the right. The first two areas are identical to what is seen in the ASCII source menu above with the exception that the UDF supports a larger number of extra variables. The UDF definition portion of the menu is composed of two sections, the UDF source definition and a measurement definition. The latter is expandable. Clicking the ADD SENSOR button will duplicate the MEASUREMENT DEFINITION block allowing multiple measurements within the selected UDF source to be imported. You can have multiple versions of this window active at one time if you are editing multiple UDF source definitions. A populated version of the menu is shown below followed by a detailed description of the menu options.
Close the menu window. The window can reopened by re-editing the variable entry.
The list of variable names associated with the input measurements. These names are used as references to the input measurements in all subsequent menus. There must be one variable name in the list for each measurement being input from this source. The first name in the list will be associated with the first measurement being imported, the second with the second measurement, and so on. Variable names are space separated and must be unique. You can use the N.M,, vec, or ten, prefixes to assign sets of measurements to sets of linked variables.
The variable name(s) associated with the input index measurements. The field should be left blank if no index measurements are being returned. When there are variables defined in this field, they will be filled on a first come first serve basis, that is, the first defined variable will be filled with the first defined index measurement, the second with the second, and so on until the defined variables are exhausted. Variable names are space separated and must be unique. When multiple index variables are being returned you can use the N.M, prefix to assign measurements to a set of linked variables.
This option is ignored in a UDF/IDFS source since the data by definition are time based having both a start and stop time tag per measurement.
Set this option to YES if you want the measurements to be stored in the system-wide time grid. If set to NO then the measurements will be kept at the resolution they are read in at. In general you will want to keep this option set to YES but if you do elect to keep the variables at their native time resolution then you will probably also want to define variables for the associated measurement time under Extra Variables.
The method used to store time based measurements in the time grid. This is a toggle option which switches between BAND and POINT. The BAND storage method spreads data across all of the cells in the grid which cover the duration of the measurement. Cells which are partially filled by a measurement are assigned a time weighted average of the value. With POINT storage, data is placed into the cell which contains the measurement starting time.
Data placed in a grid cell is added to any data which may already exist in it. Once the total data set has been accumulated each cell in the grid is normalized by the summed weighting factors of data stored in it.
Set this option to YES to use a linear interpolation to fill unfilled cells in a data grid. Unfilled cells can occur for several reasons. Time based data stored using the POINT storage method can leave unfilled cells when the measurements are stored in a grid which has a higher temporal resolution than the data itself. Data exclusion based on the next option can also result in unfilled cells in the data grids.
This field establishes a data exclusion criteria. It consists of a conditional statement, either < or >, followed by a value. All measurement values which meet the condition are omitted from storage in the time grid. It is useful in many instances for removing fill data or data spikes. Any holes in the grid caused by data removal can be filled using the Fill option above.
This option allows the beginning and ending times associated with the measurements to be stored as independent variables. The times are stored as deltas from the beginning time given in the Time Definition Menu. This field should be left blank if no time variables need to be returned. The time variables are used if the data being input may need to be gridded or regridded. When there are variables defined in this field, they will be filled on a first come first serve basis, that is, the first defined variable will be filled with the beginning time of first defined measurement, the second with the ending time of the first defined measurement, and so on until the defined variables are exhausted. Since in most cases all of the defined measurements are returned at the same time, generally two variables will suffice.
This option allows the beginning and ending phase angles associated with the measurements to be stored as independent variables. When there are variables defined in this field, they will be filled on a first come first serve basis, that is, the first defined variable will be filled with the beginning phase angles of first defined measurement, the second with the ending phase angles of the first defined measurement, and so on until the defined variables are exhausted. Since in most cases all of the defined measurements are returned at the same phase angles, generally two variables will suffice.
This option allows the beginning and ending elevation angles associated with the measurements to be stored as independent variables. When there are variables defined in this field, they will be filled on a first come first serve basis, that is, the first defined variable will be filled with the beginning phase angles of first defined measurement, the second with the ending phase angles of the first defined measurement, and so on until the defined variables are exhausted. Ideally there should be two variables per returned measurement. When there is only a center elevation for the measurement it will be returned in both the beginning and ending variables. If there is no elevation angle information available the values will be set to the current UDFAnalysis bad data value.
Disable (FREEZE) or enable (THAW) the cascade clearing of option settings below a changed UDF field. Changing the current PROJECT, MISSION, EXPERIMENT, INSTRUMENT or VIRTUAL setting causes all the lower options to be cleared (including the measurement options) and forces them to be reselected. This happens because the lower options generally depend on the current settings of the higher options. Turning on FREEZE allows you to change any UDF source option without the lower options being cleared. This is really only viable when dealing with a multi-satellite missions such as CLUSTER where the lower settings are often independent of the MISSION. (The options are the same for all spacecraft.) In this case FREEZE allows you to change the spacecraft without having to re-enter (in this case) all the same information to get to the same set of measurements.
In general use the FREEZE option with great care. Misuse can cause unforeseen problems.
The Project under which the measurements to be imported are to be found.
The Mission under the selected Project which contains the measurements to be imported.
The Experiment under the selected Mission which contains the measurements to be imported.
The Instrument under the selected Experiment which contains the measurements to be imported.
The Virtual Instrument under the selected Instrument which contains the measurements to be imported.
A list of groups of measurements. Select the group which contains the specific measurement to be imported.
A list of measurements under the selected measurement group. You will have the option to return either a single sensor from the list or all of the measurements in the list which have a common set of units. This is provided in the EXTRA SENSORS option. To return a single measurement, select that measurement. To return all measurements under the group with a common set of units select the first measurement in the list which fits your criteria.
Each of the measurements being returned must have an associated variable name in the Variables field.
This option allows you to import multiple measurements from a single measurement definition. To import only the selected measurement set the option to No Extra Sensors. To import all the measurements in the selected measurement group which have the same units as the selected measurement set the option to All in Group. To import all of the measurements in all groups which have the same units as the selected measurement set the option to all All Defined. With the latter setting you should select the base measurement from the first group in which a measurement you want imported is found.
It is important to understand when multiple measurements are being imported the order in which that takes place to correctly assign the variable names in the Variables field. Measurements are returned measurement block by measurement block where a measurement block is the defined set of options under a MEASUREMENT DEFINITION heading in the menu. There can be multiple defined blocks of measurements through the use of the ADD SENSOR button. Measurements in the first defined block are returned first followed by the measurements in the following blocks. Within each measurement block the first measurement returned is the selected base measurement. This is then followed by any additional measurements indicated by using the All in Group or All Defined option. With the All in Group option, the measurements within the group with the same units as the base measurement are returned beginning with the first measurement after the base measurement and proceeding cyclically back to the base measurement. With the All Defined option the measurements are returned as with the All in Group option except once the measurements in the selected MEASUREMENT GROUP have been exhausted the measurements in the remaining defined groups are looked at. Groups are searched cyclically beginning with the first group after the selected group.
The filled menu above illustrates the concept. Here the selected measurement group contains all three components of the magnetic field vector in the order Bx, By, Bz. By selecting All in Group all three sensors will be returned beginning with the selected sensor, Bx. These measurements will be associated with the variables C2Bx, C2By and C2Bz. Note the use of the vec, prefix to specify the components.
Return either the selected measurement(s) or data associated with the selected measurement(s). The associated data varies but always includes the quality flags associated with the measurement. Most of the time you will leave this option in its default setting.
A list of units that the measurement can be returned in. Select the appropriate units that you want the measurement returned with.
If the selected base measurement is returned as an array of values (as opposed to a single scalar value) then this option is a list of the units associated with the array indices. As an example, for a instrument which measures electrons as a function of energy this may be the energy or velocity associated with each element in the array.
Allows the selection of an alternate base measurement to that selected in Array Unit (1). This is often used when the array units being returned consist of an upper and lower measurement such as an upper and lower energy limit.
