Alternative Data Access Methods
- CDAWeb Browser interface for plots,listings and CDFs (both original and creation of sub/super-set CDFs)
- FTP access to original data CDFs
- HTTP access to original CDFs
- REST Web Services interface
- SOAP Web Services interface
- Limited SPASE QL access
- Scripts to keep up-to-date on data files for particular datasets:
- CDAWlib IDL software - reads, plots, lists and creates subset/superset CDFs (the software that underlies CDAWeb)
- Extremely easy to use IDL software that provides direct access to CDAWeb held data files.
- Autoplot.org (non-NASA) interface to public CDAWeb database
- Opendap access to CDAWeb data, with the pattern The suffixes determine the nature of the request: "data", "info", "das", "dds", "html", and "asc" for getting ASCII data; "dods" gets the binary data for use inside a client program, not a web browser.
"das" provides attribute data; "dds" provides data structure information (skeleton version of data).
A small Bash shell script below that reads the CDAWeb daily file
listing and retrieves new files (either ones matching patterns in
choosefile, or get all new files except ones matching patterns in
skipfiles). [Remove the "-p" from xargs to automate. wget can be
replaced with "curl -O" (-O has opposite meanings with curl/wget)]
ftp_cdaweb.sh:
#!/bin/sh
# FTP new files from CDAWeb; 2009 March 30 Robert.M.Candey@nasa.gov
# have to run twice the first time
# outputs new_files and deleted_files from the difference of new
and old CDAWeb file lists
# also retrieves new files, skipping any patterns in skipfiles or
selecting only
# files matching patterns in choosefiles
# remove -p from xargs command to automate
if [ -e cdaweb.diff ]; then
rm cdaweb.diff
fi
if [ -e deleted_files ]; then
rm deleted_files
fi
if [ -e new_files ]; then
rm new_files
fi
if [ -e cdaweb.curr ]; then
mv cdaweb.curr cdaweb.prev
fi
wget -O - "ftp://cdaweb.gsfc.nasa.gov/filelist.gz" | gunzip | grep "\.cdf" | sort -k4> cdaweb.curr
#curl "ftp://cdaweb.gsfc.nasa.gov/filelist.gz" | gunzip | grep "\.cdf" | sort -k4> cdaweb.curr
diff cdaweb.prev cdaweb.curr> cdaweb.diff
egrep "^> " cdaweb.diff> deleted_files
egrep "^< " cdaweb.diff> new_files
if [ -e skipfiles ]; then
if [ -e choosefiles ]; then
cat new_files|grep -v -f skipfiles |grep -f choosefiles |colrm 1 37|xargs -I{} -p -t wget "ftp://cdaweb.gsfc.nasa.gov/{}"
else
cat new_files|grep -v -f skipfiles |colrm 1 37|xargs -I{} -p -t wget "ftp://cdaweb.gsfc.nasa.gov/{}"
fi
elif [ -e choosefiles ]; then
cat new_files|grep -f choosefiles |colrm 1 37|xargs -I{} -p -t wget "ftp://cdaweb.gsfc.nasa.gov/{}"
else
cat new_files|colrm 1 37|xargs -I{} -p -t wget "ftp://cdaweb.gsfc.nasa.gov/{}"
fi
