Next: , Previous: Libraries, Up: Introduction


1.4 netCDF2/3/4 and HDF4/5 Support

netCDF version 2 was released in 1993. NCO (specifically ncks) began soon after this in 1994. netCDF 3.0 was released in 1996, and we were eager to reap the performance advantages of the newer netCDF implementation. One netCDF3 interface call (nc_inq_libvers) was added to NCO in January, 1998, to aid in maintainance and debugging. In March, 2001, the final conversion of NCO to netCDF3 was completed (coincidentally on the same day netCDF 3.5 was released). NCO versions 2.0 and higher are built with the -DNO_NETCDF_2 flag to ensure no netCDF2 interface calls are used. However, the ability to compile NCO with only netCDF2 calls is worth maintaining because HDF version 4 1 (available from HDF) supports only the netCDF2 library calls (see http://hdf.ncsa.uiuc.edu/UG41r3_html/SDS_SD.fm12.html#47784). Note that there are multiple versions of HDF. Currently HDF version 4.x supports netCDF2 and thus NCO version 1.2.x. If NCO version 1.2.x (or earlier) is built with only netCDF2 calls then all NCO operators should work with HDF4 files as well as netCDF files 2. The preprocessor token NETCDF2_ONLY exists in NCO version 1.2.x to eliminate all netCDF3 calls. Only versions of NCO numbered 1.2.x and earlier have this capability. The NCO 1.2.x branch will be maintained with bugfixes only (no new features) until HDF begins to fully support the netCDF3 interface (which is employed by NCO 2.x). If, at compilation time, NETCDF2_ONLY is defined, then NCO version 1.2.x will not use any netCDF3 calls and, if linked properly, the resulting NCO operators will work with HDF4 files. The Makefile supplied with NCO 1.2.x is written to simplify building in this HDF capability. When NCO is built with make HDF4=Y, the Makefile sets all required preprocessor flags and library links to build with the HDF4 libraries (which are assumed to reside under /usr/local/hdf4, edit the Makefile to suit your installation).

HDF version 5 became available in 1999, but did not support netCDF (or, for that matter, Fortran) as of December 1999. By early 2001, HDF5 did support Fortran90. However, support for netCDF4 in HDF5 is incomplete. Much of the HDF5-netCDF interface is complete, however, and it may be separately downloaded from the netCDF4 website. We are eager for HDF5 to complete netCDF support. This is scheduled to occur sometime in 2007, with the releases of HDF version 1.8 and netCDF version 4, which are collaborations between Unidata and NCSA. NCO version 3.0.3 added support for reading/writing netCDF4-formatted HDF5 files in October, 2005. See Selecting Output File Format for more details.

NCO version 3.9.0 added full support for all netCDF4 atomic data types in May, 2007. Support for netCDF4 features will be incremental, i.e., we will add one netCDF4 feature at a time. You must build NCO with netCDF4 to obtain this support.

The main netCDF4 features that NCO currently supports are the new atomic data types (NC_UBYTE, NC_SHORT, NC_UINT, NC_INT64, and NC_UINT64). Eight-byte integer support is especially useful improvement from netCDF3. All NCO operators support these types, e.g., ncks copies and prints them, ncra averages them, and ncap2 processes algebraic scripts with them.

netCDF4-enabled NCO handles netCDF3 files without change. In addition, it automagically handles netCDF4 (HDF5) files: If you feed NCO netCDF3 files, it produces netCDF3 output. If you feed NCO netCDF4 files, it produces netCDF4 output. Use the handy-dandy `-4' switch to request netCDF4 output from netCDF3 input, i.e., to convert netCDF3 to netCDF4. See Selecting Output File Format for more details.

Use appropriate caution while netCDF4 is beta software. Problems with netCDF4 and HDF libraries are still being fixed. NCO support for netCDF4 atomic types is relatively untested. Binary NCO distributions (RPMs and debs) still use netCDF3.

For now you must build NCO from source to get netCDF4 support. Typically, you will need to specify the root of the netCDF4-beta1 installation directory. Do this with the NETCDF4_ROOT variable. Then use your preferred NCO build mechanism, e.g.,

     export NETCDF4_ROOT=/usr/local/netcdf4 # Set netCDF4 location
     cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
     cd ~/nco/bld;./make NETCDF4=Y allinone # Old Makefile mechanism

Our short term goal is to track the netCDF4-beta releases, keep the new netCDF4 atomic type support working, and iron out any problems. Our long term goal is to utilize more of the extensive new netCDF4 feature set. The next major netCDF4 feature we are likely to utilize is parallel I/O. We will enable this in the MPI netCDF operators.


Footnotes

[1] The Hierarchical Data Format, or HDF, is another self-describing data format similar to, but more elaborate than, netCDF.

[2] One must link the NCO code to the HDF4 MFHDF library instead of the usual netCDF library. Does `MF' stands for Mike Folk? Perhaps. In any case, the MFHDF library only supports netCDF2 calls. Thus I will try to keep this capability in NCO as long as it is not too much trouble.