Sustainability of Digital Formats: Planning for Library of Congress Collections

Introduction | Sustainability Factors | Content Categories | Format Descriptions | Contact
Format Description Categories >> Browse Alphabetical List

NetCDF-4 (Network Common Data Form, Version 4), Classic Model

>> Back
Table of Contents
Format Description Properties Explanation of format description terms

Identification and description Explanation of format description terms

Full name NetCDF-4C (Network Common Data Form, version 4), Classic Model
Description

NetCDF is a set of software libraries and self-describing, machine-independent data formats for array-oriented scientific data. The first version of the format was developed in the late 1980s at the Unidata Program Center, with the objective of building a file format that would permit sharing of data among atmospheric scientists. It has found use in other scientific communities, with different communities developing discipline-specific conventions. The format was and is designed to be portable, platform-independent, scalable, and appendable. See Notes below for more detail on design objectives.

As of January 2012, there are four variants of the format. The first two, known as Classic and 64-bit Offset, are nearly identical and together are often referred to as netCDF-3. NetCDF-4, which is based on HDF5 (versions 1.8 and later) was introduced in 2008. NetCDF-4 Classic, also introduced in 2008 combines the simpler data model of netCDF-3 with the HDF5-based storage capabilities of netCDF-4. This format description is for netCDF-4 Classic.

The relationship of netCDF-4C to netCDF-4 and netCDF-3 as far as data model is concerned, is illustrated in a picture of the enhanced netCDF-4 data model, with the extensions to the Classic model used by netCDF-3 highlighted in red. NetCDF-4C excludes use of the grouping structure and additional data types introduced for netCDF-4. NetCDF-4C can also only include one dimension that is unlimited and appendable, whereas netCDF-4 can have many unlimited dimensions. However, netCDF-4 Classic files can take advantage of features resulting from the use of HDF5 as the storage layer. These include per-variable compression and data "chunking" (similar to "tiling" for raster images). A useful comparison between netCDF-3 and netCDF-4 and a discussion of interoperability considerations in relation to HDF5 are included in a November 2011 recommendation from a NASA working group (ESDS-RFC-022) for endorsement of netCDF-4 for NASA use.

Production phase Generally used for middle- and final-state archiving.
Relationship to other formats
    Subtype of HDF5, Hierarchical Data Format, version 5
    Subtype of NetCDF-4, Network Common Data Form, Version 4. Constrained to the "Classic" data model used by netCDF-3.
    Affinity to NetCDF-3, Network Common Data Form, Version 3. NetCDF-4C uses the same underlying "classic" data model as netCDF-3.

Local use Explanation of format description terms

LC experience or existing holdings None
LC preference None

Sustainability factors Explanation of format description terms

Disclosure

Fully and openly documented. NetCDF was developed by and is maintained and documented by the Unidata Program Center, a consortial program within UCAR (University Corporation for Atmospheric Research).

    Documentation

Software can be downloaded from http://www.unidata.ucar.edu/downloads/netcdf/index.jsp. Documentation is at http://www.unidata.ucar.edu/software/netcdf/docs/.

Adoption

NetCDF-4C is being adopted increasingly in atmospheric and earth sciences when the Classic data model is appropriate, but issues of scale and/or performance can be usefully addressed through per-variable compression and controlled data chunking. A common sequence of phases for upgrade for those already using netCDF-3 to store and manage data is: (1) relink applications with netCDF-4 library; (2) continue use of netCDF-3 APIs but with netCDF-4 Classic model format to get performance benefits; (3) adopt features of enhanced netCDF-4 model as needed/supported. Using the simpler classic data model and the netCDF-4C format realizes performance benefits and phased adoption is easier and less risky than a single giant leap from (classic model, classic format) to (enhanced model, netCDF-4 format).

As of early 2012, the NetCDF-4 classic model is supported in many analysis and visualization applications for reading, including Ferret, IDL, MATLAB, GrADS (Grid Analysis and Display System), ncl (NCAR command language), and Panoply. NetCDF-4C is supported in language APIs for C, C++ , Fortran, Java, and Python. The open source format conversion toolkit GDAL supports reading and writing of netCDF-4 in enhanced and classic variants. As of early 2012, OPeNDAP (Open-source project for a Network Data Access Protocol) supports netCDF-4C and some additional features of netCDF-4's enhanced model.

Unidata maintains a list of software supporting netCDF but as of early 2012 this does not distinguish support for netCDF-3 from that for netCDF-4 enhanced or classic formats.

Data providers within NASA and NOAA have begun using netCDF-4C to implement compression and chunking for performance reasons. One example is the 20th Century Reanalysis Project.

See netCDF-4 for information of adoption of netCDF-4 in general.

    Licensing and patents

No concerns.

Transparency

See netCDF-4.

Self-documentation

See netCDF-3.

External dependencies

See netCDF-4.

Technical protection considerations None.

Quality and functionality factors Explanation of format description terms

Dataset
Normal functionality

See netCDF-3 for data types supported in netCDF-4C.

Support for software interfaces (APIs, etc.)

An integral component of netCDF is a software library that provides an API (in Fortran, C, C++, Java, and other languages) to read and write files in the netCDF-4C format.

Data documentation (quality, provenance, etc.)

NetCDF-4C offers the capability to apply attributes to a file as a whole or any individual variable. There is no explicit support for embedding structured metadata using a particular schema or syntax. However, particular communities use conventions for naming variables and using attributes.

Beyond normal functionality

Multidimensional arrays can have one unlimited (appendable) dimension.

GIS images and datasets
Normal functionality

NetCDF-4C is not a geospatial format per se. However, it is widely used for geospatial data. In order to serve as a format for geospatial data that can be shared and used in different contexts, the description of the coordinate reference systems and projections employed must be recorded in a recognizable and unambiguous way. For this purpose, the CF (Climate and Forecast) Conventions are recommended.

Support for GIS metadata

There is no single or recommended way to embed metadata in a specific serialization or schema in netCDF-4C files. Since XML consists of strings, XML can be embedded in netCDF files by means of string variables or attributes; however, there is no officially recommended approach. Unidata makes available a service (ncISO) as part of its THREDDS Data Server that outputs metadata from a netCDF file in a form compliant with ISO 19115 (Geographic Information -- Metadata).

Support for grids The combination of the netCDF classic data model and the application of the CF conventions can provide explicit and flexible support for grid-based analysis. The conventions make recommendations for grid definition and mappings that allow for grids that are not based simply on latitude and longitude.
Beyond normal functionality See Dataset Quality and Functionality factors above.

File type signifiers and format identifiers Explanation of format description terms

Tag Value Note
Filename extension nc
The recommended file extension for netCDF-4C, and the default supplied by the Unidata software library is .nc, the same as for netCDF-3 and netCDF-4. However, since the files are valid HDF5 files, the .h5 extension may be used in some contexts. See Why aren't different extensions used for the different [netCDF] formats, for example, .nc3 and .nc4?
Magic numbers See related format.  See HDF5. The netCDF variants can be identified using the ncdump utility, a tool provided by Unidata.

Notes Explanation of format description terms

General

The stated objectives for the netCDF format are that it be:

  • Self-Describing. A netCDF file includes information about the data it contains.
  • Portable. A netCDF file can be accessed by computers with different ways of storing integers, characters, and floating-point numbers.
  • Scalable. A small subset of a large dataset may be accessed efficiently.
  • Appendable. Data may be appended to a properly structured netCDF file without copying the dataset or redefining its structure.
  • Sharable. One writer and multiple readers may simultaneously access the same netCDF file.
  • Archivable. Access to all earlier forms of netCDF data will be supported by current and future versions of the software.
History

As of January 2012, there are four variants of the NetCDF binary data format.

  • the classic format, used since 1989
  • the 64-bit offset format, introduced in 2004 to support larger variables
  • the netCDF-4 format, introduced in 2008 to support more powerful forms of data representation, based on HDF5
  • the netCDF-4 classic model format, also introduced in 2008, based on HDF5, but without the data modeling extensions

NetCDF-3 Classic was the only format for netCDF data created between 1989 and 2004 by the reference software from Unidata. It is still the default format for new netCDF data files, and the form in which most netCDF data is stored. The intent is to maintain support for netCDF-3 indefinitely.


Format specifications Explanation of format description terms


Useful references

URLs


Last Updated: Monday, 27-Feb-2017 09:55:31 EST