Welcome to the netCDF Operator (NCO) Homepage

Current NCO version is 3.9.8 last updated on

News & Milestones What is NCO? How to Contribute NCO→SDO Project Publications Version Summary Download NCO Source and Binaries Documentation and User's Guide README/ChangeLog/TODO FAQ Availability at UCI and NCAR Benchmarks Building from Source Known Problems People Contacting the Project

The original NCO logo…and a new contender from Tony Freeman. Which do you prefer? The Mandelblob or the patch? Have a better one? Send it in!

NCO Milestones

• 2009 Summer: NCO 4.0.0 (GSL interpolation) (TBD)
• 2009 Late Spring: NCO 3.9.9 (netCDF4 chunking, rebasing) (TBD)
• 2009 Apr 29: NCO 3.9.8 (GSL RNG/PDF functions)
• 2009 Mar 18: NCO 3.9.7
• 2009 Jan 22: NCO 3.9.6 (GSL special functions)
• 2008 Sep 24: ncap2 documentation
• 2008 Aug 31: NSF funding ends
• 2008 Aug 08: NCO Wiki!
• 2008 May 22: SWAMP Talk at CCGRID08 in Lyon, France
• 2008 May 11: NCO 3.9.5
• 2008 Apr 28: NCO overview paper in EMS
• 2008 Mar 04: NCO 3.9.4
• 2007 Dec 10: SWAMP Poster at Fall AGU, San Francisco, California
• 2007 Dec 07: NCO 3.9.3 (SWAMP!)
• 2007 Dec 04: NCO scaling paper in IJHPCA
• 2007 Aug 29: NCO 3.9.2 (_FillValues)
• 2007 Aug 27: SWAMP site spins-off from NCO
• 2007 Jun 29: NCO 3.9.1 (netCDF4 deflation)
• 2007 May 26: NCO 3.9.0 (netCDF4 atomic types)
• 2007 May 02: SWAMP Talk at GPC, Paris, France
• 2007 Apr 20: NCO 3.2.0
• 2007 Mar 29: NCO 3.1.9 x86_64 RPMs
• 2007 Mar 10: NCO 3.1.9 (i386/x86_64 debs and i386 RPMs)
• 2007 Feb 22: SWAMP Talk at OPeNDAP Developer's Workshop, Boulder, Colorado
• 2007 Jan 15: Talk and Extended Abstract on NCO Scaling at AMS IIPS, San Antonio, Texas
• 2007 Jan 09: Poster on NCO Scaling at Fall AGU, San Francisco, California
• 2006 Dec 09: Poster on SWAMP at Fall AGU, San Francisco, California
• 2006 Dec 06: NCO 3.1.8
• 2006 Nov 11: NCO 3.1.7
• 2006 Sep 18: NCO 3.1.6
• 2006 Aug 29: NCO 3.1.5
• 2006 Jun 07: NCO 3.1.4
• 2006 May 30: NCO 3.1.3
• 2006 Apr 21: NCO 3.1.2
• 2006 Jan 30: NCO 3.1.1
• 2005 Dec 02: NCO 3.1.0 (MPI!)
• 2005 Oct 28: NCO 3.0.3
• 2005 Oct 17: NCO parallel benchmarks
• 2005 Aug 22: NCO 3.0.2
• 2005 Jun 10: NCO 3.0.1
• 2005 Mar 24: NCO 3.0.0
• 2004 Sep 07: NCO 2.9.9 (ncpdq packs data quietly)
• 2004 Sep 01: NSF funding for NCO→SDO Project begins
• 2004 Aug 07: NCO 2.9.8 (ncpdq re-orders dimensions)
• 2004 Mar 04: Submit NSF proposal
• Ancient NCO News & Announcements

What is NCO?

The netCDF Operators, or NCO, are a suite of programs known as operators. They are standalone, command-line programs like, e.g., ls or mkdir. Each operator takes netCDF files as input, then operates (e.g., derives new data, averages, hyperslabs, manipulates metadata) and produces a netCDF output file. NCO primarily aids manipulation and analysis of gridded scientific data. The single-command style of NCO allows users to manipulate and analyze files interactively, with simple scripts that avoid some overhead (and power) of higher level programming environments. The NCO User's Guide illustrates their use with examples from the field of climate modeling and analysis. Note that the “averagers” are misnamed because they perform many non-linear operations as well, e.g., total, minimum, maximum, RMS:

• ncap2 netCDF Arithmetic Processor (examples)
• ncatted netCDF Attribute Editor (examples)
• ncbo netCDF Binary Operator (includes ncadd, ncsubtract, ncmultiply, ncdivide) (examples)
• ncea netCDF Ensemble Averager (examples)
• ncecat netCDF Ensemble Concatenator (examples)
• ncflint netCDF File Interpolator (examples)
• ncks netCDF Kitchen Sink (examples)
• ncpdq netCDF Permute Dimensions Quickly, Pack Data Quietly (examples)
• ncra netCDF Record Averager (examples)
• ncrcat netCDF Record Concatenator (examples)
• ncrename netCDF Renamer (examples)
• ncwa netCDF Weighted Averager (examples)

The operators are as general as netCDF itself: there are no restrictions on the contents of the netCDF file(s) used as input. NCO's internal routines are completely dynamic and impose no limit on the number or sizes of dimensions, variables, and files. NCO is designed to be used both interactively and with large batch jobs. The default operator behavior is often sufficient for everyday needs, and there are numerous command line (i.e., run-time) options, for special cases. NCO works well on most modern operating systems, including: Apple OS X, *BSD, Cray UNICOS, DEC Tru64, IBM AIX, HPUX, Linux, Microsoft Windows, NEC Super UX, SGI IRIX, and Sun Solaris.

How to Contribute

In March, 2000, NCO became an Open Source project hosted by SourceForge.net. This facilitates collaboration, code contributions, and support. There is a TODO list about 200 items long! No matter what your programming background there is a task you can help with. From re-organizing the TODO list itself, to improving this cheesy webpage, to documentation, to designing and implementing new features and interfaces, we need your help! The NCO project homepage contains mailing lists, discussion forums, and instructions to make contributing to NCO easy.

As of June, 2003, you may easily donate your hard-earned money to NCO. Send your donation to NCO developers as a reward for good behavior, or as an incentive to tackle more TODO items.

NCO→SDO Project

The National Science Foundation Grant NSF IIS-0431203 funded our NCO→SDO Project, “SEI(GEO): Scientific Data Operators Optimized for Efficient Distributed Interactive and Batch Analysis of Tera-Scale Geophysical Data” from 20040901–20080831 as part of the Science and Engineering Informatics (SEI) program. We appreciate the proposal reviewers for and staff of the Divisions of Information and Intelligent Systems (IIS) and Shared Cyberinfrastructure (SC) in the Directorate for Computer and Information Science and Engineering (CISE). Until September 2004, NCO development was completely voluntary and without institutional support. This NSF support dramatically changed the scale and pace of NCO/SDO development. This URL, http://nco.sf.net#prp_sei, points to the most up-to-date information on the NCO/SDO proposal.

The NSF project provides the human and technical resources to analyze, design, and implement novel advanced methods for distributed data analysis and reduction (DDRA) of geophysical data. These powerful methods will be “grid-enabled”, and we will demonstrate their efficacy with analyses of distributed climate prediction and observation datasets. On the human side, recruited full-time help for specific areas of SDO, alongside our current and future volunteers. We have a full-time programmer to help design, implement, and release major code changes. We have a graduate student researcher pursuing innovative Ph.D. research in using “grid-aware” distributed memory, shared memory, and client-server software technologies to improve DDRA of climate datasets. Another graduate student helps us design, implement, and release major code changes. Contact us if you are seriously interested in tackling these problems on a full-time, part-time, or pay-per-feature basis.

On the institutional and hardware side, this project connected the Earth System Modeling Facility (ESMF) to the California Institute for Telecommunications and Information Technology (Cal-IT2) OptIPuter at the San Diego Supercomputer Center. These supercomputers dedicated a TB of storage each to OPeNDAP-served climate simulation datasets for DDRA. After the proof-of-concept SDO DDRA was complete, we attempted a DDRA intercomparison of the internationally distributed global climate change simulations that reside on the Earth System Grid. Our techniques tremendously accelerated the analysis of different climate prediction scenarios for the same model, and among different climate models.

We encourage you to e-mail NCO/SDO endorsements to Charlie “my surname is zender” Zender with Subject: “NCO/SDO Proposal Endorsement”. An endorsement can be a sentence or a page, describing how NCO/SDO benefits your work or research. The information in your endorsement will be used to create supplementary documents to NSF and NASA for future NCO/SDO proposal advocacy. “What future proposals?” you ask, “Didn't you just get funded?” Yes, we are funded to optimize and “grid-enable” NCO. There are still many desirable and unexplored methods for distributed data analysis and reduction (DDRA) that would benefit geosciences. For example, have you tried analyzing multiple NASA satellite datasets in HDF5 format? Need I say more? If you send an endorsement, remember to include (at least) your Name, Title, and Institutional affiliation in the body.

Publications and Presentations

• Wang, D. L., C. S. Zender, and S. F. Jenks (2008), Compiling the uncompilable: A case for shell script compilation, Submitted to ACM Trans. Softw. Engin. Method.. BibTeX PDF (© 2008 by Springer-Verlag)
• Wang, D. L., C. S. Zender, and S. F. Jenks (2008), Efficient Clustered Remote Data Analysis Workflows using SWAMP, In Press in Earth Sci. Inform.. BibTeX PDF (© 2009 by ACM)
• Wang, D. L., C. S. Zender, and S. F. Jenks: Cluster Workflow Execution of Retargeted Data Analysis Scripts, in Cluster Computing and the Grid, 2008. CCGRID '08. 8th IEEE International Symposium on. Lyons, France, 19 22 May, 2008. IEEE Computer Society, 449–458, doi:10.1109/CCGRID.2008.69. BibTeX PDF (© 2008 by IEEE)
• Zender, C. S. (2008), Analysis of Self-describing Gridded Geoscience Data with netCDF Operators (NCO), Environ. Modell. Softw., 23(10), 1338–1342, doi:10.1016/j.envsoft.2008.03.004. BibTeX PDF (CSZ) PDF (EMS) (© 2008 by Elsevier Ltd.)
• Wang, D. L., C. S. Zender and S. F. Jenks (2007): A System for Scripted Data Analysis at Remote Data Centers. Presented at the Fall Meeting of the American Geophysical Union, San Francisco, CA, December 10–14, 2007. Eos Trans. AGU, 88(52), Fall Meet. Suppl., Abstract IN11B-0469. BibTeX PDF (DLW)
• Wang, D. L., C. S. Zender, and S. F. Jenks (2007), Server-side parallel data reduction and analysis, in Advances in Grid and Pervasive Computing, Second International Conference, GPC 2007, Paris, France, May 2–4, 2007, Proceedings. IEEE Lecture Notes in Computer Science, vol. 4459, edited by C. Cérin and K.-C. Li, pp. 744–750, Springer-Verlag, Berlin/Heidelberg. BibTeX PDF (DLW)
• Wang, D. L., C. S. Zender, and S. F. Jenks (2007), Server-side Parallel Data Reduction and Analysis. Presented by D. Wang to the International Conference on Grid and Pervasive Computing (GPC), Paris, France, May 2–4, 2007. PDF
• Zender, C. S., and H. J. Mangalam (2007), Scaling Properties of Common Statistical Operators for Gridded Datasets, Int. J. High Perform. Comput. Appl, 21(4), 485–498, doi:10.1177/1094342007083802. BibTeX PDF (CSZ) PDF (IJHPCA) (© 2007 by SAGE Publications)
• Wang, D. L., C. S. Zender, and S. F. Jenks (2007), Server-side Data Reduction and Analysis with Script Workflow Analysis for Multi-Processing. Presented by D. Wang to the OPeNDAP Developer's Workshop, Boulder, CO, February 21–23, 2007. PDF
• Wang, D. L., C. S. Zender, and S. F. Jenks (2007), Server-side Data Reduction and Analysis with Script Workflow Analysis for Multi-Processing. Presented by D. Wang to the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, San Antonio, TX, January 14–18, 2007. PDF
• Wang, D. L., C. S. Zender, and S. F. Jenks (2007), DAP-enabled Server-side Data Reduction and Analysis, Proceedings of the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, Paper 3B.2., January 14–18, San Antonio, TX. American Meteorological Society, AMS Press, Boston, MA. BibTeX PDF
• Zender, C. S., and D. L. Wang (2007), High performance distributed data reduction and analysis with the netCDF Operators (NCO). Presented to the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, January 14–18, San Antonio, TX, January 14–18, 2007. PDF
• Zender, C. S., and D. L. Wang (2007), High performance distributed data reduction and analysis with the netCDF Operators (NCO), Proceedings of the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, Paper 3B.4., January 14–18, San Antonio, TX. American Meteorological Society, AMS Press, Boston, MA. BibTeX PDF
• Wang, D. , Zender, C. S., and S. F. Jenks (2006): Server-side netCDF Data Reduction and Analysis. Presented by D. Wang to the Fall Meeting of the American Geophysical Union, San Francisco, CA, December 5–9, 2006. Eos Trans. AGU, 87(52), Fall Meet. Suppl., Abstract IN53B-0826. PDF
• Zender, C. S., H. J. Mangalam, and D. L. Wang (2006): Improving Scaling Properties of Common Statistical Operators for Gridded Geoscience Datasets. Presented to the Fall Meeting of the American Geophysical Union, San Francisco, CA, December 5–9, 2006. Eos Trans. AGU, 87(52), Fall Meet. Suppl., Abstract IN53B-0827. PDF

Version Summary

• NCO/SDO 4.0.0: (Future) GSL interpolation and fitting;
• NCO/SDO 3.9.9: (In progress) netCDF4 chunking and NC_STRING support; GSL statistics functions; UDUnits2 time-since support; ncrcat re-basing;
• NCO/SDO 3.9.8: ncap2 supports sort(), array(); better DAP/libnetcdf and UDUnits2 builds; GSL distribution/RNG functions;
• NCO/SDO 3.9.7: Improve GSL-build support; ncap2 threading improvements; ncks --no_dmn_var_nm printing;
• NCO/SDO 3.9.6: Fix OpenMP MSA bug (upgrade highly recommended); Memory management to improve threading; ncatted workaround for netCDF4 _FillValue; disable OpenMP with netCDF4/HDF; no math on type NC_CHAR; ncap2 documentation, better missing value propagation, #include files, GSL math functions;
• NCO/SDO 3.9.5: Multi-slabbing (MSA) and -X everywhere; ncecat -M; netCDF4-enabled builds on Cygwin; ncap2 builds on Solaris;
• NCO/SDO 3.9.4: File consanguinity; ncks: CF auxiliary coords, LZ stats; ncap2: masks, index arrays, print attributes, OpenMP, atan2(), pow();
• NCO/SDO 3.9.3: SWAMP release; ncecat -u; wget URLs; GCC 4.2; ncap2 OpenMP;
• NCO/SDO 3.9.2: _FillValue supplants missing_value; UDUnits2 support; ncap2 type-conversion support and printing; netCDF4 configure fixes; Change licenses to GPL3 and FDL1.2;
• NCO/SDO 3.9.1: Support lossless netCDF4/HDF deflation; Improve Intel compiler support; ncap2 for/while loops; various and sundry build fixes;
• NCO/SDO 3.9.0: Support netCDF4 types (ubyte, ushort, uint, int64, and uint64); Control ncks metadata copying; Fix ncbo packed valued bug (upgrade recommended); Improve IRIX 6.5 support;
• NCO/SDO 3.2.0: More forgiving exclusion list (-x -v var_nm) handling; Fix rmssdn normalization; Support Mac OS X on Intel; Documentation bugfixes;
• NCO/SDO 3.1.9: ncap2 in RPM packages; Quieter output; AIX configure supports ncap2; Update to DAP for netCDF 3.6.2; Fix nc[erw]a for coordinate min/max/ttl; ncecat allows files to differ in record dimension size;
• NCO/SDO 3.1.8: Support _FillValue with compile-time switch; Debian package synchronized, includes ncap2
• NCO/SDO 3.1.7: ncap2 “double-parsing”, array initialization, supercedes ncap
• NCO/SDO 3.1.6: Support stride in all hyperslabbing operators; change more WARNINGs to INFOs
• NCO/SDO 3.1.5: New ncap2 array and hyperslab features; change some WARNINGs to INFOs, add Pathscale and update PGI and Intel compiler support
• NCO/SDO 3.1.4: Fix ncbo memory problem; report timer results
• NCO/SDO 3.1.3: ncap2 automatically propagates attributes to new/derived variables; NCO handles record variables with zero records; support CF coordinates convention; ncwa -b retains degenerate dimensions; build fixes for Cray T3E, Solaris, Cygwin.
• NCO/SDO 3.1.2: ncap2 control structures (conditionals, loops, etc.); keep non-processed data packed in output file; binary operators ncbo and ncflint work when only one variable has missing_value attribute; change ncra and ncea to always average coordinate variables
• NCO/SDO 3.1.1: NCO builds (again) with C++ compilers; ncap2 development code; server-side extensions
• NCO/SDO 3.1.0: MPI operators are bug-free; fix ./configure for MPI and netCDF4; MRV algorithm speeds up ncwa 5—10 times; ncap random number generator rnd_nbr()
• NCO 3.0.3: Eliminate all critical read regions (accelerates all SMP operators); builds with netCDF4 alpha; read NETCDF4_CLASSIC (i.e., HDF) files automatically; write NETCDF4 files on request; benchmark results quantify parallelism.
• NCO 3.0.2: Fewer critical read regions; all known memory leaks plugged; better Intel compiler support; SFTP protocol support; FTP protocol supports .netrc-based passwords; better model grid-property-awareness (e.g., lat_bnds); ncatted ncks, ncrename support metadata header padding with --hdr; ncbo supports heterogeneous variable ordering; ncatted bugfix
• NCO 3.0.1: benchmarks; ncks printing tweaks; memory cleanup; ncap double-parse speedup and exponentiation bugfix; thread operators by default
• NCO 3.0.0: New ncks -P print switch; ncap optimizations; -Z option supports 64-bit offsets (requires netCDF 3.6); NEC SX build improvements; support GCC 3.4.X; nco_put_vara() overloads in libnco_c++
• NCO 2.9.9: ncpdq packs/unpacks entire files; ncbo is threaded
• NCO 2.9.8: ncpdq released; ncwa --msk_sng consolidated mask switch; C99 compiler required
• NCO 2.9.7: Read input file lists from stdin; write nco_input_file_list attributes; add nco_put_att<int,short,long> functions to libnco_c++; large file support (LFS); write nco_openmp_thread_number attribute
• NCO 2.9.6: Support -o fl_out syntax; 64-bit ABI for x86_64; Debian builds libnco_c++
• NCO 2.9.5: ncflint handles missing_value data better, is commutative when weights are equal
• NCO 2.9.4: ncecat works on pure scalar files; change to cvs.sf.net; AIX GCC builds
• NCO 2.9.3: ncra and ncea now work correctly when missing_value equals zero or any intermediate sums
• NCO 2.9.2: ncap S/V, S%V, and S^V operations fixed (S = scalar, V = variable)
• NCO 2.9.1: ncwa works again on packed NC_FLOAT data (broke when?)
• NCO 2.9.0: Complete ncap modulo operator (V%S) and generic exponentiation (V^S), scalar divided by variable (S/V); workaround stack memory problem with GCC extensions; fix make check
• NCO 2.8.8: --enable-[optimize/debug]-custom options for GCC work again (broke in 2.8.6)
• NCO 2.8.7: ncap packing works again (broke in 2.8.4); Cygwin builds work again
• NCO 2.8.6: AIX and IRIX builds work again; custom GCC builds with -Werror
• NCO 2.8.5: Solaris make install works; ncap changes attribute values correctly; Support DODS 3.4+, deprecate 3.3-; add --enable-[optimize/debug]-custom options to ./configure
• NCO 2.8.4: Index-based hyperslabs work again (broke in 2.7.3, upgrade highly recommended); ncap peak memory usage reduced still more; support for AMD Opteron x86_64 architecture
• NCO 2.8.3: Fix big ncap memory leak; fix regressions; remove scary packing warnings
• NCO 2.8.2: ncbo handles distinct missing_values correctly; improve error diagnostics
• NCO 2.8.1: “Variable wildcarding”—extended regular expressions in -v var,... lists—support in all operators
• NCO 2.8.0: New “binary” operator ncbo, obsoletes ncdiff; Debian integration
• NCO 2.7.4: Autotools build support for AIX; fix UDUnits confusion with floating point hyperslab specifications
• NCO 2.7.3: Support UDUnits (and some CF) conventions, e.g., ncks -d wvl "1 picometer","1 furlong" -d time,"2001-03-19 06:00:0.0","2001-03-19 18:00:0.0"
• NCO 2.7.2: Printing (ncks -H) plays well with multi-slabbing
• NCO 2.7.1: (Packaged) Fallback to B. Sittler's getopt() for long option support; store fixed, packed variables correctly with ncap.
• NCO 2.7.0: Support for GNU/POSIX long options; ncks introduces multi-slabbing (= multiple hyperslabs) capability
• NCO 2.6.6: Fix and document ncrename attribute renaming; add type conversion functions to ncap
• NCO 2.6.5: Cray, Sun build improvements
• NCO 2.6.4: Mac OS X, NEC, SGI build improvements
• NCO 2.6.3: Mac OS X builds work seamlessly; documentation fixed
• NCO 2.6.2: Major improvements in autotools support; ncap supports user-defined dimensions
• NCO 2.6.1: Minor fix to unpacking support
• NCO 2.6.0: All arithmetic operators (ncap, ncdiff, ncea, ncflint, ncra) support unpacking
• NCO 2.5.6: Handle missing_value correctly for integer variables in ncra, merge autoconf build
• NCO 2.5.5: Handle files without existing global “History” attribute correctly
• NCO 2.5.4: Add support NEC-SX support; update Cray support; small build changes
• NCO 2.5.3: Fix native builds under MS Windows with Cygwin ncap; hint to ncwa
• NCO 2.5.2: Add math functions to ncap; hint to ncwa
• NCO 2.5.1: ncatted deletes all attributes of att_var when att_nm not specified
• NCO 2.5.0: man(1) pages from Debian distribution
• NCO 2.4.6: Free (speech) sort routines
• NCO 2.4.5: Arithmetic speedups
• NCO 2.4.4: 64-bit AIX builds
• NCO 2.4.3: Handle non-standard “History” attributes
• NCO 2.4.2: ncap supports nesting files with #include
• NCO 2.4.0: ncks supports writing native binary output with -b switch
• NCO 2.3.1: ncap builds on all Unices
• NCO 2.3.0: Library redesign
• NCO 2.2.4: ncap fully supports packing and unpacking variables
• NCO 2.2.3: ncap supports manually unpacking variables stored with scale_factor and add_offset
• NCO 2.2.2: fix rare ncwa crash with weights
• NCO 2.2.1: fix ncap divide and fmodf bugs
• NCO 2.2.0: (RPM available) 20020203: ncap public debut (has division error, please upgrade)
• NCO 2.1.3: ncap implements LHS casting
• NCO 2.1.2: C++ API to netcdf; libnco_c++ open for business
• NCO 2.1.0: ncap full lexer/parser operational
• NCO 2.0.3: small fixes to 2.0.0 (e.g., ncks printing)
• NCO 2.0.2: Add packing/unpacking capabilities based on scale_factor and add_offset convention
• NCO 2.0.1: Add user-configurable multi-threaded capability to arithmetic-intensive operators ncwa, ncea, and ncra
• NCO 2.0.0: (RPM available) 20010507: Switched all internal calls to netCDF 3.x API
• NCO 1.2.2: 20010211: Allow weights with ncwa in min/max/ttl operations; easier building with HDF4. ncrename can now work on specific variables rather than whole file. ncks has -q option for cleaner printing
• NCO 1.2.1: 20000828: Added type-conversion of integers to floating point before arithmetic; fixed broken -v option
• NCO 1.2: 20000730: Added non-linear operations to ncra/ncea and ncwa: min,max,total,rms,rmssdn,sqrt. Added DODS compliance.
• NCO 1.1.48: 20000515: Potentially serious bugfix to ncdiff, recommend upgrade
• NCO 1.1.47: 20000510: Improved ncra and ncrcat to allow lazier specifications of input files

Download NCO Source and Binaries

NCO Source Code

• nco-3.9.8.tar.gz ( compressed tar-file)
  
  

NCO's browsable CVS Repository contains up-to-the-minute sources and is the easiest way to stay synchronized with NCO features. Retrieving NCO requires some familiarity with GNU development tools, especially CVS and Make. Sourceforge provides generic instructions for accessing their CVS servers. First log into the NCO CVS server:

cvs -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco login

cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -r nco-3_9_8 -d nco-3.9.8 nco

cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -kk nco

cd nco;cvs update -kk

Developer NCO Source Documentation

Automated source documentation, created by the Doxygen tool is available. Some developers find this documentation helpful, as it can clarify code and data relationships in the code.

• Source documentation for NCO and NetCDF4(alpha13)

Please be aware that the documentation may be slightly inaccurate and infrequently updated. Comments and suggestions are certainly welcomed on the forums.

NCO Binaries and OS-specific Packages

NCO developers are too short-handed to provide pre-built binary executables for all platforms. Our source tarballs are always up-to-date, and work on our development systems (Ubuntu-flavored Debian GNU/Linux for x86 and x86_64) and in our high-end scientific computing environments (AIX 5 for Power4/5/6). We also attempt to provide (theoretically) platform-independent sources in the most common UNIX packaging formats (Debian and RPM). Below are links to these and to packages created by volunteers who port NCO to other platforms. Volunteers willing to perform regular regression testing and porting of NCO to other platforms are welcome. Previous versions of these binaries are still available by searching the directory index here.

Debian and Ubuntu GNU/Linux

Debian NCO and Ubuntu NCO homepages. ‘aptitude install nco’ installs the standard NCO for your Debian-compatible OS. This standard package is, unfortunately, old (version 3.9.0, from about 2005).
Debian package for most recent NCO release (install with, e.g., ‘dpkg --install nco_3.9.8-1_i386.deb’):
• nco_3.9.8-1_amd64.deb (): Executables AMD64-compatible (last updated )
• nco_3.9.8-1_i386.deb (): Executables i386-compatible (last updated )
• nco_3.9.8-1.dsc (): Description (last updated )
• nco_3.9.8-1_i386.changes (): Changes since last deb package (last updated )
• nco_3.9.8.orig.tar.gz (): Upstream Source (last updated )
• nco_3.9.8-1.diff.gz (): Debian patch to upstream source (last updated )
Thanks to Daniel Baumann, Francesco Lovergine, Brian Mays, Rorik Peterson, and Matej Vela for packaging NCO for Debian over the years.

Fedora and CentOS GNU/Linux

The Fedora NCO RPMs are usually quite up-to-date so that ‘yum install nco’ might install, e.g.,
nco-3.9.8-1.fc11.x86_64.rpm: Executables for x86_64/Fedora Core 11-compatible environments
If not, try our own most recent self-built NCO RPMs (install with, e.g., ‘yum install nco-3.9.5-1.fc7.i386.rpm’):
nco-3.9.5-1.fc7.i386.rpm (): Executables for i386/Fedora Core 7-compatible environments (last updated )
nco-3.9.5-1.fc7.src.rpm (): Source (last updated )
nco-3.9.5-1.x86_64.rpm (): Executables for x86_64/CentOS 5-compatible environments (last updated )
More information on x86_64 RPMs available here
Volunteers have updated and maintained fairly up-to-date NCO packages in Fedora since it was added by Ed Hill in about 2004. Thanks to Patrice Dumas, Ed Hill, and Orion Poplawski for packaging NCO RPMs over the years.

Gentoo GNU/Linux

Gentoo GNU/Linux homepage for NCO. Portage packages by George Shapavalov and Patrick Kursawe:
• nco-3.2.0: Latest(?) Gentoo package

Mac OS X/Darwin

• Fink packages for NCO: Currently NCO 3.9.2. Maintained by Alexander Hansen.
• MacPorts infrastructure for NCO: Portfile for NCO 3.9.6. Maintained by Takeshi Enomoto.

Documentation and User's Guide

View the NCO User's Guide in any of these formats:

• DVI Device Independent (kdvi, xdvi)
• HTML Hypertext (any browser)
• Info GNU Info (M-x Info, emacs)
• PDF Portable Document Format (acroread, evince, kpdf, xpdf)
• Postscript Printing (ghostview, kghostview)
• TeXInfo Documentation Source code (emacs)
• Text Plain text (more)
• XML Extensible Markup Language (firefox)

Wiki:

• As of August, 2008, NCO has a Wiki. The Wiki contains a Cookbook of NCO examples not found in the manual. Volunteers are encourage to add their own “NCO Hacks” to this cookbook!

Other documentation:

• This abbreviation key unlocks the mysteries of the source code abbreviations and acronyms.
• Unidata netCDF for Developers Workshop NCO examples

README/ChangeLog/TODO

Files containing useful information about the current NCO distribution:

• README Platforms and software requirements for a successful NCO installation
• ChangeLog Up-to-date history of NCO changes
• TODO An unordered list describing the features and fixes we currently have in mind for NCO.

FAQ: Frequently Asked Questions

These questions show up almost as frequently as my mother. But they are more predictable:

• I still have questions, how do I contact the NCO project? The NCO project has various Q&A and discussion forums described below.
• Where can I find prebuilt NCO executables? Pre-built executables of some versions of NCO for the operating systems described above (Debian-compatible GNU/Linux, Fedora/RedHat GNU/Linux, Gentoo GNU/Linux, and Mac OS X). If you have an account at UCI or SDSC, try the locations in my personal home directories listed below (AIX, IRIX). At NCAR, first try /usr/local/bin (AIX, IRIX). Otherwise, you are on your own.
• Does NCAR support NCO? The NCAR CISL Technical Consulting Group (TCG) supports NCO like other community software packages such as lapack. No other NCAR division has coordinated NCO support. The NCAR CISL-suported executables are in /usr/local/bin. If you notice problems with the NCO installation on SCD machines, or if you would benefit from a more recent release or patch, then inform the relevant system administrators, e.g., NCAR User Support. If you have a comment, suggestion, or bug report, then contact the developers as described below.
• Is there an easy way to keep up with new NCO releases? Subscribe to the nco-announce mailing list. This list is for NCO-related announcements, not for questions. nco-announce is very low volume, averaging one message every few months.

Using NCO at UCI, NCAR, and other High Performance Computing Centers (HPCCs)

Users at UC Irvine and NCAR may find pre-built, almost up-to-date NCO executables in the following locations. These executables are unsupported and are almost guaranteed to hail from different NCO versions and thus to behave slightly differently. (They are built from the main trunk, not a tagged version). Users at NCAR should try the supported executables in /usr/local/bin first.

• CGD network (Solaris 2.x): /home/zender/bin/SUNMP
• ESS greenplanet.ps.uci.edu (Linux 2.6.x): /home/zender/bin/LINUX
• ESS esmf.ess.uci.edu (AIX 5.1.0.0): /home/zender/bin/AIX
• ESS dust.ess.uci.edu (Linux 2.6.x): /home/zender/bin/LINUXAMD64
• SCD bluefire.ucar.edu (AIX 5.3): ~zender/bin/AIX

We are benchmarking NCO in SMP (e.g., AIX) and MPI (e.g., Linux) environments. The full NCO Benchmarks (in PDF form) are available here. Please send us your suggestions for improved configurations. The NCO distribution includes all the custom benchmarking software, and is designed to allow you to contribute your own benchmarks. Contact us if you're interested.

There are two parallel versions of NCO—one uses OpenMP on SMP (including hyperthreaded) machines and the other uses the Message Passing Interface (MPI) on distributed architectures. OpenMP requires a compiler that supports it. Commercial compilers which we have used include those from IBM and the Portland Group. Intel's Linux compiler is free (beer). The MPI operators will run with most MPI implementations, including LAM, MPICH, and MPICH2. LAM installs easily with our own testing environment (latest Ubuntu).

Build-from-Source Requirements

Best Practices:
• Install ANTLR 2.7.x (required for ncap2), GSL (desirable for ncap2), netCDF (absolutely required), OPeNDAP (enables network transparency), and UDUnits (allows dimensional unit transformations) before building NCO. If possible build these all with the same compiler (and switches). NCO is mostly written in C99, and although you can mix and match compilers, this is often difficult in practice and is not recommended. The exception is ncap2 which is written in C++. ANTLR, OPeNDAP, and NCO must be built with the same C++ compiler to properly resolve the C++ name-mangling. NCO does not yet support newer ANTLR versions because the ANTLR 3.x C++ interface is incomplete. It is possible and often preferable to install most of the pre-requisite and optional software from pre-compiled packages on modern Debian- and RPM-based systems with (Debian)
  sudo aptitude install antlr # ANTLR
  sudo aptitude install libcurl3-dev libxml2-dev # DAP-prereqs
  sudo aptitude install bison flex gcc g++ # GNU toolchain
  sudo aptitude install gsl-bin libgsl0-dev # GSL
  sudo aptitude install libnetcdf4 libnetcdf-dev netcdf-bin # netCDF
  # NB: There are no Debian packages for DAP or UDUnits :(
  or (RPM)
  sudo yum install antlr antlr-c++-devel # ANTLR
  sudo yum install curl-devel libxml2-devel # DAP-prereqs
  sudo yum install libdap libdap-devel libnc-dap libnc-dap-devel # DAP
  sudo yum install bison flex gcc g++ # GNU toolchain
  sudo yum install gsl gsl-devel # GSL
  sudo yum install netcdf netcdf-devel # netCDF
  sudo yum install librx librx-devel # RX
  sudo yum install udunits udunits-devel # UDUnits

Hints to Build-from-Source:

Generic Build Hints:
• The two NCO build mechanisms: NCO has two build mechanisms: (1) GNU configure and (2)  a custom Makefile (nco/bld/Makefile) designed for hand-editing. Try the traditional GNU configure method first, i.e., ./configure;make;make check;make install. This method involves no hand-editing of makefiles, but trouble-free builds with it are less common than we would like. The file nco/configure.eg contains a table which describes the success you can expect building NCO with the configure mechanism. Please contribute patches required to get the configure mechanism working properly for NCO on your platform. The regressions archive contains gory details of successful (and failed) builds for many platforms.
  All platforms continue to work fine with the nco/bld/Makefile method: cd ~/nco/bld;make all;make test. Disadvantages are that nco/bld/Makefile (1) Usually requires editing to fix your platforms paths; (2) Does not support shared libraries; and (3) Requires a degree from Hogwarts to understand.
• Build Chain Version Incompatibilities: NCO requires GNU make version ≥ 3.78.1 to support target-specific variable values in bld/Makefile. NCO most often fails to build because ncap fails to build. Two of the most common reasons that ncap fails to build are (1) GNU bison is out-of-date (upgrade bison) and (2) --enable-optimize-* is specified to configure (do not specify it).
  
• C99-compliance & old compilers: NCO versions ≥ 2.9.8 requires compilers which support the 1999 ISO C-Standard called C99. Currently, the only C99 feature NCO takes advantage of is restricted pointers. To build NCO versions ≥ 2.9.8 without a C99-compliant compiler, simply undefine the restrict type qualifier before compiling NCO. Do this, e.g., with CPPFLAGS='-Drestrict='. More details on C99 compatibility are in the header file nco/src/nco/nco.h.
  
• Old netCDF libraries: NCO versions ≥ 3.0.0 require netCDF versions ≥ 3.6 to take advantage of the new 64-bit offset file type. The library and include directories (e.g., /usr/lib and /usr/include, respectively) for netCDF version ≥ 3.6 must be on the path searched by the compiler, and must precede the locations of any older netCDF installations. To build NCO versions ≥ 3.0.0 with older netCDF versions < 3.6, simply define the pre-processor token NC_64BIT_OFFSET to 0 before compiling NCO. Do this, e.g., with CPPFLAGS='-DNC_64BIT_OFFSET=0' or USR_TKN='-DNC_64BIT_OFFSET=0' if you use configure or bld/Makefile, respectively. Users attempting to build NCO with OPeNDAP versions ≤ 3.6.x may need this workaround because OPeNDAP did not fully support netCDF version 3.6.x functionality until OPeNDAP versions ≥ 3.7.0. (The indicator for this problem is a build error like "nco_fl_utl.c", line 30.20: 1506-045 (S) Undeclared identifier NC_64BIT_OFFSET.) All modern NCO versions require netCDF versions ≥ 3.5.0 to support nc_type NC_NAT in NCO source code. NCO versions 3.9.0 and greater (May, 2007) require netCDF versions ≥ 3.6.0 to resolve the nc_inq_format() function.
• ncap build hints: ncap is a very sophisticated and somewhat fragile program. ncap requires flex to build correctly. AT&T lex will not work because it does not understand EOF rules that ncap_lex.l uses. bison version 1.29 (the default version on early Mac OS X) is buggy and does not parse ncap_yacc.y correctly; previous and later versions of bison work correctly. bison version 1.25 will not build ncap on Solaris. Please upgrade bison to the latest version (e.g., 1.875+) before reporting ncap build problems. If ncap keeps you from building NCO, and you do not need ncap, configure with --disable-ncap (and know that ncwa will fail to build as a result).
• Build Hints with GSL:
  NCO versions 3.9.6— support automatic building and linking for GSL versions 1.4—. No earlier versions of GSL are supported.
  
• Build Hints with DAP:
  NCO versions 3.9.8— support automatic building and linking for DAP supplied from libnetcdf versions 4.1-beta1 and beyond
  NCO versions 3.1.3— support automatic building and linking for OPeNDAP versions 3.6.x and GCC versions 4.0+.
  NCO versions 3.0.1—3.1.2 support automatic building and linking for OPeNDAP versions 3.5.x and GCC versions 3.4+.
  NCO versions 2.8.5—3.0.0 support automatic building and linking for OPeNDAP versions 3.4.x and GCC versions 3.3+.
  NCO versions ?.?.?—2.8.4 support automatic building and linking for DODS versions 3.2.x and GCC versions 3.2+.
  No other version combinations are supported.
• libnco_c++ build hints: libnco_c++ requires a good C++ compiler to build correctly. Few people need libnco_c++, so the workaround is not to build it. This is done by configuring with --disable-nco_cplusplus.

Platform-Specific Build Hints:
• AIX build hints: NCO versions 2.8.7+ do not build with autotools under AIX when --enable-optimize-* is selected. The workaround is to not use --enable-optimize-*. NCO versions 2.9.7+ do not build shared libraries with AIX xlc compilers because of silly AIX library dependencies activated by OpenMP directives. The workaround is to configure with --disable-shared.
• IRIX build hints: The most common problem with building on SGI IRIX is and out-of-date toolchain.
• Mac OS X build hints: Mac OS X users report success building NCO without patches so long as they (1) Install the latest version of bison, (2) Disable shared libraries and (3) Disable extended regular expressions. This is done with ./configure --disable-regex --disable-shared Building NCO with shared libraries on Mac OS X on may be possible. To try, first re-build and re-install the netCDF library using CFLAGS=-fno-common. Apparently netCDF, unlike most codes, does not set this automatically. Then build NCO normally. Mac OS X ships with a broken version of the POSIX cut utility which causes many NCO regression tests to appear to fail. In fact the NCO commands succeed, but the cut output fails. The workaround is not to heed the regression tests, or to install a working cut, e.g., from the GNU coreutils package.
• Solaris build hints: All recent NCO versions build with autotools on Solaris 2.7—2.9. The most common problem with building on Solaris with cc/CC is that bison may be out-of-date. Install an up-to-date bison (e.g., 1.875+). Solaris 2.10, i.e., Solaris10, had the audacity to change the prototype of the ctime_r() function and so does not build NCO 3.0.1. The fix is to upgrade to the current NCO version.
• SX build hints: NEC SX does not understand the S_ISLNK macro, preferring instead the (non-POSIX) S_IFLNK macro. This macro appears in the file nco_fl_utl.c. The patch I wrote for this may be broken. Will someone with NEC SX please send us a working patch?
• Windows build hints: Building on Microsoft Windows is possible with Cygwin. The resulting executables will behave normally on local files, but will not have most of NCO's network functionality (e.g., scp, OPeNDAP). See the User's Guide for more details.

Known Problems

Generic Run-time Problems:
• Threading problems with MSA: NCO version 3.9.5 has a nasty bug that causes threaded arithmetic operators, e.g., ncea to produce incorrect results under some conditions. The problem may occur whenever OpenMP is enabled and the operators run on a multi-core CPU with more than one thread. These incorrect answers, if generated, are relatively easy to notice. The number of threads used to generate a file is, by default, recorded in the global attribute nco_openmp_thread_number which may be examined with ncks -M foo.nc | grep nco_openmp_thread_number. The only action that will correct a file that you think (or know) contains corrupted data because of this NCO bug is to re-process the file with a non-buggy NCO version. Version 3.9.5 is buggy and should be upgraded ASAP. Be careful with data processed using this NCO version on multi-core CPUs. The (one-line!) patch to fix this bug in 3.9.5 is here.

Platform-specific Run-time Problems:
• Float-valued intrinsic arithmetic functions in ncap on AIX: ncap versions through 3.9.8 have a bug that causes all float-valued intrinsic math functions to fail under AIX. Float-valued math functions are the ISO C99 functions, e.g., cosf(), fabsf(), logf(). The user does not invoke these functions directly— the user always specifies the generic function name, e.g., cos(), abs(), log(). NCO automatically calls the native single precision (i.e., float-valued) math functions when the generic function argument is a native float (e.g., naked constants like 1.0f or variables stored as NC_FLOAT). Double precision arguments cause NCO to invoke the standard (double-valued) form of the generic function, e.g., cos(), fabs(), log().

Generic Run-time Problems:
• Index-based hyperslab problems: NCO versions 2.7.3—2.8.3 have a nasty bug that causes index-based hyperslabs, e.g., -d lat,1, to behave like value-based hyperslabs, e.g., -d lat,1.0 under some conditions. Unfortunately, the incorrect answers generated may be hard to notice! This problem was most often enountered by users trying to assemble monthly averages using the stride feature of ncrcat. One common symptom is that the time-offset of the output file is incorrect. Versions 2.7.3—2.8.3 are buggy and should be upgraded ASAP. Re-do any data-processing that used index-based hyperslabbing with these versions of NCO.

Operator-specific Run-time Problems:
• Arithmetic problems with ncap division, modulo, and exponentiation: ncap versions < 3.0.1 incorrectly exponentiate variables to variable powers (V^V). We recommend that all ncap users upgrade.
  ncap versions up to 2.9.1 incorrectly handle division, modulo, and exponentiation operations of the form S/V, S%V, and S^V where first operand (S) is scalar (i.e., either typed directly in the ncap script or converted from an attribute) and the second operand (V) is a full variable (i.e., stored in a file or computed by ncap). Instead of the requested quantity, ncap returned V/S, V%S, and V^S. In other words ncap treated some non-commutative operations as commutative. This is now fixed. The V/V, V%V, V^V, V/S, V%S, V^S, S/S, S%S, and S^S operations were never affected. We recommend that all ncap users upgrade.
• Incorrect ncbo output for packed input: ncbo versions ???—3.2.0 incorrectly write differences of packed input. This only affects packed variables.
• Problems with ncflint and missing_values: The algorithm ncflint used to perform interpolation in versions up to 2.9.4 was not commutative. It returned the weighted valid datum when the other datum was missing_value, or it returned missing_value, depending on the order the input files were specified. As of version 2.9.5, ncflint always returns missing_value when either input datum is missing_value. Possible future implementations are discussed here.
• Problems with ncra and ncea when missing_value = 0.0: The algorithm ncra and ncea used to perform arithmetic in versions up to 2.9.2 breaks if missing_value is 0.0. Why, you ask? Running average (or total, etc.) algorithms must initialize the answer to 0.0. This is done since the sum accumulates in place as ncra and ncea proceeds across records and files. (Normalizing this accumulation by the total number of records is the last step). The old algorithm compared both the current running average and the new record to the missing_value. If either comparison matched, then nothing accumulated for that record. This zero-initialization led to a state where it was impossible to ever recognize valid data. As a result nothing accumulated and the answer was always zero. The record and ensemble averages would also fail (in a non-obvious) way whenever an intermediate sum equalled missing_value. The chances of the latter event ever happening are exceedingly remote. The new algorithm compares only the new record to the missing_value. This fixes both problems and is faster, too.
• Packing problems with ncwa: NCO versions ???—2.9.0 have a bug that causes ncwa to fail (produce garbage answers) when processing packed NC_FLOAT data. Version 2.9.1 fixes this problem. This problem may have been noticed most by OPeNDAP users since many netCDF climate datasets served by OPeNDAP are packed NC_FLOATs. Upgrade to 2.9.1 if you use ncwa on packed data.
• Packing problems with ncap: NCO versions 2.8.4—2.8.6 have a bug that causes the ncap intrinsic packing function pack() to fail. Version 2.8.7 fixes this problem.

People:

(Please communicate with us using the project forums rather than contacting us individually)

• Charlie Zender, Associate Professor of Earth System Science and Director, UCI Earth System Modeling Facility. Role: Project PI. Contributions: NCO core library, porting, release manager Current Research: 1. Extend empirically verified analytic model for terascale data reduction of gridded datasets to account for cluster- and network- effects. 2. Enable and optimize NCO for intra-file-level parallelism using netCDF4/HDF5 parallel filesystem features. Other Interests: Atmospheric Physics, Climate Change.
• Steve Jenks, Assistant Professor of Electrical Engineering and Computer Science and Director, Scalable Parallel and Distributed Systems Lab. Roles: Collaborator. Thesis advisor. Current Research: 1. Consulting on SWAMP design and implementation. Other Interests: Cache memory usage
• Phil Papadopoulos, Program Director, Grid and Cluster Computing, SDSC, and Adjunct Associate Professor of Computer Science, UCSD. Role: Co-PI. 1. Coordinates UCSD-end of NCO/SDO benchmark machines. Other Interests: NPACI Rocks
• Daniel Wang: Ph.D. Candidate (and now Doctor!), Electrical Engineering and Computer Science Role: Architect distributed data reduction and analysis solutions for geoscience workflows Current Research: 1. Use of compiler techniques and whole program analysis to discover and exploit parallelism in scripted workflows. 2. Resource management in combined computation-data service environments. 3. Design and implement SWAMP to provide automated server-side data analysis capabilities to the geoscience community. Other Interests: Large data, data management, programming languages and practice.
• Henry Butowsky, Henry Butowsky, software engineer. Roles: Scientific programmer Current Research: 1. Efficient complex data analysis with storage-layer constraints. 2. Develop and thread the ncap2 interpreter. Other Interests: Compilers and interpreters.
• Scott Capps, Ph.D. Candidate, Earth System Science. Current Research: 1. Test NCO on real tera-scale geoscience analysis problems and identify necessary improvements in ncap2 and in SWAMP. 2. Role of satellite-observed sub-gridscale wind variability on air-surface exchange. Other Interests: Weather, climate

Thanks also to past NCO/SDO contributors.

Contacting the Project:

Contact us through the Project Forums (rather than personal e-mail) so other users can benefit from and contribute to our exchange. Let us know how NCO is working for you—we'd like to hear. Have you read the documentation and browsed the forums to see if your question/comment has been reported before? Please read the Guide's suggestions for productive Help Requests and Bug Reports.

• Where should I ask my questions on how to use NCO? On the Help site.
• Where should I post suggestions/comments on NCO features and usage? On the Discussion site.
• Where are NCO development and bug-squashing discussed? At the Developer site.