On the Web at http://dust.ess.uci.edu/prp/prp˘sei/prp˘sei.pdf
NSF Science and Engineering Informatics (SEI) Proposal Submitted: March 4,
2004
Last modified: Saturday 21 August,
200413:17 Next Round Due: December 15, 2004
Dr. Charles S. Zender Dr. Phil Papadopoulos
Department of Earth System Science San Diego Supercomputer Center
University of California at Irvine University of California at San Diego
News/Preface: NSF funded the first NCO/SDO proposal. The next NCO/SDO proposal will tackle different issues, particularly native HDF support and possibly bioinformatics and multi-core processor extensions. This preface sketches out the second NCO proposal, and is followed by the fifteen-page body of the first proposal. Once we are ready to branch off and work on the second NCO proposal, probably around November, 2004, I will place a link to it here. So, just to be clear, this introductory information is an evolving document as we focus in on the second proposal. The next fifteen pages is the body of the first proposal and is fixed. Things got into this hybrid state because I expected to revise and re-submit the first proposal. I began the revisions before learning that the proposal was funded.
The second proposal will leverage the infrastructure and research provided by the first grant, and will develop whole new applications and/or enhancements for NCO/SDO. HDF, bioinformatics, and multi-core processor support are possibile directions. The eventual focus needs a champion to Co-PI the next proposal. If you are interested in being PI or Co-PI on the next proposal in one of these areas, let me know.
Information for potential collaborators: This is an NSF proposal to improve Distributed Data Reduction & Analysis (DDR&A) with NCO. The two main components of this proposal are NCO parallelism (OpenMP, MPI) and Server-Side DDR&A (SSDR&A) implemented through extensions to DODS/OPeNDAP and to netCDF4. This project would dramatically reduce bandwidth usage for NCO DDR&A. With this first NCO proposal out the door, the content of the next two NCO proposals is clear.
The second NCO proposal will be re-written and improved from the first. The certain changes include (1) a more convincing description of why CCSM IPCC data reduction exemplifies a computational, storage, and network-constrained problem common to many domains besides geophysics, (2) articulating the numerous required SSDR&A extensions to OPeNDAP and to netCDF4, and (3) deeper involvement (and funding) for distributed computing experts (e.g., UCSD/OptIPuter) on the parallelization and SSDR&A issues. Unfortunately, the NSF SEIII program has relatively limited funding. The average SEI three-year award size is $500,000. Our budget is already and I want to grow the SSDR&A component. If we are not funded this round, we will include these changes and re-submit to the next NSF SEIII round, due 20041215.
I anticipate sending a distinct (third?) NCO proposal to NASA. Its narrow technical focus will be NCO/SDO for native HDF speakers. I will ask Tom DeFanti and Mike Folk to clarify the issue of whether HDF needs to be modified to be OptIPuter-aware. The two likely NASA programs which might support SDO are ESTO and REASON CANN.
Other significant new directions for NCO/SDO might merit full new proposals to NSF or to other agencies (e.g., DOE, NASA, NIH). Significant extensions under consideration include a bioinformatics component (led by Harry Mangalam, TACGI?), arithmetic speed-ups on multi-processor single-core nodes (led by Steve Jenks, UCI?), or really going hog-wild with the SSDR&A idea (anyone?).
Please contact me if you wish to be involved with any future proposals. Comments on the proposals and letters of support are very welcome. 1. Senior Personnel Inquiries for Intellectual Collaboration (a) Peter Cornillon (URI) OPeNDAP netCDF CL extensions (declined until next round, provided LOS) (b) Steve Jenks (UCI) multi-processor core performance improvements (no response) (c) Harry Mangalam (TACGI) bio-informatics (declined until next round, provided text) (d) Phil Papadopoulos (SDSC, Co-PI) parallelization? (e) Russ Rew (UCAR Unidata) netCDF4, API (provided LOS) (f) Takemura Sato (Japan ESC) DDRA demonstrations (will serve 1 TB) (g) Larry Smarr (Cal-IT2) OptIPuter (provided LOS) (h) Padhraic Smyth (UCI ICS) data mining (declined) 2. Senior Personnel for Equipment Collaboration (a) Aaron Chin (Cal-IT2, Senior Personnel) OptIPuter server support 3. Other Letters of Collaboration/Support to consider (a) Rajiv Bendale contact (IBM) bio-informatics (b) John Caron (UCAR Unidata) NcML aggregation, lazy processing (c) Brian Eaton (NCAR) CF conformance (d) Ian Foster (Argonne) (e) James Gallagher (URI) OPeNDAP netCDF CL extensions (f) Dan Holloway (URI) OPeNDAP netCDF CL extensions (g) Mark Taylor (Sandia) climate SSDR&A 4. Big Budget Items (a) One month summer salary per year for Zender (b) One month salary for Aaron Chin (OptIPuter) (c) P/A or Specialist II to architect/bulletproof/coordinate/release SDO (Butowsky?) (d) Grad Student to optimize computational geometry based on processor topology (ICS?) (e) $17k for ESMFOptIPuter networking 5. Next SEI proposal due December 15, 2004
Project Summary.
Climate simulations for the Fourth Assessment Report of the IPCC will be performed and stored in netCDF format in multiple national and international HPC centers. HDF datasets from NASA, NOAA, and ESA satellites steadily accumulate in geographically disparate EOSDIS sites. These data are only as valuable as they are accessible to the working geophysicist at his or her desk. Concurrent interactive and scripted analysis of geographically distributed large scale scientific datasets is functionality critical to managing and interpreting the many streams of geophysical data.
We propose to develop a suite of Scientific Data Operators (SDO) for interactive and scripted manipulation of (locally and) widely distributed repositories of netCDF- and HDF5-formatted geophysical data. SDO’s functionality will suit distributed, network-transparent, analysis of ensemble tera-scale datasets stored at many remote locations. SDO will meet the current and future needs of geophysicists and, potentially, bio-informaticists.
SDO integrates four existing, proven, Open Source software technologies: (1) netCDF—the de facto standard format for climate model data and Earth-bound geophysical observational datasets. (2) netCDF Operators (NCO)—the de facto standard for analysis of climate model and geophysical data. (3) DODS/OPeNDAP—the de facto standard for network-transparent client/server access to geophysical data. (4) HDF-EOS—the official storage standard for NASA EOS satellite data. We have taken advantage of existing synergies and convergence among these standards, and guidance from their initiators, to plan the extension of the existing NCO toolkit into the next generation SDO toolkit. SDO will (like NCO) be an indispensable software assistant to individual researchers and research centers for distributed processing of climate and satellite data.
This SDO project has three main parts: (1) Implement shared memory (OpenMP) and Message Passing Interface (MPI) parallelism to speed up and reduce latency of local and distributed Tera-scale data processing. (2) Design and implement automatic computational geometry load-balancing algorithms to exploit the intrinsic parallelism of frequently used arithmetic operations without user intervention. (3) Add, enhance, and simplify functionality for server-side processing of distributed netCDF data with the OPeNDAP protocol.
Scientific Merit: The proof-of-concept application, distributed analysis of NCAR CCSM IPCC assessment simulations within and across national boundaries, may potentially yield otherwise undiscovered patterns among different SRES scenarios for the same model, and among different climate models. New data reduction efficiencies will be gained by automatic configuration of computational geometry to optimize for the data servers’ processor topology. The tera-scale climate datasets targeted for analysis will reveal any critical bandwidth, I/O, and client/server bottlenecks in distributed geophysical data processing.
Broader Impacts: Bringing distributed data analysis power out from the realm of the computational scientist to the desktop of the practicing geophysicist will leverage existing distributed capabilities by making their use transparent to the average scientific user. Improving tools to analyze and inter-compare geophysical model and observational data that reside in various national HPC centers increases the simulations’ scientific value and decreases time to discovery and publication. SDO helps address the problem of growing bioinformatics data sets, especially gene expression data, in ways similar to the geophysics domain. It provides a mechanism for efficient storage and manipulation of the numeric data separate from the contextual or meta-data which is normally stored in XML.
Climate simulations for the Fourth Assessment Report of the IPCC will be performed and stored in netCDF format in multiple national and international HPC centers. HDF datasets from NASA, NOAA, and ESA satellites steadily accumulate in geographically disparate EOSDIS sites. These data are only as valuable as they are accessible to the working geophysicist at his or her desk. Concurrent interactive and scripted analysis of geographically distributed large scale scientific datasets is functionality critical to managing and interpreting the many streams of geophysical data.
We propose to develop a suite of Scientific Data Operators (SDO) for interactive and scripted manipulation of (locally and) widely distributed repositories of netCDF- and HDF5-formatted geophysical data. SDO’s functionality will suit distributed, network-transparent, analysis of ensemble tera-scale datasets stored at many remote locations. SDO will meet the current and future needs of geophysicists and, potentially, bio-informaticists.
SDO integrates four existing, proven, Open Source software technologies: (1) netCDF—the de facto standard format for climate model data and Earth-bound geophysical observational datasets. (2) netCDF Operators (NCO)—the de facto standard for analysis of climate model and geophysical data. (3) DODS/OPeNDAP—the de facto standard for network-transparent client/server access to geophysical data. (4) HDF-EOS—the official storage standard for NASA EOS satellite data. We have taken advantage of existing synergies and convergence among these standards, and guidance from their initiators, to plan the extension of the existing NCO toolkit into the next generation SDO toolkit. SDO will (like NCO) be an indispensable software assistant to individual researchers and research centers for distributed processing of climate and satellite data.
This SDO project has three main parts: (1) Implement shared memory (OpenMP) and Message Passing Interface (MPI) parallelism to speed up and reduce latency of local and distributed Tera-scale data processing. (2) Design and implement automatic computational geometry load-balancing algorithms to exploit the intrinsic parallelism of frequently used arithmetic operations without user intervention. (3) Add, enhance, and simplify functionality for server-side processing of distributed netCDF data with the OPeNDAP protocol.
The proof-of-concept application, distributed analysis of NCAR CCSM IPCC assessment simulations within and across national boundaries, may potentially yield otherwise undiscovered patterns among different SRES scenarios for the same model, and among different climate models. New data reduction efficiencies will be gained by automatic configuration of computational geometry to optimize for the data servers’ processor topology. The tera-scale climate datasets targeted for analysis will reveal any critical bandwidth, I/O, and client/server bottlenecks in distributed geophysical data processing.
This proposal is organized as follows. Section 2 describes the purpose, capabilities, and functional relationships between netCDF, HDF, NCO, and OPeNDAP. Section 3 describes the results of our relevant, prior NSF-funded research. Section 4 describes the current barriers facing geo-scientists who require Distributed Data Reduction & Analysis (DDR&A) capabilities. Section 5 details the specific objectives of the proposal and details our methods for achieving them. We then describe then prototype experiment for this project, DDR&A of climate simulations datasets. Section 6 describes our software engineering plans. Section 7 presents our project coordination plan, PI responsibilities, time-line, milestones, and software engineering methods. Section 8 concludes with a presentation of the broader impacts and synergies of our project. Three letters of support/collaboration and a list of acronyms and abbreviations are included as supplementary documents.
The increasing size, number, and complexity of scientific data in the past decades has led to the development and use of self-describing data formats (SDDFs) and tools to manipulate these formats. The SDDFs replace less functional formats such as raw-binary or text-formatted data.
Two SDDFs currently dominate data archival in the geo-sciences. The first is the Hierarchical Data Format (HDF) (NCSA, 2004). HDF was developed at the National Center for Supercomputing Applications (NCSA) and adopted by NASA for Earth Science Enterprise (ESE) applications. HDF is the most commonly used archival format for ESE satellite data. This proposal does not involve any work directly with HDF and we mention HDF mainly due to its importance in observational geophysics. Although we would like one day to implement a native HDF-EOS back-end to NCO/SDO, that task would be extremely difficult and is beyond the scope of this proposal.
This proposal will, however, exploit and benefit HDF data indirectly thanks to a complementary proposal described in Section 8. Briefly, a fully-funded effort to layer the netCDF API on top of HDF5 is underway. Since NCO/SDO is completely netCDF-conformant, all NCO/SDO operations will soon work on any HDF file written with netCDF4.
The second popular SDDF is the Network Common Data Format (netCDF), developed by Unidata at the National Center for Atmospheric Research (NCAR) (Unidata, 2004). netCDF has become the most commonly used archival format for large scale geophysical models, such as climate models. netCDF is less-powerful than HDF because is lacks features such as data compression, irregular grids, threading, and parallel I/O. However, netCDF is much simpler to program than HDF, and, as a result is widely used in the geophysical and climate modeling community by practicing scientists.
Tools to manipulate and view netCDF files are relatively easy to write since the API is much simpler than, say, HDF. The netCDF Operators (NCO) (Zender, 2004) is probably the best-known toolkit for numeric and metadata analysis and manipulation of netCDF data.
Traditional processing of scientific data works with an intra-file paradigm. Users open a file, read a variable from the file, and manipulate it. The intra-file paradigm works well in cases where all the pertinent data are stored in one or a few files. In some disciplines, however, data storage requirements dictate that relevant data be spread over multiple files. Satellite-derived information, for example, may be stored in a file-per-day or file-per-orbit format. Data produced by geophysical time-stepping models is output every timestep or averaged over many timesteps. Climate models, for example, archive data once-per simulated day or month, and simulate years or even centuries producing hundreds or thousands of large files in a single simulation. In such applications, the inter-file paradigm becomes unwieldy and the optimal tool for data reduction must support an inter-file paradigm.
We developed some guidelines based on our extensive experience with geophysical and climate data and implemented them in NCO. NCO assumes that processing large numbers of geophysical data-files is most efficient and intuitive when:
Apparently NCO’s guiding philosophy, “do what a sane user would want” has succeeded! NCO runs on all modern operating systems, and its use is fully detailed in the NCO User’s Guide. To my knowledge, all established national and international climate modeling centers, including NASA, NOAA, N�