% $Header$ % Purpose: MRI proposal for Earth System Modeling Facility (ESMF) % URL: http://dust.ess.uci.edu/prp/prp_mri.pdf % fxm: pdflatex puts too much white space at top margin of Wessel's .eps figures % NSF MRI 2003: % http://www.nsf.gov/od/oia/programs/mri/cid.htm % Major Research Instrumentation (MRI) % Program manager: Jay Fein (703) 292-8527, (FAX) 292-9022 % NSF FastLane Proposal #0321380 % NSF Award # ATM-0321380 % Award period 20030801--20060731 % Budget: NSF $773543, UCI $331520 (= $268457 real + $63063 overhead absorbed) % Annual progress report deadlines: 20040502, 20050502, 20061029 % UCI proposal number: 33433 % UCI account number: 9-number-fund-sub-object = 9-123456-12345-1-1234 = 9-445925-21764-1-1234 % NSF MRI 2004: % Kuester's follow-on 2004 MRI for visualization % http://webfiles.nacs.uci.edu/xythoswfs/webui/fkuester/NSF-MRI-2004 % Usage: % cd ~/prp;make -W prp_mri.tex prp_mri.dvi prp_mri.ps prp_mri.pdf prp_mri.txt;cd - % scp ${HOME}/prp/prp_mri.dvi ${DATA}/ps/prp_mri.pdf ${DATA}/ps/prp_mri.ps ${HOME}/prp/prp_mri.tex ${HOME}/prp/prp_mri.txt dust.ess.uci.edu:/var/www/html/prp/prp_mri % # NB: latex2html works well on prp_mri.tex % latex2html -dir /var/www/html/prp/prp_mri prp_mri.tex % # NB: tth chokes on prp_mri.tex % cd ${HOME}/prp;tth -a -Lprp_mri -p./:${TEXINPUTS}:${BIBINPUTS} < ${HOME}/prp/prp_mri.tex > prp_mri.html % scp prp_mri.html dust.ess.uci.edu:/var/www/html/prp/prp_mri % # NB: tex4ht works well on prp_mri.tex % cd ${HOME}/prp;htlatex prp_mri.tex % scp prp_mri*.css prp_mri*.html dust.ess.uci.edu:/var/www/html/prp/prp_mri % # NB: tex4moz works well on prp_mri.tex % cd ${HOME}/prp;/usr/share/tex4ht/mzlatex prp_mri.tex % scp prp_mri*.css prp_mri*.html prp_mri*.xml dust.ess.uci.edu:/var/www/html/prp/prp_mri % Proposal preparation: % pdftk A=${DATA}/ps/prp_mri.pdf cat A3 output ${DATA}/prp/prp_mri/prp_mri_smr.pdf % pdftk A=${DATA}/ps/prp_mri.pdf cat A4-21 output ${DATA}/prp/prp_mri/prp_mri_dsc.pdf % pdftk A=${DATA}/ps/prp_mri.pdf cat A22-28 output ${DATA}/prp/prp_mri/prp_mri_rfr.pdf % pdftk A=${DATA}/ps/prp_mri.pdf cat A29 output ${DATA}/prp/prp_mri/prp_mri_bdg_jst.pdf % pdftk A=${DATA}/ps/prp_mri.pdf cat A30-31 output ${DATA}/prp/prp_mri/prp_mri_fcl.pdf % pdftk A=${DATA}/ps/prp_mri.pdf cat A32-33 output ${DATA}/prp/prp_mri/prp_mri_abb.pdf % pdftk A=${DATA}/ps/prp_mri.pdf cat A34-35 output ${DATA}/prp/prp_mri/prp_mri_clb.pdf % ESMF website maintenance: % scp -r ${DATA}/ps/prp_mri.pdf dust.ess.uci.edu:/var/www/html/esmf/prp_mri % scp -r ${DATA}/ps/prp_mri.pdf www.ess.uci.edu:/home/httpd/html/esmf/prp_mri % latex2html -dir /var/www/html/esmf/prp_mri prp_mri.tex \documentclass[12pt]{article} % Standard packages \usepackage{ifpdf} % Define \ifpdf \ifpdf % We are running PDFLaTeX \usepackage[pdftex]{graphicx} % Defines \includegraphics* \pdfcompresslevel=9 \usepackage{thumbpdf} % Generate thumbnails \usepackage{epstopdf} % Convert .eps, if found, to .pdf when required \else % We are not running PDFLaTeX \usepackage{graphicx} % Defines \includegraphics* \fi % endif PDFLaTeX \usepackage{array} % Table and array extensions, e.g., column formatting \usepackage{mdwlist} % Compact list formats \itemize*, \enumerate* \usepackage{longtable} % Multi-page tables, e.g., acronyms and symbols \usepackage{lscape} % Landscape environment \usepackage{natbib} % \cite commands from aguplus \usepackage{rotating} % Allows sideways figures and tables \usepackage{times} % Postscript Times-Roman font KoD99 p. 375 \usepackage{url} % Typeset URLs and e-mail addresses % hyperref is last package since it redefines other packages' commands % hyperref options, assumed true unless =false is specified: % backref List citing sections after bibliography entries % baseurl Make all URLs in document relative to this % bookmarksopen Unknown % breaklinks Wrap links onto newlines % colorlinks Use colored text for links, not boxes % hyperindex Link index to text % plainpages=false Suppress warnings caused by duplicate page numbers % pdftex Conform to pdftex conventions % Colors used when colorlinks=true: % linkcolor Color for normal internal links % anchorcolor Color for anchor text % citecolor Color for bibliographic citations in text % filecolor Color for URLs which open local files % menucolor Color for Acrobat menu items % pagecolor Color for links to other pages % urlcolor Color for linked URLs \ifpdf % We are running PDFLaTeX \usepackage[backref,breaklinks,colorlinks,hyperindex,citecolor=blue,linkcolor=blue,urlcolor=blue,plainpages=false,pdftex]{hyperref} % Hyper-references \pdfcompresslevel=9 \else % We are not running PDFLaTeX \usepackage[backref=false,breaklinks,colorlinks=false,hyperindex,plainpages=false]{hyperref} % Hyper-references \fi % endif PDFLaTeX % Personal packages \usepackage{csz} % Library of personal definitions \usepackage{dmn} % Dimensional units \usepackage{chm} % Commands generic to chemistry \usepackage{jrn_agu} % AGU-sanctioned journal title abbreviations % Commands specific to this file \makeatletter \renewcommand{\fnum@table}{\textbf{\tablename~\thetable}} % Boldface ``Table: #'' \renewcommand{\fnum@figure}{\textbf{\figurename~\thefigure}} % Boldface ``Figure: #'' \makeatother % Margins \oddsidemargin = 0.0in \evensidemargin = 0.0in \topmargin = -0.4in \textheight = 9in \textwidth = 6.5in \columnsep = 0.25in \headheight = 12pt \headsep = 12pt \footskip = 24pt % Vertical space % 20000717: None of these commands seem to have any effect here \setlength{\textfloatsep}{12pt} % Space between last top float or first bottom float and the text. \setlength{\intextsep}{12pt} % Space left on top and bottom of an in-text float. % \dbltextfloatsep is \textfloatsep for 2 column output. % \dblfloatsep is \floatsep for 2 column output. \setlength{\abovecaptionskip}{0pt} % Space above caption \setlength{\belowcaptionskip}{0pt} % Space below caption % Float placement % NB: Placement of figures is very sensitive to \textfraction \renewcommand\textfraction{0.0} % Minimum fraction of page that is text \setcounter{totalnumber}{73} % Maximum number of floats per page \setcounter{topnumber}{73} % Maximum number of floats at top of page \setcounter{dbltopnumber}{73} % Maximum number of floats at top of two-column page \setcounter{bottomnumber}{73} % Maximum number of floats at bottom of page \renewcommand\topfraction{1.0} % Maximum fraction of top of page occupied by floats \renewcommand\dbltopfraction{1.0} % Maximum fraction of top of two-column page occupied by floats \renewcommand\bottomfraction{1.0} % Maximum fraction of bottom of page occupied by floats \renewcommand\floatpagefraction{1.0} % Fraction of float page filled with floats \renewcommand\dblfloatpagefraction{1.0} % Fraction of double column float page filled with floats \begin{document} \def\prpttl{Acquisition of an Earth System Modeling Facility for\\ Coupled Climate, Chemistry, and Biogeochemistry Studies\\} % Cover page \pagenumbering{roman} %\pagestyle{headings} \pagestyle{plain} \thispagestyle{empty} {\noindent% NSF Major Research Instrumentation Proposal \hfill \today} \begin{center} \textbf{\Large\prpttl} \bigskip Dr.~Charles S. Zender \hfill Dr.~Susan E. Trumbore \\ Department of Earth System Science \hfill IGPP Director \\ University of California at Irvine \hfill University of California at Irvine \\ \end{center} \vskip 0.5 cm \begin{enumerate*} \item Recommended proposal structure: \begin{enumerate*} \item Research Activities: 10 pages (All) \item Research Instrumentation and Needs: 2 pages (Wessel/Zender) \item Impact of Infrastructure Projects: 2 pages (Prather) \item Project and management Plans: 1 page (Zender) \end{enumerate*} \item Strategic questions: \begin{enumerate*} \item Email Sandy Shore and Russ Kelz seeking support for ATM proposal \item According to Schmoltner, ``big project'' threshold is \$800--900k, we are at \$876,887 \end{enumerate*} \end{enumerate*} \clearpage %\onecolumn %\tableofcontents %\listoftables %\clearpage \pagenumbering{arabic} \setcounter{page}{0} \thispagestyle{empty} %\markleft{Mineral Dust Aerosol} %\markright{} %\begin{center} %\textbf{\Large\prpttl} %\end{center} \noindent{\large{\textbf{Project Summary.}}}\label{sxn:smr} %\enlargethispage*{0.5in} This proposal asks for funds to purchase a high-performance computer and storage system as the center of the UCI Earth System Modeling Facility (ESMF). The UCI ESMF will be devoted to the integration, synthesis, and analysis of large models and datasets required to advance fundamental understanding of the coupled physical climate, chemistry, and biogeochemical cycles of the Earth system. The ESMF represents a major enhancement in computational capability over the workstation-based and older shared-memory resources currently in use at UCI. It is tailored for the merging of ESM components (e.g., atmospheric chemistry, ocean biology, land hydrology) that normally consume the available computing resources of individual research projects. Although primarily a development facility for faculty and graduate researchers, the ESMF will produce the decade-long simulations of the coupled system that are needed for basic scientific studies. In terms of computing power, the ESMF fits between our individual PI and school-based resources and national modeling facilities (e.g., NCAR, SDSC), but in terms of development it is unique. The UCI ESMF fills a niche, allowing faculty, post-graduates, and students to work with high-performance computing in an environment where they control the computer and the code development. \textbf{Intellectual Merits.} The process of coupling existing ESM components will identify some of the more important feedbacks and help identify aspects of the models that need attention or further development. The ESMF will also be used operationally (i.e., 100-year simulations) with component subsets of the full ESM, to address some outstanding questions about feedbacks anticipated with global change in the 21st century. Three subset-ESMs are planned for the early development: aerosol/gas chemistry, physical climate, and land+ocean biogeochemistry; catchment-scale riverine nutrient transport, Aeolian erosion, and marine ecosystems; and ocean and land surface memory effects on the predictability of the hydrologic cycle. These subset-ESMs will provide new insights (and problems) as we explore scientific questions that highlight the new capability of coupling these components. The initial subset-ESMs will produce new scientific results which will be pursued in operational studies with the full ESM. The ESMF will foster interdisciplinary collaborations between research groups which bring together an extremely wide breadth of scientific expertise in developing, analyzing and improving a coupled Earth System Model. These improvements will feed back into national modeling efforts. The ESMF will be maintained as a multi-user resource as part of the UCI Department of Earth System Science (ESS). The primary users of the ESMF are UCI ESS Professors Famiglietti, Magnusdottir, Moore, Prather, Primeau, Yu, and Zender, and their associated researchers, post-doctoral fellows, students and collaborators. \textbf{Broader Impacts.} Development of ESMs is essential if the public is to understand the full implications of climate change and make the best informed political decisions regarding adaptation and mitigation. Integration of the human dimensions side of global change needs the details provided by an ESM rather than the reduced dimensions of integrated assessment models. Work at UCI's ESMF will contribute to ESMs around the globe. As an academic center for ESM development, the ESMF will train the next generation of scientists and research staff in applying high-performance computing to large, coupled models and datasets in the Earth sciences. The ESS Department's undergraduate curriculum encourages research projects, and the ESMF will sustain computing projects for several undergraduates. The ESMF will establish projects with UCI's Center for Educational Partnerships (CFEP), which links Southern California agencies including school districts, parents, and community colleges. ESMF projects will also bring in students from a nearby Minority Serving Institution, California State University at Bakersfield (CSUB), giving these students access to environmental science models, data, and mentors unavailable at CSUB. \clearpage %\begin{center} %\textbf{\Large\prpttl} %\end{center} \section{Introduction}\label{sxn:ntr} Faculty of the UCI Department of Earth System Science (ESS) propose to acquire a high performance computer and storage system to become the UCI Earth System Modeling Facility (ESMF). The UCI ESMF will be devoted to the integration, synthesis, and analysis of large geophysical models and datasets required to advance fundamental understanding of the coupled climate, chemical, and biogeochemical cycles of the Earth System. The ESMF represents a major enhancement in computational capability over the workstation-based and older shared-memory resources currently in use at UCI. It is tailored for the merging of ESM components (e.g., atmospheric chemistry, ocean biology, land hydrology) that---within each PI's research---normally consume the available computing resources of individual research projects. The ESMF is optimized for providing highly cached, shared-memory computing for each component, with more relaxed requirements for parallel computation and communication between the different ESM components. In our recent experience with national and international assessments of climate, key questions were unanswerable because of the lack of capability in integrating coupled climate system models, specifically the capability that the ESMF will provide. The scientific expertise in these individual areas of Earth system science exists within the ESS department, and its members are primary contributors to coupled Earth system models being assembled in many places. Yet there is no computational facility in the~US where a group of researchers with the breadth and depth of the PIs can easily collaborate on science arising from the coupling of Earth system climate, chemical, and biogeochemical cycles. In terms of computing power, the proposed ESMF fits between our individual PI and school-based resources and national modeling facilities (e.g., NCAR, SDSC), but in terms of development it is unique. The UCI ESMF fills a niche, allowing faculty, post-graduates, and students to work with high-performance computing in an environment where they control the computer and the code development. Although primarily a development facility for faculty and graduate researchers, the ESMF will produce the decade-long simulations of the coupled system that are needed for basic scientific studies. An important component of the ESMF is the dedicated disk space for data synthesis and analysis. In the next three years ESS researchers will require uninterrupted access to 30--100~terabytes (TB) of data to keep pace with the increases in temporal, spatial, and spectral resolutions of remote sensing measurements and coupled climate system models. This ESMF will allow ESS researchers to pursue data-intensive research utilizing the large geophysical datasets from current and next generation numerical models and satellite observations. \subsection{Personnel Directly Involved with the ESMF}\label{sxn:pis} The principle researchers making use of the ESMF are the research groups of UCI Earth System Science Professors Magnusdottir, Moore, Prather, Primeau, Yu, Zender, and Famiglietti (joint with Civil Engineering): \begin{enumerate*} \item Graduate students: J.~Abatzoglou, A.~Berg, S.~Bortz, L.-M.~Chen, M.~Flanner, E.~Kwon, Y.~Hu, J.-W.~Lee, D.~Ryu, H.~Syed, C.-C.~Wang \item Post-Docs: H.~Bian, J.~Hsu \item Researchers: S.~Holl, X.~Ma, D.~Newman, S.~Tyler, X.~Zhu \item Undergraduates: S.~Bamattre and others unnamed \end{enumerate*} The PIs have committed to coordinate modeling and experimental research around hypotheses and experiments testable using a unified Earth System Model (ESM). These faculty (see attached bio-sketches) have identified specific research projects (Section~\ref{sxn:prj}, below) that take advantage of the ESMF. Other faculty anticipate using the ESMF in support their research activities in the near future. These include Professors Blake (Chemistry, Using ESMF simulations to evaluate aircraft-measured trace gases in support of field programs), Dabdub (Mechanical \& Aerospace Engineering, Using ESMF to provide boundary conditions for nested regional air quality modeling), Druffel (ESS, Comparing ESMF simulations with long term coral \CXIV\ measurements), Friehe (MAE, Comparing measured and ESMF-modeled air sea exchange), Smecker-Hane (Physics and Astronomy), and Smyth (Information and Computer Science, Fitting hidden Markov models to observations and ESMF simulations of precipitation and storm tracks), and Trumbore (ESS, Understanding terrestrial C~cycling using ESMF-simulations and soil~\CXIV\ measurements). The ESMF will be maintained as a multi-user ESS departmental resource. \subsection{Proposal Organization}\label{sxn:org} Section~\ref{sxn:rsr} describes the research opportunities the PIs will take advantage of with the ESMF. Section~\ref{sxn:needs} describes why our current computational resources are inadequate to perform this research. Then we describe the computational requirements of an ESMF suitable for solving the coupled climate-biogeochemistry problems that are our highest priority. Section~\ref{sxn:mpc} describes the broader impacts of the ESMF on society, research training, and minority opportunities at UCI. Section~\ref{sxn:mng} concludes with the plans for ESMF administration and future maintenance. Appendices contain the Budget Justification, two vendor quotes, a parts list, a Research Facilities statement, letters of support from CSUB, the UCI IGPP and VCR, and a list of Acronyms and Abbreviations. \subsection{Results from Prior NSF Funding}\label{sxn:prv} None of the PIs has received previous NSF funding for instrumentation. \section{Impact of facility on current and proposed research}\label{sxn:rsr} \subsection{Coupled Earth System Modeling at UCI}\label{sxn:esm} To enhance coupled Earth System research at UCI, ESS researchers have agreed to coordinate modeling and experimental research around hypotheses and experiments testable using a unified Earth System Model (ESM). Many ESM components will be based on the Community Climate System Model (CCSM) coordinated by the National Center for Atmospheric Research (NCAR) \cite{BBB01}. The ESMF will be used to study and extend climate related tracers and biogeochemical sub-models in the existing CCSM. This allows us to focus on the cutting edge science while building on a state-of-the-art climate system model. The depth of Earth System modeling at UCI has reached a critical mass where we can now leverage eachother's expertise to make fundamental advances the coupled Earth system. The PIs will use the ESMF to coordinate and direct their autonomous modeling research into a fully coupled Earth System Model effort at a faster pace than can be done through collaborating through external groups such as Community Climate System Model (CCSM) working groups (WGs) \cite[]{BBB01}. These WGs decide where improvements and extensions are needed in the CCSM, coordinate scientific strategies to meet these needs, allocate computational resources to test competing physics packages, and evaluate the results. The ESMF is dedicated to the design and initial scientific application of next generation couplings (cf. Table~\ref{tbl:xpt}) that are led by UCI. ESMF researchers are active members of several CCSM WGs, including the Atmospheric Model (AMWG), Biogeochemistry (BGCWG), Climate Variability (CVWG), and Land Model (LMWG) groups. The ESMF will facilitate progress and ease student involvement and training in UCI focus areas across these WGs, so the ESMF is complementary to CCSM WGs. Our improvements to ESM components will feed back into national modeling efforts such as CCSM and NASA's Global Modeling Initiative (GMI) through our strong associations with these efforts. Thus any negative impact of the ESMF on existing national collaborations of the PIs is expected to be minimal and outweighed by the positive impact of ESMF contributions to these efforts. Why are we not pursuing the research outlined below at a remote supercomputer location (such as NCAR or SDSC) open to proposals for large scale Earth system computational projects? We will address this question directly and state our reasons for establishing the ESMF: \begin{enumerate*} \item UCI's research niche: The ESMF is for focused research on coupled next generation bio\-geo\-chem\-istry-climate problems. The PIs have unique strength in linking chemistry, biogeochemistry, and climate cycles together. \item Scale of research: ESMF experiments will leverage control simulations from external centers. \item Unfettered collaboration: The ESMF is devoted to quick turnaround (order one week) projects to enable university researchers to dynamically pursue interesting hypotheses without excessive administrative overhead. \item Different resolution, more free parameters, and quicker turnaround \item Responsiveness: Being on-site, the ESMF will be responsive to our specific science and teaching needs. Students and researchers will gain valuable experience implementing and analyzing their own experiments. \end{enumerate*} In contrast to external centers such as NCAR and SDSC, the ESMF is designed as a dedicated facility for a group of close-knit collaborators. Our mission is to develop the physical and chemical couplings required to address unanswered questions. One way we will use the ESMF is studying climate and biogeochemical feedbacks of a particular component of the Earth System where the remainder of the system is represented by the standard NCAR CCSM. The CCSM provides an efficient, well-documented base upon which to pursue innovative modeling studies. CCSM components run in both stand-alone and coupled mode, and already contain with many biogeochemical, hydrologic, and radiative improvements developed and contributed by UCI researchers. Table~\ref{tbl:mdl} shows the relations between the ESMF models, research areas, and researchers. \begin{table} \centering % \centering uses less vertical space than center-environment \begin{minipage}{\hsize} \renewcommand{\footnoterule}{\rule{\hsize}{0.0cm}\vspace{-0.0cm}} % KoD95 p. 111 \caption[Purpose and Users of Primary ESMF Models]{\textbf{Purpose and Users of Primary ESMF Models} \label{tbl:mdl}} \vspace{\cpthdrhlnskp} \begin{tabular}{r >{\raggedright}p{12.0em}<{} ccc >{\raggedright}p{7.5em}<{} r} \hline \rule{0.0ex}{\hlntblhdrskp}% Inst.\footnote{The institution primarily responsible for model development.}\ \& Model & Name and ESMF Purpose & UCI\footnote{Significant portion of model development led by ESMF Co-PI.} & CCSM\footnote{Model is fully integrated with NCAR CCSM.} & IPCC\footnote{Model has contributed to IPCC assessment in 1990, 1995, or 2001.} & Users\footnote{All ESMF researchers will use the CCSM, and thus most models in the table. Named are researchers with interests in specific physical processes described by the model in question.} & \\[0.0ex] \hline \rule{0.0ex}{\hlntblntrskp}% NCAR CCSM\footnote{CCSM is a coupling of CAM, CLM, POP, and CSIM.} & Comm. Climat Sys. Model & $\times$ & $\times$ & $\times$ & All & \\[0.5ex] NCAR CAM & Comm.\ Atmosphere Model & $\times$ & $\times$ & $\times$ & Magnus., Zender & \\[0.5ex] NCAR CLM\footnote{DEAD and RTM are components of the CLM.} & Common Land Model & $\times$ & $\times$ & & Famig., Zender & \\[0.5ex] LANL CSIM & Comm.\ Sea Ice Model & & $\times$ & & Magnusdottir & \\[0.5ex] UCI DEAD\footnote{DEAD, developed by Zender, is a component of CAM, CLM and the UCI CTM} & Dust Entrainment \& Depos. & $\times$ & $\times$ & & Zender, Moore & \\[0.5ex] NCAR POP & Parallel Ocean Program & & $\times$ & $\times$ & Moore, Prim., Yu & \\[0.5ex] UCI OEM\footnote{OEM, developed by Moore, is a component of POP.} & Ocean Ecosystem Model & $\times$ & $\times$ & & Moore & \\[0.5ex] UCI CTM\footnote{The UCI CTM, developed by Prather, will be coupled to the CCSM to allow for full chemical-climate interactions, a capability the CCSM currently lacks.} & Chem.\ Transport Model & $\times$ & & $\times$ & Prather, Zender & \\[0.5ex] UCI RTM\footnote{The RTM, developed by Famiglietti, is a component of the CLM.} & River Transport Model & $\times$ & $\times$ & & Famiglietti & \\[0.5ex] UCLA CGCM\footnote{The UCLA CGCM is developed by Yu.} & Coupled GCM & $\times$ & & & Yu, Primeau & \\[0.5ex] UCLA NCOM\footnote{The UCLA NCOM is a useful tool for accelerating climate predictions on the 10--100 year timescale.} & Upper ocean model & $\times$ & & & Yu, Moore & \\[0.5ex] \hline \end{tabular} \end{minipage} \end{table} % end tbl:mdl Most ESMF researchers will use CCSM components. Named in the table are researchers with interests in specific physical processes described by the model in question. Often these researchers are significant contributors to the associated models. Clearly UCI researchers have developed (and continue to maintain and develop) many of the core physical process models in the CCSM, including CRM, DEAD, OEM, and RTM. %Uncoupled and fully coupled Earth system simulations using these %models has already resulted in numerous scientific advances %\cite[e.g.,][]{MDG02,MDS03,AKZ03}. Other UCI models, such as the UCI CTM and the Catchment RTM represent processes (prognostic tropospheric chemistry and catchment hydrology, respectively) that are currently not represented in most Earth System Models (including CCSM). Studying the effects of these new interactions will be a prime focus of the ESMF. Studies have already begun on the next generation of processes we intend to link to our Earth system model. These includes riverine erosion, and multi-component aerosols. The ESMF is not designed for the fully-coupled multi-century simulations that many are pursuing at NCAR and SDSC. The ESMF will be a staging area for new collaborations among UCI faculty. As these projects reach maturity, we will wish to perform multi-decadal ensemble ``production runs'', e.g., to analyze decadal variability or feedback behavior assuming a variety of initial conditions (e.g., IPCC scenarios). These type of ensemble runs \textit{are} best suited to external locations or separately dedicated facilities such as NCAR, NERSC, or SDSC. \subsection{Specific Earth System Modeling Research Projects}\label{sxn:prj} We believe ESS researchers would make important scientific advances on understanding the coupled Earth system if we could join our current models and expertise in a single effort. The ESMF will provide this opportunity. We are poised for leaps in understanding in at least three distinctive aspects of the coupled Earth system. Thus three subset-ESMs are planned for early development: (1)~Aerosol/gas chemistry, physical climate, and land+ocean biogeochemistry; (2)~Catchment-scale riverine nutrient transport, Aeolian erosion, and terrestrial plus marine ecosystems; (3)~Ocean and land surface memory effects on the predictability of the hydrologic cycle. These subset-ESMs will provide new insights (and problems) as we explore scientific questions that highlight the new capability of coupling these components. Sections~\ref{sxn:chm}--\ref{sxn:rgn} describe these cross-cutting research projects which are of interest to many ESS researchers and constitute the primary collaborative foci of ESMF research. By the end of this proposal, these initial subset-ESMs will have produced new scientific results, and the development phase of the more complete ESM will have culminated with initial operational simulations. Sections~\ref{sxn:jf}--\ref{sxn:cz} describe related research projects that will benefit from the ESMF. \subsubsection{Climate-Chemistry Interaction}\label{sxn:chm} Aerosol chemistry plays important roles in the chemical composition and oxidative capacity of the atmosphere \cite[e.g.,][]{TBE01,BPT03}. Recent progress in fast inorganic aerosol thermodynamic equilibrium models \cite[]{ZSS00} makes on-line aerosol nucleation, coagulation, and growth feasible in coupled models. Aerosol indirect effects on climate \cite[e.g.,][]{KaF97,PAA01} are strongly sensitive to cloud distribution and formation processes \cite[][]{ZeK971,ATS00} which are in turn sensitive to climate \cite[]{MDS03}. Our work has focused on competition between photochemical and heterogeneous chemistry forcing due to aerosols that can lead to significant non-linear chemical feedbacks in and downwind of aerosol source regions \cite[]{BiZ03,BPT03}. The UCI CTM is now a full tropospheric aerosol-chemistry model which accounts for aerosol chemistry and uptake on internally mixed dust, sea salt, sulfate, nitrate, and carbonaceous particles. We are poised to examine links between climate and chemistry modes \cite[][]{Pra96,YuM99}, air quality, and the distribution and direct and indirect forcing of aerosols and greenhouse gases \cite[]{PrE01,YZS01,CRE02}. \subsubsection{Coupled Ocean-Land-Atmosphere Biogeochemistry}\label{sxn:bgc} ESS researchers are poised to address key questions concerning the roles of biogeochemically significant elements in ecosystem and climate change. Deposition of aeolian mineral dust is thought to provide limiting nutrients such as soluble iron to remote ecosystems such as the High Nutrient Low Chlorophyll (HNLC) regions of the Southern Oceans \cite[][]{MaF88,Mar90}. Our work has addressed preliminary questions concerning mineral dust transport and distributions \cite[][]{ZBN03,MZL02,ZNT03}, the impact of atmospheric iron deposition on ocean biogeochemical cycling \cite[][]{MDG02}, and aerosol effects on atmospheric chemistry \cite[][]{BiZ03}. Future work will examine important land-ocean links by coupling models of atmospheric aerosol transport and riverine transport with a global ocean circulation/biogeochemistry model. We seek to quantify the roles of atmospheric and riverine inputs of Fe, Si, N, and~P in driving ocean biogeochemistry and air-sea carbon dioxide flux at regional to global spatial scales. UCI is uniquely suited to the development and coupling of the models that simulate these important biogeochemical cycles. \subsubsection{Global Air-Sea Interaction and Regional Hydrological Cycle}\label{sxn:rgn} The regional hydrological cycle in the North American is influenced locally by land surface conditions and remotely by ocean conditions (sea surface temperatures---SSTs) associated with El Ni\~no-Southern Oscillation (ENSO) \cite[][]{HMY98,FaY03}. Both the land surface and ocean possess long-term memory and are capable of prolonging the useful lead time for predictions of North American precipitation beyond the limit imposed by atmospheric internal variability. A better understanding of the role of ocean and land-surface memory in determining interannual variations in North American precipitation is essential for improving extended-range forecasts and water resources planning. The land surface memory is not independent of the ocean memory. Global-scale atmosphere-ocean interactions in the Tropical Pacific can alter the paths of the Pacific storm track and lead to land surface anomalies in snow cover, soil moisture and surface vegetation in the western United States. With the proposed ESMF, we will integrate the existing coupled atmosphere-ocean GCM (CGCM), land surface model, and regional-scale hydrological model at UCI to 1)~assess the individual contributions of ocean and land-surface memory and their inter-dependence to the hydrological cycle in North America; and 2)~downscale global ENSO predictions for regional-scale hydrology forecasts in the North America. For task~(1), series of long-term (on the order of 100~years) experiments will be performed with the CGCM interactively coupled with/without the land surface model. For task~(2), the regional hydrological model will be nested within the CGCM to downscale its extended-range ENSO predictions for ensemble seasonal forecasts of regional hydrological cycle in North America. \csznote{ \subsubsection{Air-sea Exchanges Along the Californian Coast}\label{sxn:asx} Coastal oceans are dynamically distinct from the rest of oceans due to their unique air-sea characteristics and are recognized to have profound impacts on human society \cite[][]{BBC92}. The California coastal system is characterized by a persistent marine atmospheric boundary layer (MABL). The evolution of the MABL in this region is strongly controlled by the interactions between upwelling in the coastal ocean and the vertical fluxes of mass, heat, and biological materials in the atmosphere. Those air-sea exchanges are important to regional, perhaps global, and biological balances \cite[]{Wal91} and to the variations of tropical Pacific climate system \cite[]{YuM99}. A better understanding of this complex and small-scale coastal climate system will improve coastal hazard prediction and air pollution simulation. The proposed ESMF will allow us to obtain such an understanding using high-resolution satellite observations and coupled coastal ocean-atmosphere model simulations. } % end csznote \subsubsection{Famiglietti Group}\label{sxn:jf} Our group addresses the hydrologic cycle at regional and global scales and its interaction within the climate system. An ESMF would provide the computational resources to explore links between the water cycle and other Earth system components that are at the forefront of climate and hydrologic research. Given the diverse group of faculty participants in this proposal, and in the ESS department as well, the ESMF would enable frontier analyses and simulations that are simply not possible at other institutions. Active and planned projects that fall into this category include 1)~regional, high-resolution modeling of coupled ecologic-hydrologic change in the Sierra Nevadas in response to fire suppression and climate change; 2)~coupled model simulations of the role of terrestrial water in land-ocean-atmosphere interaction, including coupled biogeochemical process; and 3)~development of a high resolution global river network model that can assimilate satellite and ground-based data for improved climate and hydrologic prediction. These types of studies typically involve processing large amounts of satellite data, running higher resolution and longer time-scale climate simulations, and analysis of large volumes of model output. The computational resources of the ESMF would significantly improve our capabilities to conduct such cutting-edge studies. \subsubsection{Magnusdottir Group}\label{sxn:gm} Extra-tropical low-frequency variability in the atmosphere (beyond the two weeks that are considered predictable) exhibits strongly preferred spatial scales with almost no preferred time scales. Superposed on this ``natural variability'' are strong trends in certain climatic fields over the last couple of decades. Our research group tries to understand the forcing of the unprecedented trends. To accomplish this, the natural variability must be understood. In atmospheric general circulation models we examine forcing from below, in terms of SST and sea-ice anomalies, from above or the stratospheric polar vortex, and from within or internal forcing, which is substantial in the nonlinear chaotic system \cite[]{DMS03}. Even while restricting ourselves to the atmosphere, we have been limited in computational resources. The ESMF would allow us to examine various forcing mechanisms in unison, and to do coupled atmosphere/ocean climate runs. We would process both high temporal resolution reanalysis data derived from observations, and model output to produce eddy statistics, which is essential in any regional study such as one focusing on winter rain in Southern California under different climate scenarios. \subsubsection{Moore Group}\label{sxn:km} The research of our group focuses on biogeochemical cycling in the oceans and the linkages between ocean, land, and atmosphere. We developed a state of the art marine ecosystem model that includes explicit iron cycling and several key phytoplankton functional groups \cite[][]{MDG02}. Ongoing efforts seek to quantify the roles of these functional groups in elemental cycling in the oceans and to determine the influence of atmospheric and riverine inputs on ocean biogeochemistry. Coupled simulations of ocean circulation, ecosystem dynamics, and biogeochemical cycling have recently been developed in the context of the NCAR CCSM POP ocean model \cite[]{DLM03,MDL03}. Future research will involve coupling this global ocean biogeochemistry model with models of atmospheric and riverine transport from the land surface to the oceans for important biogeochemical elements (i.e. Iron, Silicon, Nitrogen\ldots). The model will also be used in coupled simulations of the global carbon cycle and climate change over decadal to century timescales. A second research track involves the analysis of oceanic remote sensing data from multiple sensors to study physical-biological interactions at regional to global spatial scales \cite[]{MoA00}. Both of these research foci would benefit greatly from the proposed ESMF. Currently ocean simulations must be done off-site and only small portions of the model output can be analyzed locally. \subsubsection{Prather Group}\label{sxn:mp} The demands for greatly expanded computational capability in atmospheric chemistry come from several research foci; two are specific to developing and improving the chemistry models; and the other to coupling across the Earth system. For one, the basic development of chemistry-transport models (CTMs) now depends on very high resolution models for comparison with detailed campaign measurements. UCI participated in the NASA TRACE-P campaign that collected a massive set of trace gas and aerosol measurements to study the export of ozone and other pollutants from east Asia \cite[][]{WSP03}. Use of this data set for model validation and testing requires the CTM to be run globally at a resolution of 1.8\dgr\ latitude or better, for more than forty chemical species, and for a wide range of sensitivity tests to evaluate the uncertainty in emissions. A second focus is chemical-mode studies: it is becoming clear that perturbations to global atmospheric chemistry must be studied in terms of the long-lived perturbations generated by short-lived species \cite[][]{Pra96,WiP00,OHZ00}. This requires many sensitivity and perturbation calculations made at modest resolution ($128 \times 64 \times 24$) but for about a decade to identify the modes. A major computational need within these mode studies is to run the CTM in a linearized mode to generate the necessary sensitivity functions in order to derive a reduced dimension model. Such calculations require about 100 effective years to be calculated per single model year. The third focus involves the coupling of atmospheric chemistry with the physical climate and biogeochemical models. In this case, we anticipate that a modest resolution CTM needs to be run in parallel (with hourly data exchange) with the atmospheric GCM and the terrestrial biosphere model. \csznote{ \subsubsection{Primeau Group}\label{sxn:fp} Our research is aimed at understanding the dynamics that generate low-frequency variability in the ocean and the coupled ocean-atmosphere system. Our approach is to formulate idealized models and analyze their dynamics by applying the numerical bifurcation techniques and dynamical systems theory. One of our research goals is to apply the understanding gained from the idealized studies to more realistic models. For example, it has been argued by \cite{Liu03} that the period planetary basin modes \cite[][]{CeP01,Pri02} determines the dynamical memory of the tropical ocean. Since mid- and high latitude processes gives these modes a decadal period they provide a mechanism for the extra-tropics to modulate ENSO on a decadal time-scale. Such a process would be absent in models with a limited latitude extent. We thus plan to collaborate with Yu's group to investigate this possibility with the UCLA CGCM \cite[][]{YuM011}. Access to the proposed ESMF would greatly facilitate this collaboration by making it possible to quickly test different model configurations with and without high latitude basins. } % end csznote \subsubsection{Yu Group}\label{sxn:jy} Our research focuses on understanding and predicting climate variations in the coupled atmosphere-ocean system, ranging from global-scale ENSO \cite[]{YuM011,YMM02}; to regional-scale monsoon rainfall cycle \cite[]{FaY03}. Our investigations rely greatly on long-term simulations performed with state-of-the-art, coupled atmosphere-ocean general circulation models (CGCMs) \cite[][]{YuM012} that have global coverage and high horizontal resolutions. High-performance computational facility is essential to those model simulations. With the ESMF, we will address the following research issues: 1.~Mechanisms of decadal ENSO modulation, with a focus on the inter-basin interactions between the tropical Pacific and Indian Oceans. 2.~Contributions of ocean and land-surface memory and their inter-dependence to the year-to-year variations of summer precipitation in North America. 3.~Air-sea exchanges in the coastal climate system of California. Atmospheric GCMs coupled with global oceanic models, coastal oceanic models, and land surface models with be used for this research. \subsubsection{Zender Group}\label{sxn:cz} Our research centers on predicting the radiative, dynamic, chemical, and biogeochemical interactions of natural and anthropogenic aerosols and trace gases on global scales. We create, maintain, and supply the research community an open source global mineral dust aerosol model used by UCI, NCAR, UCSB, U.~Oslo, GFDL, and GSFC to study the radiative and biogeochemical interactions of mineral dust and climate. Since this work demands simulations of complex radiative-dynamical interactions over global domains, nearly all of our research and publications depends on supercomputer simulations. The research problems that our group will attack with the ESMF in the next three years are 1.~Past, Present, and Future changes in climate and chemical composition due to direct and indirect radiative, chemical, and biogeochemical forcing of mineral dust aerosol. 2.~Assessment of the relative magnitudes of global wind and riverine erosion to long range transport of terrigenic biogeochemically significant minerals. 3.~Quantification of the roles of geomorphology, surface hydrology, mesoscale dynamics in making semi-arid landscapes vulnerable to wind erosion. Ensemble decadal offline transport and coupled global climate model numerical simulations to are required to advance research in all of these areas. The computational resources of the ESMF would allow us to discretize important aerosol radiative processes at finer temporal resolution and thus substantially reduce temporal aliasing errors in our simulations. \subsection{Integrating the ESMF with other UCI activities}\label{sxn:ntg} A major component of improving and testing models that simulate the Earth System is (1)~the inclusion of the best understanding of physical, chemical and biological controls on surface fluxes of energy, water, and nutrients; and (2)~testing of model predictions with appropriately scaled field measurements. Many UCI researchers make laboratory and field measurements of processes simulated by an ESM, and all of these participate in the UCI Institute of Geophysics and Planetary Physics (IGPP) branch. The UCI IGPP strongly endorses the founding of the ESMF (see attached letter of support from S.~Trumbore) since the ESMF effort will complement IGPP process studies and production of comparison data sets. Isotopic tracers are particularly useful in studying the global hydrological and carbon cycles because they integrate processes across a number of scales. They are a major emphasis of measurement programs at the UCI IGPP. The distribution of trace gases in the atmosphere as predicted by tracer transport models depends critically on the sources and sinks of those gases or their precursors at Earth's surface. Transfers of trace gases, energy and water at the land-air and ocean-air interface are controlled by complex interactions of atmospheric turbulence, boundary-layer development, and biological and chemical modification of air in contact with the surface. Major advances in modeling earth surface fluxes will continue to come from insight into these processes gleaned from direct eddy covariance observations of exchange (IGPP researchers Friehe, Goulden, Saltzman), in studying the factors controlling the rate of production and consumption of specific trace gases at the land and ocean surfaces (Cicerone, Druffel, Famiglietti, Goulden, Reeburgh, Saltzman, Tyler, Trumbore), and reactive chemistry of those gases in the atmosphere (Dabdub, Finlayson-Pitts, Prather, Saltzman). ESM evaluation will be facilitated by comparison of predicted spatial and temporal distributions with observations for a suite of trace gases in the atmosphere and ocean (Blake, Saltzman) and their isotopic composition (Druffel, Saltzman, Tyler, Trumbore). Specific examples where the confluence of process investigations, and large scale observations within the IGPP at UCI can make major contributions when combined with ESMF modeling are in the areas of (1)~high latitude climate change feedbacks to terrestrial ecosystems (observations by Goulden, Trumbore and postdocs); (2)~methane distribution in the atmosphere, especially why growth rates have changed in recent decades (Reeburgh, Tyler, Blake, and Rowland); (3)~the carbon cycle (Druffel, Goulden, Prather, Southon, Trumbore); (4)~links between hydrology, trace gas fluxes, and ocean circulation (Famiglietti, Goulden, Trumbore, Primeau, Yu); and (5)~links between ocean circulation, dust inputs, and chemistry of surface waters (Druffel, Primeau, Yu, Zender). This synergy between modeling and observation programs is another unique aspect of the proposed ESMF at UCI. While the data produced by IGPP facilities (the stable isotope trace gas facility and UCI Keck Carbon Cycle AMS facility) are widely available to all researchers, the presence of the ESMF in the same institution will provide impetus for increased collaboration and planning. \csznote{ Existing UCI IGPP facilities are presently focused on observations. The stable isotope trace gas facility (Tyler) and the trace gas analysis labs of Blake and Rowland are providing high precision data sets that may be used to test models of trace gas distribution and transport. The UCI Keck Carbon Cycle AMS facility (Druffel, Southon, Trumbore) is one of the few AMS labs in the US and is the only one devoted to measuring radiocarbon in support of research in the global carbon cycle. The UCI Keck lab will help map distributions of radiocarbon in \COd\ in the atmosphere and dissolved in the oceans for comparison with physical transport and biogeochemistry models; other efforts of this facility are geared toward using radiocarbon to understand processes and timescales of \C~exchange among land, atmosphere, and ocean reservoirs. While the data produced by these facilities will be widely available to all researchers, the presence of the ESMF in the same institution will provide impetus for increased collaboration and planning. } % end csznote \subsection{Computational Requirements}\label{sxn:cmp} The computational requirements of the coupled Earth system research described above are significant. At least three of the ten most powerful national and international supercomputing centers are primarily devoted to Earth system simulation \cite[][]{Meu02}. The proposed UCI ESMF will run many of the same models heavily employed by these national and international centers. Experience has shown that the best metric of computational requirements for our research is coupled model throughput in terms of simulated years per real (wall-clock) day. This ``throughput'' is always substantially less than the raw computational speed (i.e., floating point performance) of high performance computing systems because coupled Earth system models have global communication requirements (e.g., spectral transforms) which perform better with shared memory than with massively parallel systems (e.g., Beowulf clusters). The prototypical research project using the ESMF requires decadal simulations of the coupled climate system in experimental configurations (e.g., various aerosol interactions turned ``on'' and ``off'') and a control configuration (e.g., no aerosols). The computational requirements of the subset ESMs described in Sections~\ref{sxn:chm}--\ref{sxn:rgn} are summarized in Table~\ref{tbl:xpt}. \begin{table} \begin{minipage}{\hsize} \renewcommand{\footnoterule}{\rule{\hsize}{0.0cm}\vspace{-0.0cm}} % KoD95 p. 111 \begin{center} \caption[Prototype Coupled Experiments]{\textbf{Prototype Coupled Experiments} \label{tbl:xpt}} \vspace{\cpthdrhlnskp} \begin{tabular}{ >{\raggedright}p{10.0em}<{} >{\raggedright}p{7.0em}<{} lll >{\raggedright}p{9.0em}<{} l } \hline \rule{0.0ex}{\hlntblhdrskp}% Purpose & Models\footnote{Focus of scientific research and coupling development. Other models are used in ``black-box'' mode.} & \#\footnote{Number of simulations per scientific experiment} & Yrs.\footnote{Years per simulation} & CAM\footnote{Estimated equivalent number of default CAM (T42L26 $\Delta t = 30$~min.) years.} & Collaborators & \\[0.0ex] \hline \rule{0.0ex}{\hlntblntrskp}% Interactions of climate, chemistry, and aerosol & UCI CTM, DEAD, CAM & 8 & 10 & 160 & Prather, Magnusdottir, Zender & \\[0.5ex] \hline Aeolian$+$riverine mineral-nutrient impact on Ocean BGC & DEAD, RTM, CLM, UCI OEM & 4 & 20 & 320 & Moore, Zender, Famiglietti & \\[0.5ex] \hline Interaction of tropical atmosphere-ocean with catchment hydrology & UCLA CGCM, RTM, CLM & 30 & 1 & 150 & Yu, Famiglietti & \\[0.5ex] \hline \end{tabular} \end{center} \end{minipage} \end{table} % end tbl:xpt We expect our subset ESMs to run 2--4 times slower than the uncoupled CAM, and we need to simulate 60--120~years per experiment. We would like the ESMF turnaround time to be about one week per experiment. Thus we require an ESMF throughput of 150--300 years of CAM per week. \subsection{Storage Requirements}\label{sxn:str} Storage requirements of ESMF researchers are driven by three main tasks: (1) Storage of model input (forcing) datasets; (2) Archival and analysis of model simulations; (3) Archival and analysis of observations, notably satellite-derived datasets. Accounting for these three needs, we estimate that ESMF investigators will require 30--100~TB storage within 3~years. The past decade has seen a tremendous advancement in data assimilation techniques for atmospheric and oceanic data which are used to drive models. The met fields that the UCI CTM uses to run with in off-line mode typically are more than 30~GB per year, and we anticipate that 40~years need to be stored on-line at the ESMF. The NCEP/NCAR reanalysis data represent the most reliable and complete atmospheric dataset available \cite[]{Kal96}. The entire NCEP reanalysis from 1948--2002 (T62L28, 6~hour resolution) is about 500~GB. Due to its large size, many ESMF researchers can access only specific portions of this dataset. This entire datasest will be stored on-line at the ESMF, improving our potential research while avoiding duplication. A typical atmospheric circulation simulation might run for 40~years and generate a minimum of 22~gigabytes~(GB) of data. A decadal coupled climate-chemistry-biogeochemistry scenario might generate 500~GB of data. When we compare the results of just two different such climate scenarios our raw data is already in the terabyte range. ESMF researchers currently analyze, or will soon analyze, data from the following satellite instruments: Nimbus~7 and Earth Probe TOMS, NOAA 7--14 AVHRR, SeaStar SeaWiFS, Terra and Aqua CERES and MODIS, Terra MOPITT, and Aqua AMSR-E and GRACE. These instruments collect gigabytes of processed data every year. Analysis of climate variability on interannual to decadal scales therefore can require processing 100~GB or more of data. Cross-comparisons of measurements from different instruments (such as correlating and merging optical depth estimates from several radiometers) can demand more storage than personal workstations are currently able to provide. The multi-terabyte storage facility (a RAID disk farm) we are requesting will allow us to perform integrated research on these large datasets in ways that are unavailable to us at the present time. \csznote{ The ESMF will allow us to eliminate extensive duplication of identical data sets, while providing a more reliable and efficient means to archive, access and preprocess data. For example, since SST represents a forcing of atmospheric circulations and is also an indicator of oceanic circulation, we anticipate maintaining a single archive of SST data to be shared by multiple investigators. Sharing of data is also likely to encourage more serious collaboration, such as examining how SST and winds work together to govern the coupled ocean-atmosphere system. } % end csznote \section{Research Instrumentation and Needs}\label{sxn:needs} \subsection{Current Facilities and Needs}\label{sxn:crr} Over the past few years there has been a substantial increase in the number of Earth system modeling groups at UCI, but no increase in centralized computational resources. In mid-1999 the UCI acquired a 32~CPU SGI O2000 computer ("\texttt{krein}"). The O2000 is shared by the whole School of Physical Sciences (SPS) and cannot be dedicated to ESM modeling. Even if \texttt{krein} were dedicated as an ESMF, its throughput and storage system are inadequate for our purposes by factors of about 10 and 100, respectively. Two ESS groups (Magnusdottir and Zender) use \texttt{krein} for uncoupled atmospheric simulations. Other ESS groups use their own workstations (Primeau, Prather) and/or external supercomputer facilities, chiefly GSFC (Famiglietti), NCAR (Magnusdottir, Moore, and Zender), and NPACI/SDSC (Yu). This Balkanization of our simulation environments impedes effective local collaboration. The local siting of the ESMF will allow ESS faculty, post-graduates, and students to control the computer administration and code development. \subsection{ESMF Design Requirements}\label{sxn:cnf} The scientific problems the ESMF will tackle determine the optimal topology of the computational components. Figure~\ref{fgr:esmf} shows the organization of computations, information flow, and storage in an ESM experiment. \begin{figure}[!t] \centering \includegraphics*[width=0.8\hsize]{/data/zender/prp/prp_mri/prp_mri_fgr}\vfill \caption{ ESMF conceptual diagram showing interfaces among compute nodes and storage. \label{fgr:esmf}} \end{figure} In a typical experiment, five to seven computational nodes would be dedicated to specific component models (Atmosphere, Chemistry, Ocean, Land, and Sea Ice) that run independently of the other nodes. In practice, the load-balancing node would be used for additional throughput or higher resolution a particular model of interest, or for a new sub-model, e.g., catchment hydrology or aerosol thermodynamics. It is also possible to run two or more of the faster models (e.g., land and sea ice) on one node, freeing up nodes for models, such as CAM, that scale well with MPI. For experiments where the load balancing node would not improve throughput it would remain available for interactive work or use as a backup in case another node fails. Figure~\ref{fgr:esmf} illustrates a coupling configuration where all models communicate indirectly via the flux coupler. The flux coupler handles the details of regridding fluxes among the various model resolutions using a mass and energy conserving algorithm. In the CCSM, fluxes are accumulated and then exchanged between models (nodes) about once per simulation hour to day. The bandwidth requirements for this rate of coupling are not extreme but still must be taken into account. The proposed ESMF must be modular and expandable, meaning that the simulation throughput could be increased by adding multi-CPU nodes. Figure~\ref{fgr:esmf} shows why the optimal configuration for running an Earth System Model has between five and seven computational nodes. The administrative node handles overhead associated with system utilities, queue management, and interactive work. We conclude that the optimal design for the minimal ESMF is an eight node system. Two simulations running concurrently (e.g., control and experiment) would require, on average, 14 total nodes; three simulations would require~20, etc. The cost difference between highly distributed (Beowulf) and highly shared memory realization of this configuration are significant. There we give more detail on the rationale for our decision in the attached budget justification. \csznote{ At any one time, the ESMF will be running a mix of numerical experiments computationally comparable to this prototype experiment, but running at higher resolution for shorter periods, coarser resolution for longer periods, fewer or more ensembles, and with coupled and uncoupled configurations. } % end csznote \subsection{Specific Hardware Configuration}\label{sxn:hrd} IBM Corporation, an independent and reputable provider of high-speed scientific computing environments, can meet all these design requirements. IBM suggests a 64~processor p655 Power4 system similar to, but (much) smaller than, the fastest production systems at NCAR, SDSC, and LLNL. Their quote includes eight 8-way shared memory nodes based on the upcoming 1.5~GHz Power4 chips, and GB~Ethernet interconnects for each node. The theoretical peak performance of this machine is 384~GFlops, and we expect throughput of about 50~GFlops. The inter-node bandwidth demands of the coupled ESM are met by Gigabit Ethernet switches. More expensive, proprietary IBM interconnects are available but not deemed necessary for ESMF applications. The attached quote and parts list contain complete hardware details. The ESMF must be capable of simulating 60--120~\yrxwk\ of the coupled system (Section~\ref{sxn:cmp}). IBM has converted CAM benchmarks performed on a 32-processor IBM p690 at Oak Ridge National Lab (ORNL) \cite[][]{Wor02} to estimate the CAM throughput of their proposed p655-based ESMF. The ORNL p690 system, using 32~processors, simulates about 70~\yrxwk\ of CAM (T42$\times$L26, Eulerian dynamical core). IBM estimates that the proposed ESMF, using 6~compute nodes (48~processors) will simulate at least 120~\yrxwk\ of CAM (about 180~\yrxwk\ for the SLD dynamical core). This estimate accounts for the differences in interconnect and chip speeds between the ORNL and ESMF systems. Based on our experience, 120~\yrxwk\ of CAM is equivalent to at least 60~\yrxwk\ of a typical ESMF simulation from Table~\ref{tbl:xpt}. The models in Table~\ref{tbl:mdl} perform well on these IBM systems. Most ESS modelers are already familiar with the IBM run-time environments NCAR and SDSC, and using the similarly configured ESMF will be easy. Our selection of IBM Corporation equipment for the ESMF quote is influenced by the success of other Universities and National laboratories running coupled climate system models on shared memory IBM systems. However, the optimal price/performance/maintenance system for high performance scientific computing (HPSC) solutions for climate problems changes rapidly. If this MRI is funded, we will solicit updated and competing bids for systems from other vendors (e.g., SGI, HP, Beowulf) which can demonstrate comparable simulation throughput as the quoted IBM p655 system. Selecting an external storage system is relatively easy since high quality, fault-tolerant storage systems are rapidly commoditizing. Western Scientific will provide an array of ten Tornado~F4 3.2~TB IDE-SCSI devices---a total initial capacity of 32~TB. The largest logical partition on this system is about 3~TB. This is enough archival space for any one of our experiments. Western Scientific has worked closely with IBM to ensure that the their systems systems will mesh. \section{Impact of Infrastructure Plans}\label{sxn:mpc} \subsection{Outreach}\label{sxn:out} The ESMF will build upon an ongoing collaboration with the Department of Physics and Geology at California State University, Bakersfield (CSUB) coordinated by Dr. Jorge Talamantes and PI~Zender. Faculty in the Physics and Geology Department (and possibly other departments such as Chemistry and Computer Science) at CSUB will provide mentorship for CSUB undergraduates interested in environmental science and modeling, geophysical data analysis and statistics, and HPC. These students will get computer accounts on the ESMF and advice from relevant ESMF researchers. These Cal State students have no comparable opportunity to explore research projects in Earth system science and HPC. CSUB has been designated by the Department of Education as a Minority Serving Institution, with an undergraduate Latino population of 36.6\%, and an overall minority undergraduate population of 53\% which closely mirrors the regional population. Thus our existing UCI-CSUB collaboration will evolve into a major outreach component of this proposal. The ESMF will also establish links with UCI's Center for Educational Partnerships (CFEP), the campus outreach coordinator. We will provide new opportunities for undergraduate research, teacher education, mentoring, and K--12 education in the geosciences with a perspective on treating the Earth as a coupled system. CFEP links Southern California agencies whose goals are to increase academic success in students. CFEP supports a large number of programs specifically designed for students in elementary, intermediate, and high schools. These programs assist students to become eligible or competitively eligible for UC schools. %School districts, parents, community colleges, and the UCI community %join forces through CFEP programs. %CFEP's mission is to stimulate and advance cooperative educational %relationships among UCI and California schools and colleges. Through established programs and community outreach CFEP fosters community links to improve the preparation of all students, particularly those from under-served groups, for success in higher education. One program, the UCI Summer Science Institute (SSI), is an excellent outreach to high school science teachers. The SSI offers professional development programs for teachers and additional learning opportunities or all students to ensure equal access to high quality, effective and motivating science education. ESS faculty have participated in the institute, and the funding of the ESMF would provide a focus for summer course with some hands-on experience for teachers to understand the complexities and rewards of modeling the Earth system. Another program to which the ESMF will contribute is the California State Summer School in Mathematics and Science (COSMOS), established by the state legislature in 1999. COSMOS motivates the most creative minds of the new generation of prospective scientists, engineers, and mathematicians to actively participate in the business and higher educational sectors of California. COSMOS serves California high school students in grades 9--12 who are gifted and talented in mathematics and science. The program, which offers a curriculum not traditionally offered in high school, exposes them to an intensive learning experience that will enhance their academic development and shape their educational and career goals. The curriculum was designed by UCI faculty in eight subject areas: astronomy, biological sciences, isotope and atmospheric chemistry, cognitive science, computer science, engineering, mathematics and physics. We will expand the atmospheric chemistry curriculum of COSMOS in the broader area of environmental modeling and analysis. \subsection{Campus-wide benefits}\label{sxn:igpp} The ESMF will benefit investigators who are not directly involved in a number of ways. The ESMF will make extensive data sets, pre-processing, and storage capacity readily available to students and postdocs without current access to large computational facilities. This will allow these researchers to carry out interdisciplinary studies using coupled models, or to compare in situ measurements with larger global satellite and numerical model records. Such innovative studies will likely in some case greatly expand the scope of their advisors' current work. ESS faculty plan to enhance the graduate curricula by developing a set of ESM modeling exercises to which a small portion of the ESMF resources would be dedicated. Finally, the existence of a terabyte-scale data processing and archiving facility will help build the scientific infrastructure of the ESS department. This will benefit the ESS department in future recruitment of graduate students, post-docs and faculty, and improve the competitiveness of our research proposals. Campus-wide the capability of the ESMF will draw a range of non-geoscientists---such as engineers, social and political scientists, legal scholars, and others---who will want to address a wide range of technological and societal options for the future. As the capability of the ESM develops, and we are able to evaluate future impacts of global change, the interest in merging Earth system science with the human dimensions side will become stronger and benefit many academic programs on campus. \subsection{National and International Assessments}\label{sxn:ipcc} National and international assessments are the chief mechanism for providing policy-makers with consensus scientific input on the on the past and future of Earth's climate. ESS researchers are key contributors to the IPCC and other assessments \cite[][]{PDE95,OPB97,PrS99,PrE01}. Many UCI-based models (DEAD, RTM, and OEM, see Table~\ref{tbl:mdl}) are already integrated in CCSM and will be used in NCAR's upcoming IPCC simulations for 2006. Given the strong links between ESS researchers and development and use of models contributing to national and international climate assessments, it is clear that ESMF research will influence these assessments broadly and for many years. \section{Project and Management Plans}\label{sxn:mng} The ESMF will be maintained as a multi-user resource as part of the UCI ESS Department. The ESMF Director will coordinate ESMF scientific use and promote outreach linkages. PI~Zender will serve as ESMF director. Zender has 10+ years experience with HPC Earth system modeling. Dr.~Frank~J. Wessel of the UCI Research Computing Support (RCS) group will serve as ESMF technical director. Wessel has 15~years experience in HPC procurement, use, system administration, and management. Dr.~Wessel will oversee the system administrator, coordinate vendor relations, and determine appropriate facility upgrade and maintenance strategies to ensure maximum ESMF uptime and security. The Director, technical Director, and three to five ESMF investigators will form an ESMF advisory committee responsible for facility oversight and planning. We will accommodate all reasonable requests UCI researchers who desire to use the ESMF for purposes consistent with this proposal. This includes the individual purchase of additional nodes on the ESMF using startup or project funds. \subsection{System Administration and Maintenance, First 3~Years}\label{sxn:adm} Based on experience and discussions with UCI Network and Computing Services (NACS) and the Research Computing Support (RCS) Group, we have budgeted 1/2-FTE Programmer Analyst~II for ESMF system administration and support. The system Administrator will have primary responsibility for day-to-day maintenance of the ESMF, accounts, and software upgrades. Our request includes vendors' fees which fully cover the first three years of maintenance. These maintenance costs go to the vendor and are in addition to the time spent by ESMF personnel. \subsection{Facility Maintenance After Year~3}\label{sxn:mnt} The initial purchase of the ESMF covers the first three years of software and hardware maintenance and system administration costs. After year~3 (late 2006), we will cover these costs entirely with UCI funds. We estimate the annual costs of operation as \$60,000 for the 0.5~FTE system administrator, \$45,000 for hardware maintenance, and \$25,000 for software maintenance (see attached quote from IBM). The total annual cost of operation is therefore about \$130,000. The ESMF will be the nucleus of future ESS and UCI HPC activity. The University is committed to the maintenance and support of the ESMF beyond the third year through a combination of income derived from a recharge facility, support from IGPP, possible integration into the support structure of a larger scientific HPC facility, and financial support from SPS and NACS. After three years, significant users (those consuming more than 1\% of the ESMF resources) will be charged to help defray operational costs. A similar system has been used successfully for the UCI SGI Origin 2000, purchased in 1999. The UCI IGPP has committed \$20,000 per year to the ESMF facility after year~3 (see attached letter of support from Sue Trumbore). Growth in the number of ESS modelers, compounded by the increasing complexity of their models, will strongly motivate us to maintain and enhance the ESMF. \newpage \section{References}\label{sxn:rfr} \setcounter{page}{1} % Bibliography \renewcommand\refname{} \vspace{-24.0pt} \newlength{\oldbaselineskip} \setlength{\oldbaselineskip}{\baselineskip} %\setlength{\baselineskip}{13.574pt} % 1.234 X 11pt \setlength{\baselineskip}{12.0pt} % 1.234 X 11pt \setlength{\bibsep}{4pt} % Space between natbib bibliography items \bibliographystyle{jas} \bibliography{bib} \setlength{\baselineskip}{\oldbaselineskip} % 1.851 X 11pt \newpage \section{Acronyms and Abbreviations}\label{sxn:abb} \setcounter{page}{1} %\begin{longtable}{ >{\raggedright}p{7.0em}<{} >{\raggedright}p{8.0em}<{} } \begin{longtable}{ r >{\raggedright}p{25.0em}<{} l } & \kill % NB: longtable requires caption as table entry \caption[Acronyms and Abbreviations]{\textbf{Acronyms and Abbreviations}% \label{tbl:abb}} \\ \hline\hline \rule{0.0ex}{\hlntblhdrskp}% Abbreviation & Description & \\[0.0ex] \hline \rule{0.0ex}{\hlntblntrskp}% \endfirsthead % Lines between and \endfirsthead appear at top of table \caption[]{(continued)} \\ % Set label for following pages Abbreviation & Description & \\[0.0ex] \hline \rule{0.0ex}{\hlntblntrskp}% \endhead % Previous block appears at top of every page \endlastfoot % Previous block appears at end of table AIX & IBM variant of UNIX used in ESMF & \\[0.5ex] AMSR-E & Advanced Microwave Scanning Radiometer (satellite instrument) & \\[0.5ex] AMWG & (CCSM) Atmospheric Model Working Group & \\[0.5ex] AOGCM & (Coupled) Atmosphere-Ocean General Circulation Model & \\[0.5ex] AVHRR & Advanced Very High Resolution Radiometer (satellite instrument) & \\[0.5ex] BGC & Biogeochemical & \\[0.5ex] BGCWG & (CCSM) Biogeochesmistry Working Group & \\[0.5ex] CAM & Community Atmosphere Model & \\[0.5ex] CCSM & Community Climate System Model & \\[0.5ex] CERES & Clouds and Earth's Radiant Energy System (satellite instrument) & \\[0.5ex] CFEP & Center for Educational Partnerships & \\[0.5ex] CLM & Common Land Model & \\[0.5ex] COSMOS & California State Summer School in Mathematics and Science & \\[0.5ex] CPU & Central Processing Unit & \\[0.5ex] CSIM & Community Sea Ice Model & \\[0.5ex] CSUB & California State University at Bakersfield & \\[0.5ex] CTM & Chemistry Transport Model & \\[0.5ex] CVWG & (CCSM) Climate Variability Working Group & \\[0.5ex] DEAD & Dust Entrainment And Deposition Model & \\[0.5ex] ECMWF & European Center for Medium-range Weather Forecasts & \\[0.5ex] EES & Earth and Environmental Science & \\[0.5ex] ENSO & El Ni\~no-Southern Oscillation & \\[0.5ex] ESM & Earth System Model & \\[0.5ex] ESMF & Earth System Modeling Facility & \\[0.5ex] ESS & Earth System Science (Department) & \\[0.5ex] FTE & Full-time Employee & \\[0.5ex] GB & Gigabyte & \\[0.5ex] GMI & Global Modeling Initiative & \\[0.5ex] GRACE & Gravity Recovery and Climate Experiment (satellite instrument) & \\[0.5ex] GSFC & Goddard Space Flight Center & \\[0.5ex] HNLC & High Nutrient Low Chlorophyll & \\[0.5ex] HPC & High Performance Computing & \\[0.5ex] IBM & International Business Machines & \\[0.5ex] IDE & Integrated Drive Electronics (commodity PC hard disks) & \\[0.5ex] IGPP & Institute of Geophysics and Planetary Physics & \\[0.5ex] IPCC & International Panel on Climate Change & \\[0.5ex] LMWG & (CCSM) Land Model Working Group & \\[0.5ex] MABL & Marine Atmospheric Boundary Layer & \\[0.5ex] MAE & Mechanical and Aerospace Engineering & \\[0.5ex] MODIS & Moderate Resolution Imaging Spectroradiometer (satellite instrument) & \\[0.5ex] MOPITT & Measurement of Pollution in the Troposphere (satellite instrument) & \\[0.5ex] MPI & Message Passing Interface & \\[0.5ex] NACS & Network and Computing Services & \\[0.5ex] NASA & National Aeronautic and Space Administration & \\[0.5ex] NCAR & National Center for Atmospheric Research & \\[0.5ex] NCEP & National Center for Environmental Prediction & \\[0.5ex] NOAA & National Oceanic and Atmospheric Administration & \\[0.5ex] O2000 & Model of SGI shared memory supercomputer at UCI (krein) & \\[0.5ex] OEM & Ocean Ecosystem Model & \\[0.5ex] PI & Principle Investigator & \\[0.5ex] POP & Parallel Ocean Program & \\[0.5ex] POP$+$OEM & POP with Ocean Ecosystem Model embedded & \\[0.5ex] RAID & Redundant Array of Independent Disks & \\[0.5ex] RAM & Random Access Memory & \\[0.5ex] RCS & Research Computing Services & \\[0.5ex] RTM & River Transport Model & \\[0.5ex] SAN & Storage Area Network & \\[0.5ex] SCSI & Small Computer Systems Interface (communications protocol for high-performance hard disks) & \\[0.5ex] SDSC & San Diego Supercomputer Center & \\[0.5ex] SGI & Silicon Graphics Incorporated & \\[0.5ex] SLD & Semi-Lagrangian Dynamics & \\[0.5ex] SMP & Shared Memory Protocol & \\[0.5ex] SPS & UCI School of Physical Sciences & \\[0.5ex] SSI & UCI Summer Science Institute & \\[0.5ex] SST & Sea Surface Temperature & \\[0.5ex] TB & Terabyte & \\[0.5ex] TOMS & Total Ozone Mapping Spectrometer (satellite instrument) & \\[0.5ex] U160 & SCSI variety emulated by IDE RAID farm & \\[0.5ex] UCI & University of California at Irvine & \\[0.5ex] UNIX & Generic name for operating system & \\[0.5ex] VCR & UCI Office of the Vice Chancellor for Research & \\[0.5ex] WG & Working Group & \\[0.5ex] p690 & IBM computer proposed for ESMF & \\[0.5ex] \end{longtable} % end tbl:abb \newpage \section{Budget Justification}\label{sxn:bdg} \setcounter{page}{1} \subsection{Choice of Shared vs. Distributed Architecture}\label{sxn:rch} In defining our requirements we considered shared-memory and distributed-memory options. Our evaluation is based on our experience with these two types of systems, correspondence with other administrators and users in the community, and discussions with representatives of the major hardware vendors. Throughput, not cost, was the primary factor in our decision. However, given the cost differential to UCI and NSF of these two types of system, we felt it was important to describe our rationale in more detail. Highly distributed memory solutions (e.g., Beowulf clusters) are not optimal for the ESMF for two reasons. First, not all problems scale well in massively parallel environments. A prime example is throughput of atmospheric chemistry models which perform best in highly-cached, shared memory environments. Since our design requirement is driven by throughput of the coupled model system, it is important to choose a hardware configuration that optimizes the performance of the least parallelize-able component. Second, installation, assembly, and maintenance of Beowulf systems is a more risky and time-intensive proposition than would be a single vendor system. Peers at comparable institutions (U.~Washington, Colorado State University) warn us that keeping Beowulf solutions running smoothly is very complicated and requires highly skilled in-house expertise. Moreover, batch queueing software for Beowulf systems is not yet mature (this will change with time). At the other extreme, massively shared memory solutions (such as SGI Origin series) would also have problems serving as the ESMF. First, the caching on such machines is not as advanced as caching on architectures with multiple CPUs per chip, such as the Power4 CPU (this may change with time). Second, the financial viability of SGI, a main vendor, is uncertain. The above problems with the extremely distributed or shared memory architectures points to an optimal solutions between these two extremes. The optimal solution appears to be a highly cached, shared-memory computing environment for each component, with more relaxed requirements for parallel computation and communication between the different ESM components, all with a mature software base. \subsection{Cost of Proposed Facility}\label{sxn:cst} The hardware, software, and maintenance cost of the proposed ESMF is about \$1,073,000. System administration costs an additional \$61,569 per year. The total three-year cost of the ESMF, including hardware, software, maintenance one half-time FTE system administrator is about \$1,252,696. NSF is asked to provide 70\%, or \$876,887 and UCI will provide the remaining 30\%, or \$375,809. The School of Physical Sciences and the Office of the Vice Chancellor for Research will evenly split the burden of these matching funds, contributing 15\% each (see attached letter from VCR Dr.~Parker). Table~\ref{tbl:bdg} summarizes the costs and cost-sharing of the three components of the ESMF: computational, storage, and system administration. \begin{table}[!ht] \centering % \centering uses less vertical space than center-environment \begin{minipage}{\hsize} \renewcommand{\footnoterule}{\rule{\hsize}{0.0cm}\vspace{-0.0cm}} % KoD95 p. 111 \caption[Summary Budget of Earth System Modeling Facility]{\textbf{Summary Budget of Earth System Modeling Facility}% \label{tbl:bdg}} \vspace{\cpthdrhlnskp} \begin{tabular}{r >{\raggedright}p{15.0em}<{} llll} \hline \rule{0.0ex}{\hlntblhdrskp}% Purpose & Specific Request & Cost\footnote{Cost includes 7.75\% CA sales tax on goods and services and 51.5\% UCI institutional overhead on salary and benefits (52.5\% effective 7/01/05).} & Percent\footnote{Percent of total ESMF cost.} & NSF\footnote{Requested NSF share of component. The total NSF share is about \$876,887.} & UCI\footnote{UCI share is detailed as $X/Y$ where the sources of $X$ and $Y$ are the UCI Office of the Vice Chancellor for Research (VCR) and the UCI School of Physical Sciences (SPS), respectively. The total UCI share is \$375,809.} \\ & & \$ & \% & \% & \% \\ \hline \rule{0.0ex}{\hlntblntrskp}% Computational & IBM p655 System, eight nodes each with eight 1.5~GHzPower4 CPUs, 32~GB~RAM, GB~Ethernet switches & 960,240 & 77 & 82 & 15/3 \\ Storage & 32~TB RAID disk farm comprising ten Western Scientific Tornado~F4 3.2~TB IDE-SCSI devices & 107,750 & 9 & 82 & 15/3 \\ Administration & Three Years $1/2$ FTE Programmer-Analyst~II & 184,706 & 15 & 0 & 0/100 \\ \hline \hline Total & & 1,252,696 & 100 & 70 & 15/15 \\ \hline \end{tabular} \end{minipage} \end{table} % end tbl:bdg \subsection{Computational}\label{sxn:bdg_cmp} The total cost of the requested IBM $8 \times 8$ p655 System is \$960,240. This price includes all shipping, installation, and three years of $24 \times 7 \times 365$ maintenance for all hardware and software required to run the computational core of the ESMF. The attached itemized quote from IBM shows that over 81\% of this cost goes directly to hardware. The remainder, 19\%, goes to system hardware and software maintenance. IBM has included a number of educational discounts in this price. Most notably, IBM has applied has discounted the retail price of this system in recognition of the use of the ESMF facility by a University for research purposes. IBM provides nearly all of the required software for the ESMF free of charge a part of its campus research program. We are only directly charged for the annual software support and maintenance costs. This software maintenance fee is reasonable since the ESMF requires a complex suite of software and hardware to achieve the desired throughput. The maintenance fee provides the ESMF with access to IBM technical support, software upgrades, and bugfixes for the duration of the proposal. Our coupled modeling requires an advanced UNIX operating system, Fortran and C~compilers and debuggers, shared and distributed memory APIs, and batch queuing software. Our request includes standards-compliant versions of these resources. IBM includes all this software (AIX, Visual Age compilers, OpenMP, MPI, LoadLeveler, respectively) in the attached quote. \csznote{ The \url{http://www.scd.ucar.edu/computers/bluesky}{Bluesky} \footnote{ Bluesky is an IBM~p690 system with 1152~1.3~GHz Power4 CPUs with 2~\GBxCPU of~RAM. The CPUs are configured as $92 \times 8$-way and $13 \times 32$-way nodes. }% end bluesky footnote supercomputer installed at NCAR in November, 2002, is very similar to the ESMF and is driven by many of the same design requirements. The proposed ESMF has approximately $1/20$ the number of processors as Bluesky. Our coupled modeling requires an advanced UNIX operating system, Fortran and C~compilers and debuggers, shared and distributed memory APIs, and batch queuing software. Our request includes standards-compliant versions of these resources. IBM includes all this software (AIX, Visual Age compilers, OpenMP, MPI, LoadLeveler, respectively) in the attached quote. } % end csznote \subsection{Storage}\label{sxn:bdg_str} Our near storage requirements of 30--100~TB are driven by the massive data production and analysis demands of experiments such as those shown in Table~\ref{tbl:xpt}. Experience shows that computer vendors such as IBM charge a premium for integrating fast SCSI disks with their supercomputers. Thus we designed the external RAID disk farm to be cleanly segregable from the ESMF computational core. This allows us to consider less expensive IDE-SCSI RAID disks. The attached quote from Western Scientific is \$108,000 for 32~TB, or about \$3,000~\xTB, at present day (January 2003) prices. The quoted system is an assembly of ten independent RAID devices known as Tornado~F4s. Each Tornado melds sixteen 200~GB IDE drives into a single logical entity, with an unformatted capacity of 3.2~TB. IBM currently charges much more for the same amount (32~TB) of pure SCSI-based RAID. Since the disk I/O of climate models is sporadic and not rate-limiting, we are confident that the any performance penalty associated with IDE drives will be small. We note that disk storage costs change rapidly. Costs may fall to \$2,000~\xTB\ by late summer,~2003. This would allow us to purchase 50~TB for approximately the same amount. \subsection{System Administration}\label{sxn:bdg_sys} Salaries and Wages: The cost of the 1/2-FTE Programmer Analyst~II for system administration of the ESMF is estimated using UCI's academic and staff salary scales. The Programmer Analyst is not on the budget pages submitted to NSF because, as shown in Table~\ref{tbl:bdg}, those costs are being absorbed in the Cost Sharing portion which will be paid by UCI. A 3\% cost of living increase was applied each year in this proposal. Employee Benefits: Fringe Benefits were estimated using the composite rates agreed upon by the University of California Office of the President and the DHHS Audit Agency, the Cognizant Audit Agency for the University of California. Benefit rates used in this proposal are Staff:~22\%. Indirect Costs: Facilities and Administrative costs were estimated in accordance with UCI's approved indirect cost rate agreement. The indirect cost rate of 51.5\% of MTDC through 6/30/05 and 52.5\% of MTDC effective 7/1/05, was based upon the nature and location of the work proposed. UCI's indirect cost rate agreement was approved by DHHS, the Federal Cognizant Audit Agency for UCI on 12/5/01. These indirect costs are also being absorbed in the Cost Sharing portion paid by the Dean of Physical Sciences and the Office of Research Administration, as shown in Table~\ref{tbl:bdg}. \clearpage %\setcounter{section}{1} \section{Research Facilities}\label{sxn:fcl} \setcounter{page}{1} A research facilities statement is not required for MRIs. However, the requested computer system is a complex piece of equipment which requires a sophisticated mechanical infrastructure for proper operation. We felt it best to demonstrate that UCI will maintain the ESMF in a high-quality, cool, secure environment. The ESMF will be physically located in the UCI Network and Computing Services (NACS) machine room. This 4000~square foot environmentally controlled facility is linked to the grid by a 0.4~MW electrical power connection. This is more than adequate for the requested two-rack (one computational, one storage) system. Should it be necessary the electrical power connection will be upgraded to meet additional power needs. The machine room also has a 50~kW un-interruptible power supply (UPS) that provides for unattended operation of critical systems for up to 50~minutes. The entire machine room is also connected to a diesel generator to insure continuous operation in the event of extended power outages. The machine room is staffed 24/7/365 and monitored by a security-access system. \csznote{ Bruce Morgan, Director, Sponsored Projects, Office of Research Administration, 4-2897 Grant officer: Virginia Coffin, 4-7813 Main IBM contacts are Nott and Uberti Don Nott pseries tecnical specialist server consolidation San Diego dnott@us.ibm.com Ted Uberti Pseries specialist euberti@us.ibm.com Alice Wilkinson replaced Ted Uberti IBM Rep: Hank Watari 818 539 3571 (W) 213 219 2212 (C) IBM phone conference call number: 1 800 453 7841 295108 Pokil Wong Pseries sales manager Tom Hane IBM HPC Michigan 1. Organized NCAR/SDSC bids for IBM Bluesky configuration: http://www.scd.ucar.edu/computers/bluesky IBM p690 prices: http://www-132.ibm.com/content/home/store_IBMPublicUSA/en_US/eServer/pSeries/high_end/690.html http://www-132.ibm.com/webapp/wcs/stores/servlet/TopCategoriesDisplay?storeId=442405&catalogId=-840&langId=-1 CAM performance benchmarks: http://www.csm.ornl.gov/~worley/evaluation/CAM/benchmark.html CRM performance benchmarks: http://www.csm.ornl.gov/~worley/studies/pcrm.html IDE-SCSI Conversion: http://www.linuxhardware.org/article.pl?sid=02/12/19/145230&mode=thread Disk: Western Scientific Tornado IDE-SCSI RAID: Joe Lipman (800 443-6699 x 277) joelipman@wsm.com www.wsm.com Nexsan InfiniSAN ATABeast: $13k per 13 TB Design requirements: We are looking forward to getting your p600 series configuration options for at least 250 years per week of CAM simulation. Assume 2 GB RAM per CPU default, then give us the price for the same system with 1 GB RAM. Remember that we mean 250 years _throughput_. We expect that ~10--20\% of CPUs will be devoted to administration/ interactive login/disk controls/queue management. You must account for this in your configuration. The ESMF must have adequate "fast" internal (i.e., /tmp) storage to keep the CCSM/CAM jobs running on internals disks that you provide. Include 30 TB RAID disk farm as a line item that is cleanly segregable from the ESMF computational facility. The ESMF must have adequate U160 SCSI ports to output to an SCSI-to-IDE RAID disk farm which you may or may not provide yourselves. We are getting quotes of \$3k TB-1 from other vendors and are thus allocating \$100k for the disk farm. All quotes must include installation and 3 years support and maintenance (and no hidden add-ons :). Give us whatever configurations you can by Thursday afternoon 1/2. More choices are better. We will discuss your estimates on Friday 1/3 and decide whether to pursue other vendors and/or haggle with IBM at that time. We would require firm quotes (in PDF format) from IBM by Friday Jan. 10. } % end csznote \csznote{ Table~\ref{tbl:cnf} shows the configuration of the ESMF. \begin{table} \begin{minipage}{\hsize} \renewcommand{\footnoterule}{\rule{\hsize}{0.0cm}\vspace{-0.0cm}} % KoD95 p. 111 \begin{center} \caption[Summary Configuration of Earth System Modeling Facility]{\textbf{Summary Configuration of Earth System Modeling Facility}% \label{tbl:cnf}} \vspace{\cpthdrhlnskp} \begin{tabular}{rlc} \hline \rule{0.0ex}{\hlntblhdrskp}% Item & Description \\[0.0ex] \hline \rule{0.0ex}{\hlntblntrskp}% Vendor & IBM \\[0.5ex] System & p655 UNIX Server \\[0.5ex] % CPU & 64 1.5~GHz Power4 Processors ($8 \times 8$-way nodes) \\[0.5ex] Operating System & AIX~5.1 \\[0.5ex] % RAM & 128~GB (2~GB processor$^{-1}$) \\[0.5ex] % Internal SCSI Disk & 2.2~TB (280~GB node$^{-1}$) \\[0.5ex] Vendor & Western Scientific \\[0.5ex] IDE-SCSI RAID farm & 32~TB (ten 3.2~TB enclosures of 20 160~GB IDE disks) \\[0.5ex] \hline \end{tabular} \end{center} \end{minipage} \end{table} % end tbl:cnf } % end csznote \csznote{ The second option that could potentially fit the scientific needs of the ESMF is a massively parallel Beowulf computer, built from ``off-the-shelf'' PCs. Traditionally Beowulf clusters have not been ideal for climate simulations because their shared memory capacity was inadequate and their inter-node latency was too high. However, Beowulf cluster technology is stabilizing and rapidly improving the price-performance of climate applications. We feel it would be a mistake not to re-consider the price-performance benefits of selecting a Beowulf cluster at the time of the award. The Beowulf would be assembled by independent consultants based on the best bid. Aspen Systems (\url{www.aspsys.com}), and Scyld Computing (\url{www.scyld.com}) are two such consultants that have extensive experience assembling Beowulf clusters for weather forecasting for Universities and for National Laboratories. The two options are comparable in price, but are based on different hardware and software technologies. \begin{table} \begin{minipage}{\hsize} \renewcommand{\footnoterule}{\rule{\hsize}{0.0cm}\vspace{-0.0cm}} % KoD95 p. 111 \begin{center} \caption{Summary of Earth System Modeling Facility configuration options \label{tbl:bdg}} \vspace{\cpthdrhlnskp} \begin{tabular}{rlccl} \hline \rule{0.0ex}{\hlntblhdrskp}% {\bf Component} & {\bf Option~A} & {\bf Option~B} \\ \hline \rule{0.0ex}{\hlntblntrskp}% Vendor & IBM & Beowulf \\ CPU & 8-way Power4 & Xeon/Athlon \\ \# CPUs & 24 & 100 \\ RAM (GB) & 16 & 100 \\ IDE RAID storage (TB) & 100 & 100 \\ & & \\ \multicolumn{3}{c}{Cost in \$1000's} \\[0.0ex] Computer Equipment and Installation & 400 & 300 \\ % 3 Years Maintenance & ---\footnote{3 years $24 \times 7 \times 365$ % maintenance is bundled into the equipment estimate provided by IBM} & 50 \\ % 3 Years $1/2$ FTE System Admin. & 60 & 60 \\ IDE RAID disk farm & 100 & 100 & 100 \\ Total w/ 3 years maintenance & 593 & 593 & 593 \\ \hline \end{tabular} \end{center} \end{minipage} \end{table} % end tbl:bdg The Beowulf option requires licensing a Cluster Development Kit from Portland Group International (PGI), Fujitsu, or Intel. This software kit is required to take advantage of the OpenMP (shared memory) and MPI (distributed memory) capabilities of the Beowulf cluster. } % end csznote \csznote{ Note from Stern 20030108: NSF will pay careful attention to how we will to maintain and support the equipment after the third year. Since it is not be clear to me exactly what will be needed to maintain and support the new equipment in 2006 and what other high performance related equipment will be on campus in 2006, it is difficult to pin down exact dollars. Thus, I recommend that the proposal state that the University is committed to the maintenance and support of the facility beyond the third year through a combination of income derived from a recharge facility, support from IGPP, possible integration into the support structure of a larger high performance scientific computing facility, and financial support from the School of Physical Sciences and NACS. } % end csznote \csznote{ Impact of funding cuts on proposal: The negotiated award for this proposal is less than 90\% of the originally requested award, so we must provide a statement on the impact of the request-award difference on the project. The negotiated award reflects the drop of about 13\% in the vendors' bids for the requested computational facility that has occurred since the proposal was originally submitted. In other words, equipment of equivalent computational throughput to the originally requested system can be obtained with the negotiated budget. Therefore the grant awarded will not negatively impact the scientific research described in the proposal. } % end csznote \end{document}