CISL best practice: Use HPSS for long-term data storage

Store only the data that you need long-term in the HPSS tape archive. Rather than routinely copying output to HPSS right after completing simulation runs, for example, use your /glade/scratch space for analysis and save the data to HPSS only after post-processing. Using the tape archive only for long-term storage helps conserve your storage allocation and allows the HPSS system to run more efficiently for everyone.

With Yellowstone's 5-PB /glade/scratch file space, 10-TB default quota, and 90-day retention period, CISL encourages users to evaluate and simplify their workflows by cutting out intermediate, temporary data movement steps. 

See CISL best practices for other ways to make the best use of your computing and storage allocations.

HPSS outage, midnight to midnight Tuesday, September 25

Starting shortly before midnight (11:45 pm) on Monday, September 24, 2012, the HPSS system will be down for 24 hours to transition the metadata server to NWSC.

We anticipate having HPSS back up within 24 hours. Except for the downtime, the change will be transparent to users.

Call for Papers: 3rd International Workshop on Advances in High-Performance Computational Earth Sciences

The 3rd International Workshop on Advances in High-Performance Computational Earth Sciences: Applications and Frameworks (IHPCES) has an open call for papers, whose deadline is scheduled for January 15, 2013. NCAR researchers and users can participate in this workshop by submitting a paper reflecting your current research in the area of computational science. IHPCES 2013 is being held in conjunction with the 13th International Conference on Computational Science (ICCS2013) "Computation at the Frontiers of Science", and will be held in Barcelona, Spain, June 5-7, 2013.

The 3rd IHPCES workshop provides a forum for presentation and discussion of state-of-the-art research in high-performance computational earth sciences. Emphasis will be on novel advanced high-performance computational algorithms, formulations and simulations, as well as the related issues for computational environments and infrastructure for development of high-performance computational earth sciences. The workshop facilitates communication between earth scientists, applied mathematicians, computational and computer scientists and presents a unique opportunity for them to exchange advanced knowledge, insights and science discoveries. With the imminent arrival of the exascale era, strong multidisciplinary collaborations between these diverse scientific groups are critical for the successful development of high-performance computational earth sciences applications. Presentations and audience representation from the broad earth sciences community is strongly encouraged. Contributions are solicited in (but not restricted to) the following areas:

* Large-scale simulations using modern high-end supercomputers in earth sciences, such as atmospheric science, ocean science, solid earth science, and space & planetary science, as well as multi-physics simulations.

* Advanced numerical methods for computational earth sciences, such as FEM, FDM, FVM, BEM/BIEM, Mesh-Free method, Particle method, and etc.

* Numerical algorithms and parallel programming models for computational earth sciences.

* Optimization and reengineering of applications for multi/many-core processors and accelerators.

* Strategy, implementation and applications of pre/post processing and handling of large-scale data sets for computational earth sciences, such as parallel visualization, parallel mesh generation, I/O, data mining and etc.

* Frameworks and tools for development of codes for computational earth sciences on peta/exascale systems.

Authors are invited to submit manuscripts reporting original, unpublished research and recent developments/theoretical considerations in Computational Earth Sciences and related issues by January 15, 2013. Accepted papers will be printed in the conference proceedings of ICCS 2013 published by Elsevier Science in the open-access Procedia Computer Science series. After the conference, selected papers may be invited for a special issue of some major journals, such as Springer's Lecture Notes in Earth Sciences (LNES).

The workshop announcement is posted at:

Conveners: Yifeng Cui, University of California at San Diego ( Xing Cai, Simula Research Laboratory, Norway (

Scheduled Maintenance for the Week of October 8 - October 12

Bluefire: Downtime Tuesday, October 9, 6:00am-1:00pm

HPSS: Downtime Tuesday, October 9, 7:00am-9:00am

No Scheduled Downtime: DAV, GLADE, Lynx

Latest on Yellowstone and its Acceptance Testing Period

As of 8 a.m., Sept. 4, Yellowstone officially entered its acceptance test period. While this represents a major milestone, the first week was not without its challenges. The primary issue was assuring that system state can be preserved on the diskless nodes across a cold start, stabilizing the FDR InfiniBand interconnect and reducing interference in the communications as the workload approached the full 4,500-node capacity of Yellowstone.

IBM and Mellanox have resolved several sources of problems, and since 04:15 Sept. 12, CISL staff have been running the full system workload comprised of six different benchmark codes with a 99.94% success rate. IBM benchmark runs have shown compute performance very close to the expected 28.9 "Bluefire-equivalents," and GLADE benchmark performance has also shown better than 80 GB/s for reads and better than 90 GB/s for writes.

The ATP workload testing will continue for the coming weeks, and IBM and Mellanox will continue to troubleshoot problem nodes, cables, and software configurations to improve the stability and performance of the system. While it is still too early to identify a specific date for Yellowstone to pass acceptance testing, CISL remains confident that early October is the likely timeframe.

CISL best practice: Organize files for smooth transition

The upcoming transition to the new Yellowstone environment is an opportunity to implement CISL best practices if you haven’t already done so. Consider this one in particular as you prepare for the new system:

Organize your files and keep them that way. Arrange them in same-purpose trees, for example. Say you have 20 TB of Mount Pinatubo volcanic aerosols data. Keep the files in a subdirectory such as /glade/home/username/pinatubo rather than scattered among unrelated files or in multiple directories. Specialized trees are easier to share with other users and to transfer to other users or projects as necessary.

Getting organized will also help you transition smoothly to the new Yellowstone system and bring along only the files you need. Once Yellowstone is generally available, users will have an opportunity to migrate essential files to the new GLADE environment as described on our Transition from Bluefire page.

Yellowstone deployment efforts continue to pursue performance improvements

The Yellowstone timeline has continued to slip despite long hours put in by CISL, IBM and Mellanox staff. At this writing, the most optimistic timeline has the three-week acceptance test period beginning late this week (the tail end of August), which pushes first user access at least to late September.

While the compute and storage hardware looks good and has demonstrated itself to be more stable than anticipated, with little “infant mortality” observed so far, IBM and Mellanox are continuing to address challenges to achieving the expected performance of 90 GB/s between the compute and storage systems.

The performance tuning involves complex hardware, software, and firmware interactions among the more than 4,500 compute nodes on Yellowstone; the 4,500 disk drives, 76 disk controllers, and 20 GPFS servers of the GLADE resource; and the InfiniBand interconnect comprised of nine core switches, 250 leaf switches, and more than 9,500 copper and fibre cables.

CISL is monitoring the deployment process closely, with ongoing interactions with and updates from the IBM team. Given the extent of the delays thus far, CISL is watching the system's stability and performance results and looking for the earliest possible opportunity to move into acceptance testing. If IBM's performance results are not quite at the promised levels, CISL may elect to pursue acceptance despite the shortfall and discuss alternate methods of later achieving performance targets with IBM.

When the Yellowstone timeline solidifies, CISL will also re-evaluate the schedule for Bluefire. With any Yellowstone delays, users can expect Bluefire’s decommissioning date to be extended accordingly.

Object-Oriented Programming in Fortran

Date and Time: 
Aug 8th - 10th, 2012, 9:00-12:00, 13:00-16:00
ML Damon Room
Damian Rouson, Karla Morris, and Salvatore Filippone

Workshop Introduction:

Modern Fortran explicitly supports object-oriented programming (OOP). OOP aims to increase a program's maintainability in part by reducing cross-module data dependencies and to increase a program's reusability in part by providing for extensible derived types. Emerging compiler support for Fortran 2003/2008 inspires a more modern program design and implementation style. This course provides the requisite skills.

  • Day 1 introduces OOP in Fortran 2003.
  • Day 2 introduces patterns of best practice in program organization.
  • Day 3 explores several paths toward parallel OOP, including parallel numerical libraries and Fortran 2008 coarray programming.

Examples will utilize introductory-level numerical algorithms from linear algebra and differential equations inspired by multiphysics modeling that is coupled field problems common to many interdisciplinary, engineering, and physical science simulations.
Additional information, including a syllabus and schedule from the NERSC URL for previous offerings of the course can be found here:


If you are interested in purchasing the reference book Scientific Software Design - The Object-Oriented Way written by Damian Rouson(Sandia National Laboratories), Jim Xia(IBM Canada Lab, Markham), and Xiaofeng Xu(General Motors Corp.), here is a discount coupon.


Please register for the workshop from the link here.

Speaker Description: 

Damian is the manager of the Reacting Flow Research Department and a Visiting Assistant Professor at the University of Cyprus. His research interests focus on scalable scientific software design. He has recently written a book on the subject, Scientific Software Design: The Object Oriented Way . He is also the software architect of the object oriented Fortran interface to Trilinos, ForTrilinos. ForTrilinos provides Fortran applications direct access to capabilities in the Trilinos project.

Karla is a senior member of technical staff in the Reacting Flow Research Department at Sandia. She is the lead developer of ForTrilinos. Her research interests include computational fluid dynamics to multiphysics flows and scientific software architecture.

Salvatore Filippone graduated from the University of Rome "Tor Vergata" where he is currently with the Dep. of Industrial Engineering. His main research interests are in algorithms for numerical linear algebra, their implementation on high performance computers and their application to engineering domains such as fluid dynamics and electromagnetism. From 1990 to 2001 he has been with IBM Corp. where he was one of the lead developers of the numerical libraries ESSL and PESSL. He evaluates scientific projects in HPC for both the European Commission and the National Science Foundation.

Bluefire and other CISL systems down until Flagstaff Fire situation stabilizes

The Mesa Lab is closed all day today, June 27, to all staff other than critical operations staff due to the Flagstaff Fire west of Boulder. Because even these last few staff will have to leave immediately if a mandatory evacuation is ordered, Bluefire and the other CISL systems will remain down until operations staff are able to safely return. 

The Flagstaff Fire started yesterday afternoon, spreading rapidly and leading UCAR to issue an informal evacuation of NCAR's Mesa Lab. To enable operations staff to leave, CISL shut down all of its major systems, including Bluefire, at 4 p.m. June 26.

UCAR and local fire officials will continue to evaluate the fire situation. CISL will provide further updates about the status of Bluefire and other systems via the Notifier service.

CISL Help Desk and Consulting staff are working remotely today. Users may call 303-497-2400 or email with any questions.

CISL seeks users’ input

The CISL User Services Section (USS) would like your help in reviewing end-user documentation for the new Yellowstone HPC, analysis, and visualization clusters. We want to make sure we meet your needs as users of these resources.

There are two ways you can help:

  1. Comment on content that has been published already by using our online Feedback Form. We appreciate any suggestions for improvement.
  2. Review new content before it is published for general consumption. You will occasionally receive draft copy for a new web page and asked for your input.

If you can review some early drafts, let us know through the feedback form or email B.J. Smith, USS documentation writer/editor.


Subscribe to Computational & Information Systems Laboratory RSS