Daily Bulletin Archive

Mar. 20, 2013

Yellowstone, Geyser, and Caldera along with the GLADE storage system will be taken down starting Monday, March 18, at 7 a.m. (Mountain Time). The plan is to return the cluster for production work by Wednesday, March 20, at 8 a.m. CISL will begin draining the queues during the coming weekend to ensure no running jobs will be killed.

This maintenance work includes a number of software and firmware updates, but the major effort will be focused on replacing about 233 optical InfiniBand cables. We anticipate being able to return a healthier Yellowstone system to users after completing this work.

While GLADE is undergoing maintenance work, users will not be able to copy files from the old GLADE system via Mirage to the new GLADE.

We apologize for the short notice of this long downtime. We will provide updates as soon as possible if the timetable changes.

Mar. 20, 2013

The replacement of InfiniBand cables and planned software and firmware updates were completed on Yellowstone during the scheduled downtime. However, we encountered an issue related to an InfiniBand firmware update that caused issues with MPI and the Parallel Environment (PE). As a result, CISL, IBM and Mellanox are still working to bring Yellowstone to a state healthy enough for running MPI jobs. The issue has been escalated as a high priority to IBM PE team. We do not yet have estimated time for resolution.

At this time, the Yellowstone login nodes are up and jobs can be submitted, but no jobs will be dispatched until the system is healthy.

The GLADE resource has been returned to service, and users can move files from the old GLADE file spaces to new GLADE via the Mirage nodes.

We apologize for this extended outage and will keep you posted as we learn more.

Mar. 19, 2013

To add the MEXNC interface and SNCTOOLS codes, run ncstartup from your MATLAB command line. If you have not updated your script to be compatible with SNCTOOLS -r4028, you may need to invoke a previous version. Run help ncstartup at your MATLAB command line for details.

If you have questions about the MEXNC interface, please contact cislhelp@ucar.edu.

Mar. 14, 2013

Please join us March 14, 2 p.m. - 3:30 p.m. MT in the NCAR Mesa Lab Main Seminar Room for presentations from three of the Accelerated Scientific Discovery (ASD) projects that have been among the first to put CISL's new 1.5-petaflops Yellowstone system through its paces.

The presentation will be webcast for those unable to attend in person. To view the webcast, visit http://www.fin.ucar.edu/it/mms/ml-live.htm

Each of the speakers will discuss the team's experiences in using Yellowstone, where the system performed well (or not), and when available, some early results from their runs. The following speakers will present during the session:

2 p.m.: Gabriele Pfister, ACD -- Prediction of North American air quality

High-resolution simulations with the nested regional climate model with chemistry (NRCM-Chem) are being performed to study possible changes in weather and air quality over North America. The performed simulations are targeting insights into expected future changes related to air quality. The project is using NRCM-Chem over the conterminous U.S. at a resolution of 12 x 12 km2 with inclusion of atmospheric chemistry to examine impacts on air quality and other regional climate processes.

2:30 p.m.: Justin Small, CGD -- Meso- to planetary-scale processes in a global ultra-high-resolution climate model

This project is performing and assessing high-resolution Community Earth System Model (CESM) simulations (1) to investigate the climate response to the coupling of ocean and atmosphere mesoscale features, (2) to assess the ability of a high-resolution and frequently coupled ocean and atmosphere simulation to represent near-inertial waves in the ocean, and (3) to investigate the role of small-scale ice features such as polynyas in the climate system.

3 p.m.: David Richter, MMM -- Turbulence modification in the spray-laden atmospheric marine boundary layer

This project is focused on the effect that sea spray, suspended by turbulence in the high-wind marine atmospheric boundary layer, has on both the turbulence itself as well as the transfer of momentum and heat to the ocean surface. To study this problem, a fundamental approach is taken where direct numerical simulation (DNS) coupled with Lagrangian particle-tracking is employed to focus generally on how a dispersed phase (such as sea spray) modifies turbulence.

Mar. 12, 2013

Please join us March 14, 2 p.m. - 3:30 p.m. in the NCAR Mesa Lab Main Seminar Room for presentations from three of the Accelerated Scientific Discovery (ASD) projects that have been among the first to put CISL's new 1.5-petaflops Yellowstone system through its paces.

Each of the speakers will discuss the team's experiences in using Yellowstone, where the system performed well (or not), and when available, some early results from their runs.

The presentation will be webcast (URL tbd) for those unable to attend in person.

The following speakers will present during the session:

Gabriele Pfister, ACD -- Prediction of North American air quality

High-resolution simulations with the nested regional climate model with chemistry (NRCM-Chem) are being performed to study possible changes in weather and air quality over North America between present-day and two future time periods: 2020-2030 and 2045-2055. The performed simulations are targeting insights into expected future changes related to air quality and will also be used for dynamical downscaling (of meteorology and air quality) of global climate simulations performed at NCAR in support of the 2013 IPCC AR5. The project is using NRCM-Chem over the conterminous U.S. at a resolution of 12 x 12 km2 with inclusion of atmospheric chemistry to examine impacts on air quality and other regional climate processes.

Justin Small, CGD -- Meso- to planetary-scale processes in a global ultra-high-resolution climate model

In this project, the main computational objective is to perform and assess high-resolution Community Earth System Model (CESM) simulations in order (1) to investigate the climate response to the coupling of ocean and atmosphere mesoscale features, (2) to assess the ability of a high-resolution and frequently coupled (two-hour) ocean and atmosphere simulation to represent near-inertial waves in the ocean, and (3) to investigate the role of small-scale ice features such as polynyas in the climate system.

David Richter, MMM -- Turbulence modification in the spray-laden atmospheric marine boundary layer

This project is focused on the effect that sea spray, suspended by turbulence in the high-wind marine atmospheric boundary layer, has on both the turbulence itself as well as the transfer of momentum and heat to the ocean surface. To study this problem, a fundamental approach is taken where direct numerical simulation (DNS) coupled with Lagrangian particle-tracking is employed to focus generally on how a dispersed phase (such as sea spray) modifies turbulence. Systematic runs of turbulent Couette flow at multiple Reynolds numbers, each with various particle sizes, time scales, and concentrations, are being performed to identify the critical mechanisms of turbulence modification, as well as to determine the extent to which suspended particles can affect transport of heat and momentum.

Mar. 2, 2013

The old GLADE file spaces mounted on Mirage/Storm with read/write access will be set to read-only mode on Monday, March 4, to prevent new data from being generated.

The file spaces will remain available read-only via the Mirage nodes through March 31. Users are responsible for moving files to new locations in the meantime.

The Storm cluster will be retired March 4.

Note these details regarding the various file spaces:

* /glade/home :: CISL will retain an off-line copy of the data in home directories for six months after March 31, during which period users will need to contact cislhelp@ucar.edu to access their files.

* /glade/scratch :: The current 30-day retention policy remain in effect. Files will continue to be removed 30 days after last access.

* /glade/user, /glade/proj2, /glade/proj3 :: Files in these spaces will be available until March 31. Users are responsible for migrating files they wish to retain.

On April 1, CISL will remove all remaining data in /glade/scratch, /glade/user, /glade/proj2, /glade/proj3. The bfft.ucar.edu service for transferring files will be turned off at the same time.

For information about how to migrate files to the Yellowstone GLADE file spaces, please see http://www2.cisl.ucar.edu/resources/yellowstone/transition.

Lynx users: Your home directory for Lynx will change on March 4. You are responsible for copying necessary files from your GLADE home directory to your new Lynx home directory. 

If you have questions about the transition, the schedule, or the new systems at NWSC, please contact cislhelp@ucar.edu.

Mar. 1, 2013

HPSS downtime Wednesday, February 27 from 7:00am - 09:00am

No Scheduled Downtime: Yellowstone, Geyser_Caldera, NWSC GLADE,  DAV, ML GLADE, Lynx

Feb. 28, 2013

When moving files from the GLADE file spaces that were mounted on Bluefire, Lynx, and Mirage/Storm to the new GLADE file spaces, users with very full home directories may find that their files do not fit in their Yellowstone home directory.

Differences in file system block sizes between the old and new systems may cause files to require more space on the new system.

If your new home space will not accommodate all of the files that you had in /glade/home, consider transferring some of them to your larger /glade/p/work space rather than to /glade/u/home. Keep in mind that the home space is backed up but the work space is not.

See Transition from Bluefire for more information about migrating your files to the new file spaces before the March 31 deadline.

Feb. 22, 2013

The GLADE file spaces that were mounted on Bluefire, Lynx, and Mirage/Storm are accessible via the Mirage nodes through March 31. Users are responsible for moving files to new locations in the meantime.

The file spaces are mounted on Mirage/Storm with read/write access to permit users to complete some final analysis tasks. They will be set to read-only mode on March 1 to prevent new data from being generated. The Storm cluster will be retired March 1.

Note these details regarding the various file spaces:

* /glade/home :: CISL will retain an off-line copy of the data in home directories for six months after March 31, during which period users will need to contact cislhelp@ucar.edu to access their files.

* /glade/scratch :: The current 30-day retention policy remain in effect. Files will continue to be removed 30 days after last access.

* /glade/user, /glade/proj2, /glade/proj3 :: Files in these spaces will be available until March 31. Users are responsible for migrating files they wish to retain.

On April 1, CISL will remove all remaining data in /glade/scratch, /glade/user, /glade/proj2, /glade/proj3. The bfft.ucar.edu service for transferring files will be turned off at the same time.

For information about how to migrate files to the Yellowstone GLADE file spaces, please see http://www2.cisl.ucar.edu/resources/yellowstone/transition.

If you have questions about the transition, the schedule, or the new systems at NWSC, please contact cislhelp@ucar.edu.

Feb. 21, 2013

A reminder to all Yellowstone users that you can monitor your project activity and usage via the new SAM system at http://sam.ucar.edu/. As of Monday, February 11, jobs associated with overspent projects will be rejected by the LSF scheduler on Yellowstone.

All Yellowstone jobs associated with each project are visible via the SAM interface, and all jobs since December 1, 2012, have been charged.

The SAM system is under active development. To date, we have focused on SAM features that ensure users have proper access to the Yellowstone environment. Current efforts include expanding the reporting capabilities available to users and NCAR divisions.

With the end of Bluefire, we have also decommissioned the legacy ACC8 accounting system, so those email reports are no longer being sent.

Questions about using SAM or requests for SAM reports and features can be sent to cislhelp@ucar.edu.

Pages