The Daily Bulletin

July 22, 2019

Some Cheyenne users have reported frequent batch job failures with error messages containing “MPT: Launcher network accept (MPI_LAUNCH_TIMEOUT) timed out.” The root cause of the problem is not yet known but is believed to be related to Cheyenne’s InfiniBand network. CISL is working closely with HPE and Mellanox to identify and resolve the issue as soon as possible.

Until the issues are resolved, CISL suggests setting the following two environment variables to help jobs better tolerate the network issues. Users who have added these two settings have reported a significant reduction in the number of job failures due to the MPT errors.

MPI_IB_CONGEST=1
MPI_LAUNCH_TIMEOUT=40

Also, setting environment variables MPI_VERBOSE=1 and MPI_VERBOSE2=1 will generate more informative diagnostics that may help CISL’s system administrators identify the root cause of the problem. Users should note that setting these two environment variables will produce and add a significant amount of output to their jobs.

July 22, 2019

No scheduled downtime: Cheyenne, Casper, Campaign Storage, GLADE and HPSS.

July 15, 2019

Registration is open for the 2019 International Computing in Atmospheric Sciences (iCAS) Symposium, September 8-12 in Stresa, Italy. NCAR, which hosts the iCAS series, announced that keynote presentations will focus on the European Open Science Cloud and the Daniel K. Inouye Solar Telescope.

On Monday, September 9, Tiziana Ferrari from EGI and Hannes Thiemann from DKRZ will discuss "The European Open Science Cloud and its Use Cases for Climate, Weather, Earth, and Environmental System Research."

On Wednesday, September 11, Alisdair Davey from the U.S. National Solar Observatory, will present on "The Daniel K. Inouye Solar Telescope (DKIST): A Next-Generation Ground-Based Solar Telescope."

For information about the keynote talks and the speakers, and to register for the symposium, see the iCAS 2019 site. Room reservations should be made directly with the hotel before August 7. After that date, rooms will be available on a first-come, first-served basis. See the iCAS site for a link to the hotel reservation form.