HPSS to replace dual-copy class of service with user-controlled backup option

July 31, 2012

With the imminent arrival of Yellowstone and the expected large-scale data production, CISL is making changes to the HPSS environment. With these changes, HPSS will discontinue support for the dual-copy class of service, while providing users with a second-copy backup area for enhanced preservation of critical data.

These changes will happen in several steps. On August 1, 2012, a new user backup area (see below) will be made available to users. Then, on September 1, 2012, HPSS will discontinue support for the dual-copy class of service in favor of a user-controlled, second-copy strategy. Finally, and also starting September 1, the HPSS team will begin converting all data files stored in the dual-copy class of service to single-copy files.

Most existing dual-copy files are over one year old, dating from the prior Mass Storage System (MSS), which used dual-copy as the default class of service. HPSS has always used single-copy as its default, and since April 2011, very little new data has been stored as dual-copy. However, converting existing dual-copy files will recover 3 PB of space in support of Yellowstone-related data storage needs.

To replace the dual-copy class of service, HPSS will provide a backup area into which users can copy critical files for which they wish to have the extra level of protection. Files copied to the backup area will be guaranteed to be on separate tape cartridges. In addition, the backup area provides protection from user errors over and above the dual-copy class of service. Specifically, the backup copies are protected from user-initiated deletions or overwrites of the primary copies. In contrast, deleting or overwriting a dual-copy file deletes or overwrites both copies.

Interested users should submit a CISL Help request for their own directory in the backup area. Users can then use the cp command to copy data to the backup area. Files in the backup area will be charged against project allocations at the same rate as primary copies. Documentation for this new feature is available at http://www2.cisl.ucar.edu/docs/hpss/copies.

All users are welcome to take advantage of the backup area, but are encouraged to carefully consider which files need the extra protection. While data loss can happen, it is very rare. CISL's tape archive has had several media failures in the past year, but all files were recovered by the vendor. Over the tape archive's lifetime, there have been only a few instances of data being lost from media failure.

If you have questions about this change or whether your files may be affected, please contact cislhelp@ucar.edu.