UPDATE: 2024-07-29 14:00 MDT: most systems are back online, including data.nrao.edu. Some work remains to be done on internal Lustre filesystems, but largely this incident can now be considered resolved.
UPDATE: 2024-07-29 09:12 MDT: Power has been restored to DSOC. We are waiting until 09:30 MDT before resuming restoration work on the systems in the data center.
UPDATE: 2024-07-29 08:27 MDT: Power outage at DSOC has compounded the problem. Existing systems already restored will stay online for now, but restoration efforts for other systems are now on hold.
At around 14:00 MDT on July 28, the data center at the NRAO DSOC suffered a complete cooling failure, and as a result all systems therein had to be shut down rapidly. As of 06:00 MDT on July 29, cooling is now being restored, and the https://my.nrao.edu/ service is back online. Other systems such as the NRAO Archive (https://data.nrao.edu/) remain offline.