RESOLVED: DSOC Data Center Outage

UPDATE: 2024-07-29 14:00 MDT: most systems are back online, including data.nrao.edu. Some work remains to be done on internal Lustre filesystems, but largely this incident can now be considered resolved.

UPDATE: 2024-07-29 09:12 MDT: Power has been restored to DSOC. We are waiting until 09:30 MDT before resuming restoration work on the systems in the data center.

UPDATE: 2024-07-29 08:27 MDT: Power outage at DSOC has compounded the problem. Existing systems already restored will stay online for now, but restoration efforts for other systems are now on hold.

At around 14:00 MDT on July 28, the data center at the NRAO DSOC suffered a complete cooling failure, and as a result all systems therein had to be shut down rapidly. As of 06:00 MDT on July 29, cooling is now being restored, and the https://my.nrao.edu/ service is back online. Other systems such as the NRAO Archive (https://data.nrao.edu/) remain offline.

Posted in AOC, Observatory-Wide | Tagged , | Leave a comment

RESOLVED: NRAO Public Wiki Offline

UPDATE: The public wiki has been restored.

Due to technical issues, the NRAO Public Wiki is temporarily offline, as of 16:20 US/EDT.

Leave a comment

RESOLVED: SO VPN Down

2024-FEB-02 2245 Eastern: SO VPN services have now been fully restored.

The SO (AOC) VPN server is currently offline. Restoration efforts are underway.

Please use the CV VPN until service is restored.

Leave a comment