CONCLUDED: Network Switch Replacements, Charlottesville

Posted on November 8, 2024 by

UPDATE 2024-11-15 15:58 US/EST: the maintenance activities have concluded. All systems mentioned below should be back online and functioning.

UPDATE 2024-11-13 15:56 US/EST: due to failed hardware discovered while rebooting systems, gitlab.nrao.edu and some other services are still offline. Remediation is in process.

Several services will be offline or unreachable for some or all of the day on Wednesday, November 13th, 2024. These include:

  • public.nrao.edu (RESTORED)
  • science.nrao.edu (RESTORED)
  • almascience.nrao.edu (Restored, but backend services needing NGAS or database access will not work yet; see below)
  • help.almascience.org
  • login.cv.nrao.edu (aka ssh.cv.nrao.edu, aka polaris; includes fastx) (RESTORED)
  • …and other systems served out of our headquarters in Charlottesville, Virginia.

For access to ALMA data during this planned maintenance period, please instead use:

Posted in CV, Info | Tagged , , | Leave a comment

RESOLVED: DSOC Data Center Outage

Posted on July 29, 2024 by

UPDATE: 2024-07-29 14:00 MDT: most systems are back online, including data.nrao.edu. Some work remains to be done on internal Lustre filesystems, but largely this incident can now be considered resolved.

UPDATE: 2024-07-29 09:12 MDT: Power has been restored to DSOC. We are waiting until 09:30 MDT before resuming restoration work on the systems in the data center.

UPDATE: 2024-07-29 08:27 MDT: Power outage at DSOC has compounded the problem. Existing systems already restored will stay online for now, but restoration efforts for other systems are now on hold.

At around 14:00 MDT on July 28, the data center at the NRAO DSOC suffered a complete cooling failure, and as a result all systems therein had to be shut down rapidly. As of 06:00 MDT on July 29, cooling is now being restored, and the https://my.nrao.edu/ service is back online. Other systems such as the NRAO Archive (https://data.nrao.edu/) remain offline.

Posted in AOC, Observatory-Wide | Tagged , | Leave a comment

RESOLVED: NRAO Public Wiki Offline

Posted on April 19, 2024 by

UPDATE: The public wiki has been restored.

Due to technical issues, the NRAO Public Wiki is temporarily offline, as of 16:20 US/EDT.

Leave a comment