NRAO/Charlottesville EOS

Posted on January 6, 2025 by

UPDATE: Due to delays in cleanup from the storm, the NRAO facilities at Edgemont Road and CDL will open two hours late on January 7th, 2025. See advice below for costpoint.

Due to inclement weather, Emergency Operation Status (EOS) has been declared
and NRAO facilities at Edgemont Rd. and CDL will be CLOSED on Monday January 6th, 2025.
Those employees of the observatory that cannot work remotely during this period
will need to use ADM01.25.14.4511.01.00 (EOS Nonworked) as the project code in
Costpoint with your managers approval.

In the event of an unexpected change in weather an amended message will be sent
by email to increase the duration of the closure. You can also call
434-296-0200 or check : https://info.nrao.edu/staff under the “NRAO Facilities
Status” section on the left side of the page.

Our apologies for not having this page updated prior to start of business.

Tagged Leave a comment

CONCLUDED: Network Switch Replacements, Charlottesville

Posted on November 8, 2024 by

UPDATE 2024-11-15 15:58 US/EST: the maintenance activities have concluded. All systems mentioned below should be back online and functioning.

UPDATE 2024-11-13 15:56 US/EST: due to failed hardware discovered while rebooting systems, gitlab.nrao.edu and some other services are still offline. Remediation is in process.

Several services will be offline or unreachable for some or all of the day on Wednesday, November 13th, 2024. These include:

  • public.nrao.edu (RESTORED)
  • science.nrao.edu (RESTORED)
  • almascience.nrao.edu (Restored, but backend services needing NGAS or database access will not work yet; see below)
  • help.almascience.org
  • login.cv.nrao.edu (aka ssh.cv.nrao.edu, aka polaris; includes fastx) (RESTORED)
  • …and other systems served out of our headquarters in Charlottesville, Virginia.

For access to ALMA data during this planned maintenance period, please instead use:

Posted in Info, NRAO/Charlottesville EOS | Tagged , , | Leave a comment

RESOLVED: DSOC Data Center Outage

Posted on July 29, 2024 by

UPDATE: 2024-07-29 14:00 MDT: most systems are back online, including data.nrao.edu. Some work remains to be done on internal Lustre filesystems, but largely this incident can now be considered resolved.

UPDATE: 2024-07-29 09:12 MDT: Power has been restored to DSOC. We are waiting until 09:30 MDT before resuming restoration work on the systems in the data center.

UPDATE: 2024-07-29 08:27 MDT: Power outage at DSOC has compounded the problem. Existing systems already restored will stay online for now, but restoration efforts for other systems are now on hold.

At around 14:00 MDT on July 28, the data center at the NRAO DSOC suffered a complete cooling failure, and as a result all systems therein had to be shut down rapidly. As of 06:00 MDT on July 29, cooling is now being restored, and the https://my.nrao.edu/ service is back online. Other systems such as the NRAO Archive (https://data.nrao.edu/) remain offline.

Posted in AOC, Observatory-Wide | Tagged , | Leave a comment