{"id":1435,"date":"2024-07-29T07:13:11","date_gmt":"2024-07-29T12:13:11","guid":{"rendered":"https:\/\/status.nrao.edu\/?p=1435"},"modified":"2024-07-29T15:00:13","modified_gmt":"2024-07-29T20:00:13","slug":"july-28-29-2024-dsoc-data-center-outage","status":"publish","type":"post","link":"https:\/\/status.nrao.edu\/?p=1435","title":{"rendered":"RESOLVED: DSOC Data Center Outage"},"content":{"rendered":"\n<p><strong>UPDATE:<\/strong> 2024-07-29 14:00 MDT: most systems are back online, including data.nrao.edu.  Some work remains to be done on internal Lustre filesystems, but largely this incident can now be considered resolved.<\/p>\n\n\n\n<p><strong>UPDATE:<\/strong> 2024-07-29 09:12 MDT: Power has been restored to DSOC.  We are waiting until 09:30 MDT before resuming restoration work on the systems in the data center.<\/p>\n\n\n\n<p><strong>UPDATE: <\/strong>2024-07-29 08:27 MDT: Power outage at DSOC has compounded the problem.  Existing systems already restored will stay online for now, but restoration efforts for other systems are now on hold.<\/p>\n\n\n\n<p>At around 14:00 MDT on July 28, the data center at the NRAO DSOC suffered a complete cooling failure, and as a result all systems therein had to be shut down rapidly. As of 06:00 MDT on July 29, cooling is now being restored, and the https:\/\/my.nrao.edu\/ service is back online. Other systems such as the NRAO Archive (https:\/\/data.nrao.edu\/) remain offline.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>UPDATE: 2024-07-29 14:00 MDT: most systems are back online, including data.nrao.edu. Some work remains to be done on internal Lustre filesystems, but largely this incident can now be considered resolved. UPDATE: 2024-07-29 09:12 MDT: Power has been restored to DSOC. &hellip; <a class=\"more-link\" href=\"https:\/\/status.nrao.edu\/?p=1435\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":13,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8,11],"tags":[16,5],"class_list":["post-1435","post","type-post","status-publish","format-standard","hentry","category-aoc","category-observatory-wide","tag-downtime","tag-outage"],"_links":{"self":[{"href":"https:\/\/status.nrao.edu\/index.php?rest_route=\/wp\/v2\/posts\/1435","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/status.nrao.edu\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/status.nrao.edu\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/status.nrao.edu\/index.php?rest_route=\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/status.nrao.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1435"}],"version-history":[{"count":5,"href":"https:\/\/status.nrao.edu\/index.php?rest_route=\/wp\/v2\/posts\/1435\/revisions"}],"predecessor-version":[{"id":1443,"href":"https:\/\/status.nrao.edu\/index.php?rest_route=\/wp\/v2\/posts\/1435\/revisions\/1443"}],"wp:attachment":[{"href":"https:\/\/status.nrao.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1435"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/status.nrao.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1435"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/status.nrao.edu\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1435"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}