unscheduled downtime [2024.07.09]

2024-07-12 03:51:31 pm

from https://oplink.net/downloads/Oplink.net-Hurricane-Beryl-Statement.pdf

Quote:

Hurricane Beryl Statement
Our thoughts go out to anyone still dealing with the inconveniences of Hurricane Beryl. The Oplink.net team is safe and doing well minus some power outages at our homes.
Hurricane Beryl created a worst-case scenario for our Internet Backbones, LUMEN (Level3) and Cogent. LUMEN’s two diesel generators and Cogent’s generators both failed. This was unprecedented. Our network was offline a total of approximately five hours twenty-three minutes.
The Oplink.net datacenter did not have power, or cooling issues through Hurricane Beryl. We simply lost internet connectivity. TRG datacenter generator systems are SOC audited yearly by a third party company, are indoors and protected from the elements and storms. These units are tested live weekly and serviced regularly. We are very proud of the way TRG maintains their generators and facility.
Below is a description of what happened. We have detail data from LUMEN, but Cogent has not provided us full information of their downtime and restoration process.
Timestamp Logs with key information in red:
8:45 pm CDT 7/8/2024
LUMEN went down and our backup to Cogent successfully took over the load.
LUMEN reported the two diesel generators were experiencing power loading issues. Both failed.
10:42 pm CDT 7/8/2024
LUMEN informed us portable generators were in route. One was coming from Dallas Texas with a delivery ETA of 2 hours.
5:21 am CDT 7/9/2024
A 2.5 MW generator plus a 25 KW generator were brought online. LUMEN worked to bring up cooling/chillers first. As the cooling process commenced, technicians were sequentially restoring critical core equipment.
7:58 am CDT 7/9/2024
Texas 90° F+ and heat generated by turning up equipment prevented adequate cooling at LUMEN.
10:38 am CDT 7/9/24
LUMEN field personnel successfully connected additional air handlers at the site, resulting in a steady decrease in temperature. As conditions returned to normal, technicians powered up equipment. An additional generator dispatched from Tulsa arrived around 4PM Central time.
1:47 pm CDT 7/9/24
LUMEN made headway in powering up equipment, and the site began to cool down. Technicians noted power fluctuations from the 2.5 MW generator and it failed.

2:46 pm CDT 7/9/24
Cogent went offline presumably from backup generator failure.
This took the entire Oplink.net network offline due to LUMEN being offline at the same time.
3:13 pm CDT 7/9/24
LUMEN generator equipment vendor arrived on-site and began evaluation.
7:16 PM CDT 7/9/24
An additional LUMEN generator from Tulsa, arrived. Installation took 2 hours.
8:09 pm CDT 7/9/24
Cogent Internet restored.
Oplink.net services running 100% on Cogent. The total outage time: 5 hours 23 minutes.
10:33 pm CDT 7/9/24
500 gallons of fuel delivered to LUMEN and the new generator began installation.
2:07 am CDT 7/10/24
Transfer of power load to the new LUMEN generator completed. Supplementary backup batteries and site power and cooling systems are stable. Additional batteries are installed. Field Operations remained on site, and efforts to methodically restore node shelves and replace the necessary spares continued throughout the night.
4.57 am CDT 7/10/24
LUMEN technicians powered up an added AC chiller to mitigate any temperature fluctuations. An additional generator was sent from Albuquerque, NM
5:55 am CDT 7/10/24
LUMEN NOC confirmed service stability in the datacenter with power and cooling from generator power.
9:42 am CDT 7/10/24
At LUMEN, several cards failed from power fluctuations. Field Operations worked to reseat/reboot/replace each card in efforts to restore services.
10:00 am CDT 7/10/24
Our primary LUMEN backbone is restored. All Oplink.net core services and BGP routes transitioned to LUMEN from Cogent.
Oplink.net will continue providing reliable services you have been accustomed to over the years since 1999. We have never seen a double Internet Backbone outage where all generators failed at critical datacenters. We ask you consider that even major companies like Amazon, Google, Apple, etc. with all the investment and redundancies they have in place, still have rare outages like Oplink.net experienced. We are grateful that nearly 3 years ago we moved from LUMEN to our new home inside the TRG Datacenter. We continue to grow and expand every year. We will continue to provide the best reliable services.
We thank you for your business.
Oplink.net Team
support@oplink.net
281-445-9800

from https://oplink.net/downloads/Oplink.net-Hurricane-Beryl-Statement.pdf

[quote]Hurricane Beryl Statement
Our thoughts go out to anyone still dealing with the inconveniences of Hurricane Beryl. The Oplink.net team is safe and doing well minus some power outages at our homes.
Hurricane Beryl created a worst-case scenario for our Internet Backbones, LUMEN (Level3) and Cogent. LUMEN’s two diesel generators and Cogent’s generators both failed. This was unprecedented. Our network was offline a total of approximately five hours twenty-three minutes.
The Oplink.net datacenter did not have power, or cooling issues through Hurricane Beryl. We simply lost internet connectivity. TRG datacenter generator systems are SOC audited yearly by a third party company, are indoors and protected from the elements and storms. These units are tested live weekly and serviced regularly. We are very proud of the way TRG maintains their generators and facility.
Below is a description of what happened. We have detail data from LUMEN, but Cogent has not provided us full information of their downtime and restoration process.
Timestamp Logs with key information in red:
8:45 pm CDT 7/8/2024
LUMEN went down and our backup to Cogent successfully took over the load.
LUMEN reported the two diesel generators were experiencing power loading issues. Both failed.
10:42 pm CDT 7/8/2024
LUMEN informed us portable generators were in route. One was coming from Dallas Texas with a delivery ETA of 2 hours.
5:21 am CDT 7/9/2024
A 2.5 MW generator plus a 25 KW generator were brought online. LUMEN worked to bring up cooling/chillers first. As the cooling process commenced, technicians were sequentially restoring critical core equipment.
7:58 am CDT 7/9/2024
Texas 90° F+ and heat generated by turning up equipment prevented adequate cooling at LUMEN.
10:38 am CDT 7/9/24
LUMEN field personnel successfully connected additional air handlers at the site, resulting in a steady decrease in temperature. As conditions returned to normal, technicians powered up equipment. An additional generator dispatched from Tulsa arrived around 4PM Central time.
1:47 pm CDT 7/9/24
LUMEN made headway in powering up equipment, and the site began to cool down. Technicians noted power fluctuations from the 2.5 MW generator and it failed.
 
2:46 pm CDT 7/9/24
Cogent went offline presumably from backup generator failure.
This took the entire Oplink.net network offline due to LUMEN being offline at the same time.
3:13 pm CDT 7/9/24
LUMEN generator equipment vendor arrived on-site and began evaluation.
7:16 PM CDT 7/9/24
An additional LUMEN generator from Tulsa, arrived. Installation took 2 hours.
8:09 pm CDT 7/9/24
Cogent Internet restored.
Oplink.net services running 100% on Cogent. The total outage time: 5 hours 23 minutes.
10:33 pm CDT 7/9/24
500 gallons of fuel delivered to LUMEN and the new generator began installation.
2:07 am CDT 7/10/24
Transfer of power load to the new LUMEN generator completed. Supplementary backup batteries and site power and cooling systems are stable. Additional batteries are installed. Field Operations remained on site, and efforts to methodically restore node shelves and replace the necessary spares continued throughout the night.
4.57 am CDT 7/10/24
LUMEN technicians powered up an added AC chiller to mitigate any temperature fluctuations. An additional generator was sent from Albuquerque, NM
5:55 am CDT 7/10/24
LUMEN NOC confirmed service stability in the datacenter with power and cooling from generator power.
9:42 am CDT 7/10/24
At LUMEN, several cards failed from power fluctuations. Field Operations worked to reseat/reboot/replace each card in efforts to restore services.
10:00 am CDT 7/10/24
Our primary LUMEN backbone is restored. All Oplink.net core services and BGP routes transitioned to LUMEN from Cogent.
Oplink.net will continue providing reliable services you have been accustomed to over the years since 1999. We have never seen a double Internet Backbone outage where all generators failed at critical datacenters. We ask you consider that even major companies like Amazon, Google, Apple, etc. with all the investment and redundancies they have in place, still have rare outages like Oplink.net experienced. We are grateful that nearly 3 years ago we moved from LUMEN to our new home inside the TRG Datacenter. We continue to grow and expand every year. We will continue to provide the best reliable services.
We thank you for your business.
Oplink.net Team
support@oplink.net
281-445-9800[/quote]