It’s scary to think about, but the flooding and high winds associated with hurricanes can take down a data center. While 2017 was the most expensive hurricane season on record, costing over $300 billion, many data centers located within flood and hurricane zones continue to fear incidents similar to 2012 with Hurricane Sandy in the Northeast. The superstorm flooded streets and took out the electricity for days, leaving tanker trucks unable to deliver diesel fuel to power many data center generators; other data centers had to rely on backup electricity for as long as a week, costing millions.
Following the Sandy catastrophe, building hardening requirements (often consisting of using reinforced concrete) increased to better fortify data centers, and landlords began establishing improved systems for reserve fuel deliveries. Many cities around the country also made data centers priority number three for fuel delivery, following hospitals and important government agencies.
Of course, building hardening is no guarantee that a data center will remain unaffected by a hurricane, but there are steps that you can take to be sure your company-owned data center, or your third-party data center, remains constant in order to protect your data from disaster.
5 Disaster Planning Tips for Data Center Users
Organizations should be evaluating and re-evaluating their disaster recovery (DR) plans, speaking with their data center partner about DR now. Hurricane season is still over a month away, so take the time to verify that contingencies such as redundant operations are put into place.
Test and re-test your uninterruptible power supply (UPS).
UPS failure is the number one cause of data center outages, so you’ll want to be certain that generators, battery backups, and cooling can be maintained at the current maximum power load at a regulated voltage level. Experts recommend performing these tests twice per year, so if it’s been over six months, a before-season test is overdue. If you’re using a third-party data center, they should be able to produce a log of their Integrated Systems Testing as proof that it was performed.
Check your water pumping and fire suppression systems.
Be sure that water pumps are set to operate automatically in the event of flooding, and that the system is connected to a generator so it continues pumping out flood water if the power grid goes down.
While typically not associated with hurricanes, this is also a good time to check your fire suppression system, as water + electric can lead to fire hazards. Ideally, your data center, or your third-party data center should employ a dry pipe system; this way, water is kept behind a valve and only released if a fire is detected. In the server rooms, data centers should use FE-25, an extinguishant that is stored as a liquid, but discharges as a vapor; it’s non-hazardous to humans, non-conductive, non-corrosive, and it has zero ozone depletion potential.
Review internal operating procedures and SLAs.
If you maintain your own data center, maintenance operating procedures (MOPs) and standard operating procedures (SOPs) should be reviewed; also, be sure to perform a maintenance check to ensure servers are in top condition, that you have adequate fuel provision, and any tools or spare parts (such as belts and hoses) that may be needed are on hand. If you’re working with a third-party data center, you’ll also want to review your service level agreement (SLA). The SLA should define the MOPs and SOPs of the data center, and outline data protection and DR policies. There should also be a liquidated damages section which specifies the penalties the provider will incur if the terms of data protection in the SLA are not met.
Keep staff informed of their roles and responsibilities.
During a disaster, everyone should know what is expected of them. If you don’t yet have your disaster recovery processes documented, now is the time to do it so employees can review them before a disaster strikes; with proper documentation, they can feel empowered to react accordingly to situations that may arise. Make sure contact information for each employee is current, and provide everyone with a hardcopy in case power does go out. Finally, consider setting up a “control room” for staff that may need to stay on site during a hurricane, and that there is enough water and food.
Check your backup procedures.
If you’re an organization maintaining its own data center facility, but not maintaining backups at an off-site location, it’s imperative that you do so. Ideally, you should be following the 3-2-1 rule of data backup: maintaining three copies, in two independent storage mediums, with one located off-site.
Hurricane Season 2019
The Weather Channel reports that while we may witness fewer storms in 2019 versus the previous year, they won’t be any less dangerous. Meteorologists are predicting a total of 13 named storms and seven hurricanes, with two of them categorized as major hurricanes (Category 3 or stronger).
If you’re concerned about the safety of your company-owned data center, or are questioning the data protection and recovery policies of your third-party provider,
DSM can help. We are positioned high and dry in the middle of the state with geo-diverse locations, meaning if a disaster does strike our Florida locations, we can quickly move data to an out-of-state facility to keep your data safe and accessible.
Already using on-premise data management software such as Veeam, Zerto, or Commvault? No problem. You can get physical and virtual backups offsite through DSM, without the cost or complexity of building and maintaining offsite infrastructure. Contact DSM to learn more and keep your data protected during this, and every hurricane season.