We live in the age of big data. Everyone is connected to each other via digital strings be it email,social media or video streaming services. You couldn’t go a day without leaving some sort of digital fingerprint somewhere. With our societies ever more reliance on this technology,the expectations of a fast and reliable connections to these systems only increases.Data centers that push these information streams at us are becoming larger, and more complicated as engineers do more to guarantee the uptime and security of the systems remains first class.
The evolution of these power houses coming from what would have been 15 years ago simply a small group of servers placed in a cupboard somewhere off to the side of the office, to what we see now which are large climate controlled rooms. These data halls have enormous electrical supplies and bring along their own new and unique sets of risks and hazards. Power usage rates range for a typical office space from 9W to 90W per square foot with a data center dramatically increasing this power usage to 300 – 400W per square foot. It’s easy to see the immediate fire risk from having so much electrical equipment densely packed into smaller spaces.
To combat this, most data centers will employ a gas suppression type system.There are various systems available to the market, but all have the end goal of excluding oxygen from the ambient environment and extinguish the fire without the use of water or other destructive medium. These systems will operate in conjunction with a detection system and in an automated mode usually operate on a time delay from when an alarm detection occurs to when the gas system is initiated.
During this suppression activation, air handling systems will shut down and the gas dumped to over-pressurize the room, with shutters installed to prevent the external ambient air returning once the dump has completed. The end result will be a room with a hypoxic environment that is unable to support combustion. It’s important to note that there is no cooling outcomes with these types of systems and care must be taken when crews enter and exit the room to not allow the oxygen levels to rise, simply chocking open a door will reduce the effectiveness of the gas system and may allow re-ignition.
These data centers not only house the electrical equipment, but require workers to make access to perform physical maintenance. These tasks could range from simple short duration jobs to longer periods of time spent at the rack consoles. Depending on the level of focus a worker is committing to their tasks, they may not hear or notice the warning indicators from the suppression system, and therefore may become impaired because of the IDLH environment. It should never be discounted that a person or persons is within the room after a gas suppression activation, the victims may also be located in an obscure location.
Being a hypoxic environment this now makes the wearing of SCBA mandatory. All it takes takes is a couple of unlikely events to align and crews can easily be caught out when entering these rooms. While audible and visual indicators should be installed and alerting crews to the risks on the other side of the doors, there is always the risk that the system has activated and failed to display these warning cues. If in doubt, wear SCBA.
Cutting edge data centers pride themselves on the principle of redundancy. As more data centers aim to offer their clients “Fault Tolerant” services, it’s expected that no matter what type of event that occurs the clients services remain online. With Tier IV data centers offering 99.995% uptime with an allowable downtime of only 26.3 minutes a year, massive infrastructure is committed to these centers to ensure absolute redundancy is present. It’s this redundancy that may result in crews working inside still active data centers, with services unable to be isolated easily. Any outages will result in huge financial impacts for the data center.
With this said maintaining clean reliable electrical supply is arguably the data centers number one priority. Power backup systems come in all shapes and sizes, each creating their own unique hazards. One type of backup power system likely to be encountered is the battery style system. As you would expect these systems depending on the supply demand will include sets of batteries arranged in various configurations depending on the desired performance characteristics. The likely battery types to be seen are the lead-acid type and lithium ion.
A failure within the uninterruptible power supply (UPS) or the battery itself can allow the lead acid batteries to overcharge resulting in overheating and the generation of hydrogen and hydrogen sulfide gases, which can easily create an explosive atmosphere. In the absence of an explosion or fire event, this gas can create a HAZMAT incident with any exposures to workers or unprotected crew members.
Lithium ion batteries as we know with the emergence of electric vehicles behave quite differently during fault conditions. Overcharging, physical damage or even a manufacturers fault can cause the batteries to overheat and create a thermal runaway event. A simple internet search on lithium battery failures will return ample videos of this event. It has the potential to create quite a hazardous environment for fire crews.
Rotary UPS systems are beginning to make more of an appearance into these environments. The operating principle of these devices are a large flywheel will continue to spin during a power outage and this motion will support the critical load over a short period to allow the larger generator systems to start up and stabilize. These devices are limited in their ability to support critical loads over any substantial duration of time, but nevertheless they still have the ability to output substantial electrical energy and add another safety consideration for the incident management team.
It’s important to understand that regardless of the size of these battery systems, they are designed to hold power for a short amount of time until a longer source comes online, this can be as simple as holding out to determine if the grid power reestablishes itself or a diesel or gas generator start up to take over, regardless, isolation of the UPS should be done in conjunction with any other upstream devices such as these generator systems. Power should not be treated as isolated until confirmed by an onsite engineer or power authority.
Data centers generally house enormous amounts of sensitive data, whether it is consumers privacy or commercial trade secrets, access will certainly be more of a challenge than a standard office space would. Security systems such as proxy sensors or bio-metrics will no doubt be present and may require the overriding of any mag-lock system or forcing of mechanical locks.
Power systems provide redundancy with multiple power sources supplied. This is going to require careful considerations when tasking crews to make access to the fire affected areas. Where these data centers are located in each jurisdiction may vary. Most large commercial high rise buildings are likely to have one of more data centers, with little to no external indication of their presence. While some buildings exist that have the sole purpose for this type of occupancy, it could very well be the case that the first time a crew is aware of an incident occurring in these types of rooms is when an automatic detection system advises them on the panel. This means crews need to be conscious that a standard response to an automatic alarm can pivot quite quickly.
Logistically the locations of these rooms may require crews to carry large quantities of equipment to a predetermined location. This travel will likely cause a delay in tactical applications and reduce crews physical capability after any arduous process of equipping a staging point. Incident management need to consider that any significant fire event within these types of buildings is going to be labor intensive and require significant human resources.
Cooling and ventilation systems are a crucial component within data centers. These systems need to be able to remove large quantities of hot air and replace it with cooler air in order to maintain suitable temperatures for the critical equipment. Depending on the configuration and complexity of the data center, the air handling system may or may not be interfaced to the fire detection systems. An older retro fitted data room may simply reconfigure existing air conditioning systems with rack cooling configuration, which is unlikely to react to a fire event, an may cause any fire to behave differently to one that is expected if it were simply organically ventilated. More modern systems will integrate their air handling systems with any fire suppression to prevent in introduction of oxygen to the fire room.
As data centers push harder these days to achieve energy efficiency targets, there are various cooling setups used. A modern well planned data center will likely employ some sort of hot or cold air containment system, which is where chilled air is pushed into the rack from the outside and hot air is exhausted to the return air or chilled air is pushed through the racks from the inside and hot air is drawn back to the chillers via the room (Cold aisle). Either setup relies on a pressurized air flow and has the potential to direct any fire or heat further into the rack system. Just because there is a fire event, doesn’t mean to say that the IT equipment operating within the effected room will be de-energized.
Voids and concealed spaces will no doubt affect travel paths for any fire that may be present. While some data centers employ the use of overhead cabling systems, others can provide cable access via the means of a raised floor tiling system. Tiling panels that are designed for the use in these environments are generally constructed with non combustible materials such as cement core.While the tiles themselves are unlikely to promote fire growth, worker behaviors such as not reinstating fire stop packing or failing to reinstate the tiles might allow unpredictable fire behavior, all while being concealed in the under floor compartments. Considerations should be made to check these spaces at a fire incident involving data centers. Access to the affected equipment may be challenging. Ventilation is carefully controlled within the racks.
A well understood communications plan should also be put into place during an incident involving data centers. As data centers are beginning to occupy dedicated buildings, some of these building features may hinder or prevent satisfactory communications between internal crews and the incident management teams. Examples are that generally the data rooms themselves would be protected from the external and located well within the core of the building with access halls and public areas located on the perimeter of the rooms. Wall construction would certainly be masonry or some dense fire rated material with extensive metallic cabling passing through walls and ceiling spaces. These features may interfere with fire ground
radio traffic and may require pre-planning for these types of incidents.
Data centers range in size and complexity, each throw their own unique set of challenges to any incident management team. The take home point is that these data centers can be anywhere and in any size, from a full building sized facility to a small locked cupboard. Firefighting crews need to be aware scenarios and responses can change rapidly. The only way to truly exhibit some level of tactical readiness is to have pre-action plan in place, with crews familiar with the expected tactical response as well as being aware of the risks and hazards in these rooms.
Technology changes occur at a rapid break neck speed, and what crews saw in a site visit five years ago may be considered a legacy system and replaced with a new concept. All fire crews need to at least make some concerted effort to visit these sites where they can and pay attention to what is around them. Only then will we be able to meet community expectations while also maintaining our safety as well as those we work with.
Please see the following video that demonstrates the power of these systems and how quickly it could incapacitate anyone in this area.
Instructor John Black
