When the grid sags, improved distributed energy technologies bridge the gap.
The rapid growth of data centers as repositories for everything digital reflects a worldwide demand for information technology (IT). But many utilities are struggling to keep up with the demand, as individual centers now approach power consumption rates as high as 35 MW, and some, such as Microsoft’s $500-million data center in the Chicago suburb of Northlake, IL, run at 198 MW. The 198-MW capacity at Northlake is almost double Chicago’s next largest data center, owned by Digital Realty Trust, San Francisco, CA. When Microsoft announced the project, Mike Manos was director of data center services (in June 2009, he joined Digital Realty Trust), and he talked about energy during a keynote speech at the 2008 Data Center World convention, where he observed that 82% of data center construction costs were related to mechanical and electrical infrastructure.
The demand and costs for Microsoft have resulted in policies such as facility managers seeing their performance incentives and bonuses linked to power efficiency. “If I’m going to go spend $500 million on a data center and 82% of the cost is wrapped up in my power bill, I want to make sure I get every dollar of my 82%,” says Manos.
Speaking today as senior vice president of technical services at Digital Realty Trust, Manos now oversees more than 70 properties comprising roughly 12.6-million rentable square feet, and he hasn’t changed his views about the urgency of energy efficiency. “The growth is staggering,” says Manos. “The US Department of Energy described it as the fastest-growing energy segment in the US, and that’s pretty interesting because data centers wouldn’t have been seen in that light five years ago. In general, what you consider to be
smaller facilities are typically in the two-megawatt range, and, if you compare that with the average office building, it’s a substantially larger demand.” He adds that the constantly growing need for power is driving the industry to seek maximum performance from every watt of electrical consumption at data centers.
The ever-growing demand has created a new era of energy efficiency technologies to reduce consumption from IT servers and the chain of infrastructure that includes: batteries, flywheels, UPS systems, alternating current (AC)/direct current (DC) converters, and diesel backup generators. Improvements are impressive, but, as we’ll see later, the highest gains will come by employing distributed energy.
The Big Picture
Bringing efficiency to data center energy consumption requires looking at the big picture, according to Bob Davis, president of Sentilla Corporation, based in Redwood City, CA. Sentilla’s energy management software integrates energy tracking and measuring across a data center’s infrastructure to improve performance. “It’s been estimated that 90% of the energy that comes into a data center never makes it to the computers,” says Davis. “It’s lost to invisible energy waste.”
To understand the dynamics of waste at a data center, Davis recommends metering the energy flow at every critical juncture, including backup generators, UPS systems, chillers and coolers, and power distribution systems.
The metering information can be integrated with other applications associated with power consumption, and in the case of the Sentilla Energy Manager, the facility gets an energy profile that’s available as a single database with recommendations and predictive charts. For example, Davis notes that it’s common to discover wasted energy caused by idle and rogue servers. “Say I have a rack with a bunch of equipment that is running my e-mail servers, but my user base has grown, so I add new servers and move my e-mail exchange, but the old servers are still there. I don’t bother to replace them or remove them because they aren’t hurting anything and there could be a need to go back and use them. Our recommendation would say these servers are idled and should be shut off or removed.”
The same philosophy applies to cooling the server racks. Sentilla uses the term “intelligent power” to describe a method of selective cooling. “If I know where the power is running I can be more intelligent about how I manage the cooling within the data center,” says Davis. “Doubling the speed of a fan to get higher airflow through a data center increases the fan’s energy usage by a factor of three times. A lot of people turn off chillers and turn up fans, but energy usage goes up instead of down. Part of the problem is that they don't understand what their IT equipment is doing or what it needs.
“The management of power utilization requires a holistic view of IT and the cooling and the power distribution,” he continues. “Any strategy that ignores those three things can have many errors.”
Keeping Dips, Spikes, and Harmonics Away From Critical Loads
An important part of the holistic view is the role that uninterruptible power source (UPS) systems play. Uninterruptible power must be available for critical loads when the primary source (typically a utility) has fluctuations, disruptions, or total failure. Even minimal frequency changes, or small dips or spikes in voltage can harm sensitive (and expensive) data servers. Pure and efficient power from a UPS is the goal, says Jim Davis, business unit manager of the Eaton Power Quality, Raleigh, NC. “Obviously, we and every other UPS manufacturer provide specifications about what pure power means,” says Davis. “Typically, it’s anywhere from plus or minus 5% voltage regulation. Frequency is plus or minus one hertz, and we’re reducing the harmonic content, in most cases, to below 3%. UPS systems have had harmonic correction as an added option, but we have developed devices in the last five years that make harmonic correction affordable down into the midrange. The units supplied for the Miami Dolphins have harmonic correction as a standard.”
 |
Photo: Ed Ritchie
Syracuse University data center construction using insulated concrete forms |
The Dolphin Stadium has a data center, and the team operates another one at its Davis training facility. Each location uses two 80-kVA 9390 Powerware UPS systems. The 9390s gain efficiency with low-input current total harmonic distortion (THD) of less than 4.5%. With high efficiency ratings of 94% and an output power factor of 0.90, there are savings in the total cost of ownership because the power to support protected loads is reduced.
Less heat is another benefit of high efficiency ratings, and that can lower facility-cooling costs. “Efficiency has several dimensions,” says Davis. “First, it means that you use less utility power, because you're transferring more of it directly to the load rather than losing it to inefficiencies. Then, with less heat, your cooling expenses will fall. Even a 2% increase in efficiency could mean saving hundreds of thousands or millions of dollars a year, depending on the size of the power system.”
Battery Storage Evolves
If the UPS depends upon batteries, there are efficiency gains from new voltage and design configurations designed specifically for UPS applications. According to Steve Vechy, director of UPS and utility marketing at EnerSys, Reading, PA, the batteries used for UPS applications are designed to back up the primary power systems, typically supplying power for 15 minutes until the backup generator is fully operational.
Until recently, these batteries were limited to 12-V batteries. The market has since moved to “larger-sized UPS systems from 500-kilovolt-ampere, or even 750-kilovolt-ampere or 1,000-kilovolt-or-larger systems,” says Vechy. As a result, UPS developers like EnerSys understood that “the prudent choice was to develop a larger format battery with a 925-watts-per-cell size, and a larger format plate to accommodate larger systems with fewer parallel strings and 20% less space.”
The benefits of front access include: up to 50% fewer connections, simplified installation, plus easier access and maintenance. “Historically, the valve regulated battery market in the US has used top terminated batteries, but they are difficult to maintain and need additional headspace at the top of the cabinet to gain access,” explains Vechy. “Front-terminated units take up less space and are inherently much safer from a maintenance standpoint, because your connections are right there in front of you and accessible.”
The subject of maintenance could be an article in itself, but Vechy notes that, in general, front terminal designs require annual checks of connections, unit voltage, and internal ohmic measurements.
Double Conversion Makes a Triple Threat to Efficiency
The typical data center loses significant amounts of power during the “double conversion” process of converting AC utility power to DC for conditioning and storage in the UPS system, then inverting back to AC when returning the power to the actual servers and other electrical loads within the center. “You’re giving up energy in the form of heat during the conversion process, and from DC to AC there are losses again, so you have losses occurring twice within the box,” says Gary Rackow, vice president sales, at Active Power, a flywheel-based UPS manufacturer in Austin, TX. The losses may not look like much of the overall power consumption, but even small amounts can be costly.
“The reason a few more percentage points of efficiency is so dramatic is because the utility rates are so high,” says Rackow. “The rule of thumb for conservative estimates is eight cents per kilowatt-hour, but, in places like Hawaii, it’s more like 12 to 14 cents per kilowatt-hour. If you use eight cents per kilowatt-hour and 6% efficiency rating in comparing one architecture to another, you would see a savings of about $50,000 a year per megawatt of load.”
The solution, according to Rackow, can be as simple as a UPS that operates as a parallel online system—rather than a double-conversion system—and isolates the input and output to high impedance for very low power losses. Active Power’s Clean Source UPS, for example, utilizes a parallel system that operates by running another parallel power path that keeps the flywheel spinning as a kinetic-stored energy device. Additionally, there are electronics running in parallel with the utility on the load side. “We condition the utility power actively on the load side, and that saves the power from suffering losses,” explains Rackow. “It doesn’t require a lot of power to do that, so we are saving six or seven percentage points over other storage systems.”
Another benefit to flywheel systems is their smaller space requirements when compared to battery systems, and their ability to operate in temperatures as high as 104˚F (The industry rule of thumb for a stationary VRLA battery kept at a constant state of charge (float life) says there is a 50% reduction in life for every 8˚C (14.4˚F) increase in temperature above optimum 25˚C (77˚F).
Power in a Box Matches the New Modular Approach
Active Power offers an option called “PowerHouse,” a modular, portable continuous power and cooling system housed in a standard 8-foot-by-40-foot shipping container. The unit has a flywheel UPS, standby diesel generator, switchgear, and chiller. It’s designed to support containerized data center offering from Sun Microsystems and Hewlett-Packard.
It’s no surprise that the backup engines in those containers use diesel fuel. Diesel-powered backup generators have long been the overwhelming choice for data centers. They have proven to be highly reliable, and maintenance is minimal, says Gary Olson, director of power systems at Cummins Power Generation, Minneapolis, MN. “Standby applications in North America don’t run very often, so they don’t need much,” says Olson. “They should be kept clean and have regular coolant and lubrication changes. It’s similar to the kind of maintenance routines you do on your car, except it’s done a lot more scientifically. The oil is tested to find out when it’s starting to deteriorate, and that’s when we change it. Also, they need to be checked regularly to make sure they’re operating properly.”
Cummins and other diesel generator manufacturers have benefited from the rapid growth of data centers, but Olson sees the slow uptake of distributed energy as a missed opportunity. “As I look at all the alternatives, onsite cogeneration is the one that has been totally ignored, and the one that could make the biggest impact.”
Manos, of Digital Realty Trust, agrees. He has done a number of different studies about cogeneration and says it makes sense within the data center application model. The same can be heard from one of the industry’s most venerable suppliers of data server hardware, IBM. In May 2009, IBM teamed with Syracuse University and New York State to build and operate a new computer data center on the university’s campus that’s expected to slash energy usage by as much as 50% less than the current usage seen at typical data centers.
Distributed Energy for IBM’s Smarter Planet Initiative
The $12.4-million, 6,000-square-foot data center incorporates both advanced infrastructure and smarter computing technologies. A critical factor driving the energy savings is an onsite electrical cogeneration system based upon 12 natural gas–fueled microturbines that allow complete freedom from utility power.
The project is part of IBM’s “Smarter Planet” initiative, and, according to Vijay Lund, vice president for development and manufacturing operations in IBM’s Systems and Technology Group, energy consumption is a critical issue. “Energy use is becoming the largest single cost in operating data centers—with $2 billion per year wasted nationally due to inefficiencies,” says Lund. The company will contribute more than $5 million in equipment, design services, and support, plus the electrical cogeneration equipment and servers such as the IBM BladeCenter, IBM Power 575, and IBM z10 systems, and importantly, IBM’s innovative, Rear Door Heat eXchanger “cooling doors.”
The doors pull heat out of the backs of individual server racks with a liquid cooling apparatus (not unlike the concept of an automobile’s radiator), rather than the traditional less efficient method of pushing air conditioned cold air into a room of servers, then pulling the hot air out. Cogeneration will boost the efficiency because the double-effect absorption chillers in the liquid cooling system convert the exhaust heat from the microturbines into chilled water. In fact, there’s enough capacity to cool the data center's servers and provide free cooling and heating of an adjacent building.
“We can control the environment in each rack of servers,” says Mark Weldon, executive director of corporate relations at Syracuse. “This is important when you use virtualization to take down a rack of servers and move that load somewhere else. You can shut off the cold water to that particular rack and use it where the load actually is. There is an additional synergy if you want to keep some servers or racks cold and move the load to the cooling rather than the other way around. In a conventional system, moving the cooling to the load is often difficult because it’s not unusual to have hotspots and cold spots in most data centers. Now you can be much more precise in managing your resources.”
Another benefit is a lower cost to cool the building and lower staffing requirements. Weldon notes that when employees do work on the servers, they will be more comfortable than their counterparts at conventional data centers. “Typical computer room air-conditioning fans make so much noise that you can barely hear yourself talk. In most mechanical systems noise is an indication of waste. It’s a sign of inefficiency and our data centers will be much quieter.” Weldon says the project will also search for further efficiency gains with innovative power conversion methods.
The project includes the creation of a DC power distribution system. Just as companies like Active Power are reducing AC to DC double-conversion efficiency losses, Syracuse also plans to address the issue. “Why not have the turbines generate DC power directly?” asks Weldon. The question offers a great opportunity for the C65 Hybrid UPS system microturbine, from Capstone Turbine Corporation, in Chatsworth, CA.
“A modification to our standard 65-kilowatt microturbine makes it a UPS, as well as a backup generator,” says Steve Gillette, vice president of Capstone.
The company calls it a hybrid because it has the functionality of a UPS and a standby system, and three modes of operation. First, there is a UPS mode with a microturbine turned off. If there’s no need for cooling or heating and electricity prices are lower than the price of burning natural gas, there’s no sense in generating onsite power. In that case, utility power travels to the load via a normal double-conversion UPS system. Then there is a high-efficiency mode for situations when electric rates are higher than the air-conditioning load. The microturbine would generate electricity for the servers and overhead, with the exhaust connected to an absorption chiller to provide cooling. It’s saving money, because the CHP system can generate the electricity at a lower cost than buying it from the grid. Finally, there’s an emergency mode for times when the utility fails and the microturbines aren’t running. The UPS system would temporarily take power from the DC bus and batteries to keep the critical load going, until the microturbines return online.
Gillette says the Capstone’s hybrid configuration makes it adaptable to the goal of creating a DC power distribution system. Basically the modifications are relatively small, the major change is in the software that controls the whole system. “Today, we have an integrated signal processor or control system and it manages the power output and the microturbine itself. It manages the DC within the system, and with the hybrid, we’ve added a second AC and now manage the DC to AC connections, plus the microturbine. Power can flow in any direction, and that’s pretty wild. It’s a 3-phase AC connection which we can even operate as a grid connect, with protective relay functions, so it can do anti-island operations and synchronize with the grid.”
Weldon adds that turbines typically output high voltage and most servers require low voltage. But, Syracuse and IBM are working to design a system that could use high voltage straight into the servers. “There are some sweet points in the high-voltage DC, and we’ll be doing some experiments with the servers to see if we can run somewhere between 400- to 600-volt DC power with the turbines and take it straight into the servers,” says Weldon.
According to IBM’s project manager, Robert Hanson, the company has modified one of its servers to run off of DC power distribution system and expects roughly a 10% energy savings. “With this approach, there are less conversions as the power comes directly off the microturbines and less stepping down of the voltage,” says Hanson. “Once we do the initial experiment, Syracuse can bring in new equipment to expand this component of the system.”
Research and usable data is a major theme for Syracuse on this project, and, along with the DC power experiments, the university will study and analyze thermodynamic models to predict energy consumption, plant and chiller efficiency, water-cooled server racks, and the energy-saving advantages of onsite generation systems.
Completion of the data center is expected by the end of 2009, but Hanson notes that it’s already having an impact on the industry. Since the project’s announcement in May of 2009, his team has consulted on dozens of projects where customers wanted to investigate alternative sources of distributed energy. “We’re seeing clients in the middle of Manhattan who can’t get more power to their buildings,” he says. “So, it’s a high priority for these places where they need more power, but also more efficient forms of power. And doing this in the distributed fashion seems to be the solution that everybody likes at this point in time.”
In fact, IBM can point to a number of innovative distributed energy examples. And Manos points out two more issues driving the industry towards onsite power production. “You’re finding more and more that the locations that have both telecommunications and electrical capacity are becoming rarer. It’s almost like an arms race to find those good locations.”
Finally, there are concerns about the carbon footprint of data centers. “Although it’s [cap-and-trade legislation] not aimed at data centers specifically, data centers are large contributors and consumers,” says Manos. “You have to ask, ‘what are the sources of the power?’ If you’re in West Virginia, the carbon emission factor is pretty bad, because you have high sulfur coal being burned to make electricity. In the UK, they have a tax of 12 pounds per ton of carbon, and data centers will pay significant amounts. So I think it’s going to become more prominent and prevalent as an opportunity for data centers of a certain size.”