15 September 2008 Features

 

IT & Ops 2: Hot houses

Tom Groenfeldt

Traditional data centres are becoming more demanding on power, cooling and storage, resulting in new challenges.

"Over the next three years half the data centres in the US will be closed and moved into new facilities, often driven by power issues. It's mind-boggling."

That's the view of Patrick Eitenbichler, director of marketing for HP's StorageWorks division. "You used to hear about power occasionally, but now you hear about power issues every week, including running out of power at the data centre," he says.

Financial firms around the world are facing major challenges in their data centres - existing buildings are filling up with servers and they often don't have the heating or power capacity to add more computers. In addition, rising power costs are making firms more aware than ever of their energy consumption - the EU says information technology is roughly equal to the aviation industry in producing 2% of the world's CO2.

A US-based IBM consultant who visited London in the spring was surprised to find a bank IT manager concerned about summer temperatures - if they were high, would he have enough power for his computers?

The major technology providers are stepping up with plans for new data centres and ways to squeeze a little more time out of the existing centres.

"Each new server costs you $10,000 a year until the one that requires a new data centre, and that costs you $100 million," says Kosten Metreweli, vice president of product marketing at Tideway Systems, a specialist in monitoring computers and data centres.

At the same time they are adding computing power, banks are at the forefront, at least in their public relations campaigns, of going green. Wherever they are in their PR efforts, London banks are facing limits on power until after the Olympics because most EDF - the local power utility - substations are maxed out, says Mike Hill, director of products and services for Adapt in London.  Investment banks want to improve their balance sheets, so rather than build their own data centres, they looking to outsource and lease new data centre capacity, he adds.

Technology firms and consultants offer some ways that banks can improve existing operations and develop new data centres that will reduce costs and energy consumption.

Within existing data centres leading firms are using automated tools to locate unplug idle servers and virtualise underutilised machines.  A manual audit of a data centre with 1,000 servers costs approximately $75,000, says Metreweli, while Tideway's product can audit a data centre daily. A small investment bank in London using Tideway found they could virtualise 30% of their servers, which gave them a carbon savings of 968 tons of CO2 per year.

"If you don't know about your servers you can't virtualise them," Metreweli adds.

Old inefficient servers are being replaced with new processors that are more energy efficient. Banks are using virtualisation from VMware, Microsoft, Citrix and other vendors to put increase utilisation of Windows and Unix servers which often operate at 10-20% of capacity (see related feature). All the major hardware vendors such as Sun, HP, and IBM offer improved cooling services which analyse air flows and direct cooling where it is most effective. 

A few banks and some outsourcing firms are building entirely new data centres.

One of the most modern is Wachovia's new data centre near Birmingham, Alabama, in America's Deep South. While it might seem counterintuitive to put a data centre in a hot climate, HP's Ben Bauer, worldwide marketing manager for outsourcing services, says that once you fully insulate a centre, where it is located doesn't matter so much - HP has centres in Atlanta and Houston where temperatures are high but power is relatively cheap.

Jim Houghton, who was the executive for architecture and strategy in the infrastructure organisation at Wachovia during the development of the new data centre, says the bank took the opportunity to virtualise much of its operations through heavy use of grid server and fabric server technology from Data Synapse in tandem with other products.

Wachovia used Tideway to see how their systems are configured in a runtime environment, which helped them make the move to a new data centre and refresh their systems at the same time, says Metreweli. This approach led to much greater savings than would have been achieved if the systems refresh had been done after the move. Designing around SOA has led to efficiencies that help the bank meet its green goals.

Firms are understandably worried about unplugging servers because they can't be sure which servers depend on other servers. 80% of the downtime occurring in moves is the result of not knowing about dependences, according to Tideway. The problem extends to software. At least with servers, says Metreweli, you can walk around with a clipboard and count the tin; automated checking of infrastructure can find duplicate software licenses and save money by consolidating not only hardware but software, like Oracle licenses.

Wachovia has dynamic products running on a dynamic infrastructure so it can shift server capacity from batch to transactional processing during the day as demand warrants, says Houghton. If Federal Reserve chairman Ben Bernanke makes an interest rate announcement during the day, the bank could shift computing resources to understand the impact.

"It would be wonderful to recalculate positions, which is typically done overnight, on the fly during the day," says Houghton. "That would improve your risk posture and financial business intelligence will give you a leg up on your competition."

The first step in changing an IT infrastructure is understanding what already exists, as Wachovia did with Tideway. Tideway can be used to do an audit of existing servers; large firms typically find that after a month it identifies 3 to 5% of the servers in a large centre as doing nothing except consuming electricity. They can be unplugged.

"Wachovia used Tideway Foundation to see how their systems are configured in a run-time environment, which helped them move to the new data centre based on new technology platforms," explains Metreweli, Houghton says they tested Tideway against others and learned it was much better at picking up an application fingerprint even if the environment changes.

"The other tools were fairly dumb when it came to interpretation of the data," he adds.

Hougton says Wachovia's new data centre has all the bells and whistles of greenery, including non-traditional power, although he thinks there is more to be gained by improving efficiency in the facility than looking for power from windmills.

"You can change all your light bulbs, or you can teach your kids to turn the lights out." Wachovia identified unnecessary or underutilised systems and some obsolete servers that should have been decommissioned years ago.

The power savings of a few hundred servers off the floor is far better far better than the expensive cooling systems that are sometimes installed in data centres, he adds. "Folks who are spending tens of millions of dollars retrofitting existing data centres are out of their minds. The ROI simply is not there." But for a new data centre, smart cooling absolutely makes sense.

Part of the problem, says Adapt's Hill, is that many existing data centres were built for the dotcom era to support e-commerce and Internet independent service providers. They typically had a couple of servers and relatively light power demands - about 750 Watts per square meter.

Financial firms are running racks of blade servers that consume anything from 8 to 15 kW per square metre. To get the power they want banks are taking up far more space than the racks require; one US bank needed 200 square metres, but for their power, he had to allocate them 800.

"So data centres are running at 90-95% of their power capacity but only 55 to 60% of their space capacity."

Now new data centres around London are being built by commercial developers who locate them outside the M25 and offer 2,500 Watts per square metre.

"Some offer high density - 15 kW in two square meters which is impossible in London."

One bank Adapt is working with has 858 physical servers across four data centres across London. It wants to use virtualisation and HP blades to reduce that to 150 servers and achieve a 45 percent reduction in cost.

Power constraints, rising costs of electricity, board and, soon, EU-directed green initiatives and innovations in data centre cooling require senior executives to learn more about facilities management and power grids than they would ever have imagined five years ago.

IT typically didn't talk to facilities - they ordered the blades and expected the building would have the floor strength, power and cooling required. At an Intel financial services conference in New York a few years ago, representatives from two major banks - one in New York and one in London - talked about installing racks of blades and then learning they could power only half of the new installation. IT has learned a lot about power and cooling since then, but deeper understanding and better communication is still needed if banks are to succeed in managing their infrastructure.

How a data centre connects to power also makes a difference, says Adapt's Hill. Older centres built by telecommunications firms used 11 kVA, but if a data centre spends another £10 million it can connect at 133 kVA which means the centre is paying 6 pence per hour while other users are paying 27 pence.

"The banks are only starting to get their head around how to connect to the electrical grid."

Houghton, who now works in a consultancy called Adaptability, says the projected wave of new data centres offers a real opportunity.

"I find that people managing a move might focus on a particular tier and perhaps the tier above and the tier below, but not on the complete picture. This usually has to do with span of control. It is rare to find someone who grasps the broad picture from the front office business impact down to the efficiency of cooling and air flows."

But when a firm is spending $100 to $500 million for a new data centre, it should have a comprehensive plan that covers everything from environmental concerns to business demands and electrical supplies. People who understand power demands are at the bottom of the stack in a firm, while the IT manager who buys servers is a couple of levels up and they usually don't communicate. Yet the server power requirements depend on how much it is used; if a firm estimates power based on the specification plate on the back of the machine, it will wind up grossly over-provisioning. On the other side, 10 years ago no one was projecting today's high density power demands, and it doesn't make sense to spend half a billion on a data centre with power supplies that will barely last a year, much less the 15-year life expectancy of the centre.

"It's rare to find people who understand how to talk to a COO of a business unit and to the guy who is pulling power cables in the data centre. But you need to understand the end-to-end environment."

Sidebar: storage strategies

One of the places that data centres are saving on power is through sophisticated storage. Storage accounts for 14% to 17% of the energy consumed in data centres, according to Patrick Eitenbichler, director of marketing at HP's StorageWorks Division.

HP consultants can help large firms move the utilisation rates of spinning disks from an average of 60% today to 80% to 90%. Eitenbichler says that as firms undertake new projects they typically assign storage. If the project doesn't use it all, or when it terminates, the storage often still sits unused and unnoticed.

A second approach is to keep only the most accessed data on the high-speed 15,000 rpm disk drives and then after three or six months move the data to a lower speed high capacity SATA drive. Regulation has been good for the data storage industry - regulators are requiring financial firms to hold more data for longer periods. Eitenhichler says that for long storage WORM (Write Once, Read Many) tape storage is the best because it can be encrypted and protected from modification.

Steve Willson, solutions manager for long-term storage specialist firm Copan Systems in London, contends tape is too hard to search. Copan uses Massive Array of Idle Disk (MAID) which he says is 5 to 6 times more power efficient that regular disk storage and capable of far higher density - it can store a petabyte on a single disk.

"If you aren't trying to access the disk, it will turn off. Most storage doesn't do that." MAID costs are approaching tape, he adds. A tape cartridge costs $1 to $2 per gigabyte of storage and MAID is $2 to $4. Copan wins business not so much on price as on power efficiency as banks face a limit on the available power and look for ways to reduce unnecessary consumption of electricity.

Security is another attraction of MAID, he adds. "Many banks have moved away from tape because of security because tape losses have been a major issue. Some customers won't let us move a disk even if it fails; they want it destroyed on site."