Heat Transfer & Cooling

A breakthrough in data center efficiency is emerging, pioneered by Hong Kong-based Allied Control. The progressive engineering company, recently acquired by BitFury Group, a leading Blockchain infrastructure provider and transaction processing company, is reaching new heights in two-phase immersion cooling (2PIC). This new cooling technology is proving to significantly increase power density, reduce energy consumption and deployment time, and – when paired with a 3M™ Novec™ Engineered Fluid – deliver on environmental sustainability.

In 2014, with two successful 2PIC projects behind it, Allied Control was approached with its biggest 2PIC project yet – to realize and surpass a long-sought technology vision and build a scalable 40+ MW facility comprising 160 tanks with a power density of 250kW per tank running a PUE of 1.02. This third-generation project would be the most powerful 2PIC data center in the world. The client was BitFury Group, which builds its own full-custom Application Specific Integrated Circuits (ASICs), PCBs, servers and data centers. The company uses renewable energy for all its business operations and remains committed to maintaining the smallest carbon footprint possible. BitFury will leverage Allied Control’s award winning 2PIC technology to deploy its 28 nm and 16 nm ASICs.

Allied Control 2PIC Projects

	Generation 1	Generation 2	Generation 3
Date	November 2012	October 2013	October 2015
Size/Design	70kW	500kW 25kW racks scalable to 225kW per rack	40+ MW comprising 250 kW flat-lying tanks
Hardware	FPGA	ASIC compute clusters	ASIC compute clusters
Fluid	3M Novec 7100	3M Novec 7100	3M Novec 7100
PUE	1.02	1.02	1.02
Compute power per ft2	.25 kW/ft2	1.01 kW/ft2	3.23 kW/ft2

3M immersion coolants, a brief history

3M has a long-held leadership position in immersion cooling. In the 1950s, 3M introduced its first dielectric fluorochemical heat transfer fluids for direct contact cooling for military avionics. The 1970s and 1980s brought an era of direct contact cooling which helped enable the development of dense radar, power electronic and supercomputer systems.

In 1996, 3M™ Novec™ Engineered Fluids – a new class of sustainable chemistries – were introduced as replacements for ozone-depleting substances, such as CFCs and greenhouse gases such as HFCs. Novec fluids have a host of characteristics that set them apart. They’re non-conductive, non-ozone depleting and nonflammable with a low toxicity and low global warming potential.

Over the past five years, 3M engineered fluids have been used in server cooling in a series of technology demonstrations, which have been recognized by the industry for best in class energy efficiency.

2PIC with a 3M™ Novec™ Engineered Fluid

2PIC involves placing hardware in an accessible tank of dielectric fluid coolant with a low boiling point, e.g., 3M Novec 7100 Engineered Fluid boils at 61°C (142°F). As the hardware heats up in the tank, it boils the Novec fluid, and this change in state pulls the heat away from the components.

The Novec fluid becomes a vapor, rises, condenses on a water-cooled condenser coil and falls back into the tank without the use of pumps. The streamlined system can deliver as much as 95 percent cooling energy savings with minimal fluid loss.

Boards designed for 2PIC without bulky heat sinks and cold plates can be densely packed with processing hardware, creating a new path to previously unattainable performance capabilities with improved overclocking capacity – 500W ASICs have been successfully overclocked to well over 750W.

The fast-paced world of blockchain Bitcoin, a cryptocurrency and consensus network, enables a new payment system and is the first decentralized peer-to-peer payment network that is powered by its users. Bitcoin blockchain began as opensource software in 2009. At that time, hobbyists could use desktop PCs with conventional CPUs to perform the hashing required to verify transactions and publish them to the public blockchain. Soon, Bitcoin enthusiasts realized that GPUs were faster and more efficient. As the breakeven processing efficiency for blockchain processing increased, GPUs gave way to FPGA and then evolved to lowend ASIC chips. At present, leading industry technologies comprise multi-megawatt arrays with purposebuilt ASICs.

There are many benchmarks that can be used to measure the growth of this rapidly evolving industry, such as the number of new merchants accepting the cryptocurrency or the transaction volume across the network. The most impressive metric by far, however, is the astronomical increase in the processing power of the network of computers involved in processing blockchain transactions and creating new bitcoins. The ability to create densely packed circuit boards paired with highperformance during overclocking made 2PIC a natural fit for blockchain transaction processing, an industry with a need for processing power that is growing at unprecedented rates. This growth and progression can provide a roadmap for high performance computing as it evolves toward and beyond exascale.

Considering the total value of 2PIC

For BitFury and Allied Control, building a 2PIC system with 3M Novec Engineered Fluid offered a key feature that set it apart – scalability. The ability to accommodate rapidly evolving blockchain transaction processing hardware without the costs of integrating a new thermal management solution for each generation created more long-term value. “With 250 kW per rack and not having reached the limits yet, we now have the ability to grow quickly and spread the cost of the cooling system over many hardware generations,” said Kar-Wing Lau, vice president of operations at Allied Control.

Allied Control designed BitFury’s setup in a matter of months, and at present, the ground-up build is in progress. The target is to have 40MW up and running at 1.02 PUE within October 2015 and scale up to more in near future. This commitment underlines the expectation to create significant energy and cost savings in a very short amount of time.

In terms of power density per square foot, the space required to house BitFury’s system is phenomenally small – as much as ten times smaller than traditional air cooling. The system avoids additional costs associated with traditional chillers relying instead on dry coolers that efficiently manage the load at ambient temperatures of 40°C and enable waste heat capture.

2PIC with a Novec engineered fluid will enable BitFury to lower operational costs by allowing them to run ASICs more efficiently. 2PIC makes it possible to run fewer chips hotter, especially early in the product life cycle. 2PIC also provides the capability to recycle virtually all of the waste heat for local industrial or municipal use, lowering net operational costs. The effect is operating at nearly carbon neutral when using renewable energy to power the immersion cooling system.

Looking at 2PIC for the future

When fully operational, this third generation project for Allied Control marks another huge step forward for not only blockchain transaction processing, but the entire supercomputing industry. “The PUE we’ve achieved leads us to the next level of data center cooling for high performance computing,” said Valery Vavilov, BitFury CEO. “We’ve taken many steps to optimize our infrastructure for securing the blockchain and blockchain transaction processing that have led us to many different technological innovations. These experiences and technologies could be useful for next generation data center constructions of high performance computing and supercomputing.”

Throughout the private sector, the age of cloud computing has created a growing need for more powerful data centers across multiple industries. With rising energy costs, data center efficiency becomes a key part of financial success. Meanwhile in the public sector, on March 19, 2015, President Barack Obama mandated that all legacy government data centers in the U.S. reach a PUE of <1.5 and all new data centers run between 1.2 and 1.4 by 2025.2

The answer for increased data center efficiency for the public and private sector may lie in more advanced cooling systems. As evidenced in all three generations of Allied Control’s systems, using 2PIC with a Novec engineered fluid can quickly and dramatically decrease PUE to optimal rates and reduce enormous costs linked to fractional floor space requirements. Finally, and perhaps most importantly, a 2PIC system can help organizations with more powerful hardware configurations lead in a future where compute power is its own evolving form of currency.