Engineers at Huawei have unveiled a computer architecture they say could one day make up to a million artificial intelligence chips work as one, Live Science reported. The design, called Peerium, was presented Sept. 17 at the company's annual CONNECT conference. Huawei said computers powered by the architecture could train models with up to 40 trillion parameters. For comparison, the article said competing hardware such as Nvidia's GB200 NVL72 and Cerebras's CS-3 has been reported to train at the trillion-parameter scale, rather than the tens of trillions. Live Science also noted that Elon Musk's AI company, which it identified as SpaceXAI and said is soon to be SpaceXSI, describes its Colossus cluster as comprising 200,000 interconnected chips.
"Over the next two years, there will be roughly 6 or 7 frontier AI labs in China, and all will aim at training foundation language models of sizes ranging from 10 to 40 trillion parameters," Liao Heng, chief scientist for Huawei's chip design arm HiSilicon, told the Associated Press. "So you need such a scale of infrastructure to train these models." Huawei said Peerium relies on nested parallelism, unified memory addressing, and peer-to-peer interconnects. Nested parallelism breaks workloads into smaller tasks across hierarchical groups of chips. Unified memory addressing lets processors reach memory on other servers, not only their own. Peer-to-peer interconnects let components talk more directly, rather than through an intermediary.
Existing links, Huawei argued, cannot support that scale. Nvidia uses NVLink within racks and InfiniBand between racks, with inter-rack bandwidth of 200 gigabytes per second, rotating chairman Eric Xu told the AP. "With such a low inter-rack communications bandwidth, there is no way you can make one million processors work as one computer," he said. Huawei said it developed its own interconnect, UnifiedBus, with inter-rack bandwidth of up to 800 gigabytes per second. "When one million processors work as one computer, this lays the foundation for better and more efficient AI training and inference," Xu said. "I believe, in the years to come, other players would certainly follow suit."
Alongside Peerium, Huawei announced the Atlas 950 SuperPoD, a system built on the architecture and powered by more than 8,000 of the company's Ascend 950DT AI accelerator chips. Xu said testing for AI training is underway and that results show the SuperPoDs use available computing power more efficiently than clusters not based on Peerium and UnifiedBus. Huawei expects to begin supplying the SuperPoDs by early next year, at the latest, before SuperClusters that link multiple SuperPoDs come online. Representatives said one SuperCluster with 256,000 computing cards is currently being deployed and tested. The verified announcement is a design, early SuperPoD testing, and a supply timetable, not a demonstration that a million chips are already working as a single computer.