Meta is building a supercomputer for the sole purpose of training AI models, and it’s already built one that’s on par with the world’s fifth-fastest supercomputer.
Meta announced its latest creation: a 16,000 GPU-based supercomputer that can train AI models 3x faster than existing platforms. To get better context, consider this: The current record for most GPUs in a single machine (8,402) is held by China’s Sunway TaihuLight. And while it may seem like Meta went all-in on GPUs, according to the startup, its new supercomputer utilizes only 6,080 GPUs—the rest are Tesla V100s (a new breed of GPU).
In other words, Meta uses fewer but more powerful GPUs to push the boundaries of computing power. Once complete, the computer will measure 100 meters long, 50 meters wide, and 7 meters high.
Use of Supercomputers
Supercomputers are used for various tasks—from financial modeling to weather forecasting and defense simulation. Meta’s supercomputer, however, is designed for training large-scale language models, which work as the brain of neural networks that power all sorts of AI applications.
Meta expects to see significant gains in its ML (machine learning) research and development with this new supercomputer installed in Canada or Northern California, pending regulatory approval.
The advantage goes beyond just the speed; it comes down to how the data is processed on the chip. Think of a very efficient way of doing matrix multiplication; the hardware is designed to take a whole bunch of numbers, multiply them against each other and add them together all in one operation.
Development of Supercomputers
Meta is developing a new virtual machine called RSC (Reconfigurable Supercomputer). The company says it’s designed to help power the next generation of computing technologies in AI. This is part of an important effort by the social media giant to drive its technology development.
The RSC system is built around an architecture with memory and processing that can be dynamically reconfigured on-the-fly, to meet the needs of different tasks. In 2020, Meta engineers started development on a new supercomputer. As they called it, the RSC would be the world’s fastest. With this unprecedented computing power, users could run machine learning algorithms that had never been possible before.
Meta engineers have also extensively used machine learning in designing the RSC, taking advantage of data collected from more than 300 earlier deployments to create an efficient and fault-tolerant architecture. The result is one of the most accurate supercomputers ever built, delivering on its promise to offer unprecedented speed while being remarkably resilient.
Building Supercomputers
By building a supercomputer with 16,000 GPUs, Meta hopes RSC will make language models three times bigger than those produced by existing RSCs—and do it far more quickly. A larger language model would better understand the intent of speech and text and could improve things like Google’s search results, which currently rely on humans to correct its many transcription errors.
The Bigger Picture
Meta’s significant goal is to improve its products and make breakthroughs that others can apply. The company has already shared some of its research (including things like spatial mapping and 3D reconstruction) with other companies through its Meta Labs program — now, it just needs to scale those efforts up considerably.
You can also expect the technology industry to continue to debate the ethics of large language models, which some have suggested can be biased against certain groups of people. And suppose large language models become best-in-class for AI tasks. In that case, it stands to reason that tech giants will begin creating them for other applications like autonomous vehicles or chatbots that can hold long conversations with you.
We Can’t-Wait
We’re excited about RSC’s current progress and future potential. With RSC-powered artificial intelligence, the AR world will advance its language capabilities faster and make the experience better for everyone. Find internet providers that can enhance your online experience significantly; you can thank us for the uninterrupted connectivity later.