There is a shortage of GPUs as demand for generative AI for training and execution grows. Nvidia's best performing chips, according to reports, are out of stock until 2024. The CEO of chipmaker TSMC was less optimistic recently, indicating that the GPU shortage from Nvidia, as well as its rivals, could extend into 2025.
To decrease their dependence on GPUs, companies that can afford it (i.e., tech giants) are developing (and in some cases making available to customers) custom chips designed to create, iterate, and produce AI models. One of those companies is Amazon, which at its annual re:Invent conference unveiled the latest generation of its chips for model training and inference, that is, running trained models.
The first of two, AWS Trainium2, is designed to deliver up to 4x better performance and 2x better power efficiency than the first-generation Trainium, introduced in December 2020, Amazon predicts. Tranium2, which will be available on EC Trn2 instances in groups of 16 chips in the AWS cloud, can scale up to 100.000 chips in the AWS EC2 UltraCluster product.
100.000 Trainium chips offer 65 exaflops of computing, Amazon says, which is equivalent to 650 teraflops for a single chip. “Exaflops” and “teraflops” measure how many computing operations per second a chip can perform. There are likely complicated factors that make that simple math not necessarily so accurate. But assuming a single Tranium2 chip can deliver roughly 200 teraflops of performance, that means are above the capacity of Google's custom AI training chips from around 2017.
Amazon says a cluster of 100.000 Trainium chips can train a large AI language model with 300 billion parameters in weeks instead of months. (“Parameters” are the parts of a model learned from training data and essentially define the model's ability at a problem, such as generating text or code.) That's about 1,75 times the size of OpenAI's GPT-3, the predecessor to the GPT-4 text generator.
“Silicon underpins every customer workload, making it a critical area of innovation for AWS,” AWS Vice President of Computing and Networking David Brown said in a press release. “With increasing interest in generative AI, Tranium2 will help customers train their ML models faster, at lower cost and with better energy efficiency.”
Amazon did not say when Trainium2 instances will be available to AWS customers, other than "sometime next year."
The second chip that Amazon announced, the based on ARM Graviton4, is intended for inference. The fourth generation of Amazon's Graviton chip family (as implied by the "4" attached to "Graviton"), is distinct from Amazon's other inference chip, Inferentia.
Amazon claims that Graviton4 provides up to 30% more computing performance, 50% more cores, and 75% more memory bandwidth than a previous generation Graviton processor, Graviton3 (but not the newer Graviton3E), which runs on Amazon EC2. In another update to Graviton3, all of Graviton4's physical hardware interfaces are "encrypted," Amazon says, apparently better protecting AI workloads and training data for customers with higher encryption requirements.
"Graviton4 marks the fourth generation we have delivered in just five years and is the most powerful and energy-efficient chip we have ever built for a wide range of workloads," Brown continued in a statement. By focusing our chip designs on real workloads that matter to customers, we can offer them the cloud infrastructure more advanced.
Graviton4 will be available on Amazon EC2 R8g instances, which are already available in preview and are scheduled for general availability in the coming months.
