Invited Speakers:
Training Large Language Models on Cerebras Wafer Scale Clusters
Natalia Vassilieva (Sr. Director of Product, Machine Learning - Cerebras Systems)
Large Language Models (LLMs) are shifting “what’s possible”, but require massive compute and massive complexity of distributed training across thousands of accelerators with traditional hardware. Cerebras Wafer Scale Clusters make training LLMs faster and easier compared to GPUs due to near-perfect linear scaling and simple data-parallel distribution strategy for models of any size. In this talk we will share our experience and insights from training various LLMs, including open-sourced family of Cerebras-GPT models, on the Cerebras hardware.
Codesigning Computing Systems for Artificial Intelligence
Suvinay Subramanian (Staff Software Engineer - Google)
The rapid advancement of artificial intelligence (AI) has ushered in an era of unprecedented computational demands, necessitating continuous innovation in computing systems. In this talk, we will highlight how codesign has been a key paradigm in enabling innovative solutions and state-of-the-art performance in Google's AI computing systems, namely Tensor Processing Units (TPUs). We present several codesign case studies across different layers of the stack, spanning hardware, systems, software, algorithms, all the way up to the datacenter. We discuss how TPUs have made judicious, yet opinionated bets in our design choices, and how these design choices have not only kept pace with the blistering rate of change, but also enabled many of the breakthroughs in AI.
Building a Cloud-Native Platform for the Future of AI: Foundation Models
Carlos Costa (Principal Research Staff Member - IBM)
Foundation Model is an emerging inflection point in the creation of powerful, very high dimensional data representations, triggered by advances in AI. Foundation Models in AI are billion-parameter-scale neural networks, powered by novel architectures which are trained using a technique called self-supervision. This new paradigm imposes unprecedented opportunities and challenges across the full computing stack. Hear how IBM Research is expanding and realizing the value of Foundation Models, from building a cloud-native supercomputing infrastructure and a simplified, cloud-native common stack to train and deploy Foundation Models in an multicloud environment, to applying this full stack to enable advances in natural language domain and beyond, including time series and code generation.
Modeling and Mitigating Communication Bottlenecks for Large Model Training at Scale
Tushar Krishna (Associate Professor - Georgia Institute of Technology)
The unprecedented success of large language models (LLMs) — such as Open AI's GPT-3 and GPT-4, Google's Bard, Meta's LLaMa , Cerebras-GPT and others — is emphasizing the ever-growing demand to efficiently train them. These models leverage billions to trillions of model parameters and this trend continues to increase at an unforeseen rate. The large model size makes it impossible for their parameters to fit within a single accelerator device, whose memory is usually capped at tens of GBs. Furthermore, even if we succeed to fit the model into a single device, their tremendous compute requirement leads to almost impractical training time. For example, GPT-3 consists of 175B parameters and takes 355 GPU-years to train with a single NVIDIA V100 GPU. This has led to a growing interest in distributed training, which is the idea of sharding model weights and/or data samples across multiple accelerator devices. However, this comes at the expense of communication overhead to exchange gradients and activations, and it has already become a key bottleneck for distributed training. We identify that the communication challenge will get exacerbated in future systems that are expected to leverage multi-dimensional networks with heterogeneous bandwidths due to diverse fabric technologies (e.g., chiplets, rack-scale, and scaleout). We present our recent works on (i) modeling future training platforms to identify such bottlenecks, and (ii) a novel runtime scheduling policy to enhance network bandwidth utilization.