Improved system aids developers in creating more effective simulation and artificial intelligence prototypes.
AI models, such as those used in medical imaging and speech recognition, work with massive, complex data structures that demand significant computational power to process. MIT researchers developed an automated system that allows developers to harness two types of data redundancies in deep learning algorithms, resulting in less computation, bandwidth, and memory storage needed for machine learning operations.
Existing optimization methods for algorithms can be intricate and typically only capitalize on either sparsity or symmetry, leaving ovaction to boost computations by nearly 30 times in some experiments by enabling a developer to build an algorithm that takes advantage of both redundancies simultaneously.
This user-friendly programming language system could potentially optimize machine learning algorithms for a wide array of applications, even in the hands of those who aren't experts in deep learning. It could also aid scientists in enhancing the efficiency of AI algorithms they use to process data, with potential applications in scientific computing.
In machine learning, data is often represented as multidimensional arrays known as tensors. Engineers can boost the speed of a neural network by cutting out redundant computations. Sparstity, where data has many zero values, and symmetry, where certain patterns repeat in data, offer significant reductions in computational costs and improvement in model efficiency.
To simplify the process, the researchers built a new compiler, SySTeC. This compiler can optimize computations by automatically taking advantage of both sparsity and symmetry in tensors. SySTeC performs optimizations using symmetry, like calculating only half of the output tensor if it is symmetric, reading only one half of the input tensor if it is symmetric, and skipping redundant computations if intermediate results are symmetric. It then optimizes the program for sparsity by only storing non-zero data values.
Utilizing symmetry can yield even more savings on computation as the tensor has more dimensions. The researchers demonstrated speedups of nearly a factor of 30 with code generated automatically by SySTeC. As it is an automated system, it could be particularly advantageous in situations where a scientist wants to process data using an algorithm they are writing from scratch.
In the future, the researchers aim to integrate SySTeC into existing sparse tensor compiler systems to create a seamless interface for users and optimize code for more complex programs. This research is funded in part by Intel, the National Science Foundation, the Defense Advanced Research Projects Agency, and the Department of Energy.
Additional research in this domain includes the use of deep reinforcement learning for code optimization and the broader applications of AI in engineering and decision-making. Leveraging sparsity and symmetry in deep learning models could enable faster training times, reduce memory usage, and improve model accuracy, making them suitable for real-time applications or where computational resources are limited. These optimized models could also be applied in robotics and autonomous systems for improved performance in real-time decision-making tasks and in scientific computing and data-intensive fields for breakthroughs in physics, chemistry, and life sciences.
- The automated system developed by MIT researchers allows developers to capitalize on both sparsity and symmetry in deep learning algorithms, leading to less computation, bandwidth, and memory storage needed for machine learning operations.
- By optimizing computations using symmetry and sparsity, the newly developed compiler, SySTeC, can significantly reduce computational costs and improve model efficiency.
- In the future, SySTeC could potentially be integrated into existing sparse tensor compiler systems for a seamless interface, aiding scientists in enhancing the efficiency of AI algorithms used in scientific computing.
- Leveraging sparsity and symmetry in deep learning models could open up opportunities for faster training times, reduced memory usage, and improved model accuracy, making them suitable for real-time applications, robotics, autonomous systems, and data-intensive fields like physics, chemistry, and life sciences.
