Ax4DNNs: Integrating Approximate Computing with Deep Neural Networks

Neural networks (NNs) have now become a ubiquitous and fundamental aspect of modern computing. At their core, they are computational models designed to recognize patterns, classify data, and make decisions based on input information. Inspired by the way the human brain works, neural networks are made up of layers of interconnected neurons that process and transform input data; their ability to learn patterns and hierarchical features from large amounts of data has revolutionized fields such as computer vision, natural language processing, robotics, image recognition, and more~\cite{KrizhevskyJun17}.
Despite their widespread and undisputed utility, however, NNs are inherently extremely resource-intensive, and training large models, especially for deep neural networks (DNNs), requires enormous computational power. The energy consumption associated with training and inference in these networks has become a growing concern, with recent studies suggesting that training large-scale models is contributing to significant environmental impact~\cite{SchwartzJan20}, and, specifically, that training a single state-of-the-art language model can emit as much carbon dioxide as five cars over their entire lifetimes~\cite{StrubellJan19, StrubellFeb20}.

An obvious and urgent research question arises: can we continue to reap the benefits of DNNs, but at a much reduced cost? Can we design custom arithmetic components, specifically tailored to DNNs, and hence still capable of wide generalization,

while carefully limiting the resulting accuracy loss \emph{and} at the same time boosting energy saving? Can we research custom architectures that are approximation-configurable at at the same time power-efficient? Approximation can help to create lighter, faster models that require less memory and processing power. While this is particularly crucial for deploying models in resource-constrained environments, such as on mobile devices or embedded systems, where energy efficiency is paramount, it is extremely important in non-battery-operated devices as well, because of the need to limit environmental impact.

The challenge of approximation is, of course, to maintain the network’s predictive accuracy while in search of power efficiency. This challenge is multi-faceted, requiring simultaneous research in related but distinct fields such as exploration of the myriad of approximate arithmetic designs that could be employed for neuron computation; the study of differentiability of such operators, for efficient backpropagation; the tailoring of operators to the specific input distributions typically found in networks; the deployment of such novel approximate designs on suitable, potentially reconfigurable, fabrics.

Exploiting the knowhow and experience gained by my research in the field, in particular in Approximate Logic Synthesis, Design Space Exploration, and Reconfigurable Computing, and stemming from our recent encouraging initial results in NN approximation, this project will explore novel avenues for deploying approximate neural networks at scale that carefully balance the tradeoff between a limited loss in accuracy for a substantial energy saving.