AI Daily - Artificial Intelligence News

View Original

New MIT Architecture May Lead To Smaller Carbon Footprints For Neural Networks

At the (ICLR) International Conference on Learning Representations 2020, this paper was presented. According to the paper there is a problem with conventional approaches to DNNs (Deep Neural Network). This is that they require data scientists to manually design, or use NAS (neural architecture search), to find a specialised neural network and “train it from scratch for each case”.

Researchers at the MIT have developed an optimization strategy for DNNs (Deep Neural Network) that prepares them for deployment on thousands of different edge devices. This reduces the carbon footprint of inference as well as training.

Massachusetts Institute of Technology is a private research university in Cambridge, Massachusetts with an endowment of $17.57 billion (2019)

In 2019  University of Massachusetts at Amherst study  found that a single large (213 million parameters) Transformer-based neural network built using NAS (commonly used in machine translation) has produced around 626,000 pounds of carbon dioxide. This is comparable to the gas produced by 5 cars during their lifetimes. The papers authors authors endorse the OFA or ‘once-for-all’ approach which reduces the number of GPU hours required to train some types of models by “orders of magnitude”. However it maintains a similar, or even higher levels of accuracy. This is good as fewer GPU hours means the system consumes less electricity, and as a result, leads to lower carbon dioxide emissions.

Using the OFA approach, it’s also possible to create a specialised sub-network for a particular device from the main network without additional training. The authors say that models created using OFA perform better on edge devices than best NAS-created models, performance improved by up to 2.6 times in terms of speed in internal tests.

 As part of the architecture, the authors propose a progressive shrinking algorithm that they say can reduce model’s size more effectively than the conventional methods (network pruning). In 2019, the team behind the OFA won both competitions at the 4th Low Power Computer Vision Challenge. This being an annual event held by the IEEE that aims to improve the energy efficiency of computer vision for running on systems with stringent resource constraints.