Neural Architecture Search

Neural Architecture Search (NAS) is a systematic and automatic way of learning high-performance model architectures, rather than designing them by human experts alone, according to this survey-style overview. It cites an earlier proposal of automatically learning network topologies by Stanley and Miikkulainen in 2002, and credits Zoph and Le 2017 and Baker et al. 2017 with attracting attention to the field.

The report adopts the characterization by Elsken et al. 2019, which defines NAS as a system with three major components. The search space defines a set of operations such as convolution, fully-connected and pooling, and how operations can be connected to form valid network architectures; its design usually involves human expertise and biases. The search algorithm samples a population of architecture candidates and receives child model performance metrics as rewards, optimizing to generate high-performance candidates. The evaluation strategy measures, estimates or predicts the performance of proposed child models to give the search algorithm feedback; this process can be expensive.

Among search spaces, the report describes sequential layer-wise operations used by Zoph and Le 2017 and Baker et al. 2017, where the serialization requires hardcoded layer-specific parameters and validity rules, and is large and costly to cover exhaustively. The NASNet cell-based representation from Zoph et al. 2018 learns normal cells and reduction cells, with each cell's predictions grouped into B blocks (B=5 in the paper) and five softmax classifiers per block; it introduced ScheduledDropPath, a DropPath variant with a linearly increasing path-dropping probability during training.

Other search spaces include the hierarchical structure of Hierarchical NAS (Liu et al 2017), which recursively composes primitive operations into motifs, and the tree-structure path-level network transformation of Cai et al 2018b. Brock et al 2017 proposed a memory-bank representation in SMASH.

Search algorithms covered are random search; reinforcement learning, including the RNN controller of Zoph and Le 2017 trained with REINFORCE and MetaQNN's Q-learning agent (Baker et al 2017); evolutionary algorithms such as NEAT (Stanley and Miikkulainen 2002), AmoebaNet's tournament selection with aging evolution, and HNAS; and the progressive decision process of PNAS (Liu et al 2018), which uses Sequential Model-based Bayesian Optimization like A* search and starts with B=1.