Evolution Strategies
Lilian Weng published a technical overview of Evolution Strategies (ES), a class of black-box optimization algorithms in the evolutionary algorithms family. The post notes that while stochastic gradient descent is a common choice for optimizing deep learning models, black-box methods such as Simulated Annealing, Hill Climbing and the Nelder-Mead method can evaluate a target function without knowing its analytic form or computing gradients.
The article traces several ES variants. Simple Gaussian Evolution Strategies models the search distribution as an n-dimensional isotropic Gaussian tracking only a mean and standard deviation, iterating through sampling, fitness evaluation and elite selection. Covariance Matrix Adaptation Evolution Strategies (CMA-ES) addresses the limitation that vanilla ES cannot rapidly adjust its exploration space, instead tracking pairwise dependencies via a covariance matrix. The post details CMA-ES parameter updates including the mean learning rate, step-size control through an evolution path compared against its expected length under random selection, and covariance matrix adaptation using rank-min and rank-one update routes.
Natural Evolution Strategies (NES), attributed to Wierstra et al, 2008, optimizes a search distribution using the natural gradient, with the Fisher Information Matrix discussed as a means of estimating KL divergence between distributions. The post also covers NES heuristics such as rank-based fitness shaping and adaptation sampling using the Mann-Whitney U-test.
On applications, the post describes OpenAI ES for reinforcement learning (Salimans et al. 2017), which uses NES as a gradient-free black-box optimizer to find policy parameters maximizing a return function, adding Gaussian noise to parameters. It notes virtual batch normalization, mirror sampling and fitness shaping. It also covers Novelty-Search ES (NS-ES; Conti et al, 2018), which updates parameters to maximize a novelty score based on a domain-specific behavior characterization function.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Stochastic gradient descent is a universal choice for optimizing deep learning models. However, it is not the only option. With black-box optimization algorithms, you can evaluate a target function $f(x): \mathbb{R}^n \to \mathbb{R}$, even when you don’t know the precise analytic form of $f(x)$ and thus cannot compute gradients or the Hessian matrix. Examples of black-box optimization methods include Simulated Annealing , Hill Climbing and Nelder-Mead method .