Advanced Cooperative NeuroEvolution for Autonomous Control

There are many real-world problems for which programming a solution directly is not feasible. Typically, these problems involvecontrolling some system by repeatedly measuring its state and selecting the best action from a set of possible choices. Because theenvironment can be very complex (i.e. non-linear, high-dimensional, noisy, unstable, etc.) it can be dificult, if not impossible, to knowin advance the consequences of a sequence of such actions.
The ?eld of Reinforcement Learning (RL), has contributed many methodsthat, in principle, can solve these types of problems. However, in practice, they have not scaled well to large state spaces or environments where the state is not completely observable by the agent. This is a serious problem because the real world is continuous (i.e. an infinite number of states) and artificial agents, like natural organisms, have imperfect sensory systems.
More recently, an approach called neuroevolution (NE), where artificial neural networks are "evolved", has shown promising resultson continuous, partially observable tasks. Instead of have a single agent that learns by trial-and-error interaction with the environment(i.e. standard RL), a population of candidate agents is adapted to the task through a process inspired by natural selection: each network inthe population is evaluated on the task, assigned a fitness, and then new, hopefully better, networks are created by combining parts of themost fit networks. These new, "offspring" networks are then evaluated, and the cycle is repeated until a network that solves the task isdiscovered. The key advantage of this approach is that the networks do not have to be trained in the conventional way, which is especiallyproblematic if the networks have memory---a requirement for coping with partial observability.
We have made important contributions to this area with methods that use cooperative coevolution were networks are built from subpopulations of components that are coevolved to work together to solve a given task. Our Enforced SubPopulations (ESP) and CoSyNE methods are particularly e?ective examples of this approach that has been a applied to many diverse reinforcement learning problems including ?nless rocket control, and supervised time-series prediction.
While these results are encouraging, NE to date has only been applied successfully to control tasks with a limited number of inputs (typically less than 100). However, to solve sophisticated tasks in real-world domains such autonomous robotics, much richer sources of sensory information such as vision are required. High-dimensional inputs imply much larger networks and a correspondingly large search space, which can cause evolution to converge before ?nding a solution.
To scale neuroevolution to these conditions, we propose the following multi-faceted approach:
1. Develop new, more powerful neuroevolution algorithms based on our own previous methods using insights from an empirical analysis of cooperative coevolution.
2. Reduce the dimensionality of the input space by evolving non-redundant codes using the principle of predictability minimization, that can be used to preprocess inputs to evolved the evolved controllers.
3. Explore new methods based on our recent groundbreaking work to delay convergence in the space of large networks by sustaining diversity in the behavior of candidate controllers instead of in their genotypes.