Mihika Kamat

Rapid discovery of robot designs via model predictive control

Paper (PDF)    

Legged robot designs are often inspired by animals in nature. However, these designs may not be optimized for task-specific performance, and only encapsulate a small fraction of the design space. Animals have developed distinct morphological characteristics that enable them to compete and survive in their environment. Can we automatically search the space of robot designs in a similar fashion? In this work, we propose an efficient evolutionary algorithm that designs a robot from scratch, given a task and a simulation environment.

Background

Morphology representations. Soft robots are usually represented with a 3D grid of voxels with varying physical properties. Rigid robot designs are represented with kinematic trees, which consist of “bodies” interacted with “joints”. Physics simulators such as MuJoCo, enable us to physically simulate robots from a kinematic tree. We adopt the latter formulation, following the approach from Sims [3].

A rendered MuJoCo Ant beside the kinematic tree describing it: a head with four hinge joints, each leading to a limb
Figure 1: The kinematic tree describing the “MuJoCo Ant”.

Design agnostic control. Because legged robot designs can be very different structurally, a unique controller is required to score each morphology to its true ability. Thus, robot evolution, has so far required the co-design of morphologies and their respective controllers. We formulate the control problem as a Markov decision process, pictured below.

Diagram of a Markov decision process: an environment emitting state and reward into a policy, which emits an action back to the environment
Figure 2: A Markov decision process consists of an environment, which outputs a state and reward, and a policy which outputs an action.

Reinforcement learning optimizes a controller which maps from a state to a policy. Due to their stability, policy gradient methods such as PPO have been used in prior works to evaluate each candidate in the evolutionary process [1]. However, it typically requires billions of simulation steps per morphology, making it unsustainable for large population sizes and generation counts.

Model predictive control optimizes a policy online, during the controller rollout, by periodically updating an action plan based on “imagined” rollouts. While it doesn’t result in a reusable controller at the end of optimization, it enables sample efficient evaluation of candidate robot morphologies in specific scenarios.

We hypothesize that a general physics simulator, combined with model predictive control can enable efficient comparison and discovery of robot designs based on task success.

Method

We begin with a population of 128 procedurally generated morphologies. Each morphology starts with a head body, and receives a randomly chosen sequence of mutations. At each generation, every member of the population is evaluated for fitness. The best performing morphology per group of 4 is mutated and used to replace the oldest members of the population.

The evolution cycle: a population is divided into groups and evaluated, the best in each group is cloned and mutated, and the oldest members are deleted
Figure 3: Evolution cycle on sample population. (1) Population → (2) Divide into groups and evaluate → (3) Clone and mutate best → (4) Delete oldest members.
Seven mutation primitives shown as small robot renders: grow limb, delete limb, limb size, gear ratio, limb material, joint flexion, and joint degrees of freedom
Figure 4: List of mutation primitives used for initializing and modifying designs.

Experiments

To test our hypothesis, we design four tasks: rough terrain locomotion, flat terrain locomotion, transporting a cube to a goal, and reorienting a cube to a goal orientation. Their individual performance as reward, with key components outlined below.

Rough terrain locomotion

Flat terrain locomotion

Transport cube to goal

Reorient cube to goal

Results

Four box-plot panels showing task fitness rising across generations 0 to 40 for rough locomotion, flat locomotion, cube transport, and cube reorientation
Figure 5: Distribution of fitness score for morphologies versus evolution generations completed for each task.

Over 50 generations, design performance improves. However, diversity in designs reduces in terms of skeletal structure, but continues to exhibit variance in limb length.

Two charts comparing EPS with DERL: normalized return against simulation steps, and evaluations per 100M steps
Figure 6: Comparing EPS to DERL (an RL baseline).

EPS achieves 80x more evaluation throughput than DERL as measured in simulation steps. In terms of improvement per generation, these two methods are comparable.

Future Work

Overcoming sparse rewards. Predictive sampling relies on dense rewards, due to a finite planning horizon. While this method works well for locomotion tasks, it is more challenging to reward manipulation tasks. Future work could explore the use of value functions to overcome these limitations.

Hybrid Tasks. To make EPS useful for real world robot design, morphologies could be evaluated on multiple tasks simultaneously.