Brittle robots, metacognitive loops
Brittleness, in a robot or program, is an inability to deal with unexpected developments, an important problem in AI. For examples of brittleness see the DARPA Grand Challange, $1 million competition for racing a robot across 142 miles of Mojave Desert in under 10 hours, which not one robot came close to doing, with some running into obstacles, others having navigation problems, and with one which couldn’t start itself. Th solution proposed by Michael L Anderson and Donald R Perlis is a metacognitive loop, a mechanism whereby the robot or program can notice whether or not it is achieving its goals, and if not, it can then try something else or just perform random variations in behavior until it gets better results. No one wants a brittle robot.
Logic, self awareness and self-improvment; The metacognitive loop and the problem of brittleness, Michael L Anderson and Donald R Perlis, Journal of Logic and Computation 14, 2004