Quantitative Assessment of Navigation Solutions
for Mobile Robots

Background:

As mobile robots become more ubiquitous, their utility will rely on the ability of the robotic system to safely operate in dynamic, unstructured environments. These systems will need to explore new environments, generate maps that identify obstacles and hazards through exploration, and use these maps to safely navigate to any location. They will also need the ability to intelligently adapt to momentary changes in the environment. Central to the realization of this vision of mobile robots is the system's ability to develop a stable navigation solution, which we define as the ability of the system to sense the environment, create internal representations of its environment, and estimate pose (where pose consists of position and orientation) with respect to a fixed coordinate frame.

Commonly, characterizing the performance of navigation solutions is based on the qualitative analysis (i.e. visual inspection) of the robot-generated maps. While this type of analysis provides some indications of the overall performance, it does not allow researchers to understand what errors a specific system is prone to and how these errors impact the overall performance of the system. The absence of standardized methods for evaluating emerging robotic technologies has caused segmentation in the research and development communities for robotic technologies. This lack of cohesion hinders the attainment of robust mobile robot navigation, in turn slowing progress in many domains, such as manufacturing, service, health care, and security.

At the National Institute of Standards and Technology (NIST), we developing tools and standardized test methods to classify the performance characteristics of navigation solutions that facilitate the inter-comparison of experimental results. The development of a de facto standard testbed for evaluation of navigation solutions will provide a baseline for comparison and the means to target specific aspects of the navigation solution, allowing researchers to assess the performance of various systems in different scenarios and environmental conditions. Providing the research community access to standardized tools, reference data sets, and an open-source library of navigation solutions, researchers and consumers of mobile robot technologies will be able to evaluate the cost and benefits associated with various navigation solutions.

Through this collaborative forum, we hope to bring together researchers, vendors, and end-users of the robotic technologies to compile a reference guide that documents lessons learned and the performance characteristics of various navigation solutions. This will enable end users to select the "best" possible method that meets their needs and will also lead to the development of the adaptive systems that are more technically capable and at the same time are safe thus permitting collaborative operations of man and machine. This, in turn, will not only improve the utility of mobile robots in already established application areas, such as vacuum cleaning, robot surveillance, and bomb disposal, but will enable the proliferation and acceptance of such technologies in other emerging markets.