Banner National Institute of Standards and Technology
ISD Research Areas
ISD home About ISD ISD Research Areas ISD's Products and Services What's New in ISD Search ISD
Manufacturing Engineering Laboratory Skip navigation

 

Home

Overview

Preliminary Requirements Report

Standards Working Groups

Feedback

Events

News

Contact

 


Response Robot Informal Evaluation Exercise #3
Montgomery County Fire Rescue Training Academy
Rockville, MD
August 19-21, 2006

Click on the links below for more information:

 








Event Event Introduction

The third in a series of response robot informal evaluation exercises for DHS/FEMA US&R teams will be hosted at the Montgomery County Fire Rescue Training Academy in Rockville, Maryland (near Washington DC). This event will finalize the test methods targeted for the initial (Wave 1) set of standards as well as initiate experimentation with onboard payloads, especially for Chemical, Biological, Radiological, Nuclear, and Explosive (CBRNE) sensing. Therefore, emphasis will be on (a) robots that can address the deployment categories relevant to Wave 1 standards and (b) deploying CBRNE sensors on these robots. The three robot deployment categories selected by responders to be emphasized in Wave 1 are: ground peek robots that are small and throwable, ground wide-area survey robots that can traverse non-collapsed structures or areas external to the collapse, and aerial survey or loiter robots. Manufacturers of robots, purchasable and/or developmental, that can address these areas, are invited to take part in this exercise, which will highlight operationally relevant US&R scenarios.

Background

Response robot evaluation exercises for US&R teams introduce emerging robotic capabilities to emergency responders while educating robot developers regarding the necessary performance requirements to be effective, along with the associated environmental conditions and operational constraints to be useful. Standard test methods and usage guides for US&R robot performance are under development within the ASTM International E54.08 Subcommittee on Operational Equipment, which is under the Homeland Security Committee. These events help refine the proposed standard test methods and artifacts that developers can use to practice critical capabilities and measure performance in ways that are relevant to the end user, i.e. responders. These events are conducted in actual US&R training scenarios to help correlate the proposed standard test methods with envisioned deployment tasks and to lay the foundation for the usage guides, which will provide guidance on which robots are best suited for which response situations.

NIST will be working directly with CBRNE sensor vendors to make sensors available for mounting on robots. Robot vendors are encouraged to bring their own sensors, if available. In addition to the CBRNE sensors, the robots envisioned for use in these scenarios should deploy any or all appropriate capabilities such as: color cameras, microphones, speakers, thermal imagers, 3D mapping, GPS/GIS location, and/or manipulators. Robots sought are, but are not limited to:

  • Ground based robots that can circumnavigate large unknown situations and carry sensor payloads to assess the presence of chemical, biological, radiological, nuclear, or explosive materials as well as provide visual imagery to the responders at a standoff distance.
  • Confined space accessible robots for deployment into sub-human size voids or be thrown into/over inaccessible areas

This event will include three days of robot evaluations in all the available US&R training props. The event will also provide evaluations of the proposed Wave 1 standard test methods and artifacts. This will be the final validation step prior to submitting these test methods to the ASTM balloting process. All stakeholders will be able to provide feedback on the proposed standard test methods. There will be ASTM working group meetings to discuss and incorporate any proposed changes to the test methods. In the late afternoon of August 21st, the training site will be opened up to tour groups to observe the responders and robots in action. Visitors will include representatives from DHS and other government agencies, as well as attendees at the Performance Metrics for Intelligent Systems and the Safety, Security, and Rescue Robotics conferences, which are being held at the National Institute of Standards and Technology.

These practice incident response exercises are as close as we have to the real thing, and provide an important opportunity to introduce the potential benefits of robots and advanced sensors into US&R responses. So consider this an open invitation to present your applicable robot or sensor to those that can benefit, and let them practice in a relevant training environment, so that they can be better equipped, better trained, and better prepared to save lives at the next disaster.

All participants in the response robotics exercise are encouraged to attend the conferences being held at NIST August 21-24th. The Performance Metrics for Intelligent Systems Conference is the sixth in a series targeted at defining measures and methodologies of evaluating performance of intelligent systems, with an emphasis on applications of performance measures to practical problems in commercial, industrial, homeland security, and military applications. The IEEE Safety, Security, and Rescue Robotics is dedicated to identifying and solving the key issues necessary to field capable robots across a variety of challenging applications. This fourth workshop in the series will emphasize objective performance testing and evaluation of robotic systems to satisfy user-defined requirements. For more information, see

www.isd.mel.nist.gov/PerMIS_2006
www.isd.mel.nist.gov/ssrr2006

[Return to Top]

Scenarios

Proposed Robot Test Methods and Operational Scenarios

Site Overview



Test Method: Visual Acuity (with variable Illumination)

Test Method: Visual Acuity (with variable Illumination)

Metrics:

  • Smallest readable line (far field, near field)
  • Time

Key Features:

  • Far field and near field charts with labels
  • Indexed distances to measure range
  • Lights on/off to measure illumination
  • Assorted flooring to measure pointing:
    • Flat
    • Pitch ramps
    • Half-cubic step-field (orange)

Test Method: Random Step-Fields (Cubic Unit Pallets - Red)

Test Method: Random Step-Fields

Metrics:

  • Number of step-field pallets traversed
  • Time

Key Features:

  • Cubic unit (red) topologies to test mobility
  • Scaleable sizes:
    • Small-size made of 2x2s
    • Mid-size made of 4x4’s (shown)
    • Large-size made of clusters of (4) 4x4’s
  • Variable topologies:
    • Random flat pallets
    • Random hill pallets
    • Random diagonal hill pallets

Test Method: Random Step-Fields (Confined Space Pallets - Red)

Test Method: Random Step-Fields (Confined Space Pallets - Red)

Metrics:

  • Number of step-field pallets traversed
  • Time

Key Features:

  • Covered cubic unit (red) topologies to test confined space maneuverability
  • Scaleable sizes:
    • Small-size made of 2x2s
    • Mid-size made of 4x4’s (shown)
    • Large-size made of clusters of (4) 4x4’s
  • Variable topologies (above and below):
    • Random flat pallets
    • Random hill pallets
    • Random diagonal hill pallets

Test Method: Random Step-Fields (Half Cubic Unit Pallets - Orange)

Metrics:

  • None (embedded into other test methods only)

Key Features:

  • Half-cubic (orange) topologies for orientation complexity
  • Scaleable sizes:
    • Small-size made of 2x2s
    • Mid-size made of 4x4’s (shown)
    • Large-size made of clusters of (4) 4x4’s
  • Variable topologies:
    • Random flat pallets
    • Random hill pallets
    • Random diagonal hill pallets

Test Method: Directed Perception (With Assorted Sensor Targets)

Test Method: Directed Perception (With Assorted Sensor Targets)

Metrics:

  • % target boxes cleared at each level
  • Time

Key Features:

  • Box stacks of assorted targets
  • Four levels performed sequentially
  • 0° and 90° approaches
  • Assorted flooring:
    • Flat floor
    • Pitch ramps
    • Roll ramps
    • Half-cubic step-fields (orange)

Test Method: Manipulator Dexterity (With Assorted Grasping Tasks)

Test Method: Manipulator Dexterity (With Assorted Grasping Tasks)

Metrics:

  • % grasping objects removed at each level
  • Time

Key Features:

  • Shelf stacks of grasping objects
  • Four levels performed sequentially
  • 0° and 90° approaches
  • Assorted flooring:
    • Flat floor
    • Pitch ramps
    • Roll ramps
    • Half-cubic step-fields (orange)

Tower and Burn Building: VTOL Model Aircraft Station Keeping and Wall Climbing

Tower and Burn Building: VTOL Model Aircraft Station Keeping and Wall Climbing

Burn Building Exterior: Beyond Line of Sight Radio Testes

Burn Building Exterior: Beyond Line of Sight Radio Testes

Burn Building Exterior: Manipulator Dexterity Tests (Doors, Mailbox, etc.)

Burn Building Exterior: Manipulator Dexterity Tests (Doors, Mailbox, etc.)

Burn Building (Stairwell): Five Levels with Two Landings Between Each

Burn Building (Stairwell): Five Levels with Two Landings Between Each

Burn Building (Level 3): Rooms/Hallways with Obstacles

Burn Building (level 3): Rooms/Hallways with Obstacles

Burn Building (Level 4): Three Crawling Mazes with Obstacles and Features

Burn Building (Level 4): Three Crawling Mazes with Obstacles and Features

Rubble Pile and Pits

Rubble Pile and Pits

Rescue Mall: Two Level Facade with Various Insertions and Props

Rescue Mall: Two Level Facade with Various Insertions and Props

Vehicle Accident with Simulated Chemical Spill

Vehicle Accident with Simulated Chemical Spill

K-9 Training Course

K-9 Training Course

Test Method: Ramps (Resurfaced and Adjustable)

Test Method: Ramps (Resurfaced and Adjustable)

Test Method: Logistics - Cache Packaging

Test Method: Logistics - Cache Packaging

[Return to Top]

Schedule

Schedule for the Response Robot Exercise at the Montgomery County (MD-TF1) Training Academy:

Friday, August 18
Travel day

Saturday, August 19
0830 Bus departs Courtyard Marriott hotel to Montgomery County (MD-TF1) Training Academy
0900 Site check-in and safety briefing
1000 Orientation
1100 Practice scenarios for downrange situational assessments of chem/bio/nuclear hazards, rubble pile voids, building clearing using wide-area survey robots, throwable robots, and vertical take-off and landing aircraft.
1700 After action briefing
1900 Dinner outing

Sunday, August 20
0830 Bus departs hotel to training site
0900 Site check-in and safety briefing
1000 Continue scenario practice. Rotate robots/responders/scenarios.
On-site informal ASTM standards working group meetings (Mobility, Sensors, Human Factors, Logistics, Communications)
1700 After action briefing and site check-out

Monday, August 21
Morning Events at NIST
PerMIS begins with a plenary by Henrik Christensen, Kuka Chair of Robotics and a Professor of Computing with the College of Computing, Georgia Institute of Technology (talk title: Evaluation of Robots for Human-Robot Interaction) followed by ASTM standards meetings for test method development working groups: Mobility, Sensors, Human factors, Logistics, Communications (potentially sequentially scheduled so people can attend more than one). You will be warmly welcomed on any committee you choose. Or you can just sit in and informally contribute to discussions.

Afternoon Events at MD-TF1 training site
Final practice of best scenarios for each robot/responder/developer. We'll have a possible inclusion of canine teams and other mock incident response operations. Attendees from PerMIS/SSRR conferences will be invited out to see practiced, operationally significant, robot deployments starting at 1600.

Tuesday August 22
PerMIS continues through the 23rd and SSRR begins with compelling plenary presentations from Shigeo Hirose, Tokyo Institute of Technology, a brilliant developer of biologically inspired robots and Hugh Durrant-Whyte, University of Sydney, Australia, will present information systems involving mixed air and ground autonomous systems. There will be technical presentations and demonstrations of robot prototypes not quite ready for field exercises at an evening welcome reception for the combined conferences. Social events such as these throughout the week will allow extensive interaction between our assembled emergency responders and the international developers focusing on response robots.

Wednesday, August 23
In the morning plenary presentation given by Martin Buehler, Boston Dynamics, will discuss efforts to develop dynamic legged robots. There will be technical presentations, more robot demonstrations and a panel discussion of emergency responders talking about recent deployments and opportunities for robots. There will also be an evening banquet at the Gaithersburg Hilton Hotel.

Thursday, August 24
The last of the technical sessions will wrap up quite a week of events!!

A draft of the program in pdf can be downloaded below:

Program

Please be sure and check the response robot exercise and conference web sites for the latest information and program schedules.
www.isd.mel.nist.gov/US&R_Robot_Standards/montgomery_county/eventintro.htm
www.isd.mel.nist.gov/PerMIS_2006/
www.isd.mel.nist.gov/ssrr2006

[Return to Top]

How to get Involved

Thanks for your interest in participating in our upcoming Response Robot Exercise at the Maryland Task Force 1 training facility.

Here is a short checklist to facilitate your registration and support necessary logistics:

1) Make your hotel reservation
(check-in Friday, August 18th, event starts Saturday 0800)

Courtyard Marriott Washingtonian Center
204 Boardwalk Place
Gaithersburg, Maryland 20878
Telephone Number: 301-527-9000
Room Block: NIST Robots Deadline: August 9, 2006
Rate: $99/night + 12% tax

This high-end Courtyard Marriott just opened three weeks ago and is conveniently located in the heart of the Washingtonian Center Rio with access to many restaurants, stores, entertainment.... and close to the Fire Rescue Training Academy.

2) Fill out the robot description form below
We want to familiarize the emergency responders with your robot in advance of the event. Please email this form with your robot(s) appropriate file name to usar.robots@nist.gov by deadline 7/31/06. You may email additional specification sheets or brochures as well, and pass out any literature you like on site.

Click either on the .ppt file name or the image to down the forms.

When you send in your robot description form to usar.robots@nist.gov we'll consider you "registered" for the event, and add you to the email list for updates on logistics and such.

3)Fill out the expected attendees and foreign visitor forms below:

Expected Attendees.ppt

Foreign Visitors.xls

Please e-mail back to usar.robots@nist.gov.

4) Read the attached safety presentation [click to download safety presentation]
Get familiar with the rules imposed by the training facility. Adherence to these personal protective clothing rules is essential while within any of the scenarios at the site.

What Personal Protective Equipment (PPE) is needed to participate in this event?

Personal protective equipment (PPE) is a must for working within any scenario at the site. People in street clothes or without helmets/gloves/etc as shown below are limited to paved roads only. If you are working in/near a scenario, you must wear ALL the equipment shown below. Compliance with these personal protective equipment rules are mandatory - it is standard practice for US&R environments.

Personal Protective Equipment (PPE)

 

5)Optional demonstration/exhibit at the International Workshop on Safety, Security, and Rescue Robotics (SSRR) at NIST starting the day after
Please fill out the additional robot demonstration form below (focused on robot capabilities, not the specifications) and send any appropriate videos (10 megs each please) to ssrr2006@nist.gov. We'll use this form and the videos to advertise your demonstration throughout the SSRR event. Submission of this form will ensure that you have a booth with tables and power to display your robot between demonstrations and inclusion in the demonstration agenda.

SSRRDemo_Organization_RobotName.ppt

Thanks again for your interest. Looking forward to seeing you.

Please be sure and check the response robot exercise and conference web sites for the latest information and program schedules.
www.isd.mel.nist.gov/US&R_Robot_Standards/montgomery_county/eventintro.htm
www.isd.mel.nist.gov/PerMIS_2006/
www.isd.mel.nist.gov/ssrr2006

[Return to Top]

Shipping Robots to the Training Academy

If you need to ship a robot to the Training Academy, please use this address:

Montgomery County Fire Rescue Training Academy
Attn: Tyrone DeMent
9710 Great Seneca Highway
Rockville, MD 20850

Please inform Jeanenne Salvermoser of expected delivery date.

Please make sure your robots and equipment are on-site on/before Friday, August 18, 2006.

[Return to Top]

Directions

Directions to the NIST/DHS Response Robot Exercise
Montgomery County Fire Rescue Training Academy
9710 Great Seneca Highway, Rockville, MD

Click on the map to download directions to the site.

or visit their website at: www.montgomerycountymd.gov/firtmpl.asp?url=/content/firerescue/psta/index.asp

[Return to Top]

Frequently Asked Questions


How can you have a standard robot for urban search and rescue?

Can I participate in the standards process?

Where can I learn more about the urban search and rescue robot standards program?

What is the goal of these Responder/Robot Evaluation Exercises and how will they lead toward sales of my robot?

Are you grading my robots at the Responder/Robot Evaluation Exercises?

Who gets to attend these events?

I want to demo my new whizbang gadget but I don't want it to get damaged. Can I bring it and just show it?

I have a new whizbang technology that responders should know about, but it's doesn't really have anything to do with robots? Can I bring it?

Will this event produce purchase decisions for my robot?

What Personal Protective Equipment (PPE) is needed to participate in this event?


How can you have a standard robot for urban search and rescue?
The goal of this program is NOT to develop standards to govern robot technologies. Rather, this program is working toward generating standard test methods to quantify robot performance related to specifically articulated requirements. This program will and generate measures of performance that emergency responders can relate to their task requirements. These test methods will measure performance in an objective, well defined way, using easily fabricated test apparatus so that manufacturers and developers can practice the test methods in their own facilities. Each test method will have an advertised range of acceptable performance levels defined by the emergency responders, from a minimum threshold to an objective level of performance, so that developers understand the useful range as defined by the ultimate customers. The results of the test methods will quantify such performance, without necessarily indicating good or bad results. The emergency responders will then be able to make informed trade-offs between their key variables such as performance, size, weight, cost, etc., to best address their envisioned task.

Can I participate in the standards process?
Anyone with a stake in the development and use of robots, especially for emergency response, is encouraged to join the standards committee.   This includes, but is not limited to: robot vendors, manufacturers, end users, technology developers, researchers, local, state, and federal response agencies, and other government agencies.   The standards are being developed by ASTM International, under the E54.08.01 subcommittee.  

Where can I learn more about the urban search and rescue robot standards program?
The best place to start is the web site:  http://www.isd.mel.nist.gov/US&R_Robot_Standards
This site contains the preliminary requirements report, calendar of events, reports from past events, including standards meetings, and other pertinent information.

What is the goal of these Responder/Robot Evaluation Exercises and how will they lead toward sales of my robot?
Below is an outline of the expected path to procurement for robotic technologies that can be effectively applied to urban search and rescue tasks. Note, however, that this is not exclusively the case, since all the FEMA Task Force leaders and responders at these Responder/Robot Evaluation Exercises are members of local organizations every other day of the year; some from NY City, some from LA, etc.

0. Emergency responders on our advisory panel produced an initial set of performance requirements for robots which could improve/augment their existing capabilities based on their experiences performing urban search and rescue. This was prior to much exposure, if any, to robotic technologies, so the emphasis was on gaps in their current capabilities that could potentially be addressed with robotic technologies. The 100 or so initial requirements can be found at www.isd.mel.nist.gov/US&R_Robot_Standards (Preliminary Report).

1. Responder/Robot Evaluation Exercises are being conducted to introduce emerging robotic technologies to responders within relevant training environments. The scenarios used in these exercises are opportunities to highlight the utility or unique advantages that particular technologies may provide, and how they must be deployed to be useful.

2. Responders will augment their defined set of requirements with additional needed capabilities, adding requirements for particularly helpful robotic technologies, such as 3-D mapping of confined space voids for example, based on their assessment of these emerging technologies within their training environments. Prior to these events, they either didn't know about that particular technology, or never saw it working in their environments.

3. Particularly "ripe" technologies demonstrated to be useful and survivable within responder training environments will get performance thresholds and objectives assigned by the responders as a group after each event, giving technology providers an envelope in which to provide solutions -- since differing levels of cost/performance may appeal to different response organizations or support specific roles/tasks (i.e. initial reconnaissance vs. structural assessment). Less "ripe" technologies that are considered on the path toward fieldable will get more general requirements written to help guide development. Both technologies will benefit from ongoing participation in these responder/robot evaluation exercises to guide development .

4. Performance test methods will be developed for these "ripe" technologies to capture key performance criteria necessary to quantify/compare implementations. These test methods will not identify good/bad performance, rather they will objectively capture actual performance in a known (and practiced) test method to help guide applicability and purchasing decisions, etc. These performance test methods will ultimately be standardized in waves within ASTM International's Homeland Security portfolio. The first wave of standard test methods covering "ripe" technologies is due to be published in 2006.

5. Robots and technologies that use these performance test methods to quantify their capabilities will be considered available for purchase with DHS funds. They will be included in a compendium which captures their performance test results and disseminated to FEMA and other response organizations to help guide trade-off and applicability decisions for purchasing. Note, however, that NIST cannot represent DHS's intent regarding the timing or funding of any grants to purchase robotic equipment.

Are you grading my robots at the Responder/Robot Evaluation Exercises?
Robot performance is not formally captured at these events. The Disaster City event and others like it are meant to achieve several goals. (1) Responders are being introduced to robots and supporting technologies and are learning about what is and isn't feasible presently and what new technologies look promising and need associated performance requirements to be defined. (2) Robot developers and vendors are being exposed to the needs of the urban search and rescue community and are able to learn firsthand what responders like and don't like, and do so in realistic training environments. (3) Draft test methods for performance requirements are evaluated and refined by responders and robot developers prior to finalization within the standards process. (4) Performance objectives and threshold values under various deployment circumstances are captured for key requirements (initially focusing on Wave1 of the standards process).

That said, video of successful robots in challenging responder training scenarios is captured at these events so that responders everywhere can start to envision how these assets might work for them.

Who gets to attend these events?
In general, these events are working exercises in somewhat dangerous environments. Casual observers are not allowed to attend. The exercises are also not currently open to all responders. FEMA US&R Task Force members who comprise the advisory panel on robot performance requirements and standards are the key participants and robot operators. Robot manufacturers, developers, funding agencies, and researchers who have relevant technology that can be shown to be applicable to at least a subset of the scenarios are welcome to come. Active members of the standards committee are welcome to come as well.

I want to demo my new whizbang gadget but I don't want it to get damaged. Can I bring it and just show it?
Standalone demonstrations are strongly discouraged. It is our goal to have the technologies be run by responders in realistic situations. Note that there are certain scenarios and subsets of scenarios that may not be as harsh as others. There are also the proposed standard test methods that need to be exercised, so there is a wide range of deployment opportunities for many levels of robot development. But in general, your technology has to be demonstrably relevant and functional to get involved in these events.

I have a new whizbang technology that responders should know about, but it's doesn't really have anything to do with robots? Can I bring it?
This event is about robots for urban search and rescue. If your technology has no possibility of ever being deployed on or by a robot, this is not the event for you.

Will this event produce purchase decisions for my robot?
We understand that many robot developers hope for or need an imminent purchase decision to hang on every outing. Unfortunately, we can't say with confidence that this event will necessary lead directly to a purchase. This event is intended to close the loop on your technologies and their intended applications, to identify necessary changes in your approach to be maximally useful to responders, and to provide exposure of your technology to the people who guide purchasing decisions for this community. See questions above for addition discussion regarding the envisioned path to procurement.

What Personal Protective Equipment (PPE) is needed to participate in this event?

Personal protective equipment (PPE) is a must for working within any scenario at the site. People in street clothes or without helmets/gloves/etc as shown below are limited to paved roads only. If you are working in/near a scenario, you must wear ALL the equipment shown below. Compliance with these personal protective equipment rules are mandatory - it is standard practice for US&R environments.

Personal Protective Equipment (PPE)

On Site Safety Comes First
Be cautious, use common sense, and watch out for others!

Safety of all personnel participating in this event is our first concern. The fact that we have robotics personnel on site, generally unaccustomed to working within hazardous scenarios, is particularly problematic. First, all appropriate personal protective equipment (PPE) must be worn at all times while on site (see associated page on PPE) -- which means wear all of it when you are working in/near a scenario. People in street clothes or without helmets, gloves, etc., are limited to paved roads only. Compliance with these personal protective equipment rules are mandatory.

  • The rubble piles present the most risk to novices. If you are operating a robot on a rubble pile or other difficult scenario and it needs to be extracted, please ask your associated emergency responder to retrieve it (hopefully, he/she was the one driving when it got stuck anyway!).
  • Always maintain awareness of others working within your scenario and communicate your intentions *before* doing whatever you have in mind.
  • Understand that robots can do unpredictable things; the bigger/heavier the robot the more space you should allow it. And don’t stick your fingers into places it might not like. Always familiarize yourself with the emergency stop procedures first… and last…. before interacting with or operating robots. Some implementations are more predictable than others, and it has nothing to do with shininess!
  • If you see anything you consider unsafe in our environment, please inform the incident commander or any emergency responder on site, and let’s discuss it at the daily after action briefing (or the next morning safety briefing).

[Return to Top]

 

 



 

 

isd-webmaster@cme.nist.gov
Date Created: 06/19/2006
Last updated: 08/04/2008