Terrain Perception Sensors for Autonomous Off- Road Navigation
Larry Matthies
Supervisor, Machine Vision Group
Jet Propulsion Laboratory
Point of Departure: Demo III and Perceptor Programs
QuickTime™ and a Video decompressor are needed to see this picture.
Basic System Loop
Sensing
Representation
Planning
Control
Today’s Presentation
Sensing
Representation
Planning
Control
Sensing
• What do we need to perceive?
– Terrain geometry – Material type
• Hard terrain: soil, rock, concrete, etc
• Soft stuff: primarily vegetation; maybe mud
• Water
• Barbed wire – Road locations – Other race vehicles
• Under what conditions:
– Daylight
– Atmosphere: dusty, maybe rain
• How do we perceive; ie. what sensors will do this job?
• What sensor specs and performance is required?
– Angular resolution (eg. milliradians/pixel) – Field of view of each sensor
– Field of regard of sensor system (including any sensor pointing platform) – Range resolution
– Exposure time, depth of field – Frame rate
– Etc.
Sensing: Today’s Material
• What do we need to perceive?
– Terrain geometry – Material type
• Hard terrain: soil, rock, concrete, etc
• Soft stuff: primarily vegetation; maybe mud
• Water
• Barbed wire – Road locations
• Under what conditions:
– Daylight
– Atmosphere: dusty, maybe rain
• How do we perceive; ie. what sensors will do this job?
• What sensor specs and performance is required?
– Angular resolution (eg. milliradians/pixel) – Field of view of each sensor
– Field of regard of sensor system (including any sensor pointing platform) – Range resolution
– Exposure time, depth of field – Frame rate
– Etc.
Perceiving Terrain Geometry
• The usual candidates:
– Stereo vision (visible or thermal) – Ladar (one or two axis scanning) – Radar
– Sonar
– Structured light
• Likely choices for our budget:
– Stereo vision (visible) – Ladar (one axis scanning)
– Maybe radar (for seeing through dust and detecting barbed wire)
JPL Stereo Vision
512x480 Firewire stereo pair Downsample to
256x240 Rectify
SAD 7x7 template, 25 &12 disparities
Consistency checks and subpixel interpolation
256x240 and 128x120 disparity maps Bandpass filter
SAD 11x11 template, 11x11 search
Consistency checks and subpixel interpolation
128x120 flow field Downsample to
128x120 and bandpass
QuickTime™ and a Animation decompressor are needed to see this picture.7 Hz on 750 MHz P3 laptop
MovieBrief Digression: Moving Object Detection On- the-Move Built on JPL Stereo System
QuickTime™ and a BMP decompressor are needed to see this picture.
QuickTime™ and a Photo - JPEG decompressor are needed to see this picture.
QuickTime™ and a Photo - JPEG decompressor are needed to see this picture.
One Axis Scanning Ladar
SENSOR: SICK LMS200
PURPOSE: Obstacle Detection and Avoidance COST: $6,000 each
KEY SPECIFICATIONS:
• 1-axis laser scanner
• 0.9µm, 3mrad divergence, class 1
• 0 - 20m range, ±0.02m accuracy
• Line scan: 180º at 75Hz (selectable)
• Data interface - RS-232 or RS-422
Perceiving Vegetation
• Main options:
– RGB color image classification
– Visible/near-infrared (VNIR) image classification – Thermal infrared (TIR) image analysis
– Ladar range data scatter analysis
Perceiving Vegetation: RGB Imagery
50 0 150 100
250 200 0
50 100
150 200 0
20 40 60 80 100 120 140 160 180
Green Red
Blue
50 0 150 100
250 200 0
50 100
150 200 0
20 40 60 80 100 120 140 160 180
Green Red
Blue
From training examples, determine clusters in color space
• Mathematical model: mixture of Gaussians
• Parameters estimation: Expectation Maximization
• C++ implementation benchmarks (Motorola 2400): 40 ms for 320x240 image
green vegetation
green vegetation dry vegetationdry vegetation soil/rocksoil/rock
Perceiving Vegetation: RGB Imagery
Range map Original
image
QuickTime™ and a Video decompressor are needed to see this picture.
Terrain classification
Classified obstacles
green vegetation
dry vegetation
soil rock
outlier
Perceiving Vegetation: VNIR
0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
foliage
soil dry grass
Wavelength (µm)
Reflectivity
Reflectivity
Wavelength
Perceiving Vegetation: VNIR
: =
650nm image 800nm image 650:800 ratio image
Classified image
Perceiving Vegetation: VNIR
Classification error rates with simple Gaussian classifier:
• 7% with red-green
• 0% with red-NIR Errors in shadow with red-green up to 20%
Blue (450) Green (550) Red (650)
NIR (800) RGB R,NIR,B
Perceiving Vegetation: Thermal Imagery (TIR)
2:30 pm 10:30 pm
• TIR may also be useful for water and road detection; may also see a little better
through dust
Perceiving Vegetation: Ladar Analysis
laser
Self-occlusions
sees rocks 1.5 m away through grass Near-zero false positives
in sparse and dense grass with robust detection of solid material.
Ex. 1. Dense grass vs. ground Ex. 2. Sparse grass vs. rock
Perceiving Vegetation: Camera Vendors
• RGB: same as stereo
• Ladar: same as for terrain geometry
• VNIR: eg. Duncantech
• TIR: uncooled is all that affordable; Raytheon “ControlIR 2000B” and Indigo Systems “Omego” are candidates for roughly $6K to $15K each
• My bias:
– Baseline RGB imagery as the lost-cost starting point
– Ladar and TIR as options, because we probably need them for other things as well – VNIR probably doesn’t add enough to be worth the time and cost
Perceiving Water
• Main options:
– RGB image classification – Ladar lack of return
– Thermal image classification
– Short wave infrared image classification – Polarization
Perceiving Water: RGB Imagery
• Goal: develop/understand approaches for (1) detection of water bodies in imagery and (2) placing it in the map. Important cases to distinguish:
1. Out in the open, reflecting sky 2. Out in the open, reflecting trees 3. Under tree canopy
4. Large vs small water body 5. Day vs night
• Begun by addressing (1) with a simple classifier based on brightness, color saturation, and local image variance.
Scene containing water Detected water Saturation vs. brightness
scatterplot from training data
Combined with RGB Soil and Vegetation Classifier
Rippled water
Smooth water
Results for 24-Hour Image Sequence
QuickTime™ and a Animation decompressor are needed to see this picture.
