ISSN 1055- 1425
November 2007
This work was performed as part of the California PATH Program of the
University of California, in cooperation with the State of California Business,
Transportation, and Housing Agency, Department of Transportation, and the
United States Department of Transportation, Federal Highway Administration.
The contents of this report reflect the views of the authors who are responsible
for the facts and the accuracy of the data presented herein. The contents do not
necessarily reflect the official views or policies of the State of California. This
report does not constitute a standard, specification, or regulation.
Final Report for RTA 65A0150
CALIFORNIA PATH PROGRAM
INSTITUTE OF TRANSPORTATION STUDIES
UNIVERSITY OF CALIFORNIA, BERKELEY
Transit Integrated Collision Warning System
Volume II: Field Evaluation
UCB- ITS- PRR- 2007- 20
California PATH Research Report
California PATH Program
Carnegie Mellon University - Robotics Institute
CALIFORNIA PARTNERS FOR ADVANCED TRANSIT AND HIGHWAYS
Transit Integrated Collision Warning System,
Volume II: Field Evaluation
Prepared by:
University of California at
Berkeley
PATH Program
1357 South 46 th Street
Richmond, CA 94804
Carnegie Mellon University
Robotics Institute
5000 Forbes Ave
Pittsburgh, PA 15213
Prepared for:
California Department of Transportation
U. S. Department of Transportation
Federal Transit Administration
Final Report for RTA 65A0150
ii
Acknowledgements
This report presents the results of a research effort undertaken by the California PATH
Program ( PATH) of the University of California at Berkeley, Carnegie Mellon University
( CMU) Robotics Institute, San Mateo County Transit District ( SamTrans), and Port
Authority of Allegheny County ( PAT) under funding provided by the Federal Transit
Administration under Federal, California Department of Transportation ( Caltrans) and the
Pennsylvania Department of Transportation. ( PennDOT) under federal ID#
250969449000 through RTA 65A0150.
The people who participated directly in this research include ( in alphabetical order):
PATH: Joanne Chang, Susan Dicky, Bart Duncil, Scott Johnston, Paul
Kretz, Thang Lian, Xiaoyun Lu, David Marco, David Nelson,
Steven Shladover, Wei- Bin Zhang, Yongquan Zhang
CMU: Dave Duggins, Jay Gowdy, Martial Hebert, John Kozar, Rob
MacLachlan, Christoph Metz, Aaron Steinfeld, Arne J Suppe,
Chuck Thorpe
SamTrans: Frank Burton ( Project Manager)
PAT: Dan DeBone, Rick Snyder
The direction of Sébastien Renaud and Brian Cronin of the Federal Transit
Administration are gratefully acknowledged. Special thanks are also due to Sonja Sun of
Caltrans and Chris Johnson of PennDOT, who provided contractual assistance and were
instrumental in the progress of this work. The technical assistance provided by Eric
Traube of Mitretek has been very beneficial to this research and evaluation program.
Also this work would not have been possible without the cooperation of the local transit
agencies. Specifically, the contributions from the dispatchers and bus operators of
SamTrans and PAT were essential for the success of the field tests and ultimately the
project.
iii
iv
Abstract
This evaluation report examines the performance of the Integrated Collision Warning
System prototype developed by the University of California PATH Program and the
Carnegie Mellon University Robotics Institute. The evaluation was based on testing the
sensors, processing algorithms, and driver- vehicle interfaces in both controlled and real
world operational environments. Evaluation metrics and methodologies were used to
evaluate the effectiveness of the system. The effort for this evaluation was based on the
following tasks:
Task 1: Develop Evaluation Scenarios
Task 2. Perform Closed Course System Testing Under Controlled- Environment
Task 3. Conduct Detection Analysis
Task 4 Analyze Driving Behavior Data
Task 5 Surveys and Interviews
Keywords: Integrated Collision Warning System, low speed collision warning, Transit
bus safety
v
vi
Executive Summary
Transit bus crashes cost agencies money, cause service interruptions and personal
injuries, and adversely affect transit reliability and public image. Over the past five
years, 30,000 bus crashes have caused 17,000 deaths and injuries, accounting for $ 800
million in annual insurance claims. i Sudden stops or swerves to avoid a crash can cause
passenger falls which result in additional passenger injuries and liability. Insurance
claims reflect only some of the costs to transit agencies. In addition to the financial cost
of insurance claims and vehicle repairs, there are issues with staff resources consumed to
process claims, and ( more importantly), the issue of lost future rider ship due to adverse
public sentiment regarding transit reliability and safety.
Effective collision warning systems for transit buses could address many of these
incidents. This project used commercially available sensors with custom developed
algorithms to determine the suitability of these types of systems for transit- specific
operating conditions. The evaluation of these systems demonstrated that:
• Current sensors for forward collision warning work reasonably well in a
typical urban transit operating environment, although some modification
will be required
• Side obstacle sensors and algorithms also work reasonably well, but have
some issues with appropriate threat detection, which require further
development of software algorithms
• Under- the- bus detection functions did not work well enough in the
configuration tested to be enabled for revenue service and would require a
higher leap in technology to be useful.
This project involved test track verification of sensor capabilities and software algorithms
and a year of testing in revenue service. Driver reaction to the system in revenue service
was generally positive. Thousands of hours of data were collected that could be used for
further analysis in future research.
A preliminary cost- benefit analysis of the systems tested indicates that these systems
have significant promise. Comprehensive analysis of crash and incident data from 35
California transit agencies ( operating a total of 1758 revenue service buses) collected
between 1997 and 2001 revealed a total of about 10,000 crashes and incidents, averaging
more than one incident per bus per year. Total costs of these crashes and incidents were
$ 36 M ($ 23 M crash related; $ 13 M passenger injury related), averaging $ 4000 per bus
per year. Based on these statistics, if 30% - 50% of transit bus accidents could be
prevented by deploying ICWS at a cost of $ 5,000 per bus, the liability savings alone
could pay for the systems in two to four years. This analysis clearly shows that transit
ICWS could be cost effective.
Currently available off- the- shelf collision warning systems are designed primarily for
highway use by passenger cars and heavy commercial vehicles ( trucks). The highway
operating environment represents a less complex threat assessment scenario than the
vii
urban and suburban arterial environments in which transit buses generally operate.
Commercially available systems are generally designed to operate at highway speeds on
roads with primarily moving targets and clearly marked lane boundaries. The urban
driving environment represents a particularly complex threat environment for the
collision warning systems. Transit buses need to operate in close proximity to many
stationary and moving objects, including pedestrians, bus stops, parked cars, moving cars,
bicyclists, etc. and often need to make sharp turns with minimal clearance to nearby
objects. These factors add to the challenge of making a collision warning system for
transit with accurate threat assessment.
This project built on previous research conducted under DOT’s Intelligent Vehicle
Initiative. Previous research developed and tested frontal, side and rear transit
collision warning systems separately. This project tested an integrated frontal and
side transit collision warning system. As part of this IVI program, two research teams
from California and Pennsylvania, composed of transit agencies, state departments of
transportation, research universities, and a bus manufacturer, engaged in the development
of Frontal Collision Warning Systems ( FCWS) and Side Collision Warning Systems
( SCWS). Under that Phase One project ( 2000- 2002), preliminary requirement
specifications and prototype FCWS and SCWS were developed. This project represented
the Phase Two effort to develop an Integrated Collision Warning System ( ICWS).
This ICWS project included the following major efforts:
1. Development of interface requirements
2. Development of two prototype ICWS, including integrated Driver Vehicle
Interface ( DVI)
3. Test track verification tests of ICWS
4. Pilot tests and data collection on ICWS in the San Francisco Bay Area, CA
and in Pittsburgh, PA for 12 months
5. Analysis of field data before and after ICWS activation to analyze any driver
behavior changes.
The development of the ICWS interface requirements and two prototype ICWS were
reported in the Transit ICWS Interface Control Document [ FHWA- JPO- 04- 097] and
Integrated Collision Warning System Final Technical Report [ FTA- PA- 26- 7006- 04].
This evaluation report provides results of the test track verification and field tests. The
verification tests of the FCWS and SCWS elements of the ICWS were conducted
separately due to differences in their respective system characteristics.
The verification tests for the FCW system showed that the obstacle detection
function provided adequate longitudinal measurements in a transit operational
environment, but the quality of the measurements of the lateral distance to targets
in front of the bus still needed improvements. Test results showed that, under the
tested scenarios, the FCWS could correctly identify hazardous targets and generate
warnings when driver action was needed. However, errors in lateral position
measurements could potentially cause false detections of targets that were not threats,
thereby resulting in false positive warnings. Time delays in the sensing and signal
viii
processing functions also reduced the effectiveness of the frontal collision warning
system.
Tests under controlled conditions showed that the SCWS had no missed warnings
or false negatives under specific staged crash scenarios, but there were some issues
with false positives. These tests also showed that the false positive warning rates for
both contact and under- the- bus incidents were unacceptably high for any reasonable
performance requirement, and therefore warnings for these two conditions were not
activated and displayed for the operational testing in revenue service. Analysis of field
test data showed that of the warnings issued by the SCWS, about 2/ 3 of the alerts and 1/ 3
of the imminent warnings were correct warnings. Most of the incorrect imminent
warnings were caused by incorrect velocity estimates. Curb detection reduced the
nuisance alarms and false warnings on the right side by 30%. The analysis also showed
that the remaining nuisance alarms and false warnings were caused by a variety of
reasons including vegetation, false or no velocity, and ground returns, etc.
Two buses instrumented with the prototype ICWS were tested in revenue service in
the San Francisco Bay Area and Pittsburgh. Data for a total of seven bus operators
were analyzed, dealing with issues of driver behavior in general, as well as issues
specific to the collision warning systems. The database developed in this project contains
both engineering data and video recording of operating conditions and driving behavior.
These data represent a valuable asset for evaluation in future research. The data analysis
compared drivers’ behavior during the period when the ICWS was turned on with the
baseline ‘ before’ data ( when the systems were active and collecting data, but not issuing
any alerts or warnings).
The field data collection and analysis of the usage of the collision warning systems
by bus operators have shown that the ICWS increased consistency of driving
behavior and had the most noticeable effects on the most aggressive drivers. The
general trends in bus operator behavior after activation of the frontal warning system
were more cautious or conservative driving, at larger car following gaps and with reduced
braking severity. The data also showed that changes in driver behavior with regard to the
SCWS were also towards safer driving, but the changes were less evident than for the
frontal collision warnings. There were some hints that the SCWS was also used in
unintended ways such as driving closer to the guardrails.
In addition to the above main findings, the research team also learned the following
lessons:
• The existing commercially available collision warning systems, which were
developed for highway applications, are not suitable for transit operations in
urban and suburban environments without significant modification. Data
collected using instrumented buses in revenue service showed that the transit
operation environment involves complex threat scenarios that existing
commercial CWS were not designed for.
ix
• The advanced ICWS developed specifically for this project addressed some of the
limitations in existing commercial CWS for transit- specific operating
requirements. However, improvements are still needed to overcome the limited
ability of the systems to detect, classify and track target objects, so that false and
nuisance alerts can be further reduced without additional false negatives.
• The verification tests indicated that the sensing approaches used for both frontal
and side collision warning systems need refinement to meet transit requirements.
Specifically, the FCWS required additional sensing means and sensor fusion to
determine the lateral position of obstacles relative to the vehicle path and their
threat levels and to compensate for sensor and processing delays and errors. The
SCWS may also need to employ additional sensing means and improved
algorithms to classify objects as vegetation or ground, and to improve velocity
measurements.
• An integrated Driver Vehicle Interface ( DVI) for FCWS and SCWS was
developed in order to make sure that warnings were intuitive and effective for the
drivers. A warning synthesizer to present fewer warnings to the operator was not
implemented for several reasons. It was thought that a false positive could
potentially suppress a true positive. In operation, very few examples of frontal
and side alerts occurred in close proximity to each other during the field tests,
indicating very limited potential usefulness of a warning synthesizer for
prioritizing the warnings.
• The need for integration of forward and side collision subsystems will depend in
part on whether the integrated system will be significantly different in cost or
performance from independent ones. In discussions with transit operators and bus
manufacturer/ suppliers, operators generally prefer to have an integrated ICWS
unless the cost is almost as much as the combined cost of two independent
systems. If the cost is the same for one integrated system or two single function
systems, some operators would prefer separate subsystem options.
• As today’s bus manufacturers have already implemented selected standards for in-vehicle
communication networks ( J- 1939 data buses) and electronic interfaces, it
would be desirable if the collision warning systems are integrated with the transit
bus electronics through these already standardized electronic interfaces on buses.
From these lessons learned and as a result of the data analysis, the following topics are
recommended for future research:
• Additional evaluations of warning strategies in a bus driving simulator
• Further improvements of FCWS and SCWS threat assessment algorithms in a
representative transit operating environment
x
• Larger Field Operational Tests, with more drivers and more buses for a longer
duration
• Outreach to transit operating agencies regarding cost/ benefit potential of transit
ICWS
• Development of an effective under- the- bus warning system
• Additional analyses of existing data.
In conclusion, the verification tests were valuable in establishing parameters for
acceptable performance of ICWS in transit- specific urban environments, and the
ability of current technologies to meet those parameters. Both raw sensor capability
and threat assessment algorithms were verified in the test track work. This work will be
quite valuable as a foundation work for development of an ICWS for a larger field
operational test or commercial system. In other words, this project provided the
foundation work on what can and cannot be done with currently available sensor,
threat assessment, and data fusion capabilities to meet typical transit operating
requirements. This project also performed initial driver acceptance testing of systems
using current capabilities, as well as pioneering integrated DVI work. The field testing in
revenue service provided useful lessons that could be used as the basis for larger scale
field tests.
Transit operators participating in this project were generally enthusiastic about the
potential of these systems. We believe that the ICWS technologies developed under this
project have great potential for improving safety of transit operations and could
contribute to the effective performance of ICWS systems for other vehicle platforms in
urban/ suburban scenarios. However, more work is needed on the threat assessment
algorithms and sensor suite to develop an ICWS that is suitable for a typical transit
operational environment in terms of accurate threat detection and driver
acceptance. We therefore recommend that the Federal Integrated Vehicle Based
Safety Systems ( IVBSS) initiative be expanded to include a transit IVBSS FOT.
xi
xii
TABLE OF CONTENTS
Executive Summary ............................................................................................................................... ............ vi
1 Introduction ............................................................................................................................... ................ 1
1.1. ICWS Program Need ......................................................................................................................... 1
1.2. ICWS Goals ............................................................................................................................... ........ 2
1.3. ICWS Evaluation Report Scope ........................................................................................................ 2
1.3.1 Task 1: Develop Evaluation Scenarios ........................................................................................ 2
1.3.2 Task 2: Closed Course System Testing in a Controlled Environment....................................... 3
1.3.3 Task 3: Conduct Warning Analysis ............................................................................................. 3
1.3.4 Task 4: Analyze Driving Behavior Data...................................................................................... 3
1.3.5 Task 5: Surveys and Interviews.................................................................................................... 4
1.4. Document Organization................................................................................................................... . 5
1.5. ICWS Overview ............................................................................................................................... . 6
1.5.1 ICWS Architecture................................................................................................................... .... 6
1.5.2 FCWS Hardware Overview.......................................................................................................... 7
1.5.3 SCWS overview ............................................................................................................................ 9
1.5.4 ICWS Driver Vehicle Interface.................................................................................................. 12
1.5.5 ICWS Sensors Field of View ..................................................................................................... 12
1.6. ICWS Reference Documents........................................................................................................... 13
2 Engineering Verification of ICWS under Controlled Environments .............................................. 16
2.1. Terminology for Warning System Processes ................................................................................. 18
2.2. Verification and Validation of FCW System ................................................................................. 19
2.2.1 Methodology.................................................................................................................... ........... 20
2.2.2 Sensor Calibration and Validation ............................................................................................. 22
2.2.3 System Testing Under Controlled Environment Scenarios ...................................................... 31
2.2.4 Evaluation of FCWS Test Results.............................................................................................. 39
2.3. Verification and Validation of SCW System ................................................................................. 41
2.3.1 Methodology.................................................................................................................... ........... 42
2.3.2 Sensor Calibration and Validation ............................................................................................. 43
2.3.3 System Testing in a Controlled Environment ........................................................................... 47
2.3.4 Evaluation of the SCWS Test Results ....................................................................................... 55
2.4. Summary of FCWS and SCWS Test Results ................................................................................. 55
2.4.1 Findings ............................................................................................................................... ....... 55
2.4.2 Recommendations ....................................................................................................................... 56
3 Evaluation of ICWS in Revenue Service .............................................................................................. 58
3.1. Testing Procedures ........................................................................................................................... 59
3.1.1 Field Testing Routes ................................................................................................................... 59
3.1.2 Participants ............................................................................................................................... .. 59
3.1.3 Schedule....................................................................................................................... ............... 60
3.1.4 Weather ............................................................................................................................... ........ 60
3.1.5 Training....................................................................................................................... ................ 60
3.1.6 Data Collection..................................................................................................................... ...... 61
3.2. Evaluation of Frontal Collision Warning System ( FCWS) ........................................................... 61
3.2.1 FCWS Measures of Effectiveness.............................................................................................. 61
xiii
xiv
3.2.2 Data Processing Tools and Procedures ...................................................................................... 62
3.2.3 Analyses of Driving Behavior .................................................................................................... 64
3.2.4 FCWS Technical Issues .............................................................................................................. 74
3.2.5 Summary of Key Findings about Driving with FCWS............................................................. 76
3.3. Evaluation of Side Collision Warning System ( SCWS)................................................................ 77
3.3.1 SCWS Measures of Effectiveness.............................................................................................. 77
3.3.2 Data Processing Tools and Procedures ...................................................................................... 78
3.3.3 Analyses of Driving Behavior .................................................................................................... 80
3.3.4 False Positive Rate during Normal Operation........................................................................... 85
3.3.5 Reduction of Nuisance Alarms through Curb Detection.......................................................... 89
3.3.6 SCWS Technical Issues .............................................................................................................. 89
3.3.7 Summary of Key Findings about Driving with SCWS............................................................. 95
3.4. Key Hardware Problems .................................................................................................................. 96
3.5. Collision Warning System Integration Issue: Simultaneous Warnings ........................................ 97
3.5.1 Warning Integration Issues ......................................................................................................... 98
3.6. Operator Feedback ......................................................................................................................... 100
3.6.1 Operator Acceptance of the System......................................................................................... 100
3.6.2 Operator Suggested Changes.................................................................................................... 100
3.6.3 Agencies’ Feedback .................................................................................................................. 101
3.7. Conclusions Regarding ICWS in Revenue Service ..................................................................... 101
3.7.1 Benefits of the System .............................................................................................................. 102
3.7.2 Issues that Need Further Attention........................................................................................... 102
3.7.3 Remarks on Other Findings ...................................................................................................... 102
3.7.4 Recommendations ..................................................................................................................... 103
4 Conclusions and Recommendations .................................................................................................... 104
4.1. Conclusions ............................................................................................................................... .... 104
4.2. Recommendations .......................................................................................................................... 107
4.2.1 Driving Simulator Studies ........................................................................................................ 107
4.2.2 Further Improvements of FCWS and SCWS .......................................................................... 107
4.2.3 Need for Larger Scale Field Operational Tests ....................................................................... 107
4.2.4 Outreach to Transit Agencies for ICWS.................................................................................. 108
4.2.5 Improve Cost and Performance of Laser Scanner................................................................... 108
4.2.6 Add a Dedicated Under- the- bus Sensor................................................................................... 108
4.2.7 Perform Additional Data Analysis ........................................................................................... 108
4.2.8 Refine the SCWS Measures of Effectiveness ......................................................................... 108
4.2.9 Recommendation Summary ..................................................................................................... 109
References..................................................................................................................... .................................... 110
Appendix A DVI Improvement Testing............................................................................................... A- 1
A. 1 Introduction ............................................................................................................................... ... A- 1
A. 2 Experimental Set- up...................................................................................................................... A- 2
A. 3 Results........................................................................................................................ ................... A- 3
A. 4 Conclusion ............................................................................................................................... ... A- 10
Appendix B Data Analysis for Verification Tests for Chapter Two ................................................ B- 1
B. 1 Terminology for Verification Test Data Analysis ....................................................................... B- 1
xv
xvi
B. 2 Sensor Verification and Calibration Tests .................................................................................... B- 2
B. 2.1 Inter- vehicle Distance Measurement Error.............................................................................. B- 2
B. 2.2 Static Object Lateral Distance Measurement, Prediction / Estimation Error ........................ B- 5
B. 2.3 Time Delay Test Data Analysis................................................................................................ B- 9
B. 2.4 Gyro Rate Angle Measurement Tests .................................................................................... B- 14
B. 3 Scenario- based System Verification ........................................................................................... B- 15
B. 3.1 Vehicle Following................................................................................................................... B- 15
B. 3.2 Detection of Moving Target in Adjacent Lane ..................................................................... B- 21
B. 3.3 Cut- in and Cut- out Test .......................................................................................................... B- 21
B. 3.4 Low Speed Approaching / Crashing into a Static Object ..................................................... B- 22
Appendix C Detailed Data Plots for Chapter Three .......................................................................... C- 1
C. 1 Database Records of Driving Statistics for Seven Bus Operators .............................................. C- 1
C. 2 Table of Vehicle Following Time Gap Percentile Values........................................................... C- 5
C. 3 Cumulative Distributions of Brake Pressure Applied by Operators ........................................... C- 5
C. 4 Cumulative Distribution of Accelerations .................................................................................... C- 7
C. 5 Cumulative Distributions of Time To Collision ( TTC)............................................................... C- 9
C. 6 Cumulative Distributions of Required Deceleration Parameter................................................ C- 13
Appendix D Questionnaires and Direct Operator Feedback ........................................................... D- 1
D. 1 Questionnaire Methodology ......................................................................................................... D- 1
D. 1.1 Respondents - SamTrans ......................................................................................................... D- 1
D. 1.2 Respondents - Port Authority .................................................................................................. D- 2
D. 2 Ride- Along Methodology............................................................................................................. D- 2
D. 3 Operator Acceptance of the ICWS............................................................................................... D- 2
D. 3.1 Questionnaire Results .............................................................................................................. D- 2
D. 3.2 Ride- Along Results .................................................................................................................. D- 5
D. 4 Did the System Prevent a Crash? ................................................................................................. D- 6
D. 5 Operator Reliance....................................................................................................................... .. D- 6
D. 6 Alarm Rates ............................................................................................................................... ... D- 6
D. 6.1 Nuisance Alarms and False Alarms ........................................................................................ D- 6
D. 6.2 Missing Alarms ........................................................................................................................ D- 7
D. 6.3 Multiple Alarms ....................................................................................................................... D- 7
D. 6.4 Fog Alarms ............................................................................................................................... D- 8
D. 7 Operator Sensitivity Ratings / Reports ........................................................................................ D- 8
D. 8 Operator Suggested Changes........................................................................................................ D- 8
D. 9 Training Type and Amount for Users of an ICWS..................................................................... D- 9
D. 9.1 Questionnaire Results .............................................................................................................. D- 9
D. 9.2 Ride- Along Results – Operator Questions ........................................................................... D- 10
D. 10 Long- term Effects of Using a Collision Warning System........................................................ D- 11
D. 11 Relaying of Passenger Queries and Comments......................................................................... D- 12
D. 12 Final Notes.......................................................................................................................... ........ D- 12
D. 13 Discussion..................................................................................................................... .............. D- 13
xvii
xviii
D. 14 Transit Agency Feedback ........................................................................................................... D- 14
D. 15 Conclusion ............................................................................................................................... ... D- 16
Appendix E Collision Warning System ( CWS) Evaluation Questionnaire .................................... E- 1
Appendix F Conversion Tables.............................................................................................................. F- 1
LIST OF TABLES
Table 2- 1 - Measured errors in forward target vehicle prediction ................................... 24
Table 2- 2 - Static Object Lateral Position Prediction Error in Azimuth Angle................ 26
Table 2- 3 - Time Delay Analysis for Overall System..................................................... 28
Table 2- 4 - Measured errors in forward target lateral estimation .................................... 32
Table 2- 5 - Measured errors in forward target vehicle estimation................................... 33
Table 2- 6 - Parameter estimation for moving target in adjacent lane .............................. 34
Table 2- 7 - Target Detection/ Estimation/ Prediction Characteristics Including Accuracy 41
Table 2- 8 - Range and Resolution of Laser Line Striper................................................. 43
Table 3- 1 - Summary of data available for seven bus operators...................................... 60
Table 3- 2 - Format of time gap data file......................................................................... 64
Table 3- 3 - Warnings triggered by mixed true and false targets...................................... 75
Table 3- 4 - Durations and distances of driving used for the MOE analysis..................... 79
Table 3- 5 - Warning rate for different side warning levels, by side and transit agency ... 79
Table 3- 6 - True and false positive warnings. ................................................................ 86
Table 3- 7 - Rates of simultaneous warnings per hour..................................................... 98
Table 3- 8 - Warning rates for the front, right and left side ............................................. 98
Table 3- 9 - The percentages of warnings that would be simultaneous if uncorrelated..... 98
Table A- 1 - Overall Mean Reaction Times ( msec) ....................................................... A- 8
Table C- 1 - Lower percentiles of car- following time gaps in seconds with std dev....... C- 5
Table C- 2 - Upper Tails of Brake Pressure Distributions, psi with standard deviations C- 7
Table C- 3 - Lower percentiles of driving acc. distribution ( m/ s/ s) with std dev............ C- 9
Table C- 4 - Lower Percentiles of Time- to- Collision Distribution ( sec) and std dev.... C- 13
Table C- 5 - Upper Percentiles of Req’d Deceleration Parameter, m/ s/ s, with std dev. C- 16
Table D- 1 - Summary of Operator Responses to Questionnaire ................................... D- 4
Table D- 2 - Summary of Operator Responses to Questionnaire ................................. D- 10
Table D- 3 - Summary of Operator Questions About the System................................ D- 11
Table D- 4 - Summary of Transit Agency Feedback to the Questionnaire ................... D- 15
xix
xx
LIST OF FIGURES
Figure 1- 1 - System Architecture..................................................................................... 6
Figure 1- 2 - Layout of the Frontal portion of the ICWS computer enclosures .................. 8
Figure 1- 3 - Layout of FCWS sensors, cameras, DVI and SCWS curb detector ............... 9
Figure 1- 4 - Layout of the Side portion of the ICWS computer enclosures....................... 9
Figure 1- 5 - Right and Left Side Collision Warning System Sensor Layout ................... 10
Figure 1- 6 - Front bumper with the laser ( blue arrow) and camera ( red arrow) visible ... 11
Figure 1- 7 - Left Side Camera on SamTrans Bus 601 .................................................... 11
Figure 1- 8 - ICWS DVI ................................................................................................. 12
Figure 1- 9 - Integrated system spatial coverage on SamTrans bus.................................. 13
Figure 1- 10 - Integrated system spatial coverage on the PAT bus .................................. 13
Figure 2- 1 - Bus is driven along the left lane marker instead of the lane center .............. 21
Figure 2- 2 - Photos of the test instrumentation and setups.............................................. 22
Figure 2- 3 - Tests for longitudinal measurements wrt a frontal moving target................ 24
Figure 2- 4 - Parked cars on both sides, with all car doors closed.................................... 27
Figure 2- 5 - Parked cars on both sides, with one door open............................................ 27
Figure 2- 6 - Verification of Gyro yaw rate accumulation error test ................................ 30
Figure 2- 7 - Vehicle following, without use of string pot ............................................... 32
Figure 2- 8 - Moving target vehicle in adjacent lane ....................................................... 35
Figure 2- 9 - Cut- in to test lateral movement detection ................................................... 36
Figure 2- 10 - Cut- out to test lateral movement detection................................................ 37
Figure 2- 11 - Low Speed Crash Test.............................................................................. 38
Figure 2- 12 - Distributions of the errors in velocity ....................................................... 44
Figure 2- 13 - Density ( log scale) of warnings around the PAT bus ................................ 46
Figure 2- 14 - Density ( log scale) of warnings around the SamTrans bus ........................ 46
Figure 2- 15 - Snapshot of the bus driving past four boxes.............................................. 49
Figure 2- 16 - One of the boxes is pulled parallel to the driving direction of the bus ....... 49
Figure 2- 17 - The box in the front is pulled perpendicular to the bus.............................. 50
Figure 2- 18 - The POC distribution of all curves of all objects from one run ................. 50
Figure 2- 19 - Medium sensitivity warning area graph with an example of a POC curve. 51
Figure 2- 20 - A three- second image sequence of a person falling under the bus............. 53
Figure 3- 1 - Data sorting procedure ............................................................................... 63
Figure 3- 2 - Time gap cumulative distribution ( Operator A) .......................................... 66
Figure 3- 3 - Time gap cumulative distribution ( Operator B) .......................................... 67
Figure 3- 4 - Time gap cumulative distribution ( Operator C) .......................................... 67
Figure 3- 5 - Time gap cumulative distribution ( Operator D) .......................................... 67
Figure 3- 6 - Time gap cumulative distribution ( Operator E)........................................... 68
Figure 3- 7 - Time gap cumulative distribution ( Operator F)........................................... 68
Figure 3- 8 - Time gap cumulative distribution ( Operator G) .......................................... 68
Figure 3- 9 - Post- activation evolution of sensitivity versus warnings/ alerts ( Op A)........ 73
Figure 3- 10 - Post- activation evolution of sensitivity versus warnings/ alerts ( Op B) ...... 73
Figure 3- 11 - Post- activation evolution of sensitivity versus warnings/ alerts ( Op C) ...... 74
Figure 3- 12 - Range and range rate of target tracks and tracks rejected.......................... 75
Figure 3- 13 – Probability distribution of side warning durations.................................... 79
Figure 3- 14 - Ratios/ differences of the alert/ imminent warning rates for the right side... 81
Figure 3- 15 - Ratios/ differences of the alert/ imminent warning rates for the left side..... 82
xxi
xxii
Figure 3- 16 - Yaw rate after a warning was issued......................................................... 83
Figure 3- 17 - The average yaw rate after an imminent warning...................................... 84
Figure 3- 18 - Measure of the fluctuations. ..................................................................... 85
Figure 3- 19 - Overhanging bush is close enough to trigger an imminent warning .......... 87
Figure 3- 20 - The velocity of the vehicle is slightly off, leading to an alert .................... 87
Figure 3- 21 - Ground returns seen as an object in the left front of the bus ( red box)....... 88
Figure 3- 22 - Distribution of precipitation ..................................................................... 90
Figure 3- 23 - Distribution of average daily temperature................................................. 91
Figure 3- 24 - Distribution of alert rate ........................................................................... 92
Figure 3- 25 - Alert rate versus precipitation................................................................... 92
Figure 3- 26 - Alert rate versus temperature.................................................................... 93
Figure 3- 27 - Example of a splash of water appearing on the right side of the bus.......... 95
Figure A- 1 - Integrated DVI ........................................................................................ A- 1
Figure A- 2 - Set- up for DVI experiment ...................................................................... A- 3
Figure A- 3 - Histograms of response time to a high intensity signal ( 20 msec step) ..... A- 4
Figure A- 4 - Histograms of response time to a low intensity signal ( 20 msec step) ...... A- 5
Figure A- 5 - Histograms of response times to a low intensity signal ( 10 msec step)..... A- 6
Figure A- 6 - RMS Error in Lane Keeping Task ........................................................... A- 7
Figure A- 7 - Test subject 1 - Response Time ............................................................... A- 8
Figure A- 8 - Test subject 2 - Response Time ............................................................... A- 8
Figure A- 9 - Test subject 3 - Response Time ............................................................... A- 9
Figure A- 10 - Test subject 1 – RMS Error ................................................................... A- 9
Figure A- 11 - Test subject 2 – RMS Error ................................................................. A- 10
Figure A- 12 - Test subject 3 – RMS Error ................................................................. A- 10
Figure B- 1 - Lateral and longitudinal position and relative speed for Scenario 2.......... B- 3
Figure B- 2 - Relative distance tracking error [ m]; Estimate and prediction are equal ... B- 3
Figure B- 3 - Relative speed tracking error with estimation and prediction ................... B- 4
Figure B- 4 - Front moving target absolute acceleration error ....................................... B- 4
Figure B- 5 - Lateral, longitudinal and speed estimation/ prediction of first firm track... B- 5
Figure B- 6 - The same as Figure B- 5 with zoomed middle plot ................................... B- 6
Figure B- 7 - Lateral, longitudinal and speed est / prediction of second firm track ........ B- 7
Figure B- 8 - The same as Figure B- 7 but with zoomed middle plot.............................. B- 7
Figure B- 9 - Lateral, longitudinal and speed estimation/ prediction of third firm track.. B- 8
Figure B- 10 - The same as Figure B- 9 with zoomed middle plot.................................. B- 8
Figure B- 11 - Target speed: fifth wheel ( red), estimation ( green) and prediction ( blue) B- 9
Figure B- 12 - Zoomed from Figure B- 11................................................................... B- 10
Figure B- 13 - Zoomed from Figure B- 11................................................................... B- 10
Figure B- 14 - Zoomed from Figure B- 11................................................................... B- 11
Figure B- 15 - Zoomed from Figure B- 11................................................................... B- 12
Figure B- 16 - Zoomed from Figure B- 11................................................................... B- 13
Figure B- 17 - Yaw angle estimate from yaw rate measurement: radians vs. seconds.. B- 14
Figure B- 18 - Zoomed from Figure B- 17 to see the error ........................................... B- 14
Figure B- 19 - Lateral position, longitudinal relative distance and speed for Scen 1 .... B- 15
Figure B- 20 - Relative speed tracking error wrt the fifth wheel of the target vehicle .. B- 16
Figure B- 21 - Second firm track lat. position, relative long. position and speed ......... B- 17
xxiii
xxiv
Figure B- 22 - Lat. and relative long. position and speed with middle plot zoomed..... B- 18
Figure B- 23 - Lat. position, long. relative position, and target speed estimate ............ B- 19
Figure B- 24 - The same as in Figure B- 23 but with zoomed middle plot.................... B- 19
Figure B- 25 - The Arq parameter for Scenario 1 ........................................................ B- 20
Figure B- 26 - Left adjacent lane moving target ( vehicle) detection ............................ B- 21
Figure B- 27 – First firm track lat. and long. position, speed estimate and prediction.. B- 22
Figure B- 28 - Front target lateral and longitudinal distance estimation....................... B- 23
Figure B- 29 - Low- Speed Crash Test, Left and Right Warning Levels....................... B- 23
Figure C- 1 - Driving statistics by day for bus operator A............................................. C- 1
Figure C- 2 - Driving statistics by day for Bus operator B............................................. C- 2
Figure C- 3 - Driving statistics by day for Bus operator C............................................. C- 2
Figure C- 4 - Driving statistics by day for Bus operator D ............................................ C- 3
Figure C- 5 - Driving statistics by day for Bus operator E............................................. C- 3
Figure C- 6 - Driving statistics by day for Bus operator F ............................................. C- 4
Figure C- 7 - Driving statistics by day for Bus operator G ............................................ C- 4
Figure C- 8 - Brake Pressure Cumulative Distribution ( Operator A) ............................. C- 6
Figure C- 9 - Brake Pressure Cumulative Distribution ( Operator B) ............................. C- 6
Figure C- 10 - Brake Pressure Cumulative Distribution ( Operator C) ........................... C- 7
Figure C- 11 - Acceleration Cumulative Distribution ( Operator A)............................... C- 8
Figure C- 12 - Acceleration Cumulative Distribution ( Operator B) ............................... C- 8
Figure C- 13 - Acceleration Cumulative Distribution ( Operator C) ............................... C- 8
Figure C- 14 - Time to Collision Cumulative Distribution ( Operator A) ..................... C- 10
Figure C- 15 - Time to Collision Cumulative Distribution ( Operator B) ..................... C- 10
Figure C- 16 - Time to Collision Cumulative Distribution ( Operator C) ..................... C- 11
Figure C- 17 - Time to Collision Cumulative Distribution ( Operator D) ..................... C- 11
Figure C- 18 - Time to Collision Cumulative Distribution ( Operator E)...................... C- 11
Figure C- 19 - Time to Collision Cumulative Distribution ( Operator F)...................... C- 12
Figure C- 20 - Time to Collision Cumulative Distribution ( Operator G) ..................... C- 12
Figure C- 21 - Required Deceleration Parameter Cumulative Distribution ( Op. A) ..... C- 14
Figure C- 22 - Required Deceleration Parameter Cumulative Distribution ( Op. B) ..... C- 14
Figure C- 23 - Required Deceleration Parameter Cumulative Distribution ( Op. C) ..... C- 14
Figure C- 24 - Required Deceleration Parameter Cumulative Distribution ( Op. D) ..... C- 15
Figure C- 25 - Required Deceleration Parameter Cumulative Distribution ( Op. E)...... C- 15
Figure C- 26 - Required Deceleration Parameter Cumulative Distribution ( Op. F)...... C- 15
Figure C- 27 - Required Deceleration Parameter Cumulative Distribution ( Op. G) ..... C- 16
xxv
1
Introduction
1.1. ICWS Program Need
Bus crashes have been a major concern for transit operators. Over the past five years,
30,000 bus crashes have caused 17,000 deaths and injuries, accounting for $ 800 million
in annual insurance claims. Bus crashes have resulted in property damage, service
interruptions and personal injuries; they also affect transit efficiency, revenue and
image. In addition to collision damage, passenger falls resulting from emergency
maneuvers also contribute to an increased potential for passenger injuries and liability.
Comprehensive analysis of crash and incident data from 35 California transit agencies
( operating a total of 1758 revenue service buses) collected between 1997 and 2001
revealed a total of ~ 10,000 crashes and incidents, averaging more than one incident per
bus per year. Total costs of these crashes and incidents were $ 36 M ($ 23 M crash
related; $ 13 M passenger injury related), averaging $ 4000 per bus per year. Furthermore,
a transit collision ripples through the agency and consumes additional resources to settle
claims and results in significant loss of good will. The study showed that if 30% - 50%
of transit bus accidents could be prevented by deploying ICWS at a unit cost of $ 5,000,
the liability savings due to crashes and incidents could pay for the system in two to four
years. These results clearly show that transit ICWS can be cost effective.
Existing work including SAE and ISO standards, have all been focusing on collision
warning for highway applications ii iii . Currently available off- the- shelf collision warning
systems are also designed for highway use, primarily for commercial vehicle operations.
The highway operating environment is much simpler than the urban and suburban arterial
environments in which transit buses generally operate. Transit buses need to operate in
close proximity to many stationary and moving objects, including pedestrians, bus stops,
parked cars, moving cars, bicyclists, etc. and often need to make sharp turns with
minimal clearance to nearby objects. Because of sensor limitations, the commercially
available collision warning systems tend to give too many warnings to the drivers when
used in urban / suburban environments, causing drivers to ignore the system or disable it.
These factors add to the challenge of making a collision warning system that contributes
to safety and that transit operators will accept. The critical issue is to improve the
accuracy of the warnings in order to be effective in advising drivers to take corrective
action.
Under the Transit Intelligent Vehicle Initiative ( Transit IVI) program sponsored by the
U. S. Department of Transportation and based on recommendations from transit
stakeholders, the Federal Transit Administration ( FTA) initiated development efforts on
transit collision warning technologies. Two research teams from California and
Pennsylvania, composed of transit agencies, state departments of transportation, research
universities, and a bus manufacturer, have engaged in the development of Frontal
Collision Warning Systems ( FCWS) and Side Collision Warning Systems ( SCWS).
Under the Phase One program ( 2000- 2002), preliminary requirement specifications and
prototype FCWS and SCWS were developed. FTA, with the advice of the transit IVI
2
stakeholder group, decided to move forward with integrating the FCWS and SCWS into
an Integrated Collision Warning System ( ICWS) in Phase Two ( 2003- 2005).
The Integrated Collision Warning System evaluated herein was built and integrated on
two transit buses operating in revenue service. They were operated in the San Francisco
Bay Area, CA and in Pittsburgh, PA for about one year in order to collect adequate data
for evaluation of the effectiveness of the ICWS.
1.2. ICWS Goals
The goals identified by the ICWS team were as follows:
1. Develop a Functional ICWS
2. Create System Acceptable to Operators ( Drivers & Operations)
3. Demonstrate a Potential for Reduction in the Severity and Frequency of
Collisions
4. Prove Technical Feasibility Through Field Test of Prototype System( s)
1.3. ICWS Evaluation Report Scope
This evaluation report examines the performance of the Integrated Collision Warning
System prototype in order to verify if the integrated system achieved these goals. The
evaluation was based on testing the sensors, processing algorithms, and driver- vehicle
interfaces in both controlled and real world operational environments. Evaluation metrics
and methodologies for testing advancement towards these goals were generated in order
to evaluate the effectiveness of the system against the goals. The effort for this evaluation
was based on the following tasks, which are described in more detail in the following
paragraphs.
1. Task 1: Develop Evaluation Scenarios
2. Task 2. Perform Closed Course System Testing Under Controlled Environment
3. Task 3. Conduct Detection Analysis
4. Task 4 Analyze Driving Behavior Data
5. Task 5 Surveys and Interviews
1.3.1 Task 1: Develop Evaluation Scenarios
As the first step of this evaluation, the ICWS team developed two sets of evaluation
scenarios and refined the metrics and methods for the subsequent tasks. The first scenario
set was used to quantitatively evaluate the performance of the integrated system including
the sensing, detection, and warning functions ( for Tasks 2 & 3). The second set included
scenarios designed for examining driver behavior for baseline ( none), independent ( left,
forward, right), and integrated warnings ( for Task 4). Specific survey questions were also
developed to examine driver acceptance and system performance ( for Task 5).
3
1.3.2 Task 2: Closed Course System Testing in a Controlled
Environment
Certain scenarios do not occur frequently enough in real world driving to adequately test
how the system handles specific events. Events of key interest are actual frontal and side
collisions, pedestrian under bus warnings, and bicycle side collisions. Closed course
testing allowed tests to be run using staged scenarios to gather data that would not be
possible with the bus in revenue service.
Controlled testing of this nature also allowed evaluators to collect accurate system
performance data to identify sensor bias, misclassifications, and other subtle system
errors. Independent measuring systems were established in order to identify the sensor
and system errors and delays.
1.3.3 Task 3: Conduct Warning Analysis
Perhaps the largest concern for an integrated collision warning system operated in an
urban environment is that the system will be susceptible to false alarms and unable to
consistently identify real threats. Using manually encoded real threats from recorded
video data, the system warning outputs were examined and classified. Metrics for this
task included:
1. True positives: when the system correctly identifies a real threat.
2. False negatives: when the system does not identify a real threat.
3. True negatives: when the system does not identify a threat when none is present.
4. False positives: when the system identifies a threat when none is present.
5. Fault tree distribution: for false positives and false negatives, where does the fault
originate?
6. Scenario parsing: Under what driving scenarios do false and nuisance alarms
occur? False alarms may be caused by faults ( system malfunctions) or incorrect
classification of a safe situation as a threat, while nuisance alarms are situations
when the system functions correctly, but the driver finds the alarm annoying.
1.3.4 Task 4: Analyze Driving Behavior Data
On- board collection of driver behavior data provided insights to the use of an assistance
system and the potential for safety benefit. Such data were valuable because they were
collected during field- testing in revenue service.
The analysis of these data included a longitudinal human factors analysis of driving
behavior. The periods of data collection were:
( A) Baseline - DVI off, but system on and recording
( B) Full System - DVI on and system on and recording
Metrics used in evaluating driver behaviors were:
4
1. Behavior when within CWS DVI activation range: does time gap change, and in
what way, when drivers are following a lead vehicle and the DVI is activated? Do
drivers alter their lateral behavior as a result of DVI activation?
2. Normal following distances: do drivers alter their following distances as a result
of the system?
3. Time within each CWS DVI category ( alert, warn): the quantity of time drivers
occupy activated DVI categories. This includes analysis of whether drivers try to
exit such threat regions earlier than when DVI is not present.
4. Braking rate: there is concern that the DVI may lead to more hard braking events
and therefore increase risk of passenger falls. This is an attempt to determine if
the system increases such risk.
5. Swerving rate: this is similar to braking behavior but focused on lateral behavior.
6. Frequency of warnings over time: this is a measure of how overall driver behavior
may or may not shift towards safer driving habits.
1.3.5 Task 5: Surveys and Interviews
Driver perceptions of the system were quantified through carefully constructed surveys
and interviews.
Metrics for this task included:
1. False and Nuisance alarms: the false positives, as well as true positives that
drivers find annoying.
2. Driver sensitivity ratings/ reports: survey or discussion based data collection that
quantified driver opinion on the appropriateness of system sensitivity.
3. Driver perception of safety benefit: these data include subjective reporting of
safety improvements or degradations for the whole system, and specific events
( e. g., simultaneous warnings). This line of data collection included driver
perception of system impact on their workload.
4. Self- reports of alterations in driving behavior: these data involved documentation
of behavior shifts as a result of system use.
5. Satisfaction with system performance: This metric involved documentation of
how drivers perceived the system with respect to overall performance of the
whole system, and specific factors ( e. g., reliability in inclement weather, details
relevant for training, etc.).
6. Perception of system accuracy: This metric is related to feedback on false and
nuisance alarms but is more general. For example, the system may accurately
detect threats but incur an unacceptable delay before issuing a warning. Another
example is that drivers may feel the system improperly elevates certain threats
from an alert to a warning.
7. Relaying of passenger queries and comments: the team fully expected passengers
to notice the DVI and external sensors. Documentation of their comments and
opinions via the drivers and existing rider feedback options permitted an initial
read on how riders perceive the system.
5
1.4. Document Organization
This first chapter of this document describes the program need, goals and scope. It
provides a summary of the tasks accomplished in the evaluation process, document
organization and presents a high level ICWS Overview describing the system
architecture, hardware, sensors, operator interface and the areas of coverage around the
bus. It also includes a list of reference documents for additional information on the
Frontal, Side and Integrated Collision Warning System Programs.
Chapter Two describes the closed course testing and results, which involved separate
testing of the forward and side looking components of the ICWS. The Frontal Collision
Warning System testing involved driving the equipped bus through scenarios featuring
static objects and other vehicles in known positions, and evaluating the correctness of the
responses of the warning system. The side collision warning testing involved staged
scenarios of collisions and near collisions to calibrate and evaluate the performance of the
system, including its curb detection and object- under- bus detection capabilities.
Chapter Three describes the field testing of the buses in revenue service. This includes
descriptions of the test conditions and data acquisition, and the results of the analysis of
the data, including measures of changes in safety- related driver behavior. Also included
is a summary of the operator feedback and analysis.
Chapter Four describes in more detail the conclusions, lessons learned and
recommendations as a result of building, testing and evaluating this system
The appendices provide additional technical details and are organized by the chapters that
they refer to. Specifically:
Appendix A provides data, results and recommendations after testing the ICWS
Driver Vehicle Interface display in simulation.
Appendix B provides additional data and analysis for Chapter 2: “ Engineering
Verification of ICWS under Controlled Environments”. These data include
backup for the sensor verification and calibration closed course tests:
• Verification of inter- vehicle distance measurement error
• Verification of static object lateral distance measurement, prediction /
estimation error
• Time Delay Test Data Analysis
• Gyro rate angle measurement tests
And the scenario based system verification data for
• Vehicle Following
• Detection of moving target in adjacent lane
• Cut- in and cut- out test
• Low speed approaching/ crashing to a static object
6
Appendix C contains more detailed data plots and analysis for Chapter 3:
“ Measurements of Driver Usage of Collision Warning Systems”
• Database records of driving statistics for seven bus operators
• Cumulative distributions of brake pressure applied by operators
• Cumulative distribution of accelerations
• Cumulative Distributions of Time to Collision ( TTC)
• Cumulative Distributions of Required Deceleration Parameter
Appendix D describes the feedback from transit operators and transit agencies,
obtained from questionnaires, emails, meetings, phone calls, and demonstrations,
as well as the feedback received from drivers on ride- alongs during the course of
the field testing.
Appendix E contains the questionnaire used for obtaining the operator feedback.
Appendix F contains the metric conversion tables and formulas.
1.5. ICWS Overview
1.5.1 ICWS Architecture
Figure 2- 1 shows the architecture of the Integrated Collision Warning System ( ICWS)
Prototype.
Figure 2- 1 - System Architecture
7
The overarching design philosophy was to integrate the frontal and side collision warning
systems through information integration. In implementing the integrated prototype
hardware, we wanted to ensure that each system could operate even if the others go
down. With separate computing systems this dictated a level of independence that does
not need to be reflected in the end commercial product.
The three computers which are executing the warning algorithms are integrated together
through a FCWS- SCWS serial communication link. This link was used to synchronize
the time basis for data collection, to pass warnings between the frontal and side systems
and was proposed to pass obstacle data at the boundaries between the frontal and side
systems. The time stamps and the warnings were used extensively for post processing
data analysis, but the obstacle data were not shared in this program. It remains to be
shown whether the data sharing is useful in an integrated collision warning system.
An integrated DVI displays the warnings from both the FCWS and the SCWS. The DVI
and Driver Interface control box are responsible for presenting integrated warnings to the
transit operator.
A common coordinate system was used to enable the integration of the frontal and side
areas of coverage.
This integration at the higher level facilitated the ICWS development and testing
activities, building on prior research on the separate FCWS and SCWS. However, future
generation systems for commercial use are likely to be integrated at lower levels to
economize on component costs, volume and weight. The next steps in this program
should include developing an initial commercial prototype which would integrate the
hardware subsystems, overlapping sensor fields of view and developing common
software modules between the frontal and side collision warning systems.
1.5.2 FCWS Hardware Overview
1.5.2.1 FCWS Computer Enclosure Layouts
Figure 2- 2 shows the layout of the FCWS computer enclosure.
8
Figure 2- 2 - Layout of the Frontal portion of the ICWS computer enclosures
1.5.2.2 FCWS Sensors
Figure 2- 3 shows the layout of FCWS object sensors and video cameras as well as the
SCWS Curb Detector on the front face of SamTrans bus 601. The positions of each
sensor/ camera are measured in a FCWS reference frame. The frame is originated on the
ground under the center point of the front bumper with positive directions of x-, y- and z-axes
pointing to driver- side, upward, and forward respectively.
Vehicle speed is recorded from the vehicle’s SAE J1939 interface on the SamTrans bus
and the J1708 interface on the PAT bus and also by measuring the analog speed signal
directly from the transmission. A rate gyro is mounted in a waterproof enclosure on the
underside of the bus floor near the rear axle and a yaw rate accelerometer is mounted
within the electronics area. Brake pressure is measured using a pressure transducer
mounted on a spare port of the air brake system under the floor of the driving area. A
proximity sensor mounted near a universal joint on the drive shaft is used to determine if
the bus is moving at speeds lower than 2- 3 miles per hour. Turn signal activation and
backing light status are recorded by tapping off the existing turn signal circuit and
backing lights. A DINEX module was added to read the door open status, turn / hazard
flashers and as a time delay after power up to enable power to the Collision Warning
System hardware. Windshield wiper activation is determined with a proximity sensor
mounted on the windshield wiper mechanism. The GPS antenna is mounted on the rear of
the roof near the exhaust for the HVAC, and the GPS computer is mounted in a
waterproof enclosure near the HVAC evaporator unit in the rear of the bus. The GPS and
CDPD modem antenna are mounted on the rear of roof near the exhaust for the HVAC,
while the GPS and CDPD modem computers are mounted in a waterproof enclosure near
the HVAC evaporator unit in the rear of the bus.
9
o
z
x
y
Figure 2- 3 - Layout of FCWS sensors, cameras, DVI and SCWS curb detector
1.5.3 SCWS overview
1.5.3.1 SCWS Computer Enclosure Layouts
Figure 2- 4 shows the layout of the SCWS computer enclosure.
Figure 2- 4 - Layout of the Side portion of the ICWS computer enclosures
10
1.5.3.2 SCWS Sensors
Figure 2- 5 shows the right ( top drawing) and left side ( bottom drawing) of the transit bus.
The SCWS object sensors are SICK laser scanners mounted on the left and right sides of
the transit bus and a curb detector mounted in the right side of the front bumper. The
SICK laser scanners sit approximately 24 inches above the ground.
The Curb Detector is mounted inside the front bumper as shown in Figure 2- 6. The
underside of the front bumper is shown, with the blue arrow pointing to the laser and the
red arrow pointing to the camera.
Figure 2- 7 shows the forward part of the left side of SamTrans bus number 601. The data
collection camera that looks toward the rear of the bus can be seen in the upper left
corner of the figure. There are four of these cameras, whose locations are shown in
Figure 2- 5.
Figure 2- 5 - Right and Left Side Collision Warning System Sensor Layout
Front of Bus
Scanner
Camera Camera looking towards rear looking towards front
Camera looking towards front Camera looking towards rear
Scanner
Curb Tracker
Front of Bus
11
Figure 2- 6 - Front bumper with the laser ( blue arrow) and camera ( red arrow) visible
Figure 2- 7 - Left Side Camera on SamTrans Bus 601
12
1.5.4 ICWS Driver Vehicle Interface
The main components of the DVI are two LED assemblies – one on the left- hand A- pillar
and the other on the center pillar. Both assemblies are constructed identically, with seven
LED segments filling the top and two LED segments filling the bottom ( See
Figure 2- 8). All LEDs in the displays have the capability to be either amber or red. The
upper LEDs are 3 x 2 cm and the lower LEDs are 3 x 3 cm, with a triangular mask
pointing towards the side for which it is displaying the warning. The total assembly
dimension is 4 x 22 cm. The LEDs have a maximum luminance intensity of 90/ 60 mcd
and a viewing angle of 100 degrees.
Figure 2- 8 - ICWS DVI
1.5.5 ICWS Sensors Field of View
Figure 2- 9 and
Figure 2- 10 illustrate the Fields of View of the two buses equipped with the ICWS
system. The farthest detectable range for the FCWS in the same lane is 100 m ( 330 ft)
and the closest detectable range in the same lane is no greater than 3 m ( 10 ft). The
maximum detectable side- looking angle from the front bus corners is 30 degrees on
SamTrans bus 601 and 20 degrees on the PAT bus. The detectable lateral position for the
forward sensors is over 6 m ( 20 ft). The side looking sensors can closely track objects
that are within 3 meters of the bus however, objects can be detected as far as 50 meters
away.
13
Samtrans ICWS BUS
6m
3m
100m
30d
1m
6 m
3m
2m
: Uncovered Area
Figure 2- 9 - Integrated system spatial coverage on SamTrans bus
PAT ICWS BUS
6m
3m
100m
20d
1m
6 m
3m
2m
: Uncovered Area
Figure 2- 10 - Integrated system spatial coverage on the PAT bus
1.6. ICWS Reference Documents
The “ Integrated Collision Warning System” ( ICWS) project was preceded by two
projects, one concerning frontal ( FCWS) and the other concerning side ( SCWS)
collisions. This section lists the documents which were produced by these three projects.
The journal articles, conference papers, etc. related to these projects are shown at the end
of this document. Most of the documents are available at
http:// www. ri. cmu. edu/ projects/ project_ 324. html ( SCWS) and
http:// www. ri. cmu. edu/ projects/ project_ 498. html ( ICWS).
14
Side Collision Warning System:
1. “ A Summary of Commercially Available Side Collision Warning Systems”,
AssistWare Technology, Inc., 1998
2. “ A New Focus for Side Collision Warning Systems for Transit Buses”, S.
McNeil, C. Thorpe, and C. Mertz, ITS2000, Intelligent Transportation Society of
America's Tenth Annual Meeting and Exposition, May, 2000.
3. “ Side Collision Warning Systems for Transit Buses”, C. Mertz, S. McNeil, and C.
Thorpe, IV 2000, IEEE Intelligent Vehicle Symposium, October, 2000.
4. “ Side Collision Warning Systems for Transit Buses: Functional Goals”, D.
Duggins, S. McNeil, C. Mertz, C. Thorpe, and T. Yata, Technical Report - CMU-RI-
TR- 01- 11, Robotics Institute, Carnegie Mellon University, 2001.
5. “ Facts and Data Related to Bus Collisions”, Carnegie Mellon University Robotics
Institute, April 2002
6. “ Functional Goals”, Carnegie Mellon University Robotics Institute, April 2002
7. “ Assessment of Technologies”, Carnegie Mellon University Robotics Institute
8. “ State of the Art of Technology”, Carnegie Mellon University Robotics Institute,
April 2002
9. “ Side Collision Warning System ( SCWS) Performance Specifications”, Carnegie
Mellon University Robotics Institute, May 2002
10. “ A Performance Specification for Transit Bus Side Collision Warning System”, S.
McNeil, D. Duggins, C. Mertz, A. Suppe, and C. Thorpe, ITS2002, proceedings
of 9th World Congress on Intelligent Transport Systems, October, 2002
11. “ Development of the Side Component of the Transit Integrated Collision Warning
System”, A. M. Steinfeld, D. Duggins, J. Gowdy, J. Kozar, R. MacLachlan, C.
Mertz, A. Suppe, C. Thorpe, and C. Wang, IEEE Conference on Intelligent
Transportation Systems ( ITSC), 2004
12. “ A 2D Collision Warning Framework based on a Monte Carlo Approach”, C.
Mertz, Proceedings of ITS America's 14th Annual Meeting and Exposition, April,
2004.
13. “ Collision Warning and Sensor Data Processing in Urban Areas”, C. Mertz, D.
Duggins, J. Gowdy, J. Kozar, R. MacLachlan, A. M. Steinfeld, A. Suppe, C.
Thorpe, and C. Wang, Proceedings of the 5th international conference on ITS
telecommunications, June, 2005, pp. 73- 78.
Front Collision Warning System:
1. " Preliminary Safety Analysis of Frontal Collision Avoidance", El Miloudi El
Koursi, Ching- Yao Chan, Wei- Bin Zhang, 3rd IEEE International Conference on
Intelligent Transportation Systems, Dearborn, MI, Oct. 1- 3, 2000
2. " Develop Performance Specifications for Frontal Collision Warning System for
Transit buses", Wei- Bin Zhang, et al. 7th Intelligent Transportation Systems
World Congress Turin, Italy, November 6- 11, 2000
3. " Integrated Multi- Sensor System: A Tool for Investigating Approaches for
Transit Frontal Collision Mitigation", Xiqin Wang, Wei- Bin Zhang, Scott
Johnston, Dan Empey, and Ching- Yao Chan, ITS World Congress, Sydney,
Australia, 2001
15
4. “ Functional Analysis of Frontal Collision Warning System”, M. El Koursi, E.
Lemaire, Ching- Yao Chan, Wei- Bin Zhang, ITS World Congress, Sydney,
Australia, 2001
5. " Studies of Accident Scenarios for Transit Bus Frontal Collisions", Ching- Yao
Chan, Kun Zhou, Xi- Qin Wang and Wei- Bin Zhang, ITS America Annual
Meeting, Orlando, Florida, 2001
6. " Scenario Parsing in Transit Bus Operations For Experimental Frontal Collision
Warning Systems", Ching- Yao Chan, Xi- Qin Wang, Wei- Bin Zhang, IEEE
Intelligent Vehicle Conference, Tokyo, Japan, 2001
7. " A new maneuvering target tracking algorithm with input estimation", Kun Zhou,
Xiqin Wang, Masoyashi Tomizuka, Ching- Yao Chang, and Wei- Bin Zhang,
American Control Conference, Anchorage, Alaska, 2002
8. “ Development of Requirement Specifications for Transit Frontal Collision
Warning System,” California PATH program, March 2002.
9. " Development of Requirement Specifications for Transit Frontal Collision
Warning System", Xiqin Wang, Joanne Lins, Ching- Yao Chan, Scott Johnston,
Kun Zhou, Aaron Steinfeld, Matt Hanson, Wei- Bin Zhang, PATH Technical
Report, UCB- ITS- PRR- 2003- 29, November, 2003
10. " Development of Requirement Specifications for Transit Frontal Collision
Warning System- Final Report", Xiqin Wang, Joanne Chang, Ching- Yao Chan,
Scott Johnston, Kun Zhou, Aaron Steinfeld, Matt Hanson, and Wei- Bin Zhang,
PATH Technical Report, UCB- ITS- PRR- 2004- 14, May 2004
11. " Studies of Accidents and Cost data for Transit Buses", Kun Zhou, Wei- Bin
Zhang, Gary Glenn, Xiqin Wang, and Ching- Yao Chan, ITS World Congress,
Nagoya, Oct. 2004
Integrated Collision Warning System:
1. “ Transit Bus Integrated Collision Warning Systems Performance Specifications
( Draft)”, joint publication with Carnegie Mellon University Robotics Institute
and California PATH program, December 2002
2. “ Integrated Collision Warning System Interface Control Document”, joint
publication with Carnegie Mellon University Robotics Institute and California
PATH program
3. “ Integrated Collision Warning System Final Technical Report”, FTA- PA- 26-
7006- 04.1, joint publication with Carnegie Mellon University Robotics Institute
and California PATH program
16
2 Engineering Verification of ICWS under Controlled
Environments
An ICWS needs to provide threat warnings to the driver correctly and in time. Correctly
means that the system only provides warnings to the driver in situations when an object
in the path of the bus could potentially cause a frontal or side collision. To achieve this, a
transit ICWS system needs to be able to accurately detect obstacles, to determine their
threat level and to provide warnings early enough to allow the driver to react. Nuisance
warnings, which violate the driver’s expectations about the necessity of the warnings,
need to be minimized.
These basic principles for the design of a warning system are simple enough to state in
qualitative form, but it is not straightforward to turn them into quantitative system
requirements. The top- level performance requirements for a collision warning system
have to be defined based on considerations of acceptability to drivers and compatibility
with their driving behavior, because the driver is an essential component of the combined
driver/ vehicle safety system. At the same time, these requirements have to be tempered
by realistic constraints based on the limitations of available components, especially
sensors.
The field testing element of this project, to be described in Chapter 3, provides a good
opportunity to observe the effects of the collision warning system on driver behavior and
the responses of the drivers to warnings. The test- track testing under controlled
conditions reported in this Chapter provides complementary information about the
capabilities of the sensors and the warning system software to distinguish hazards from
non- hazards. The combined results from both sets of tests improve our understanding of
how to improve the performance of the collision warning system iteratively, rather than
in a top- down design process driven by a priori system requirements. The extensive
work of CAMP for passenger car collision warning systems has shown how challenging
it can be to define such a priori requirements.
The objectives of the controlled- condition tests reported here were:
1. to understand the error characteristics of the measurements and parameter
estimations based on the vehicle on- board sensors;
2. to calibrate the measurements;
3. to evaluate the ability of the ICWS to issue warnings in known hazardous
conditions and avoid issuing warnings in known non- hazardous conditions.
This chapter describes the results of tests that have been conducted for multiple scenarios
under controlled conditions, apart from the field tests in public service, and which have
been designed to represent situations that could be encountered by a bus driven in a real
urban or suburban environment. Since the ICWS is operated autonomously and warnings
are completely based on real- time detection/ estimation from measurement by remote
sensors such as LIDAR ( laser radar), three factors are crucial for the system to have good
performance:
1. tracking of objects that have relative motion with respect to the bus
17
2. detection, estimation and prediction of the motions of the objects – their position,
speed and acceleration with respect to the bus
3. short time delays associated with these processes.
The original FCWS specification iv mainly concentrated on system hardware
characteristics, including sensors and vehicles. There was no specification of warning
system functional requirements such as false negative or false positive warning rates.
However, two aspects of the original specification are closely related to the quantitative
testing:
1. System operation environment: Along bus routes on urban streets, objects such
as trees, poles, traffic signs, parked cars, pedestrians, bicycles, motorcycles, and
other vehicles, will be encountered. This motivated the quantitative tests to
include typical representatives of those static and moving objects.
2. Time delay: The processing delay from system input to output should be no
longer than 0.5 s ( this includes the maximum 0.3 s sensor delay).
From sensor detection to warning issuance, there are several complicated processes:
Sensor detection 􀀂 tracking 􀀂 prediction 􀀂 warning ( threat assessment)
algorithm + warning threshold 􀀂 warning issuance
It would be desirable to have quantitative specifications for the warning issuance such as
false negative or false positive warning rates. Errors in any of the intermediate processes
would affect this performance. It would be difficult to specify the error level in advance
to satisfy the end requirement for the following reasons:
1. Sensor measurement limitations in precision: most sensor manufacturers
specify their products under ideal situations. For example, when LIDAR and
radar sensors are mounted to a solid pole on the ground, their measurement
accuracy can satisfy the error specifications. However, if they are mounted on
a moving bus with random vibrations and rotational movements caused by
unevenness of the road, the target angles will be distorted significantly;
2. Some processes in the chain are algorithm dependent, and alternative
implementations would lead to different error magnitudes;
3. Proper algorithms for tracking and filtering would reduce error magnitude,
while improper algorithms would magnify one error or the other;
4. Many factors would affect the a priori specification of those intermediate
parameters. In fact, much work would be necessary to quantitatively
determine how the error bound specification of each factor in the chain would
affect the end performance.
Since there is no way to specify the error bound in advance for all those intermediate
parameters, the quantitative tests can identify the magnitudes of the errors without a
priori criteria to compare to. An iterative design process is necessary to improve the end
18
performance through the refining of each of the intermediate processes. The
development and testing of the warning system in this project are part of this iterative
process.
The key results of the testing under controlled conditions include the performance of the
obstacle detection system’s sensors, i. e., their ability to discriminate hazardous obstacles
from non- hazardous ones, and the performance of the collision warning system, including
the ability to generate correct warnings under staged crash situations and the rate of
incorrect positive and negative warnings.
Because of the different characteristics of FCWS and SCWS, two different approaches
were taken for the verification tests:
• For the frontal system, it is possible to validate the obstacle detection system with
reference to ground truth and to verify the overall system performance through a
limited number of scenarios that will cover most of the possible situations the
system will be exposed to. Staging these scenarios and comparing the system
outputs with ground truth will give the desired information.
• For the side system, there is a much greater variety of possible situations,
including a greater diversity of objects and a greater variety of dynamic
arrangements. It was therefore necessary to find situations that are likely to cause
false warnings by first examining operational data and then staging appropriate
situations.
2.1. Terminology for Warning System Processes
The process of using a transducer inside a sensor system to represent aspects of the
physical environment in electronic form is observation. The process of determining
whether an object exists or not, is defined as detection. The process of measuring the
object status, such as location and velocity, from the observations, is defined as
estimation. The estimated parameters are random variables, because they are calculated
from observations and the observations are random samples from a probabilistic set. The
results of detection and estimation are called measurements in this report. A
measurement may come from single or multiple observations.
The results of detection and estimation of objects are called tracks or target tracks, and
the process to initiate, manipulate and end tracks is called tracking. A track is a
stochastic process generated by a sensor to represent an object. Tracks from different
sensors may represent the same object, but these tracks must be fused into one track in
order to be useful. Threat assessment is the process whereby the current situation is
projected into the future to assess the severity of a potential encounter with an object.
The detection is an internal process for sensors, which usually has some time delay.
Tracking may also introduce extra data when the tracks have been built. To reduce the
overall time delay from detection to warning issuance, a technique called prediction is
introduced, which is based on algorithms such as Kalman filtering, which predict ( in real
time) the parameter( s) to be measured at the next time step. Although prediction may
reduce time delay, it may also produce extra measurement errors at the same time. In this
19
collision warning system, prediction of parameters is used for threat assessment and thus
will be emphasized.
2.2. Verification and Validation of FCW System
In contrast to frontal collision warning systems designed for highway applications, a
transit collision warning system needs to perform obstacle detection and threat
assessment and to determine the need for warnings in complex urban environments where
a significant number of targets is always present. In order to correctly detect hazardous
situations and to minimize false positives, it is essential that the obstacle detection
function in an FCW system accurately detects all obstacles near the vehicle path and
discriminates the obstacles that may potentially cause threats to the vehicle from the ones
that do not.
The FCWS obstacle detection system consists of a combination of sensing and data
analysis processes. The range sensors detect various targets within their range and build
numerical ‘ target tracks’. The tracking process determines the consistency of the
detected obstacles and selects those that are most relevant as firm tracks. Because the
transit FCWS threat assessment algorithm is built upon the estimation of the distance
between the target vehicle and the bus and the estimation and prediction of the velocity
and acceleration of the target vehicle, it is critical to understand the characteristics of the
measurements and estimations relevant to obstacle detection. The most effective way to
evaluate these characteristics is to conduct a set of tests in a known environment, which
involves setting up targets in predetermined locations and allowing the target vehicles
and the instrumented bus to travel in a predetermined manner without disturbances.
Additional sensors are used to establish ground truth measurements so that performance
of the system can be quantitatively characterized.
Certain scenarios may not occur frequently enough in real world driving to adequately
test how the system handles specific events, such as collisions which are very unlikely to
be encountered during the limited testing period in revenue service. The controlled
closed- course testing allows tests to be run using staged obstacles, which the bus can
crash into without causing any problems.
The verification tests of FCWS were conducted at Crows Landing, an abandoned NASA
airfield, which provided multiple straight lanes ( runways) without extra disturbances. A
number of test scenarios were defined to represent the majority of the urban driving
environment. The tests were designed and conducted to quantitatively measure several
aspects of system performance:
( 1) Sensor measurement errors and time delays: The sensors that require calibration
and verification include the range sensor ( LIDAR in this case), speedometer and
yaw rate Gyro. It is critical to understand the accuracy and time delays of the
range and azimuth measurements obtained from the range sensors. Because the
tracking algorithm also uses speed, yaw angle and yaw rate measurements,
disturbances generated from minor yaw movements ( even on straight roads)
would affect the sensor detection accuracy. Such disturbances become prominent
20
when the bus is driven on an uneven/ bumpy road. Similar to the obstacle
detection sensors, vehicle status sensors also introduce measurement errors and
time delays.
( 2) Target tracking reliability and robustness: Target missing may occur in the
process of sensing, target detection or tracking. Causes of target missing may
include the following: ( a) the sensors themselves do not detect the target at all,
which may happen to both LIDAR and radar; and ( b) incorrect algorithm and/ or
improper threshold values may cause target missing. Even if a target track for an
object is established, tracking errors may still cause nuisance and / or unnecessary
warnings. For example, the target position may be miscalculated/ misestimated
due to measurement errors, or tracking, filtering and/ or fusion algorithm
problems.
( 3) System estimation/ prediction error and processing time delay: The quality of
estimation and prediction of range, range rate, target vehicle speed and
acceleration would be affected by the sensor errors and delays. Since these
parameters are essential for target tracking, threat assessment and warning
issuance, it is critical to understand the errors and time delays associated with
these measurements.
( 4) Warning characteristics: The verification of warning characteristics will focus
on crash scenarios in order to evaluate the performance of the warning algorithm,
including the correctness of the warning and delay factors.
2.2.1 Methodology
The design of the verification tests includes defining the arrangement of static objects and
planning the target vehicle and bus trajectories in a known environment. A static target
may be either a parked car or a cardboard box put in known places with respect to the
center of the road, which are placed to represent roadside parked vehicles, mail boxes,
traffic signs, etc. To represent different objects, cardboard boxes of different sizes were
chosen. In order to make them radar / LIDAR sensitive, the boxes were wrapped with
reflective covering materials. Moving vehicle targets were represented by a passenger
car driven along a known course in the same or opposite direction along the bus driving
course, or as a lead vehicle in front of the bus.
Both the bus and the target vehicle were driven along predetermined straight paths
defined in the coordinate system shown in Figure 2- 1 with the origin at point O. For
measurement consistency, each bus run always started from a known position. Based on
the ground position of the targets and the running distance of the bus, one can calculate
the relative position between the bus and the targets.
21
Figure 2- 1 - Bus is driven along the left lane marker instead of the lane center
The test involved the SamTrans ICWS bus with the following additional instrumentation:
• A test car as a target vehicle equipped with data acquisition system and a
wireless communication system
• An AMETEK Rayelco Position Transducer with a maximum range of 50 ft
( string pot) installed on the rear end of the target vehicle and connected to the
front bumper of the bus for measuring the distance between the bus and the
target vehicle.
• A fifth wheel was mounted on the target vehicle to measure true vehicle speed
and running distance, free from any tire slip and tire pressure variations
• A wireless communication system for synchronization and to pass the
measurements of the target vehicle to the bus.
The true bus speed was obtained through the following process. Since the bus did not
have a fifth wheel and the bus tachometer could only provide a wheel speed, several test
runs were conducted at different speeds to collect data used to calibrate the wheel speed
measurements. The distance traversed on each run was measured precisely and
compared with the integral of the wheel speed measurements. The relative error after
calibration could be as small as 0.3~ 0.5%.
In the discussion throughout this chapter, the true measurement means use of one of the
ground truth references listed above.
22
Figure 2- 2 shows the instrumented target vehicle and static target placements. The
placement of the static obstacles and the instrumentation provide means for collecting
independent and ground truth data regarding range, range rate and lateral displacement of
the obstacles. This information is compared with the data collected and processed within
the FCWS to independently determine the soundness of the warning signals.
Figure 2- 2 - Photos of the test instrumentation and setups
2.2.2 Sensor Calibration and Validation
Two sets of calibration and verification tests were conducted, including a set of tests
aiming at validating and calibrating the characteristics of the sensors and processing
algorithms and scenario- based tests to verify the performance of the system.
2.2.2.1 Sensor Verification and Calibration Tests
The following tests were designed to validate and calibrate ( a) error characteristics of
inter- vehicle distance measurement, ( b) error characteristics of lateral distance
measurement, prediction and estimation from the obstacle detection sensor, ( c) time delay
associated with obstacle detection sensor, and ( d) error characteristics of gyro
measurement.
23
2.2.2.1.1 Error characteristics of inter- vehicle distance
measurement
The longitudinal distance between the subject vehicle and the target vehicle, their relative
speed and relative acceleration are essential for determining the threat level. The
longitudinal distance is obtained from the range sensors. Some sensors can provide
relative speed ( range rate) as well. The relative acceleration, however, needs to be
estimated based on the range and range rate measurements. In many cases, the FCWS
algorithm derives predictions from these measurements in order to compensate for sensor
delays. In the prototype FCWS algorithm tested under this project, an intermediate
parameter Arq ( required deceleration parameter) is used for estimating the threat level.
Arq is closely related to, but not equivalent to, the inverse of time to collision. If the Arq
exceeds the threshold, a warning will be issued.
Tests were designed to verify and validate the range measurements, relative speed and
relative acceleration of a moving target acquired by the ranging sensor, and their
prediction. In order to verify the error characteristics of the sensor measurements and
predictions based on them, independent measurements were collected using a string pot
connected between the rear end of the target vehicle and the bus, a fifth wheel mounted
on the target vehicle and wireless communication transceivers installed on both target
vehicle and the bus. In order to minimize interference for target track processing, no
other targets were placed in the field of view of the sensors. There is no accelerometer on
either the bus or the target vehicle. The true acceleration of the forward target vehicle is
obtained using a fifth wheel and through linear filtering and numerical differentiation of
the fifth wheel speed measurements. Acceleration of the bus is obtained using similar
processing of the calibrated wheel speed measurements on the bus. Based on the
difference between those two measurements, the “ true” relative acceleration is obtained,
which is used to compare with the prediction using the tracking algorithm.
The tests were conducted with the bus following the target vehicle along a straight path
defined by reference lines. The target vehicle accelerated to predetermined speeds ( 5
mph, 10 mph, 15 mph, 20 mph) for a short duration and then decelerated at
approximately 0.2 m/ s 2 , 0.5 2 m/ s , or 0.8 2 m/ s . Because the total length of the string
was 16 m, the tests were conducted to limit the range variations within 6.4 m in order to
avoid breakage. Figure 2- 3 depicts the configuration of this set of tests.
24
Figure 2- 3 - Tests for longitudinal measurements wrt a frontal moving target
Data analysis is shown in Table 2- 1, with results based on test data shown in the
Appendix B, Figure B- 1 - Figure B- 3.
Table 2- 1 - Measured errors in forward target vehicle prediction
Parameter Prediction Errors
Longitudinal distance Directly used the measurement; No prediction.
Longitudinal relative speed 8%
Longitudinal relative acceleration
( RMS)
0.2280 m/ s2
The time points for speed error calculation were selected at t = 25, 50, 75, 100, and 125
seconds of the data in Figure B- 1 - Figure B- 3. Relative speed error was calculated at
each of these points and then averaged. Here the Root Mean Square ( RMS) value was
used for acceleration error calculations, while relative error was used for speed. The
acceleration error calculation has been averaged over the whole time interval ( Figure
B- 4).
25
The test results are compared with the preliminary specifications defined for the FCWS
system by this project team in the previous phase of the project. v The preliminary
specifications specified the closest and farthest detectable range in the same lane to be
greater than 3 m ( 10 ft) and less than 100 m ( 330 ft) respectively, with a resolution to be
finer than 1 m ( 3.3 ft). The test results show that the LIDAR can effectively detect
objects between 0.5 ~ 120 m, which therefore satisfies this specification. The
preliminary specifications also specified the relative speed or range rate measurements to
be valid from - 44 m/ s (- 100 mph, approaching) to + 20 m/ s (+ 45 mph, separating). The
test results show that the LIDAR can satisfy these requirements as well. Note that the
preliminary specifications did not specify the absolute accuracy of the parameters. Since
inaccurate measurements would cause false detections, which in turn would result in false
positive warnings or false negative detections, the acceptable levels of false positives and
false negatives will determine the sensor and processing requirements. Therefore,
extensive field operational tests need to be conducted to first determine the system level
performance requirements and then the requirements on the acceptable level of error
tolerance for the sensor measurements.
2.2.2.1.2 Error characteristics of lateral distance measurement,
prediction and estimation from the obstacle detection sensor
Roadside parked cars can create challenges for transit FCWS. It is necessary to
understand how well the forward obstacle detection sensors detect a static side target
along the roadside and distinguish it from those in the path of the bus. In most cases, the
static side targets are not hazardous. In less frequent cases, side targets may present
hazards when a car door is opened or a car begins to move out of a parking space. In
order to determine if a side target is potentially hazardous to the bus, it is necessary to
have accurate knowledge of the target lateral distance from the bus. The lateral distance
is derived from an azimuth angle measurement by the forward ranging sensor.
In order to verify lateral distance measurements, static targets were placed along the
vehicle path. Two parked cars and a box were staged on the right hand side and left hand
side at predetermined distances with respect to the center of the bus path, as shown in
Figure 2- 4 and Figure 2- 5. The bus was driven straight ahead at speeds of 5 mph, 15
mph, 27 mph, 30 mph and 35 mph. The left door of a car parked on the RHS was opened
occasionally ( Figure 2- 5). The open car door detection scenario was included among the
tests based on feedback from bus drivers. They considered that the suddenly opened door
of a roadside parked car was a real threat to the bus and should be detected if possible.
Our experience shows that this is extremely difficult to achieve using current sensors.
Side static target distance measure relative error is calculated as in Table 2- 2 based on
data corresponding to Figure B- 22: The distance of the closest target edge line in the
ground coordinate with respect to the Y axis ( Figure 2- 3) is 3.0 m.
26
Table 2- 2 - Static Object Lateral Position Prediction Error in Azimuth Angle
Data Source Average Prediction Error Note
Azimuth Error in
Figure B- 22
0.025 rad ( 1.17 deg)
Averaged at time points
t= 161, 162, 163, 164, 165;
The object is placed 3 m from
the center of the bus path
Azimuth Error in
Figure B- 23
0.0163 rad ( 0.93 deg)
Averaged at time points:
t= 156.4,156.8,157.2,157.6,158.
0;
The calculation of the parameters is based on several randomly selected time points as
noted in the table. Data analysis showed that the azimuth error is sufficient small for
identifying targets within the vehicle path. This error may still be larger than desired for
estimating the lateral position of roadside targets that are located at the boundary of the
vehicle path. The test results show that the LIDAR sensor has difficulties to distinguish
the door opening situation for vehicles parked immediately near the path of the bus. This
could be attributed to the yaw motion and vibration of the subject vehicle ( the bus) and
the azimuth resolution of the LIDAR, which was designed for a less demanding
application. Future improvements could be investigated by using a video camera to assist
the radar or LIDAR to detect the target. A video sensor could potentially provide better
knowledge of the target location relative to the vehicle path. Although video cameras
may also be subject to disturbances, fusion of the vision and LIDAR/ radar could help to
achieve robust performance.
The preliminary requirements developed under the previous phase of the project specified
that the maximum detectable side- looking angle from the front bus corners should be at
least 30 degrees and the maximum lateral position should be at least 6 m ( 20 ft). The
LIDAR tested satisfies these requirements. However, the accuracy requirements were
not yet given in the previous phase. Because urban operating conditions are complex, it
is recommended that further quantitative tests be conducted under a variety of conditions
in order to determine the correlation between the accuracy requirements for azimuth
angle or lateral position measurements and the overall system performance.
27
Figure 2- 4 - Parked cars on both sides, with all car doors closed
Figure 2- 5 - Parked cars on both sides, with one door open
2.2.2.1.3 Verification Test of Time Delay Associated with Obstacle
Detection Sensors
Time delays exist in a variety of processes, including sensor detection, prediction,
tracking and warning generation. Delays for sensor detection are mainly contributed by
28
the physical properties of the sensor detection principle and the front end processing
algorithm, which is specific to sensor design. Additional time delays can be generated
through target parameter prediction and can be FCW algorithm dependent. These time
delays can introduce difficulties for threat assessment
This verification test is to quantify time delays associated with the sensor and the
processing of the target tracking algorithm. The target vehicle was driven with sinusoidal
speed variations, with maximum speeds of 10 mph, 15 mph, and 20 mph. The frequency
of the sine wave was between 0.1 ~ 0.5 Hz and the magnitude of the variation was as
large as 40% of the maximum speed. The sinusoidal speed profile would not be
encountered directly in normal urban driving, but this scenario was chosen based on the
following considerations:
• Urban bus driving typically involves many alternations between accelerator and
brake pedals. The sinusoidal speed profile is an approximation to these speed
variations;
• It was hoped that, by using a sinusoidal speed profile, the maximum and
minimum speed points could be identified in order to measure the phase shift
between true speed trajectory and predicted speed trajectory. Such a phase shift
would be a strong indication of time delay.
• The following cases are easy for prediction: constant speed ( zero acceleration)
and constant acceleration / deceleration, which are impossible to achieve in
practice. The challenging cases, which need to be tested, are variable
accelerations.
The bus followed the target vehicle at a reasonable distance, with a variation within the
range of the string pot. The time delays were to be identified from the phase shift
between recorded ( from on- vehicle sensor), detected ( raw data), estimated and predicted
distance/ speed/ acceleration. Due to the difficulty of using other analytical methods for
data analysis, some representative points are selected for peak and valley points as well
as points on up/ down slopes. It is expected that those selected points can represent most
speed change situations.
In the data analysis as shown in Appendix B, overall time delay is composed of two
parts: sensor internal measurement delay and target parameter prediction delay. The
results are shown in Table 2- 3.
Table 2- 3 - Time Delay Analysis for Overall System
Sensor internal
measurement
delay
Signal
processing
delay
Combined
delay,
average
Combined delay,
Standard
Deviation
Prediction 0.5 s 0.5 s 1.0 s 0.17 s
It should be noted that the results about delays shown in Table 2- 3 may involve
observation errors. The initial test plan called for the bus to be operated in such a way
that the distance between the bus and the target vehicle would follow a sinusoidal profile.
The bus ranging sensor response time delays would be quantified by the phase shift of the
29
LIDAR sensor outputs with respect to the driver input “ truth” measurements from the
string pot. The difficulty, however, is that the bus driver cannot adjust the distance
between the bus and the target vehicle to precisely follow a sinusoidal profile. Instead of
analyzing the phase shift of the sensor output, the time delay is achieved by manually
selecting comparable time points and calculating the delay values at the selected points.
This selection may not be objective and can create observation errors. Nevertheless, the
measurement magnitude of the delay is still very significant, enough to degrade the
performance of the collision warning system.
In the initial FCWS specification stage, the analysis recommended that the sensor delay
not exceed 0.3 seconds and the overall processing delay not exceed 0.5 seconds. Table
2- 3 shows that that the tested prototype system cannot meet these requirements, in part
because the sensor front- end internal processing delay is about 0.5 seconds instead of 0.3
seconds. The additional 0.5 second delay is likely attributed to the following signal
processing processes. The first time period is from the instant of receiving sensor data to
processing, which is determined by the sensor system update rate. The FCWS sensors
have an update interval of 0.075 seconds. The tracking process takes 3 samples to build
the firm track, which resulted in a 0.225 second delay. Additional procedures such as
transformation to ground coordinates and transformations back also take additional time.
The processing delays may be attributed to the prediction method, which may produce
some over- shoots when the target accelerates or decelerates. Recovery from the over-shoots
can cause additional time delays and errors.
Although the sensor delay is large and the crash tests in the controlled environment
( described in 2.2.3.1) showed that the prototype system would not be effective for
hazards that involve stationary obstacles and a very short detection time, the field testing
data revealed that the FCWS was still effective at warning drivers in most cases. This is
because the transit drivers are trained to drive at large time gaps. Warnings, though later
than desired, can still be received by drivers and reacted upon. In practice, there will be
inherent delays regardless of what type of sensor or warning system is used. The extent
to which drivers can tolerant warning delays in the urban driving environment needs to be
further studied through serious human factors studies. Furthermore, alternative designs
of sensing and signal processing approaches can reduce this delay. Examples of these
approaches include implementing the tracking processing directly from raw data from the
sensor front end and/ or sensor fusion using sensors that can provide additional lane and
target information. However, the unavailability of sensor front end data and project
resource limitations did not allow the project team to investigate these approaches.
2.2.2.1.4 Error characteristics of yaw angle measurements
Steering angle measurements are used in conjunction with obstacle detection and lane
detection to determine whether forward obstacles are within the vehicle path and if they
pose any threat. Steering angle measurements can be achieved through a number of
means, including direct measurements of ground wheel angle using displacement sensors,
measurement of steering wheel angle using a potentiometer, or through indirect
estimation using a gyroscope. The earlier prototype system developed under the FCWS
project used a ground wheel displacement sensor. Tests showed that the ground wheel
30
sensing can achieve a high degree of accuracy when it is well calibrated. However, due
to its contact nature, the displacement sensor is very easy to be out of calibration or
malfunction, therefore a non- contact method was selected for the prototype system tested
under the ICWS project. The gyro readings provide the yaw rate of the bus, and the yaw
angle is obtained by integrating the yaw rate,
The evaluation tests focused on the error characteristics of the gyroscope, which is
typically presented in the form of accumulated error. To test the error accumulation, the
bus was driven in irregular circles as shown in Figure 2- 6 and finished each run by
returning to its initial parked position. Physically, the bus had turned 720 degrees with
respect to the original position and returned to its starting position. Through the circular
driving, the accumulated errors were obtained and potential errors from other sources
were cancelled. The tests were conducted at maximum speeds of 5 mph and 15 mph.
The data analysis in Appendix B shows that after the bus completed a 720 degree turn,
the error in the accumulated gyro yaw angle estimation was within 0.1% ( Figure B- 17
and Figure B- 18) compared to the known accumulated angle change of 720 degrees.
Figure 2- 6 - Verification of Gyro yaw rate accumulation error test
In the initial transit FCWS specifications, we defined that the measurement range of the
front wheel angle should be at least 50 degrees to both right and left, though it is
preferable if all possible front wheel angles are covered. The yaw rate b 􀀁 &
of the bus
should be known to within +/- 1 deg/ sec.
31
The test results show that the gyroscope can provide yaw rate measurements with a
resolution of less than +/- 1 deg/ sec and it can support accurate estimation of the steering
angle beyond the specified range. Tests show that the gyro yaw angle measurement is
adequate for supporting the intended target identification purpose. Furthermore, the
results obtained from this verification test provide a basis for the refinement of the
requirement specifications.
2.2.3 System Testing Under Controlled Environment Scenarios
The scenario- based tests were performed to verify the performance of the FCWS in
several scenarios that are typical of urban bus driving conditions. Several basic scenarios
were identified, including: ( a) vehicle following with static target ( such as parked car) in
adjacent lane, ( b) moving target in adjacent lane, ( c) target vehicle cut- in and cut- out
movements, and ( d) low speed approaching/ crashing to a static object.
2.2.3.1 Vehicle following
Vehicle following, as represented in Figure 2- 7 is one of the primary scenarios in bus
operation. Assessment of the threat posed by a forward moving target vehicle is mainly
determined by the relative distance, relative speed, and in some algorithms, relative
acceleration of the two vehicles. The accuracy of the estimation and prediction of these
parameters is essential. The vehicle following test is designed to focus on the evaluation
of dynamic measurement, estimation and prediction of the lateral position of the target
vehicle and side static targets and longitudinal relative distance, speed and acceleration
between the host vehicle and the target vehicles and side static targets.
The setup involves a target vehicle equipped with fifth wheel, wireless communication
between the host bus and target vehicle, and static targets located to the left and right of
the vehicle path. Because the string pot was not connected, the bus could operate at
much higher speeds, with higher relative speed and larger variations. During the tests,
the target vehicle ran at constant speeds of 5 mph, 10 mph, 27 mph, 40 mph, or 50 mph.
It was up to the bus driver to determine a safe and comfortable inter- vehicle distance
compatible with vehicle speed and relative speed. The moving target vehicle accelerated
or decelerated at rates of 0.2, 0.8, or 1.5 m/ s2 . The maximum relative speed recorded
was 4.6 m/ s or approximately 10 mph.
32
Figure 2- 7 - Vehicle following, without use of string pot
Table 2- 4 showed the average prediction error for side static target converted into
azimuth based on LIDAR measurement of Figure 2.17 – Figure 2.22 in the appendix. It is
noted that, unlike the front moving target, tracking for side static target only lasted for a
shorter period of time. This might be due to the relatively small size of boxes used as
target: a small target at longer distance is more difficult for LIDAR and radar to detect.
Table 2- 4 - Measured errors in forward target lateral estimation
Parameter
Average
prediction error
Standard
deviation
Maximum error
Lateral Azimuth
Error ( RMS)
0.107 rad
( 6.09 deg)
0.168 rad
( 6.96 deg)
0.305 rad
( 17.7 deg)
Table 2- 5 shows the errors in the measurements of the relative speed of the frontal
moving target vehicle, which is calculated based on the data shown in Figure B- 19 and
Figure B- 20.
33
Table 2- 5 - Measured errors in forward target vehicle estimation
Parameter
Average prediction
error
Note
Longitudinal relative
speed
11.3 %
The calculation is derived from the
integration of the relative speed error
over the time interval and averaged
over time on the interval.
The calculation of the average prediction error is derived from the integration of the
relative speed error and averaged over time for the selected data set. The test results
show that the tracking, estimation and prediction algorithms can correctly track all the
moving and static obstacles within a reasonable range. Note that there is always a trade-off
between the prediction error and time delay. The prediction is intended to reduce time
delay but could also induce additional errors, particularly in situations when relative
speed varies. The results show that the estimation and prediction errors for longitudinal
relative distance, relative speed and relative acceleration are of similar magnitude to
those of the measurements obtained from the sensor verification testing. The errors in
these measurement predictions may directly affect the correctness and timeliness of the
warning issuance. However, it is not possible to draw quantitative conclusions about the
impact of the prediction errors on the overall system performance with the limited set of
testing conducted under this project. Further tests and data analysis will be necessary.
Meanwhile, future improvements of measurement accuracy, delay characteristics and
robust warning algorithms will be needed. Recommendations from the project team
include adaptation of sensors that can require shorter track acquisition time or direct
range rate measurement sensors ( such as Doppler radars) and sensor fusion.
2.2.3.2 Detection of moving target in adjacent lane
Moving targets in adjacent lanes are another main cause of false positives, particularly if
the moving vehicle is too close to the bus. It is necessary to understand how well the
obstacle detection sensors and tracking algorithm can distinguish and properly track
moving targets in the adjacent lanes in the field of view of the obstacle detection sensors.
Data analysis was focused on detection, estimation, and prediction of range, relative
speed, relative acceleration and lateral offset of the target vehicle.
In this scenario, a target car was running in the left lane adjacent to the bus path at a fixed
lateral distance. Tests were conducted with the car traveling in the same and opposite
directions as the bus traveled, with no other obstacles along the bus path. The maximum
speeds of the car for test runs were 10 mph and 30 mph. The bus ran at approximately
the same speed as the car, but with slight speed variations ( non- constant) so that there
was moderate relative movement between the two vehicles ( Figure 2- 8 shows the test
setup).
34
Table 2- 6 - Parameter estimation for moving target in adjacent lane
Parameter Prediction error Explanation
Average
azimuth angle
error
6.7 % Time points chosen are: t = 36, 40, 44, 48, 52 s
The calculation in Table 2- 6 is based on the data corresponding to Figure B- 26 in
Appendix B. Note that the error is measured with respect to the center of the target
vehicle. Also noted is the fact that the LIDAR used for the prototype system can directly
provide lateral position of the target. However, we used azimuth angle instead of lateral
position for evaluation because the magnitude of error for lateral measurement is
proportional to the distance of the bus to the target because the sensor measurements are
based on detecting azimuth angle. The azimuth errors are calculated using the lateral and
longitudinal measurements at five time points, which are then averaged.
Target lateral distance is used to discriminate detected non- hazardous objects from
hazardous ones. Under the tested condition, the obstacle detection can correctly
recognize the front target. The error characteristics obtained from this set of tests also
suggest that, when a forward obstacle is placed very close to the vehicle path and is
combined with slight road curvature, it is easy for the obstacle detection algorithm to
misjudge the location of the obstacle at a distance.
35
Figure 2- 8 - Moving target vehicle in adjacent lane
2.2.3.3 Target vehicle cut- in and cut- out movements
Cars cutting in and out in front of a bus is a very common maneuver encountered in
urban and suburban operation. A cut- in vehicle suddenly decelerating may potentially
cause a threat to the bus. It is thus necessary to test if the obstacle detection sensor and
tracking algorithm are capable of detecting and properly tracking the cut- in target. From
an algorithm point of view, quickly building a target track for the cut- in vehicle,
estimating its relative distance, speed and acceleration, and ending the tracking when it
cuts out ( leaving the field of view of the sensor) are critical for enabling correct threat
assessment and warning.
The cutting- in test involved a target vehicle driven in an adjacent lane in the same
direction as the bus at a known lateral distance, at speeds of 10 mph, 20 mph, and 35 mph
for a short period of time before accelerating to overtake the bus. Figure 2- 9 shows the
test scenario. The target vehicle then moves out of the bus path as shown in Figure 2- 10.
The speed of the target vehicle varied, and the bus driver had to decide the appropriate
inter- vehicle distance for car following. The test was set up to evaluate whether tracking
can be established as soon as the target vehicle cut in, if tracking continues while the
target vehicle is lane changing on both sides, and whether the tracking ends at an
appropriate time.
36
Figure 2- 9 - Cut- in to test lateral movement detection
The detection of cut- in and cut- out maneuvers involves detection of vehicles in adjacent
lanes ( left/ right), keeping a tracking record of those vehicles, and measuring and
predicting their behavior based on previous and current information. Data analysis in the
Appendix shows that track building starts when the inter- vehicle distance is about 5 m,
while the target vehicle is still completely in the left lane early in the cut- in maneuver.
The target track is dropped at about 7 m inter- vehicle distance after the target vehicle has
completely moved out to the right lane for the cut- out maneuver. This detection is quite
effective, fully tracking the cut- in and cut- out motions of the target vehicle. This is
consistent with the LIDAR lateral measurement characteristics in the results obtained
from the testing described in the sensor verification section.
37
Figure 2- 10 - Cut- out to test lateral movement detection
In order to discriminate non- hazardous vehicles from hazardous vehicles, it is necessary
to have reasonably good lateral position or azimuth measurements and predictions. The
test results show that the LIDAR sensor and obstacle detection algorithm are adequate for
tracking the behavior of lane changing vehicles in adjacent lanes. The system is able to
build up a tracking record when the vehicle is in the field of view of the sensor and to
keep a record of the movement of the detected vehicle until it disappears from the field of
view.
2.2.3.4 Low speed approaching/