University of California Berkeley Phone: ( 510) 642- 4522
2105 Bancroft Way, Suite 300 Fax: ( 510) 642- 0910
Berkeley, CA 94720- 3830 http:// www. calccit. org
CALIFORNIA CENTER FOR INNOVATIVE TRANSPORTATION
CCIT Task Order 11
Systems Engineering
Final Report – December 2006
Prepared by:
California Center for Innovative Transportation
For:
California Department of Transportation
Division of Transportation Planning
ii
Project Fact Sheet
Title: Systems Engineering
Sponsor: Caltrans Division of Transportation Planning
Office of Policy Analysis and Research
Executing Organization: California Center for Innovative Transportation
2105 Bancroft Way
Berkeley, CA 94720
Phone: ( 510) 642- 4522 – Fax: ( 510) 642- 0910
Execution Period: 6/ 1/ 2004 – 6/ 30/ 2006
Contract Amount: $ 224,980
Principal Investigator: Dr. Martin Wachs
UC Berkeley
Center Director: Dr. Hamed Benouar
Director, CCIT
Project Manager: Erik Alm, AICP
Senior Development Engineer, CCIT
Administrative Officer: Anne Crowe
Assistant Director, CCIT
iii
Acknowledgements
The project team thanks the following people who assisted with the project by
committing their time and knowledge, particularly the Consultant Team ( Mike Krueger –
ASE Associates LLP and Ron Ice – Ron Ice & Associates) and the Project Management
Team:
Caltrans
Reza Navai, HQ Planning
Bill Tournay, HQ Planning
Darlene Tigner, HQ Planning
Fred Dial, HQ Operations
Randy Woolley, HQ DRI
Mohammed Al- Kadri, HQ DRI
Doug Kempster, HQ IT
CCIT
Hamed Benouar, Director
Angela Sugihara, Student Assistant
FHWA
Frank Cechini
We also thank the Technical Advisory Working Group and the Steering Committee for
their vision and support, whose members are listed in Appendix 4. Lastly, many thanks
to the CCIT staff whose help is fully appreciated every day.
iv
Report Content
The California Center for Innovative Transportation ( CCIT) under the University of
California, Berkeley ( UCB) led this Systems Engineering Evaluation (“ Evaluation”)
project under a contract with Caltrans Division of Transportation Planning ( CCIT Task
Order 11).
This report is organized as a collection of stand- alone documents. The project summary
outlines the project steps and results; the rest of the documents are deliverables developed
as part of the project: the Work Plan developed for Task 1, Data Collection Review for
Task 2, and the Final Systems Engineering Evaluation for ITS Report for Tasks 3- 5. All
were delivered to the project sponsor, Caltrans’ Division of Transportation Planning.
Appendix 4 lists the contacts for this Systems Engineering Evaluation, including the
Steering Committee, Technical Working Group and Project Management Team.
1
Project Summary
Why was this research undertaken?
The purpose of TO 11 was to implement Systems Engineering best practices and
capabilities for Intelligent Transportation Systems ( ITS) projects within Caltrans and
integrate these best practices and capabilities within the existing Project Development
process. Implementation of Systems Engineering practices for ITS projects is required
by the Federal Highway Administration ( FHWA) Final Rule on ITS ( 23 CFR 940) and
the related Federal Transportation Administration ( FTA) Final Policy.
What was done as part of this research?
This purpose of this research would be accomplished by documenting best practices in
systems engineering for ITS projects and evaluate the benefit of the recommended
Systems Engineering processes for ITS to other system development processes within the
department. During the course of the TO 11 Evaluation project, the project team
conducted extensive communication and coordination with significant stakeholders at all
levels, including thirteen Divisions, three Districts, key offices, and individual experts.
Over 100 key project development documents were reviewed as well. Findings from the
project were presented during followup meetings ( including Caltrans Executive
Management). The result was the “ Systems Engineering Evaluation for ITS Projects”
report released June 2006.
What can be concluded?
The Evaluation’s key observations were:
• For capital development projects, the principles of systems engineering have been
practiced for many years and the maturity of these practices is very high.
• For information technology projects, the application of systems engineering is still
developing and processes are currently being documented.
2
• For ITS projects, the application of systems engineering is in the very beginning
phases.
The Evaluation’s findings and recommendations received significant attention from
stakeholders. As a result, the concept and importance of Systems Engineering has gained
attention.
What do researchers recommend?
The Evaluation included recommendations that addressed different aspects of Systems
Engineering process improvement. The recommendations were divided into 3 sections:
1) Systems engineering capability enhancements and process improvements
2) Organizational enhancements to create an environment for systems engineering to
work
3) Integration of system engineering activities between traditional capital projects
and ITS projects
Process Improvements and Capability Enhancements
• Review, refine, and adopt a systems engineering life cycle model.
• Establish Department policies and a documented systems engineering process for
ITS project development, building on existing processes and the systems
engineering lifecycle model. Implement the documented systems engineering
processes in a phased approach that leverages the IT process development effort
whenever possible. The processes should be initially piloted in selected projects
and lessons learned should be used to improve the documented processes, prior to
full- scale implementation across the organization.
• Establish an ITS Academy that will include systems engineering and ITS project
management elements.
Organizational Enhancements
• Establish policies and key initial management processes including configuration
management, risk management, systems engineering management planning, etc.
Utilize existing PMI- based policies and processes already established by the
Project Management Division.
• Establish a systems engineering organization that has a focus on ITS processes
across the Department.
• Establish an ITS technology center – a forum for the discussion of technologies
with ITS practitioners from around the state.
• Establish an on- line systems engineering repository that includes systems
engineering directives, best practices, templates, processes, guidebooks, case
3
studies and tools ( such as requirement management, modeling, and decision
support tools).
Integration Activities
• Extend the existing project development team concept to form integrated teams
for ITS projects at the beginning of Phase 0, and include Information Technology,
Maintenance, ITS and Systems Engineering expertise.
• Perform Phases 0- 2 of ITS project development as part of the capital development
process, funded through the STIP/ SHOPP.
• Use the artifacts that are developed in phases 0- 2 to develop the Feasibility Study
Report ( FSR) for ITS applications. Coordinate an agreed- to format with
Department of Finance ( DOF) that supports DOF oversight requirements using
artifacts that are natural by- products of the Caltrans process.
• Involve FHWA ITS staff in the Concept phase and then at the decision gates for
phases 0- 3.
In addition to these recommendations, a broader audience of Caltrans staff needs to be
made aware of how Systems Engineering Evaluation recommendations will affect their
work, as well as how new System Engineering processes should be applied to their unit’s
business practices.
Implementation Strategies
The evaluation results, systems engineering model, and recommendations should be
broadly reviewed and revised to reflect the vision of the Department. The agreed to plan
should be reviewed and coordinated with FHWA and other agencies ( such as DOF) to
ensure that all parties support the plan.
Contacts
Project Manager: Erik Alm, AICP ( ealm@ calccit. org)
California Center for Innovative Transportation
University of California at Berkeley
2105 Bancroft Way, Berkeley, CA 94720- 3830
Telephone: ( 510) 642- 3150
Project Monitor: Darlene Tigner ( Darlene_ tigner@ dot. ca. gov)
Caltrans, Division of Transportation Planning
Office of Policy Analysis and Research
1120 N Street, Sacramento, CA 95814
Telephone: ( 916) 653- 9255
Appendixes
1. Final Work Plan
2. Data Collection Review
3. SEA Final Report
4. Team Contacts
PLAN OF WORK
SYSTEMS ENGINEERING EVALUATION FOR
ITS
UNIVERSITY OF CALIFORNIA
SUBAGREEMENT SA4645
CALIFORNIA DEPARTMENT OF TRANSPORTATION
INTERAGENCY AGREEMENT 51A0255, TASK ORDER 11
MARCH 20, 2005
C A L I F O R N I A C E N T E R F O R I N N O VA T I V E T R A N S P O R T A T I O N
I N C O L L A B O R A T I O N W I T H
A S E C O N S U L T I N G L L C A N D
R . C . I C E A N D A S S O C I A T E S
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
i
PLAN OF WORK
SYSTEMS ENGINEERING EVALUATION FOR ITS
INTRODUCTION ............................................................................................................................... .... 1
Project Objectives ............................................................................................................................... ........................... 1
Task Overview....................................................................................................................... ......................................... 2
Project Organization................................................................................................................... ................................... 5
Deliverable Reviews........................................................................................................................ ............................... 7
Schedule ............................................................................................................................... ............................................ 9
TASK 1: DEVELOP WORK PLAN .......................................................................................................... 12
Task 1.1: Define Goals and Objectives ..................................................................................................................... 12
Task 1.2: Develop Work Plan ............................................................................................................................... ..... 14
TASK 2: CAPTURE BEST PRACTICES ................................................................................................. 15
Task 2.1: Capture Caltrans Project Development Practices................................................................................... 16
Task 2.2: Capture Industry Standard Best Practices ................................................................................................ 18
TASK 3: EVALUATION..................................................................................................................... .... 20
Task 3.1: Identify Gaps between Current and Needed Processes Capabilities ................................................... 20
Task 3.2: Recommend Processes and Capabilities................................................................................................... 21
TASK 4: DEVELOP SYSTEMS ENGINEERING MODEL ................................................................. 22
Task 4.1: Define Tailored Systems Engineering Model .......................................................................................... 22
TASK 5: DEVELOP ROADMAP............................................................................................................. 24
Task 5.1: Define Roadmap for including SE Processes and Capabilities into Existing Caltrans Processes... 24
PROJECT DELIVERABLES .................................................................................................................. 26
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 1
INTRODUCTION
This work plan defines the tasks that will be performed to support the Systems Engineering Evaluation
project ( Caltrans Contract # 51A0256, Task Order 11, CCIT Subagreement SA4645). All work to be
performed under the CCIT Subagreement, including development of this work plan, is defined in detail
herein.
PROJECT GOALS AND OBJECTIVES
The tasks to be performed for this project support the goals and objectives of the project sponsor. The
broader Caltrans strategic goals and objectives that are supported by this project are:
Strategic Goals and Objectives
Long Term Goal: Implement Systems Engineering best practices and capabilities for Intelligent
Transportation Systems Projects within Caltrans and integrate these best practices and capabilities
within the existing Project Development Process.
Primary Objective: Document and implement best practices in systems engineering for ITS
projects and at a minimum meet the 23 CFR 940 part 11 of the FHWA Final Rule on the
application of the systems engineering analysis for ITS projects.
Secondary Objective: Evaluate the benefit of the recommended Systems Engineering Processes
for ITS to other system development processes within the department.
Detailed project objectives that build on the strategic goals and objectives were identified during the
Preliminary Meeting on January 5, 2005 and documented in the meeting minutes.
Detailed Project Objectives
1. Utilize the best aspects of the existing processes applicable to ITS project development,
without impacting existing non- ITS project development processes.
o The new systems engineering process should capitalize on the best parts of the
existing process and complement the best practices applicable to ITS projects.
o Systems engineering applies specifically to ITS projects, but could be applied to all
systems. Other departments may benefit from some of the new practices of systems
engineering.
2. Review the existing ITS project development process and IT process, identify gaps, and
recommend a best practice process that follows the systems engineering process.
o ITS projects differ from IT projects, both these project development processes
should either follow one process or the other, but not both.
o Not only identify gaps between ITS and IT, but also overlap between departments.
3. Develop a roadmap on how to achieve the implementation of the recommended process,
assess the existing capabilities and resources needed to implement such a process.
o Key is getting Advisory Group to commit to implementation in the field in order to
exhibit capabilities for systems engineering and meet federal requirements.
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 2
TASK OVERVIEW
To support these objectives, five major tasks will be performed as shown in figure 1:
• Task 1: Develop Work Plan
• Task 2: Capture Best Practices
• Task 3: Evaluation
• Task 4: Develop Systems Engineering Model
• Task 5: Develop Roadmap
Task 1
Develop
Work Plan
PPllaann ooff WWoorrkk
Task 4
Develop
Systems
Engineering
Model
Task 2
Capture
Best Practices
Best Practices
• Caltrans
• Industry
Best Practices
• Caltrans
• Industry
Task 3
Evaluation
Strengths &
Weaknesses
• Processes
• Capabilities
Strengths &
Weaknesses
• Processes
• Capabilities Task 5
Develop
Systems Roadmap
Engineering
Model
Systems
Engineering
Model
Implementation
Plan
Implementation
Plan
Scope
Constraints
Matrix of existing and
needed processes
and capabilities
High leverage
processes and capabilities
vs. ease of
Implementation
Figure 1: Task Overview
Task 1 - Defines the tasks, deliverables, and schedule associated with the project, based on a common
understanding of the project objectives and desired outcomes. This task ensures the project team, CCIT,
Caltrans, and other stakeholders ( Advisory, Management and Technical teams) agree on the end goal of
the project and the required tasks to achieve the goal.
Tasks 2 - Identifies and gathers existing and needed best practices and capabilities, the sources initially
will be from the non- ITS e. g. Capital Development best practices, Information Technology department,
System Engineering Guidebook for ITS, Industry Standards such as Capabilities Maturity Model
Integrated ( CMMI), FHWA Final Rule, and others.
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 3
Task 3 - Evaluate the existing best practices, where they are performed within the organization, and the
required best practices and capabilities to perform Systems Engineering for ITS. Then analyze the
strengths and weaknesses of each existing process and capability.
Task 4 - Construct an SE process model and a capabilities model that will best fit Caltrans in the
development of ITS projects. This model may be an evolving model, in that there will be an initial set of
processes and capabilities that will serve Caltrans in the short term and a longer term model that will
integrate more of the organization processes together with recommendations on existing process
improvements within Caltrans.
Task 5 - Develop a roadmap with recommendations on the activities, capabilities, for both the short
term SE Model and the Longer term SE Model for Caltrans.
The tasks are related to the key statements in the project objectives as shown in the following table.
􀀹 Follows Systems Engineering
Process
Recommend Best Practice Process 􀀹 􀀹
Develop Roadmap 􀀹
Identify Gaps 􀀹
Review ITS/ IT Processes 􀀹
Utilize existing processes 􀀹
5
Roadmap
4
Model
3
Evaluate
2
Capture
Tasks
Objectives
The five major tasks each produce one or more deliverables; this work plan is the deliverable associated
with Task 1. Each major task includes one or more subtasks with intermediate deliverables, milestones,
and reviews that are elaborated in the remainder of this work plan.
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 4
PROJECT ORGANIZATION
Figure 2 illustrates the project organization. As shown, the Department of Planning is the Project
Sponsor, acting on behalf of Caltrans. Within Caltrans, a Steering Committe will provide strategic
direction and executive support and a Working Group will provide technical review and support to the
project. The Federal Highway Administration is a key stakeholder in the project, and will advise the
Project Sponsor on the project direction, strategic goals and objectives. CCIT will manage the project
and use a Technical Advisors panel of systems engineering experts to provide technical review and
consultation. The Project Management Team, including representatives from all involved organizations,
will provide the day to day oversight of the project, review all deliverables, and participate in regular
working meetings and milestone reviews.
Caltrans Planning
Reza Navai
Caltrans Planning
Reza Navai
ASE Consulting LLC
Michael E. Krueger
Ice & Associates
Ronald Ice
ASE Consulting LLC
Michael E. Krueger
Ice & Associates
Ronald Ice
CCIT Coordination
Angela Sugihara
CCIT Coordination
Angela Sugihara
Project Administrator
Darlene Tigner
Project Administrator
Darlene Tigner
Steering Committee
( Division Chiefs/ Deputy Level)
Steering Committee
( Division Chiefs/ Deputy Level)
Technical Advisors
( Systems Engineering)
Technical Advisors
( Systems Engineering)
Project Management
Team
Reza Navai
Darlene Tigner
Hamed Benouar
Angela Sugihara
Frank Cechini
Fred Dial
Bill Tournay
Randy Woolley
Mohamed Alkadri
Project Management
Team
Reza Navai
Darlene Tigner
Hamed Benouar
Angela Sugihara
Frank Cechini
Fred Dial
Bill Tournay
Randy Woolley
Mohamed Alkadri
CCIT
Hamed Benouar
CCIT
Hamed Benouar
Project Sponsor
Project Director
Consultant Team
FHWA
Frank Cechini
FHWA
Frank Cechini
Working Group
( Office Chiefs/ Appointees)
Working Group
( Office Chiefs/ Appointees)
Figure 2: Project Organization
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 5
The project will involve the following participants:
Caltrans: Reza Navai will serve as principal point of contact for Caltrans. Darlene Tigner is the project
administrator for this project and will help to coordinate meetings and serve as a focal point for
communications with Caltrans. Two oversight committees will provide strategic direction and technical
input:
Steering Committee: Consists mainly of HQ Division Chiefs and District Planning or Operations
Deputy Directors. Also Deputy Directors of: Planning and Modal Programs, Maintenance and
Operations, Information Technology, Project Delivery, as well as District Directors.
Working Group: Consists of District and Headquarters personnel. Senior or Supervisory staff as
selected by Division Chiefs and Deputy Directors; a representative sample of district personnel
involved with advanced technology planning and development.
FHWA: Frank Cechini will participate in project reviews and review and comment on project
deliverables to ensure that FHWA Rule 940 requirements are adequately addressed by the project
analyses and recommendations.
CCIT: Hamed Benouar is the CCIT project director and will be responsible for reviewing and approving
consultant work and coordinating appropriate university staff participation in the overall project scope of
work. Angela Sugihara will provide daily support to the project, and will coordinate CCIT’s support of
the technical tasks as identified in this work order. Angela will be the CCIT contact point for the
consultant team. She will oversee the work progress and manage the project under the direction of
Hamed Benouar. A group of technical advisors, expert in system engineering, will be selected and used
by CCIT to review the project approach and deliverables.
Consultant Team: Michael Krueger ( ASE Consulting LLC) and Ron Ice ( R. C. Ice and Associates) are
responsible for all technical effort in support of CCIT according to the CCIT Subagreement, including
development of all deliverables and presentation at all project reviews and other meetings as approved by
CCIT and Caltrans. Michael is the project manager for the project. Both Michael and Ron can be
contacted directly by others on the project team.
Michael E. Krueger, ASE Consulting LLC
Phone: ( 949) 706- 9914 and email: michael. krueger@ ase- consult. com
Ron Ice, R. C. Ice and Associates
Phone: ( 714) 692- 0180 and email: ronice@ ronice. com
As the Working Group, Steering Committee, and Technical Advisor group membership is defined, it is
recommended that there be a single point of contact for each of these groups to liaison with the project.
Ideally these points of contact would become part of the Management Team to ensure the highest level
of stakeholder involvement.
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 6
DELIVERABLE REVIEWS
The project work products will require multiple levels of review and comment from the multi- tiered
project organization. This review and comment cycle will promote stakeholder involvement in the
process and provide needed guidance for the project. Figure 3 illustrates the consultant teams
recommended approach to work product review and comment.
Figure 3: Review and comment process
All of the levels of review shown in the figure will not be required for every deliverable. The Project
Management Team will review all of the work products produced by the consultant team including
presentations and initial draft deliverables. When the review is limited to the Project Management Team,
a one ( 1) week turnaround time is recommended. At the discretion of the Management Team, selected
work products will also be made available to the Working Group and Technical Advisors for additional
review. When a work product is reviewed by the Working Group, a minimum of two ( 2) weeks will be
required to accommodate the broader review. Comments will be provided back to the Management
team, who will filter and consolidate the comments before they are provided back to the consultant team
for incorporation. It is envisioned that the major deliverables ( e. g. work plan, evaluation results, SE
model, and final report) may also be reviewed by the Steering Committee, supported by summary
presentations at major project milestones. The involvement of each of the groups in review of the
project work products is identified in the final “ Project Deliverables” section of this work plan.
All work products will be submitted to the CCIT coordinator, who will distribute the work products to
the appropriate groups for review. All review comments will similarly be assembled by the CCIT
coordinator and delivered to the Consulting Team.
Consultant
Team
Work
Products
Management
Team
Review
Working
Group
Review
Advisory
Group
1 week
Review/ Comment
-
Consolidate
Workgroup
and
Advisory Group
comments
Presentations
Draft Products
1 - 2 week
update
1 week
Review
and
Comment
1 week
Review
and
Comment
Technical
Advisors
Steering
Committee
Review
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 7
SCHEDULE
The Systems Engineering Evaluation schedule was extended. The updated schedule information that was
coordinated with CCIT, Caltrans, and the contractor in February 2004 is documented below.
Here are the planned schedule dates for the five major tasks described in this work plan.
Task Start Date End Date
Task 1: Develop Work Plan December 8, 2004 February 9, 2005
Task 2: Capture Best Practices January 5, 2005 May 27, 2005
Task 3: Evaluation May 9, 2005 August 19, 2005
Task 4: Develop Systems Engineering
Model
July 26, 2005 September 30, 2005
Task 5: Develop Roadmap September 7, 2005 December 16, 2005
The major meeting and delivery milestones are identified in the following table.
Meeting Date Purpose
Initial December 15, 2004 Initial meeting. Present preliminary work plan.
Preliminary January 5, 2005 Initial meeting of the project management team. Discuss and
clarify project goals and objectives. Discuss draft work plan.
Present work plan and example products.
Kick- off January 26, 2005 Meet with advisory group and present objectives and work
plan. Concurrence on goals/ objectives/ scope/ approach and
project team roles and responsibilities.
Data Collection March ??, 2005 Meet with Working Group to collect data and capture best
practices.
Best Practices
Review
April 21, 2005 Review interim data collected and draft existing and needed
processes and capabilities
Evaluation
Review
June 8, 2005
July 21, 2005
Present draft strengths and weaknesses
Technical Paper
Model Review August 11, 2005
September 1, 2005
Present Draft Model
Technical Paper
Implementation
Plan Review
October 4, 2005
November 17, 2005
Present Draft Integration Plan
Technical Paper
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 8
ID Task Name
1 Systems Engineering Evaluation
2 Task 1 Work Plan
3 Prepare Draft Work Plan
4 Preliminary Plan Presentation
5 Final work Plan
6 Task 2 Capture Best Practices
7 Collect Best Practices
8 Present Interim data
9 Technical Paper of Best Practices
10 Task 3 Evaluation
11 Current and Needed Process and Capabilities
12 Draft Process and Capabilities Recommendations
13 Present Recommendations
14 Final Technical Paper
15 Task 4 Systems Engineering Model
16 Develop SE Model
17 Present Interim Model
18 Technical Paper on Final Model
19 Task 5 Implementation Plan
20 Develop Implementation Plan
21 Present Draft Plan
22 Technical Paper on Final Plan
1/ 26
3/ 3
12/ 5 12/ 12 12/ 19 12/ 26 1/ 2 1/ 9 1/ 16 1/ 23 1/ 30 2/ 6 2/ 13 2/ 20 2/ 27 3/ 6 3/ 13 3/ 20 3/ 27 4/ 3 4/ 10 4/ 17 4/ 24 5/ 1 5/ 8 5/ 15 5/ 22 5/ 29 6/ 5 6/ 12 6/ 19 6/ 26
cember January February March April May June Ju
Caltrans Systems Engineering Assessment Schedule
Schedule Revised. See previous tables for current schedule information.
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 9
TASK 1: DEVELOP WORK PLAN
In Task 1, the team will develop a shared understanding of the project goals and objectives and define the
tasks to be performed to realize the project goals and objectives. The primary deliverable associated with
Task 1 is this Project Plan of Work.
In Task 1.1, we begin with the project objectives stated in the introduction to the work plan. These
project objectives are based on broader Caltrans strategic goals and objectives that are supported by this
project.
Long Term Goal: Implement Systems Engineering best practices and capabilities for Intelligent
Transportation Systems Projects within Caltrans and integrate these best practices and capabilities
within the existing Project Development Process.
Primary Objective: Document and implement best practices in systems engineering for ITS
projects and at a minimum meet the 23 CFR 940 part 11 of the FHWA Final Rule on the
application of the systems engineering analysis for ITS projects.
Secondary Objective: Evaluate the benefit of the recommended Systems Engineering Processes
for ITS to other system development processes within the department.
Building on these objectives, we define “ desired outcomes” that clearly state what Caltrans wishes to
achieve through this project.
Outcome 1: An impartial assessment of the strengths and weaknesses of existing Caltrans project
development processes, recognizing where existing processes already meet systems engineering
requirements and identifying areas where there is opportunity for systems engineering process and
capability improvement.
Outcome 2: A recommended, tailorable Systems Engineering process for ITS Project Development that
is compatible with Caltrans planning and project development cycles.
TASK 1.1: DEFINE GOALS AND OBJECTIVES
INPUT
Sources of Information
• Sub- agreement SA4645 ( Project Goal, Primary/ Secondary Objectives)
• Meeting Minutes – Preliminary Meeting held 1/ 5/ 2005
PROCESS
Key Activities
• Develop strawman short- term and long- term desired outcomes
• Review with CCIT/ Caltrans
• Revise
OUTPUT
Results from Process
• Consensus objectives, desired outcomes, deliverables and schedule
documented in Project Plan of Work
RESOURCES
People, Tools, Meetings
• Consultant
• Project Review Team
• Review at Kick- off Meeting ( January 26, 2005)
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 10
Outcome 3: A plan to implement the recommended systems engineering process into Caltrans standard
development practices ( Gold Book) and acquire necessary systems engineering capabilities.
The strategic goals and objectives and desired outcomes are documented in this Plan of Work. The
project team will review these project goals and objectives, along with the remainder of the work plan.
Comments will be incorporated until we have defined a consensus set of strategic goals, objectives and
outcomes that will set the direction for the remainder of the project.
In Task 1.2, the sequence of tasks required to meet the project objectives and desired outcomes is
defined. Each task is fully characterized in terms of required subtasks, associated deliverables, supporting
meetings and milestones, roles and responsibilities, and required resources. A draft Work Plan is
reviewed by CCIT and Caltrans personnel, comments are incorporated, and a final version is generated
once all comments have been addressed. This Plan of Work is the principle deliverable of task 1.2.
TASK 1.2: DEVELOP WORK PLAN
INPUT
Sources of Information
• Subagreement SA4645 ( Task Order 11 Task 1 and Task 2 descriptions)
• Project Goals, Objectives, and Desired Outcomes ( Task 1.1)
PROCESS
Key Activities
• Develop draft work plan
o Identify all project tasks and subtasks
o Define process/ approach for each subtask
o Define roles and responsibilities for each subtask
o Define deliverables
o Identify reviews
o Set schedule
• Review work plan
• Revise and submit final work plan
OUTPUT
Results from Process
• Presentation - Viewgraphs
• Consensus plan of work
RESOURCES
People, Tools, Meetings
• Consultant ( Prepare work plan)
• Project Review Team ( Review work plan and provide comments)
• Preliminary Meeting - Deliver Draft
• Kick- off Meeting – Review Draft
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 11
TASK 2: CAPTURE BEST PRACTICES
In Task 2, the team will identify and gather systems engineering best practices and capabilities. Within
Caltrans, sufficient data will be collected to define current ITS project development, capital project
development, and information technology ( IT) project development practices. Broader systems
engineering best practices will be collected from industry standards such as the Capabilities Maturity
Model Integrated ( CMMI), FHWA Final Rule, and others. The output of this task will identify the
existing Caltrans Systems Engineering processes and capabilities and the needed processes and
capabilities based on current systems engineering standards.
The scope of this data collection effort is based on the project planning and development processes that
are covered and the number of organizations that are involved. The data collection effort will collect
relevant process information for ITS projects, information technology ( IT) projects, and Capital projects,
gathering data from the following Caltrans Divisions:
• Transportation Planning
TASK 2.1: CAPTURE CALTRANS PROJECT DEVELOPMENT PRACTICES
INPUT
Sources of Information
• Subagreement SA4645
• Project Work Plan
• Caltrans project planning and development process documentation
• Sample Caltrans ITS Project Documentation
• Caltrans Personnel ( Interviews – HQ and District)
PROCESS
Key Activities
• Gather current Caltrans process documentation
• Gather sample ITS Project Information
• Gather Organizational Roles and Responsibilities in ITS projects
• Conduct interviews to supplement collected documentation
• Present existing processes and capabilities to Caltrans
• Incorporate comments and revise
OUTPUT
Results from Process
• Repository of collected data
• Presentation w/ Viewgraphs - Caltrans State of the Practice
o Summary of project planning and development roles and
responsibilities
o Summary of existing processes and capabilities
o Differences between ITS and IT or Non- ITS ( Capital projects)
- Technical, organizational and funding
RESOURCES
People, Tools, Meetings
• Consultant ( Assemble and analyze data. Create presentation.)
• CCIT Staff ( Support data collection)
• Caltrans ( Provide requested data.)
• Project Management Team ( Validate collected data.)
• Meeting – Present Interim Data Review
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 12
• Traffic Operations
• Transportation Systems Information,
• Information Technology
• Research and Innovation
• Design
• Maintenance
• Project Management
The Caltrans project planning and development processes will be characterized based on analysis of the
information gathered, highlighting aspects of the existing processes that are already consistent with good
systems engineering practice. Documentation review will be supplemented with interviews where
necessary to confirm our characterization of the existing processes for capital projects, IT projects, and
ITS projects.
ITS projects will be compared and contrasted with IT projects and Capital projects. Technical,
organizational, and funding differences between ITS projects and the other two project types will be
identified.
In addition to review of the documented processes, the data collection effort will also collect process and
deliverable documentation from two example ITS projects:
• Northern California Project ( e. g., Headquarters CATMS – To Be Determined)
• Southern California Project ( e. g., District 11 Reversible Lanes Project or another candidate that is
further along in it’s development - To Be Determined)
The project data that is collected will be used to further characterize current Caltrans ITS project
development practices. This is standard practice in a systems engineering assessment where the
documented processes are corroborated by collection and review of project deliverables and other actual
artifacts of the project development process.
A summary of current project development practices will be documented in a “ Caltrans State of the
Practice” presentation. A review of this material will be used to refine the collected data and summary
and verify the data collection results.
All data collected will be logged and made available to project team members. All collected data and
process characterization results will be held in confidence and will not distributed outside the project
team.
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 13
Systems engineering processes and capabilities are defined in several industry standards. In this task,
these standards and other ITS- specific resources will be reviewed and a set of needed systems engineering
processes and capabilities for ITS projects will be identified.
The industry- standard Capabilities Maturity Model Integrated ( CMMI) will be reviewed and the systems
engineering capabilities that apply to Caltrans ITS project planning and development will be identified.
Target maturity levels for each relevant systems engineering capability will be identified, with different
capability levels identified based on the organization’s anticipated role in ITS project development. This
work will build on page 191 of the Systems Engineering Guidebook for ITS, which suggests systems
engineering capability maturity levels for organizations that procure, manage, and develop ITS projects.
The identified capabilities will be compared with the systems engineering requirements identified in
FHWA Rule 940. Capabilities necessary to support the final rule will be highlighted.
The needed systems engineering processes and capabilities will be included in the Caltrans State of the
Practice presentation that will be distributed and presented to the project team. Feedback from the
presentation will be used to refine the needed process and capability information. The revised results of
task 2 will be documented in a “ Caltrans State of the Practice” technical report.
TASK 2.2: CAPTURE INDUSTRY STANDARD BEST PRACTICES
INPUT
Sources of Information
• FHWA Rule 940
• Capabilities Maturity Model Integrated ( CMMI)
• Systems Engineering Guidebook for ITS
PROCESS
Key Activities
• Gather identified standard references
• Identify needed processes and capabilities, tailored for ITS
• Review and revise
OUTPUT
Results from Process
• Presentation w/ viewgraphs – Needed Systems Engineering processes and
capabilities
• Technical Paper ( covering Task 2.1 and 2.2 findings)
RESOURCES
People, Tools Meeting
• Consultant ( Collect and analyze data)
• Project Management Team ( Validate needed processes and capabilities)
• Meeting – Interim Data Review
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 14
TASK 3: EVALUATION
In Task 3, we will evaluate the existing best practices used for Caltrans project development and compare
these with recommended best practices and capabilities to perform Systems Engineering for ITS. The
strengths and weaknesses of each existing process and capability will be documented.
In Task 3.1, the Caltrans project planning and development processes and systems engineering
capabilities are evaluated with respect to the needed systems engineering processes and capabilities for
ITS projects, identified in task 2.2. Both strengths and weaknesses of the current processes will be
documented. Apparent gaps in systems engineering process and capabilities will be identified and gaps
related to FHWA systems engineering requirements will be highlighted.
Preliminary findings will be included in view graphs and presented to the project team. Feedback will be
incorporated and a draft Caltrans Systems Engineering Assessment technical report will be generated that
includes the strengths and weaknesses information from this task and the findings from Task 3.2. The
deliverable will be reviewed and revised based on project team comments.
TASK 3.1: IDENTIFY GAPS BETWEEN CURRENT AND NEEDED PROCESSES AND
CAPABILITIES
INPUT
Sources of Information
• State of the Practice ( from Task 2.1)
• Industry Standard Best Practices ( from Task 2.2)
PROCESS
Key Activities
• Define recommended systems engineering capabilities per organization and
process.
• Identify strengths and weaknesses of state of the practice with respect to
recommended capabilities
• Identify gaps, highlighting issues that must be resolved to meet federal
requirements
OUTPUT
Results from Process
• Presentation w/ Viewgraphs
• Documented “ Strengths and Weaknesses”
RESOURCES
People, Tools, Meetings
• Consultant ( Identify strengths, weaknesses, gaps)
• CCIT Staff ( Support Data Collection)
• Project Review Team ( Validate collected data)
• Meeting – Present draft Compliance Recommendations
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 15
In Task 3.2, process and capability improvements will be defined that address the gaps, if any are
identified in task 3.1. A presentation of preliminary recommendations will be developed, distributed, and
presented to the team. Feedback will be incorporated to ensure that the final process and capability
recommendations are acceptable to the project sponsor.
TASK 3.2: RECOMMEND PROCESSES AND CAPABILITIES
INPUT
Sources of Information
• Identified Gaps ( Task 3.1)
PROCESS
Key Activities
• Identify specific process recommendations to address gaps
• Identify recommended systems engineering capability enhancements
OUTPUT
Results from Process
• Presentation w/ Viewgraphs
• Technical Report – Caltrans Systems Engineering Evaluation
o Documented Process recommendations
o Documented Capability recommendations
RESOURCES
People, Tools, Meetings
• Consultant ( Develop recommendations)
• Project Review Team ( Review recommendations)
• Present Process and Capability Improvement Recommendations
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 16
TASK 4: DEVELOP SYSTEMS ENGINEERING MODEL
In Task 4, we will construct a Systems Engineering process and capabilities model that will best fit
Caltrans in the development of ITS projects. These models may be an evolving model, in that there will
be an initial set of processes and capabilities that will serve Caltrans in the short term and a longer term
model that will integrate more of the organization processes together with recommendations on existing
process improvements within Caltrans.
TASK 4.1: DEFINE TAILORED SYSTEMS ENGINEERING MODEL
INPUT
Sources of Information
• Industry Standards ( e. g., EIA 632, CMMI)
• Systems Engineering Guidebook for ITS
• Process and Capability Recommendations ( Task 3.2)
PROCESS
Key Activities
• Define Caltrans Systems Engineering Model
• Define range of tailoring options for different ITS projects
• Review and revise
OUTPUT
Results from Process
• Presentation w/ Viewgraphs
• Draft Technical Report - “ Caltrans Systems Engineering Model”
• Final Technical Report - “ Caltrans Systems Engineering Model”
RESOURCES
People, Tools Meetings
• Consultant ( Develop model)
• Project Review Team ( Review model)
• Present Interim Model ( April 30, 2005)
• Present Final Model ( May 15, 2005)
A systems engineering process model will be tailored so that it is as consistent as possible with existing
Caltrans project planning and development processes, but incorporates process and capability
recommendations from Task 3.2 for ITS projects. The recommended process and capability
improvements ( if any) will be integrated with the existing Caltrans ITS Project planning and development
process in a way that minimizes impact to existing processes.
The model for systems engineering should be based on industry standards. This systems engineering
process model will use the Vee model presented in the Systems Engineering Guidebook for ITS as a
starting point, which tailors EIA 632 and other industry- standard systems engineering standards
specifically for ITS projects. In this task, the general ITS Systems Engineering model presented in the
Guidebook will be tailored specifically to minimize impact to existing Caltrans ITS project planning and
development processes.
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 17
TASK 5: DEVELOP ROADMAP
In Task 5, we will develop a roadmap with a time- sequenced series of recommendations for process
improvement and systems engineering capability enhancement.
After the Systems engineering model has been completed, a plan for incorporating systems engineering
process improvements and capability enhancements into the Caltrans Project planning and development
processes for ITS will be developed. The plan will be a roadmap, or series of steps, that recognizes
existing processes that already support systems engineering best practices, processes and capabilities that
require change, and new processes and capabilities that may have to be documented and implemented,
based on the results of previous tasks. The recommendations will be sequenced to focus first on specific
changes necessary to meet systems engineering requirements for ITS projects. Subsequent
recommendations will support broader Caltrans objectives. The draft recommendations will be
documented in a Systems Engineering Implementation Plan deliverable which will be reviewed by the
project team, including the Project Management Team, Working Group, and Advisory Group, to verify
that the prioritized set of recommendations is appropriate and acceptable to the project sponsor.
TASK 5.1: DEFINE ROADMAP FOR INCLUDING SE PROCESSES AND CAPABILITIES
INTO EXISTING CALTRANS PROCESSES
INPUT
Sources of Information
• Process and Capability Enhancement Recommendations ( Task 3.2)
• Caltrans Systems Engineering Model ( Task 4.1)
PROCESS
Key Activities
• Identify process recommendations to support broader systems engineering
objectives
• Recommend approaches to enhance systems engineering capabilities
OUTPUT
Results from Process
• Presentation w/ Viewgraphs
• Draft Technical Report - Systems Engineering Implementation Plan
• Final Technical Report - Systems Engineering Implementation Plan
RESOURCES
People, Tools, Meetings
• Consultant ( Develop recommendations)
• Project Review Team ( Review recommendations)
• Present Draft Plan
• Present Final Plan
PLAN OF WORK SYSTEMS ENGINEERING EVALUATION FOR ITS
CCIT/ ASE CONSULTING LLC/ R. C. ICE AND ASSOCIATES PAGE 18
PROJECT DELIVERABLES
The following table identifies all project work products, including presentations ( view graphs) and reports. The
initial due date for every work product and the project group( s) responsible for review are identified for each
work product as follows:
• M – Project Management and Technical Advisors Team Review
• W – Working Group Review
• S – Steering Committee Review
Note that the Advisory Group review may be based on a summary presentation of the deliverable contents in
lieu of review of the actual deliverable, as appropriate.
Review
Deliverable Due Date
M W S
Plan of Work ( Task 1)
Preliminary – Preliminary Plan w/ Viewgraphs
Draft – Draft Plan/ Viewgraphs
Final – Work Plan
January 5, 2005
January 26, 2005
February 9, 2005
􀁺
􀁺
􀁺
􀁺
􀁺
􀁺
Caltrans Project Development Processes and Industry Best
Practices ( Task 2)
Draft – Viewgraphs and Presentation
Technical Paper – Caltrans and Industry Best practices
March 3, 2005
March 24, 2005
􀁺
􀁺
􀁺
􀁺
Evaluation and Process/ Capability Recommendations ( Task 3)
Preliminary - Presentation/ Viewgraphs
Draft – Technical Paper
Draft – Presentation/ Viewgraphs
Final – Technical Paper
March 3, 2005
March 31, 2005
April 7, 2005
April 24, 2005
􀁺
􀁺
􀁺
􀁺
􀁺
􀁺
􀁺
􀁺
Systems Engineering Model ( Task 4)
Draft – Technical Paper - Caltrans Systems Engineering Model
Draft – Viewgraphs and Presentation
Final – Technical Paper - Caltrans Systems Engineering Model
May 12, 2005
May 19, 2005
June 2, 2005
􀁺
􀁺
􀁺
􀁺
􀁺
􀁺
Implementation Plan ( Task 5)
Draft – Implementation Plan
Draft – Viewgraphs and Presentation
Final – Implementation Plan
June 2, 2005
June 9, 2005
June 28, 2005
􀁺
􀁺
􀁺
􀁺
􀁺
􀁺
1
1
ITS Systems Engineering Evaluation for
Caltrans
Data Collection Review
Tasks 2.1 and 2.2
February 7, 2006
2
2
Agenda
• Introduction
– Project approach
– Current status
• Best Practices Findings ( Task 2)
– Repository of collected information and interviews
– Preliminary Observations
• Approach for Evaluation ( Task 3)
– Brief background on CMMI
– Initial Qualitative Assessment
– Next Steps
3
3
Project Approach
Task 2
Capture
Best Practices
Best Practices
• Caltrans
• Industry
Best Practices
• Caltrans
• Industry
Task 3
Evaluation
Strengths &
Weaknesses
• Processes
• Capabilities
Strengths &
Weaknesses
• Processes
• Capabilities
Task 4
Develop
Systems
Engineering
Model
Systems
Engineering
Model
Systems
Engineering
Model
Task 5
Develop
Roadmap
Implementation
Plan
Implementation
Plan
Scope
Constraints
Matrix of existing and
needed processes
and capabilities
High leverage
processes and capabilities
vs. ease of
Implementation
Task 1
Develop
Work Plan
PPllaann ooff WWoorrkk
4
4
Current Status
• Task 2 Data Collection Effort Completed
– Draft Document Delivered
– ( This) Preliminary Findings Presentation on Feb 7
• Expedited Approach to Tasks 3 – 5 Initiated
– Develop Final Report
• Evaluation Results
• Systems Engineering Model
• Roadmap
– Submit Draft for Review
– Submit Final Report
5
5
Best Practices Findings ( Task 2)
Industry Best
Practices
Interviews Documents
Capital
IT Development
ITS
SA Smart
Acquisition
Caltrans Best
Practices
Documents
Hardcopy, softcopy, hyperlinks
Interviews
Standard structured interviews; on- call interviews; follow- up interviews
Primarily exploratory questions targeted at manager and practitioners
Industry Best Practices
Capabilities Maturity Model Integration ( CMMI)
Presentations/ Workshop
Validate data collected and preliminary findings
6
6
Task 2 Data Collected
• Repository of collected data
– Over a 100 documents relating to processes, capabilities,
roles goals, work products and responsibilities.
Example Inventory
Division of Transportation Planning
Overview of Caltrans Transportation Planning Activities
Bible for Transportation Planning Managers
Transportation Planning Academy manual
California Department of Transportation 2005/ 2005 Program Level Action Plan
GoCalifornia Presentation ( Downloaded from the Web)
Intelligent Transportation Systems ( ITS) Mainstreaming Charter
Intelligent Transportation Systems ( ITS) Interdepartmental Coordination
Unleashing Technology's Benefites in California Transportation System
Updated Vee model
7
7
Task 2 Interviews
• Headquarters interviews
• Additional experts
• District interviews
• Compilation of released interview notes provided
Headquarters interviews
Twelve ( 12) Divisions, 1 office, and 5 follow- up interviews
Planning, Design, DRI, TSI, Local Assistance, Construction, Environmental,
IT, PPMD, PM, Maintenance, Traffic Operations, Engineering Services
Additional Experts
Value Analysis, Maintenance
District interviews - Three ( 7, 8, 3) Districts for this project
All twelve ( 12) districts will be interviewed as part of the CATMS Requirements
Project
Already interviewed districts 1, 2, 3, 6, 10
Remaining districts will be interviewed in 2006
Compilation of released Interview Notes provided
8
8
What was gained from the Data Collection
• Documentation
– Documented processes
– Work artifacts
– Roles & responsibilities for project development
– Direct and In- direct evidence on processes
• Interviews
– Undocumented processes
– Organizational issues
– Distinction between Headquarters and District roles
– Gaps and overlaps between Divisions
– Affirmation evidence on processes and capabilities
Context & rationale for requirements on project development process,
articulation & understanding of roles & responsibilities, experts & capabilities,
and management support
9
9
Industry Best Practices
• Capabilities Maturity Model Integrated ( CMMI)
– Identifies practices and processes
– Measures capabilities and maturity
• CMMI Integrates the following disciplines
– Systems engineering
– Software engineering
– Integrated product and process development & project
management
– Acquisition
• Smart Acquisition – Relationship Maturity Stages for
Integrated Product Teams ( IPT)
– Maturity stages with a focus on IPT relationships
SA Smart
Acquisition
10
10
Systems
Engineering
• Systems engineering is an
interdisciplinary approach and
means to enable the
realization of successful
systems
• Capability is the ability and
capacity of an organization to
achieve a desired result.
Capability
Definitions
Systems Engineering – Complete definition
Systems Engineering is an interdisciplinary approach and means to enable the realization of
successful systems. It focuses on defining customer needs and required functionality early in the
development cycle, documenting requirements, and then proceeding with design synthesis and
system validation while considering the complete problem:
Operations
Cost & Schedule
Performance
Training & Support
Test
Disposal
Manufacturing
Systems Engineering integrates all the disciplines and specialty groups into a team effort forming a
structured development process that proceeds from concept to production to operation. Systems
Engineering considers both the business and the technical needs of all customers with the goal of
providing a quality product that meets the user needs.”
( from INCOSE - http:// www. incose. org/ practice/ whatissystemseng. aspx )
Capability – Complete definition
In this context, “ capability” is the ability and capacity of an organization to achieve a desired result.
Systems engineering requires a number of capabilities. For example, the capability to develop
requirements, manage requirements, etc. Different organizations have different levels of capability,
depending on the processes and practices that they have in place, and the ability of the organization
to use and apply those processes and practices.
11
11
General Definition of Process
• A process is a set of practices performed to achieve
a given purpose; it may include tools, methods,
materials, and/ or people.
• While process is often described as a leg of the
process- people- technology triad, it may also be
considered the “ glue” that unifies the other aspects.
CMMI Tutorial
12
12
Everyone realizes the importance of
having a motivated, quality work force
but...
• ... even our finest people
can’t perform at their best
when the process is not
understood or operating
“ at its best.”
PEOPLE
PROCESS
TECHNOLOGY
Major determinants of product
cost, schedule, and quality
Processes Tie People and Technology Together
13
13
Preliminary Observations
14
14
Issues
• ITS is not fully mainstreamed
into the Capital Project
development process
• Disconnect in ITS between field
elements and TMC
• Minimal leverage is currently
realized between Capital, IT,
and ITS
Capital ITS/ IT
Capital
IT
ITS
ITS is not fully mainstreamed into the Capital Project Development Process
Intelligent Transportation Systems – elements are fragmented
Capital project development processes are used for field devices
TMC elements are being merged with IT processes
No leverage is currently realized between Capital, IT, and ITS
15
15
Leveraging Systems Engineering Capability
• Capital development process
is very mature performs
many SE process activities
• ITS development process is
Ad Hoc
• IT development process is
Emerging
Unified
Systems Engineering
Process
Capital Development Process is very Mature
Performs many systems engineering processes
Highly institutionalized
ITS Development Process is Ad Hoc
Performed at the district level
Not institutionalized
No documented development processes in place
IT Development Process is Emerging
Performed at Headquarters level
Partially institutionalized
16
16
Current ITS Projects Responsibilities
Capital
Development ITS IT
Field Equipment
• Controllers
• Communications
• Devices
Centers
• Networks
• Web
• Desktop
& Apps
511…???
TMC WS
( Some Districts)
Application
Software
TM CAD
Etc. Capital tools
support
Administrative
Support
email
Structures
& Highways
Issues
- ITS systems are fragmented – different processes for Field elements and
Transportation management elements
-* Legacy software, small applications – TM CAD, SOCCS,
17
17
Headquarters' and District’s Roles
• Capital development projects
have clear roles &
responsibilities
• IT projects are driven by DOF
policies with primarily HQ
definition
• ITS project definition is at the
district level, deployment
constrained by HQ DOF –
many project are developing
“ under the radar” of DOF
Headquarters
• Sets policy
• Defines processes
• Allocates funding
• Training
• Supports district projects
Districts
• Project implementation
• Project definition
• Operations & Maintenance
• Data collection and reporting
Funding
Policy
Process
Project status
Data reporting
18
18
Quick and initial review of the state of
Systems Engineering within the Department
• Capital Development Projects
– Most CMMI processes are addressed and performed at a high level of
maturity ( Some CMMI processes are not applicable to Capital
development) – COTS
• Intelligent Transportation Projects
– Basic processes are incomplete and/ or are not performed
• Information Technology Projects
– Basic processes are incomplete and/ or are not performed
– Some CMMI processes are documented and are in- place and are
emerging ( SIMM, FSR, Oversight documentation)
19
19
Initial Observations
Example Highlights - Capital Development
• Staff – highly experienced, competent and dedicated
• Strong management support
• Well developed technical processes
• Well documented project management processes
• Process adaptable to wide range of projects and institutional issues such as new
regulations and state legislation
• The staff just knows what to do –– roles and responsibilities are well understood
• Staff at all levels that we observed were well trained
• Strong emphasis on metrics, modeling, decision support
• Processes are well integrated into the organizational culture
20
20
Initial Observations
Example Issues Capital Development
• Formal documented process ends at delivery of project does not include
entire life cycle
• Some key stakeholders are not included early in the planning phase ( e. g.
Environmental, ITS and Maintenance)
• Processes are not continuously updated
• Processes are not formalized at the Division level as they are at the
Department level
– What is done that is visible to the external world tends to be well documented
– Local assistance, Gold book, SIMM, FSR, Project Management Handbooks
– In some cases what is done internal to the Division is not as well documented
Notable exceptions include Overview of Caltrans Planning Activities and Bible
for Caltrans Planning Managers, a number of TSI processes
• Succession plans for key people are not formalized
21
21
Initial Observations
Example Highlights – IT Process for ITS
• Initiative for developing a life cycle process is underway
• Most Districts have excellent relationships with IT
regarding ITS support
• Most Districts roles and responsibilities are clear IT in
TMC support
• Headquarters in process of consolidation that will clarify
the roles and responsibilities
• IT has a good relationship with DOF
22
22
Initial Observations
Example Issues IT Process for ITS
• Using the FSR process – Needs & Requirements are at
risk of having to be revisited when projects start due to
the time lag of the process
• The IT oversight framework is an fairly new process and
has not been tested on ITS projects ??
• Focus of IT process is on COTS software applications,
integration and customization
23
23
Initial Observations
Example Highlights – ITS project development
• Staff – HQ & District level champions, dedicated, Heroes, high pride
of ownership in implementation at minimal cost, very efficient –
bare bones implementation.
• TMC operations are being formalized – TOPS, TMC master plan,
TMS standardization plan, BPR reports, Pool Fund Studies
• District 7 – has develop an informal integrated development and
management team that has worked for them at the District level
and may be used as a model for other districts
• District level management support
• Multiple divisions are involved with ITS projects
24
24
Initial Observations
Example Issues for ITS Project Development
• Virtually no process documentation
• Lack of requirements
• Turf battles and issues
• Roles and Responsibilities are not clearly defined and vary
• Lack of a ITS technology clearing house or technology center for ITS
• Cannot get development project through DOF
• For the most part, districts would rather do ITS for themselves
• Field maintenance lacks the training & resources
• ITS projects a tough sell to DOF
• Lack of a life cycle model for ITS elements ( district comment)
• Funding for ITS Maintenance lags ITS implementation
• Lack of dedicated funding for ITS project development, operations, and
maintenance
• Field Elements implementation through capital development process – the
TMC Elements that are needed for control is implemented through DOF
process
• Issues with local agencies being part of the district network to share
information or control
Virtually no process documentation on ITS project development from Traffic Operations, IT provided
a life cycle tool that since has been abandoned.
Lack of requirements, conops, stakeholder involvement, verification, validation, project planning.
Turf battles and issues ( across divisions and between headquarters’ and districts)
Roles and Responsibilities are not clearly defined
Lack of a ITS technology clearing house or technology center – favorite technology solutions emerge
around the state -
Cannot get development project through DOF – lack of confidence – DOF is requires and IT Project
Manager for ITS projects
For the most part, districts would rather do ITS for themselves – prefer not to have HQ involvement
Field maintenance lacks the training, resources to maintain ITS field elements
ITS projects a tough sell to DOF and to the Department at large compared to Capital development –
Benefits not as obvious.
Lack of a life cycle model for ITS elements ( district comment)
Funding for ITS Maintenance lags ITS implementation
Lack of dedicated funding for ITS project development, operations, and maintenance.
Field Elements implementation through capital development process – the TMC Elements that are
needed for control is implemented through DOF process
Issues with local agencies being part of the district network to share information or control
25
25
Approach for Evaluation ( Task 3)
Industry Best
Practices
Presentations
Compare
SA Smart
Acquisition
Caltrans Best
Practices
Documents
Hardcopy, softcopy, hyperlinks
Interviews
Standard structured interviews; on- call interviews; follow- up interviews
Primarily exploratory questions targeted at manager and practitioners
Industry Best Practices
Capabilities Maturity Model Integration ( CMMI)
Presentations/ Workshop
Validate data collected and preliminary findings
26
26
CMMI: A Brief Introduction
27
27
Scope of Capabilities Maturity Model Integrated
( CMMI)
• Defines a set of integrated capabilities for:
– Systems engineering
– Software engineering
– Integrated Product Development ( IPT) & project management
• Defines two representations of a model of capabilities &
maturity
– Staged – 5 levels
– Continuous – 6 levels
Background: Developed by Software Engineering Institute at Carnage Mellon –
Initially for Software development ( CMM) now extended to Systems & Integrated
Product development.
CMMI is a model with two representations
Staged – Maturity levels 1- 5 ( 1= Initial to 5 = statistically managed &
continuously improved)
Continuous – Capabilities Levels 0- 5 – 0 = incomplete or not performed, to 5
= Quantitatively Managed - uses profile of Specific Processes
28
28
Which representation to use?
Initial
Managed
Defined
Quantitatively
Managed
Optimizing
Staged Continuous
REQM
PP
PMC
SAM
MA
PPQA
CM
-
-
-
CL1 CL2 CL3 CL4 CL5
SP
Generic & Specific Goals
Target Capability Level
Maturity Levels
1
3
2
4
5
29
29
CMMI Staged Representation
Maturity Level 5
OID, CAR
Maturity Level 4
OPP, QPM
Maturity Level 3
REQD, TS, PI, VER,
VAL, OPF, OPD, OT,
IPM, RSKM, DAR
Overview
Introduction
Structure of the Model
Model Terminology
Maturity Levels, Common Features, and Generic Practices
Understanding the Model
Using the Model
Maturity Level 2
REQM, PP, PMC,
SAM, MA, PPQA, CM
Appendixes
CMMI- SE/ SW
Staged
Benefits
• Pre- determined and
proven improvement
path
• Focus is on a set of
process areas for
improvement for
attaining the next
maturity level
• Provides a single
maturity level rating 1- 5
30
30
CMMI Continuous Representation
Engineering
REQM, REQD, TS,
PI, VER, VAL
Project Management
PP, PMC, SAM
IPM, RSKM, QPM
Process Management
OPF, OPD, OT,
OPP, OID
Process Management
PAs
- Goals
- Practices
Support
CM, PPQA, MA,
CAR, DAR
Appendixes
Overview
Introduction
Structure of the Model
Model Terminology
Capability Levels and Generic Model Components
Understanding the Model
Using the Model
CMMI- SE/ SW
Continuous
Benefits
• Visibility of capabilities in
individual process areas
• Freedom to select the
order of improvement
• Allows selecting the order
of improvement that best
meets Caltrans objectives
Recommended for the
Caltrans evaluation
31
31
CMMI Continuous Representation
A process area capability profile may be
represented by a set of points in two dimensions.
– the process dimension
• “ What” you do
– the capability dimension
• “ How well” you do it
Capability
( How well)
Process Area
( What you do)
REQM
PP
PMC
SAM
MA
PPQA
CM
CL1 CL2 CL3 CL4 CL5
32
32
Capabilities a Continuous Representation
• CMMI defines capabilities in 6 levels
– Level 0 – incomplete
– Level 1 – Performed
– Level 2 – Managed
– Level 3 – Defined
– Level 4 – Quantitatively Managed
– Level 5 – Optimizing
• Process area specific
– Based on organizational goals, select the process areas that are
needed and the level of competency
Level 0 – incomplete – Processes that are either not performed or partially
performed.
Level 1 – Performed – Processes satisfies the specific goals of the process
area
Level 2 – Managed – Performed processes which has the basic infrastructure
in place to support the process – they are planned & executed in accordance
with policy
Level 3 – Defined – Managed processes that are tailored from the
organization’s set of standard processes according to tailoring guidelines
Level 4 – Quantitatively Managed - Defined processes that are controlled
using statistical and other quantitative techniques.
Level 5 – Optimizing – Quantitatively Managed Processes that are improved
based on an understanding of the common causes of variation inherent in
the process.
33
33
Requirements Management ( Example)
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Engineering
Project
Management
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Integrated Project Management( IPPD)
Integrated Supplier Management ( SS)
Integrated Teaming ( IPPD)
Risk Management
Quantitative Project Management
Organizational Process Focus
Organizational Process Definition
Organizational Training
Organizational Process Performance
Organizational Innovation and Deployment
Process
Management
Configuration Management
Process and Product Quality Assurance
Measurement and Analysis
Causal Analysis and Resolution
Decision Analysis and Resolution
Organizational Environment for Integration ( IPPD)
Support
Organization of Process Areas
Category Process Area
34
34
Example - Purpose of
Requirements Management
Manage the requirements of the project’s
product & product components.
Requirements Management identifies
inconsistencies between those requirements
and the project’s plans & work products.
35
35
Example Capability Levels
for Requirements Management
• Specific goal of the process Manage Requirements:
Requirements are managed and inconsistencies with the
project plans and work products are identified
• Specific practices
– SP 1.1- 1 - Obtain understanding of requirements
– SP 1.2- 2 - Obtain commitment to requirements ( Level 2+)
– SP 1.3- 1 - Manage changes
– SP 1.4- 2 - Maintain bi- directional traceability ( Level 2+)
– SP 1.5- 1 - Identify inconsistencies between project work and
requirements
36
36
REQM - Capability Levels 1 & 2
Requirements Management
Specific practices ( CL1 - “ base”)
SP1.1- 1: Obtain an Understanding of
Requirements
SP1.3- 1: Manage Requirements Changes
SP1.5- 1: Identify Inconsistencies Between
Project Work and Requirements
Generic practices ( CL1)
GP1.1: Perform Base Practices
Specific practices ( CL2 - “ advanced”)
SP1.2- 2: Obtain Commitment to Requirements
SP1.4- 2: Maintain Bidirectional Traceability of
Requirements
Generic practices ( CL2)
GP2.1: Establish an Organizational Policy
GP2.2: Plan the Process
GP2.3: Provide Resources
GP2.4: Assign Responsibility
GP2.5: Train People
GP2.6: Manage Configurations
GP2.7: Identify and Involve Relevant Stakeholders
GP2.8: Monitor and Control the Process
GP2.9: Objectively Evaluate Adherence
GP2.10: Review Status with Higher Level Management
37
37
REQM - Capability Level 3
Requirements Management
Specific practices ( CL1 & CL2)
SP1.1- 1: Obtain an Understanding of
Requirements
SP1.2- 2: Obtain Commitment to Requirements
SP1.3- 1: Manage Requirements Changes
SP1.4- 2: Maintain Bidirectional Traceability of
Requirements
SP1.5- 1: Identify Inconsistencies Between
Project Work and Requirements
Generic practices ( CL1 & CL2)
GP1.1: Perform Base Practices
GP2.1: Establish an Organizational Policy
GP2.2: Plan the Process
GP2.3: Provide Resources
GP2.4: Assign Responsibility
GP2.5: Train People
GP2.6: Manage Configurations
GP2.7: Identify and Involve Relevant Stakeholders
GP2.8: Monitor and Control the Process
GP2.9: Objectively Evaluate Adherence
GP2.10: Review Status w/ Higher Level Management
Specific practices ( CL3)
All the CL1 & CL2 Specific Practices
Generic practices ( CL3)
All the CL1 & CL2 Generic Practices plus(+):
GP3.1: Establish a Defined Process
GP3.2: Collect Improvement Information
38
38
REQM - Capability Levels 4 & 5
Requirements Management
Specific practices ( CL4)
All the CL1 & CL2 Specific Practices
Generic practices ( CL4)
All the CL1 & CL2 & CL3 Generic Practices plus(+):
GP4.1: Establish Quantitative Objectives for the Process
GP4.2: Stabilize Subprocess Performance
Specific practices ( CL5)
All the CL1 & CL2 Specific Practices
Generic practices ( CL5)
All the CL1 & CL2 & CL3 & CL4 Generic Practices plus(+):
GP5.1: Ensure Continuous Process Improvement
GP5.2: Correct Root Causes of Problems
39
39
Improving a Process Area
No GPs or SPs exist CL0 Not performed, incomplete
GP1.1
CL1 ( base) SPs
GP1.1 through GP2.10
CL1 + CL2* SPs
GP1.1 through GP3.2
CL1+ CL2*+ CL3* SPs
GP1.1 through GP4.2
CL1+ CL2*+ CL3* SPs
GP1.1 through GP5.2
CL1+ CL2*+ CL3* SPs
CL1
Performed Perform the work
CL2
Managed
Adhere to policy, follow documented plans and processes,
apply adequate resources, assign responsibility and
authority, train people, apply CM, monitor, control, and
evaluate process, identify and involve stakeholders,
review with management
CL3
Defined
Project’s process is tailored from organization’s
standard processes, understand process qualitatively,
process contributes to the organizations assets
CL4
Quantitatively
Managed
Measure process performance,
stabilize process, control charts,
deal with causes of special variations
CL5
Optimizing
Defect prevention, proactive improvement,
innovative technology insertion and deployment
* Advanced practices exist only in the Engineering PAs.
40
40
IPT Relationship Maturity Stages
• Integrated Product Team ( IPT) relationship
Excelling Model for others
Resilience to personnel changes, success
spreads beyond key areas
High Performing
Performing Trust & joint goals lead to results
Developing Respect and developing trust
Beginning Them & us
Stage Description SA Smart
Acquisition
The stages relate to the ability to:
1) Involve stakeholders
2) Perform Processes
3) Define the Relationship of who is involved
Example
Beginning – Involve stakeholder – Stakeholders Identified, but reluctance/ low priority given to
make contact with all stakeholder representatives and developing the team
Perform Processes - Ineffective team identification, Requirements limited to the major
reviews, no stakeholder engagement process
Relationship define Them and Us, Stakeholders not engaged at all levels,
Individuals do not know the lines of communication.
Excelling - Involved Stakeholder - Working as one team is a reality, Goals developed from the
outset, Changing needs of stakeholders easily accommodated
Perform Processes - Process collects information about future changes and matches
with opportunities, Manages open dialog with all stakeholders
Relationship defined - Whole team is involved, everybody is aware of what is going
on, partnership ethos ( guiding principle, distinguishing characteristic)
41
41
Summary
• CMMI models were developed with broad
participation and review.
• Process areas identify “ what you do.”
• Capability levels identify “ how well you do it.”
• Relationship stages of maturity – “ how well the
team works together”
42
42
Initial Qualitative Evaluation
and Next Steps
43
43
State of Practice is Systems Engineering within the Department
Requirements Management
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Engineering Project
Management
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Integrated Project Management( IPPD)
Integrated Supplier Management ( SS)
Integrated Teaming ( IPPD)
Risk Management
Quantitative Project Management
Organizational Process Focus
Organizational Process Definition
Organizational Training
Organizational Process Performance
Organizational Innovation and Deployment
Process
Management
Configuration Management
Process and Product Quality Assurance
Measurement and Analysis
Causal Analysis and Resolution
Decision Analysis and Resolution
Organizational Environment for Integration
Support
Process Areas Capital Development Information Technology ITS activities FHWA Rule
44
44
Requirements Management
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Engineering Project
Management
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Integrated Project Management( IPPD)
Integrated Supplier Management ( SS)
Integrated Teaming ( IPPD)
Risk Management
Quantitative Project Management
Organizational Process Focus
Organizational Process Definition
Organizational Training
Organizational Process Performance
Organizational Innovation and Deployment
Process
Management
Configuration Management
Process and Product Quality Assurance
Measurement and Analysis
Causal Analysis and Resolution
Decision Analysis and Resolution
Organizational Environment for Integration
Support
Division Roles and Responsibilities
for Capital Development
Process Areas IT DD LA DRI DPM TSI DC TO EA DTP DES DM
For Capital Development a large number of documents have been identified, Gold
book, Red book, Local Assistance are available at the Department level.
45
45
Requirements Management
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Engineering Project
Management
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Integrated Project Management ( IPPD)
Risk Management
Integrated Teaming ( IPPD)
Integrated Supplier Management ( SS)
Quantitative Project Management
Organizational Process Focus
Organizational Process Definition
Organizational Training
Organizational Process Performance
Organizational Innovation and Deployment
Process
Management
Configuration Management
Process and Product Quality Assurance
Measurement and Analysis
Causal Analysis and Resolution
Decision Analysis and Resolution
Organizational Environment for Integration
Support
Division Roles and Responsibilities
for IT Development
Process Areas IT DD LA DRI DPM TSI DC TO EA DOTP DES DM
For IT projects, these are the areas that are currently being addressed – SIMM,
FSR, Oversight document Driven from DOF - One or more activities are being
performed in these process areas, as identified in the documentation. Also, a
complete life cycle is emerging that will be more encompassing is being developed
by IT.
46
46
Requirements Management
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Engineering Project
Management
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Integrated Project Management( IPPD)
Integrated Supplier Management ( SS)
Integrated Teaming ( IPPD)
Risk Management
Quantitative Project Management
Organizational Process Focus
Organizational Process Definition
Organizational Training
Organizational Process Performance
Organizational Innovation and Deployment
Process
Management
Configuration Management
Process and Product Quality Assurance
Measurement and Analysis
Causal Analysis and Resolution
Decision Analysis and Resolution
Organizational Environment for Integration
Support
Division Roles and Responsibilities
for ITS Development
Process Areas IT DD LA DRI DPM TSI DC TO EA DTP DES DM
For ITS , one or more activities are being performed in each of these process areas
by the identified division. Ad hoc documentation has been developed as part of
various procurements and scopes of work, other than the IT and DRI processes ( SE
Guidebook) no formal development process documentation or policies has been
identified.
47
47
Task 3 Evaluation Approach
Next Steps
• Complete Evaluation Matrix for all 25 Process Areas
– Separate Evaluations for ITS, IT, and Capital Development
processes
– The following are examples of the evaluation to be performed.
The data shown is for illustration only – they have not been
verified.
48
48
Capital Development Evaluation Example
Requirements Management
Specific practices ( CL1 - “ base”)
SP1.1- 1: Obtain an Understanding of
Requirements
SP1.3- 1: Manage Requirements Changes
SP1.5- 1: Identify Inconsistencies Between
Project Work and Requirements
Generic practices ( CL1)
GP1.1: Perform Base Practices
Specific practices ( CL2 - “ advanced”)
SP1.2- 2: Obtain Commitment to Requirements
SP1.4- 2: Maintain Bidirectional Traceability of
Requirements
Generic practices ( CL2)
GP2.1: Establish an Organizational Policy
GP2.2: Plan the Process
GP2.3: Provide Resources
GP2.4: Assign Responsibility
GP2.5: Train People
GP2.6: Manage Configurations
GP2.7: Identify and Involve Relevant Stakeholders
GP2.8: Monitor and Control the Process
GP2.9: Objectively Evaluate Adherence
GP2.10: Review Status with Higher Level Management
Incomplete
Excelling
High Performing
Performing
Developing
Beginning
Unknown ( Blank)
Assessment Codes
49
49
IT Evaluation Example
Requirements Management
Specific practices ( CL1 - “ base”)
SP1.1- 1: Obtain an Understanding of
Requirements
SP1.3- 1: Manage Requirements Changes
SP1.5- 1: Identify Inconsistencies Between
Project Work and Requirements
Generic practices ( CL1)
GP1.1: Perform Base Practices
Specific practices ( CL2 - “ advanced”)
SP1.2- 2: Obtain Commitment to Requirements
SP1.4- 2: Maintain Bidirectional Traceability of
Requirements
Generic practices ( CL2)
GP2.1: Establish an Organizational Policy
GP2.2: Plan the Process
GP2.3: Provide Resources
GP2.4: Assign Responsibility
GP2.5: Train People
GP2.6: Manage Configurations
GP2.7: Identify and Involve Relevant Stakeholders
GP2.8: Monitor and Control the Process
GP2.9: Objectively Evaluate Adherence
GP2.10: Review Status with Higher Level Management
Incomplete
Excelling
High Performing
Performing
Developing
Beginning
Unknown ( Blank)
Assessment Codes
50
50
ITS Evaluation Example
Requirements Management
Specific practices ( CL1 - “ base”)
SP1.1- 1: Obtain an Understanding of
Requirements
SP1.3- 1: Manage Requirements Changes
SP1.5- 1: Identify Inconsistencies Between
Project Work and Requirements
Generic practices ( CL1)
GP1.1: Perform Base Practices
Specific practices ( CL2 - “ advanced”)
SP1.2- 2: Obtain Commitment to Requirements
SP1.4- 2: Maintain Bidirectional Traceability of
Requirements
Generic practices ( CL2)
GP2.1: Establish an Organizational Policy
GP2.2: Plan the Process
GP2.3: Provide Resources
GP2.4: Assign Responsibility
GP2.5: Train People
GP2.6: Manage Configurations
GP2.7: Identify and Involve Relevant Stakeholders
GP2.8: Monitor and Control the Process
GP2.9: Objectively Evaluate Adherence
GP2.10: Review Status with Higher Level Management
Incomplete
Excelling
High Performing
Performing
Developing
Beginning
Unknown ( Blank)
Assessment Codes
51
51
Cross- Discipline Comparison
Example
Generic practices ( CL1)
GP1.1: Perform Base Practices
Generic practices ( CL2)
GP2.1: Establish an Organizational Policy
GP2.2: Plan the Process
GP2.3: Provide Resources
GP2.4: Assign Responsibility
GP2.5: Train People
GP2.6: Manage Configurations
GP2.7: Identify and Involve Relevant Stakeholders
GP2.8: Monitor and Control the Process
GP2.9: Objectively Evaluate Adherence
GP2.10: Review Status with Higher Level Management
Capital
Development
IT
ITS
Incomplete
Excelling
High Performing
Performing
Developing
Beginning
Unknown ( Blank)
Assessment Codes
FINAL REPORT
SYSTEMS ENGINEERING EVALUATION FOR
ITS
UNIVERSITY OF CALIFORNIA
SUBAGREEMENT SA4645
CALIFORNIA DEPARTMENT OF TRANSPORTATION
INTERAGENCY AGREEMENT 51A0255, TASK ORDER 11
JUNE 15, 2006
C A L I F O R N I A C E N T E R F O R I N N O VA T I V E T R A N S P O R T A T I O N
I N C O L L A B O R A T I O N W I T H
A S E C O N S U L T I N G L L C A N D
R . C . I C E A N D A S S O C I A T E S
i
Executive Summary
This is the final report for the “ Systems Engineering Evaluation for ITS Projects” study. This
final study report includes evaluation results, a systems engineering model that has been adapted
so it fits with the existing Caltrans project development process, and a series of recommendations
for how to implement the systems engineering approach within Caltrans
The Strategic goal and objectives that guided this study were:
Long Range Goal: Implement Systems Engineering best practices and capabilities for
Intelligent Transportation Systems Projects within Caltrans and integrate these best
practices and capabilities within the existing Project Development Process.
Primary Objective: Document and implement best practices in systems engineering for
ITS projects and at a minimum meet the 23 CFR 940 part 11 of the FHWA Final Rule on
the application of the systems engineering analysis for ITS projects.
Secondary Objective: Evaluate the benefit of the recommended Systems Engineering
Processes for ITS to other system development processes within the department.
Data Collection
The Consultant team obtained and reviewed over 100 documents as part of this effort. In
addition to the Project Development Procedures Manual ( the “ Gold Book”), many key documents
were collected from each division that provided real insight into the Caltrans Project
Development Process. A significant number of the documents are available from the Caltrans
website and many additional documents were identified during the interviews and passed along
by interview participants. The task 2 interim report includes a complete list of the documents that
were collected, sorted by the Division that provided the document.
The Caltrans coordinator did an excellent job of making a range of domain experts from the
department available for interviews. Interviews were conducted with thirteen ( 13) Divisions,
three ( 3) Districts, key offices, individual experts, and five ( 5) follow- up interviews. The
interviews were critical to the data collection effort since they allowed the consultant team to
move beyond the documents and learn first- hand how the project development process really
works within Caltrans. The interviews provided valuable context for the documentation review,
affirming the documented process and providing information on the roles and responsibilities of
each of the divisions at headquarters and in each of the districts. A complete list of interview
participants and a comprehensive set of interview notes is included in the task 2 interim report.
General Observations
The documentation review and expert interviews resulted in a number of key observations on the
application of systems engineering for Capital, IT, and ITS projects:
1) For capital development projects, the principles of systems engineering have been
practiced for many years and the maturity of these practices is very high. The project
development processes are well documented and followed.
2) For information technology projects, the application of systems engineering is still
developing and processes are currently being documented. Due to the relative
immaturity of the process, few completed project artifacts were available for review. The
Department is currently aligning the IT project development process so that it supports
the DOF project oversight requirements defined in the State Acquisition Manual ( SAM).
ii
3) For ITS projects, the application of systems engineering is in the very beginning phases.
There is an awareness of the need for a systems engineering process, but few examples of
good systems engineering practice were identified. There are pockets of excellence and
in general, the quality of people that are involved in project development are excellent.
Currently, systems engineering processes have not been documented specifically for ITS
project developments, but a number of systems engineering practices are being performed
within the department at different levels of the organization.
During the interviews and data collection effort, it became clear that organizational relationships,
both within the department and between the department and other agencies, are critical to
successful ITS project development. Developing productive working relationships were some of
the most formidable challenges identified during the interviews. For Capital development
projects, the key relationships and organization roles and responsibilities have been well
established. For ITS Developments, these relationships should be expanded to include a closer tie
to the Division of Information Technology and the Department of Finance.
Systems Engineering Process Requirements
While the FHWA systems engineering analysis requirements provided the initial impetus for this
study, there are a number of process- related requirements that apply to ITS projects that are
levied by other federal and state agencies, in addition to FHWA. Since ITS projects are so varied,
ranging from field equipment procurement through major system integration and software
development projects, different requirements will apply to different projects. As shown in Figure
5, ITS projects may be subject to requirements associated with the Capital Development process
as well as the IT project development process. ITS projects have been deemed by the Legislature
as IT projects and are subject to the same SAM requirements.
CTC,…
• PID/ PSR
• Environmental
• Etc.
DOF
• FSR
• Project Oversight
Framework
• PIER
FHWA
• 23 CFR 940
SE Requirements
Capital
Development ITS IT
Figure 1: ITS, IT, and Capital Development Process Requirements
The systems engineering process that is developed must be tailorable so that it applies to widely
varied projects that are subject to different requirements. The ultimate objective is a single
tailorable process and set of deliverables that satisfies all of the process- related requirements
levied on the Department for ITS projects ranging from simple signal systems to highly complex
ITS projects like CATMS.
Systems Engineering Model
A life cycle model that integrates the systems engineering approach into the Caltrans project
development process is shown in Figure 12. This model addresses the FHWA 23 CFR 940
systems engineering requirements as well as the process requirements for IT projects that are
iii
levied by the SAM. The model is based on the model developed for the Department in the
Systems Engineering Guidebook for ITS project Version 1.9. This initial model was updated
based on the detailed evaluation of the Caltrans processes that was performed for this project.
Recommendations
Beneficial implementation of the systems engineering approach depicted in Figure 2 requires a
documented process, organizational support, skilled personnel, and appropriate resources and
tools. This report includes a series of recommendations that addresses each of these aspects of
process improvement. The recommendations are divided into 3 sections:
1) Systems engineering capability enhancements and process improvements
2) Organizational enhancements to create an environment for systems engineering to work
3) Integrate the activities of traditional capital projecs and ITS projects together.
These recommendations build on initial steps that the department has already taken in systems
engineering process improvement. For example, the Intelligent Transportation Systems
Interdepartmental Coordination agreement, signed by most divisions and the District 4 director,
provides critical management support and visibility for the recommended process and capability
improvements. It is the foundation for the departmental policy, organizational enhancements and
allocation of responsibilities, process definition and integration, and staff capacity building that
are recommended in this study. The Department has also already implemented systems
engineering training and the Caltrans Systems Engineering Guidebook for ITS.
Process Improvements and Capability Enhancements
The following recommendations were identified to improve the systems engineering process and
capabilities for ITS projects within Caltrans.
1) Review, refine, and adopt a systems engineering life cycle model, beginning with the
recommended systems engineering model identified in Figure 2.
2) Establish Department policies and a documented systems engineering process for ITS
project development, building on existing processes and the systems engineering
lifecycle model. Implement the documented systems engineering processes in a phased
approach that leverages the IT process development effort whenever possible. The
processes should be initially piloted in selected projects and lessons learned should be
used to improve the documented processes, prior to full- scale implementation across the
organization. For example, the CATMS project is being developed using a systems
engineering process. Also, District 7 has established an integrated team approach to ITS
project development.
3) Establish an ITS Academy that will include systems engineering and ITS project
management elements. This has to potential to be a model for other states and could be
premiere center for training ITS and systems engineering and project managers in ITS.
Organizational Enhancements
Organizational enhancements will provide an environment for systems engineering to mature and
to improve over time. The organizational enhancements will address one of the most important
keys to success – management support for system engineering. The organizational
recommendations are:
iv
Figure 2: Caltrans Integrated Systems Engineering Lifecycle Model
v
1) Establish policies and key initial management processes including configuration
management, risk management, systems engineering management planning, etc. Utilize
existing PMI- based policies and processes already established by the Project
Management Division.
2) Establish a systems engineering organization that has a focus on ITS processes across the
Department. This could be a standing committee that represents the existing
organizations that are involved with ITS and has the authority to implement process
change.
3) Establish an ITS technology center – Create a forum for the discussion of technologies
with ITS practitioners from around the state. ( Phase 3) – This would allow the key ITS
technologist to work together and collaborating on ITS technology, standards, and other
key issues.
4) Establish an on- line systems engineering repository that includes systems engineering
directives, best practices, templates, processes, guidebooks, case studies and tools ( such
as requirement management, modeling, and decision support tools). The repository
would be a resource that project managers and project systems engineers could access
and use.( phase 1- 3)
Integration Recommendations
The following initial steps are recommended to integrate systems engineering into the capital
development process. Refer to Figure 2 to see the ITS project phases that are referenced here.
1) Extend the existing project development team concept to form integrated teams for ITS
projects at the beginning of Phase 0, using District 7’ s approach as a starting point.
Include Information Technology, Maintenance, ITS and Systems Engineering expertise.
2) Perform Phases 0- 2 of ITS project development as part of the capital development
process, funded through the STIP/ SHOPP. ITS Field element projects would continue
through the capital development process and the ITS Application development would
proceed until the end of phase 2.
3) Use the artifacts that are developed in phases 0 through 2 to develop the FSR for ITS
applications. Coordinate an agreed- to format with DOF that supports DOF oversight
requirements using artifacts that are natural by- products of the Caltrans process.
4) Involve the FHWA ITS staff in the Concept phase and then at the decision gates for
phases 0- 3.
Note in particular that the FSR would be developed after the studies and analyses in Phases 0- 2
are performed since studies and analyses can be performed without involvement of DOF or
securing funds through the FSR process. Using this approach, the process is under the control of
the Department while the basic systems analyses are performed.
Next Steps
The evaluation results, systems engineering model, and recommendations that are identified in
this final report should be broadly reviewed and revised as necessary to ensure they reflect the
vision of the Department. The agreed to plan should be reviewed and coordinated with FHWA
and other agencies to ensure that all parties support the plan. Many studies have shown the
importance of using a systems engineering approach, but the approach must be implemented so
that it fits with the way the Department does business.
Final Report
SYSTEMS ENGINEERING EVALUATION FOR ITS
1 INTRODUCTION ............................................................................................................................... 2
1.1 THE BENEFITS OF SYSTEMS ENGINEERING .................................................................................... 2
1.2 A FEW DEFINITIONS ...................................................................................................................... 3
1.3 SYSTEMS ENGINEERING PROCESS REQUIREMENTS........................................................................ 7
1.3.1 Federal Highway Administration ............................................................................................. 7
1.3.2 Department Of Finance ............................................................................................................ 9
1.3.3 Caltrans Capital Development Requirements ........................................................................ 13
1.3.4 Objective – Unified Requirements .......................................................................................... 14
1.4 TASK 2 OVERVIEW – DATA COLLECTION AND PRELIMINARY ASSESSMENT ................................ 14
1.4.1 Caltrans Documents ............................................................................................................... 15
1.4.2 Caltrans Interviews................................................................................................................. 15
1.4.3 Industry Best Practices and Standards................................................................................... 16
1.4.4 CMMI Model Selection........................................................................................................... 17
2 EVALUATION..................................................................................................................... ............ 18
2.1 CALTRANS PROCESSES AND ACTIVITIES...................................................................................... 18
2.2 TAILORING CMMI FOR THE CALTRANS EVALUATION................................................................. 19
2.2.1 Which CMMI representation to use?...................................................................................... 20
2.2.2 Process Area Focus for this Evaluation ................................................................................. 20
2.2.3 CMMI Capability Levels......................................................................................................... 21
2.2.4 Evaluation of Organizational Relationships........................................................................... 22
2.3 SYSTEMS ENGINEERING EVALUATION......................................................................................... 23
2.3.1 Project Management............................................................................................................... 24
2.3.2 Support ............................................................................................................................... ... 28
2.3.3 Engineering ............................................................................................................................ 33
2.3.4 Process Management.............................................................................................................. 37
2.4 RELATIONSHIPS ........................................................................................................................... 42
2.4.1 Departmental ( External) Relationships .................................................................................. 43
2.4.2 Divisional ( Internal) Relationships ........................................................................................ 44
3 SYSTEMS ENGINEERING MODEL............................................................................................. 53
3.1 KEY OBSERVATIONS................................................................................................................... 53
3.2 PHASE - 1: TRANSPORTATION PLANNING AND ARCHITECTURE DEVELOPMENT ........................... 54
3.3 PHASE 0: CONCEPT EXPLORATION AND BENEFITS ANALYSIS ...................................................... 54
3.4 PHASE 1: PROJECT PLANNING AND CONCEPT OF OPERATIONS .................................................... 54
3.5 PHASE 2 - SYSTEM LEVEL REQUIREMENTS AND HIGH- LEVEL DESIGN ........................................ 57
3.6 PHASE 2- PREPARE PROCUREMENT AND ADVERTISE PROJECT. ................................................... 59
3.7 PHASE 3 - DETAILED DESIGN AND SYSTEM IMPLEMENTATION.................................................... 59
3.8 PHASE 4- OPERATIONS AND MAINTENANCE ................................................................................ 61
3.9 PHASE 5- RETIREMENT/ REPLACEMENT........................................................................................ 61
3.10 CROSS- CUTTING ACTIVITIES ....................................................................................................... 62
4 ROADMAP FOR IMPLEMENTATION........................................................................................ 65
4.1 CAPABILITY AND PROCESS IMPROVEMENTS................................................................................. 65
4.2 ORGANIZATIONAL RECOMMENDATIONS...................................................................................... 68
4.3 INTEGRATING SYSTEMS ENGINEERING AND CAPITAL DEVELOPMENT PROCESSES ...................... 68
4.3.1 Capital and ITS Lifecycle Comparison................................................................................... 68
4.3.2 Integration Recommendations ................................................................................................ 71
APPENDIX A: LIST OF DOCUMENTS COLLECTED....................................................................... 73
APPENDIX B CAPABILITY MATURITY MODEL INTEGRATION ( CMMI)............................... 77
APPENDIX C – IPT RELATIONSHIP MATURITY STAGES............................................................ 92
APPENDIX D: DEPARTMENT OF FINANCE INTERVIEW NOTES .............................................. 96
1
1 Introduction
This is the final report for the “ Systems Engineering Evaluation for ITS Projects” project.
Through this project, the Department is assessing how well it is performing systems engineering
on Intelligent Transportation Systems Projects. The project supports the overall objectives of the
department:
Project Strategic Goals and Objectives
Long Term Goal: Implement Systems Engineering best practices and capabilities for
Intelligent Transportation Systems Projects within Caltrans and integrate these best
practices and capabilities within the existing Project Development Process.
Primary Objective: Document and implement best practices in systems engineering for
ITS projects and at a minimum meet the 23 CFR 940 part 11 of the FHWA Final Rule on
the application of the systems engineering analysis for ITS projects.
Secondary Objective: Evaluate the benefit of the recommended Systems Engineering
Processes for ITS to other system development processes within the department.
This report builds on the interim task report produced at the conclusion of Task 2. It provides
evaluation results, a systems engineering model that has been adapted so it fits with the existing
Caltrans project development process, and a series of recommendations for how to implement the
systems engineering approach within Caltrans as shown in Figure 7.
Task 2
Capture
Best Practices
Task 2
Interim
Report
Task 2
Interim
Report
Task 3
Evaluation
Strengths &
Weaknesses
• Processes
• Capabilities
Strengths &
Weaknesses
• Processes
• Capabilities
Task 4
Develop
Systems
Engineering
Model
Systems
Engineering
Model
Systems
Engineering
Model
Task 5
Develop
Roadmap
Implementation
Plan
Implementation
Plan
Scope
Constraints
Matrix of existing and
needed processes
and capabilities
High leverage
processes and capabilities
vs. ease of
Implementation
Task 1
Develop
Work Plan
WWoorrkkPPllaann
FINAL
REPORT
Figure 1 Project Approach
2
1.1 The Benefits of Systems Engineering
What every ITS project manager wants is a successful system at the end of the project, where
“ success” is measured by:
1. how well the system satisfies the needs of the people who use it.
2. the cost and schedule performance of the project
The primary benefit of doing systems engineering is that it will reduce the risk of schedule and
cost overruns and increase the likelihood that the system will meet the user’s needs. Other
benefits include:
􀂃 better system documentation
􀂃 higher level of stakeholder participation
􀂃 system functionality that meets stakeholders’ expectations
􀂃 shorter project cycles
􀂃 systems that can evolve with a minimum of redesign and cost
􀂃 higher level of system reuse
􀂃 more predictable outcomes from projects
Many studies have shown the importance of using systems engineering principles. Better systems
engineering has been correlated with shorter project schedules and lower development costs.
Perhaps the most frequently cited report, and one that is based on a broad cross- industry survey of
more than 250,000 IT projects, is the Standish Group Chaos Report ( published in 1994 and 2000).
Other more focused studies have been performed by the International Council of Systems
Engineering ( Eric Honour, “ Understanding the Value of Systems Engineering”, 2004.), and
Boeing ( John D. Vu. “ Software Process Improvement Journey: From Level 1 to Level 5”). IBM
( IBM Commercial Products, Bruce Barker) also did an interesting analysis in 2003 that
determined that incorporating systems engineering practices into their organization – through
organizational changes, process documentation, and training – substantially improved their
project development efficiency. As shown in Figure 3, the IBM study indicated that adopting
systems engineering cut their project costs per “ point” ( a standard complexity measure used at
IBM) in half from 2000 ( no systems engineering) to 2002 ( systems engineering fully
implemented). Unfortunately, we don’t have enough experience with systems engineering for
ITS to have amassed solid quantitative proof for ITS projects - yet.
1999 2001 2002
500
1000
1500
2000
2500
2000
SE organization created
SE Process documented
SE Formal Training Started
Project w/ o SE
Project With SE
Yearly Avg
Cost per “ Point”
Figure 3: Systems Engineering Yields Productivity Improvements at IBM ( © IBM Corp 2003)
3
1.2 A Few Definitions
This final report and, in fact, the entire project, relies on a shared understanding of a few key
terms. In this project, the consultant team will perform an evaluation of the systems engineering
process and capabilities that are used by Caltrans for ITS projects. The definition for Systems
Engineering that we use was developed by the International Council of Systems Engineering
( INCOSE). The definition goes beyond the specific requirements in FHWA Rule 940.11 to
provide a more comprehensive view of systems engineering’s application to systems
development.
What is Systems Engineering?
“ Systems Engineering is an interdisciplinary approach and means to enable the realization of
successful systems. It focuses on defining customer needs and required functionality early in the
development cycle, documenting requirements, then proceeding with design synthesis and system
validation while considering the complete problem:
• Operations
• Cost & Schedule
• Performance
• Training & Support
• Test
• Disposal
• Manufacturing
Systems Engineering integrates all the disciplines and specialty groups into a team effort forming
a structured development process that proceeds from concept to production to operation. Systems
Engineering considers both the business and the technical needs of all customers with the goal of
providing a quality product that meets the user needs.”
( from INCOSE - http:// www. incose. org/ practice/ whatissystemseng. aspx )
What is an ITS Project?
In order to apply systems engineering to ITS projects and satisfy the Code of Federal
Regulations, Chapter 23, Section 940 ( 23 CFR 940), it is important to define “ ITS Project”. 23
CFR 940 defines ITS projects quite broadly:
ITS Project means any project that in whole or in part funds the acquisition of
technologies or systems of technologies that provide or significantly contribute to the
provision of one or more ITS user services as defined in the National ITS Architecture.
This definition encompasses a wide range of projects. Smaller ITS projects might be limited to
purchase and installation of field equipment – controllers, ramp meters, signals, etc. Larger ITS
projects support integration of multiple systems and development of custom software – for
example, CATMS and 511 system developments. These varied ITS projects overlap with
traditional capital development projects and IT projects, as shown in Figure 4.
4
Capital
Development ITS IT
Field Equipment
• Controllers
• Communications
• Devices
Centers
• Networks
• Web
• Desktop
& Apps
511…?
TMC WS
( Some Districts)
Application
Software
TM CAD
Etc. Capital tools
support
Administrative
support
email
Structures
& Highways
Figure 4: ITS, Capital, and IT projects
A traditional capital development project that includes an ITS element such as a traffic signal or a
ramp meter, is an ITS project. This is represented by the overlap between the Capital
Development and ITS projects in the figure. The traditional capital development process is
generally used for these types of projects, but the ITS elements within the project are also subject
to the systems engineering requirements in 23 CFR 940.
Similarly, there is significant overlap between IT projects and ITS projects. The state defines an
IT project in the State Administrative Manual Section 4819.2 as:
A project that encompasses computerized and auxiliary automated information handling,
including systems design and analysis, conversion of data, computer programming,
information storage and retrieval, data transmission, requisite system controls,
simulation, and related interactions between people and machines.
This very broad definition for IT projects could be interpreted to include any information
processing system. In practice, ITS projects that include center application software, computers,
and networks are classified as IT and would be subject to IT ( DOF) processes and requirements.
Systems Engineering and the ITS Architecture
A systems engineering approach for ITS projects requires up- front planning and system definition
so that project requirements are identified and documented, before technology choices are made
and the system is implemented. A regional ITS architecture, required by 23 CFR 940, is a
framework that supports this up- front planning since it allows ITS projects to be viewed in the
broader context of the regional transportation system. Among other benefits, US DOT promotes
development of a regional ITS architecture
because it helps integrate operational
considerations into the strategic planning
process. The best opportunity for using the
regional ITS architecture is early in the systems
engineering process, when the project is
initiated and preliminary engineering is
performed. The architecture is most valuable as
a scoping tool that allows a project to be broadly
defined and shown in a regional context. In
later steps in the systems engineering process,
when the project scope is more firmly
established and the project is defined in
increasing detail, there is less opportunity to use
the high- level definitions included in the
5
regional ITS architecture.
What is Process?
A process is a set of practices performed to achieve a given purpose. It may include tools,
methods, materials, and/ or people. While process is often described as a leg of the process-people-
technology triad, it may also be considered the glue that unifies the technology with
people.
“ The quality of a product is largely determined by the quality of the process that is used to
develop and maintain it.” This quote is based on Total Quality Management ( TQM) principles as
taught by Shewhart, Juran, Deming and Humphrey. ( CMMI V1.1 and Appraisal Tutorial) SEI.
What is Capability?
In this context, “ capability” is the ability and capacity of an organization to achieve a desired
result. Systems engineering requires a number of capabilities. For example, the capability to
develop requirements, manage requirements, etc. Different organizations have different levels of
capability, depending on the processes and practices that they have in place, and the ability of the
organization to use and apply those processes and practices.
1.3 Systems Engineering Process Requirements
There are a number of process- related requirements that apply to ITS projects that are levied by
FHWA and other federal and state agencies. Since ITS projects are so varied, ranging from field
equipment procurement through major system integration and software development projects,
different requirements will apply to different projects. As shown in Figure 5, ITS projects may be
subject to requirements associated with the Capital Development process as well as the IT project
development process.
CTC,…
• PID/ PSR
• Environmental
• Etc.
DOF
• FSR
• Project Oversight
Framework
• PIER
FHWA
• 23 CFR 940
SE Requirements
Capital
Development ITS IT
Figure 5: ITS, IT, and Capital Development Process Requirements
:.
1.3.1 Federal Highway Administration
The FHWA systems engineering requirements for ITS projects are included in 23 CFR 940
( commonly referred to as “ the final rule”). Part 940.11 of this rule defines seven specific
requirements that must be included at a minimum in the systems engineering analysis. Part
940.13 of the rule requires compliance with the systems engineering requirements to be
demonstrated prior to authorization of highway trust funds. This compliance is monitored by the
FHWA division offices under existing federal oversight procedures.
6
Currently, the FHWA oversight of Caltrans ITS projects is handled informally on a project- by-project
basis. The Department would like to be self certified in ITS project development with
respect to FHWA oversight. FHWA would like to work with Caltrans to document a procedure
for FHWA oversight of Caltrans ITS projects based in part on the outcome of this Systems
Engineering evaluation project.
Caltrans Local Assistance Procedures
It is likely that the oversight procedures for Caltrans ITS projects would be similar to the
procedures that FHWA uses to provide oversight for ITS projects that are developed by local
agencies in California. In 2004, the Caltrans Local Assistance Procedures were revised to address
the requirements in 23 CFR 940 for local agency projects. The revised Local Assistance
Procedures include three significant new elements that address the systems engineering
requirements:
1.) Systems Engineering Review Form ( SERF) – Completed at beginning of a project as part
of the Field Review Form, this form summarizes activities needed to meet the
Architecture Rule requirements that all ITS projects must undertake a Systems
Engineering Analysis. The form includes seven questions that exactly correspond to the
seven requirements in 23 CFR 940.11
2.) Systems Engineering Management Plan ( SEMP) – Completed in early phases of system
development, to serve as part of the Project Management Plan, the SEMP identifies the
“ best professional practices” to manage and undertake the technical tasks.
3.) Systems Engineering Process – As defined in the SEMP, this process establishes the
design, implementation, and testing steps necessary to accomplish the system
implementation in a manner that is scaled to the size and risk of the project.
SYSTEMS ENGINEERING REVIEW FORM
This form needs to be filled out for all ITS projects. For all major ITS projects, this completed form needs to be
submitted to FHWA for review and approval prior to PE authorization ( Phase 1 PE authorization).
For all major ITS projects, a Systems Engineering Management Plan ( SEMP), which includes the seven items below,
must be submitted to FHWA for review and approval, prior to PE authorization for final or detailed design ( Phase 2
PE authorization). The 2- phased PE authorization only applies to major ITS projects.
For guidance in filling out the seven items below, see last part of this exhibit.
1. Identification of portions of the Regional ITS Architecture being implemented:
__________________________________________________________________________________________
2. Identification of participating agencies roles and responsibilities:
__________________________________________________________________________________________
3. Requirements definitions:
__________________________________________________________________________________________
4. Analysis of alternative system configurations and technology options to meet requirements:
__________________________________________________________________________________________
5. Procurement options:
__________________________________________________________________________________________
6. Identification of applicable ITS standards and testing procedures:
___________________________________________________________________________________________
7. Procedures and resources necessary for operations and management of the system:
___________________________________________________________________________________________
7
The FSR Process: Perceptions and Reality
Perception: The FSR must be done at the
beginning of project development.
Reality: Caltrans has considerable latitude in
establishing when the FSR is prepared in the
project development process. For example, an
FSR could be generated after the analysis and
design steps in the systems engineering process.
The only limitations are that Caltrans would
have to find a way to fund the pre- FSR work
and the FSR must be submitted and approved
before operational hardware/ software is
developed/ procured.
Perception: DOF will only approve projects
that will yield hard Person Year ( PY) savings.
Reality: Hard PY savings are not required for
project approval; broader economic/ social
benefits are also acceptable justification for a
project. DOF has approved projects based on
projected service to the public.
Perception: DOF requires the Project Manager
to be from the IT Division.
Reality: DOF focuses on qualifications and
expects the project managers to have relevant
experience. For example, the project manager
for a software- intensive project should have
experience in managing software acquisition.
The experienced individual can come from any
division within Caltrans, per DOF. The
Caltrans IT Division is actually levying this
requirement.
Figure 6: SERF Form Supporting 23 CFR 940.11 in California
Currently, local agencies are required to complete the SERF for all ITS projects. The SERF only
requires approval from Caltrans Local Assistance and FHWA for higher risk ITS projects that
include new system functionality ( e. g., ITS projects that include substantial new software
development). The SEMP must also be prepared and submitted to Caltrans Local Assistance and
FHWA for these higher risk projects. The project sponsor ( the local agency) determines whether
the project is a higher risk project that requires the additional oversight based on guidelines in the
Local Assistance Procedures and supporting guidance.
If a similar process were to be used for Caltrans ITS projects, then the “ higher risk” projects that
include additional FHWA oversight would often be the same projects that also are subject to DOF
oversight requirements. One of the key objectives of this evaluation is to identify ways to
minimize the impact of this “ double jeopardy” oversight of the systems engineering processes by
two different agencies.
1.3.2 Department Of Finance
The initial scope of work for this project focused on systems engineering requirements levied by
FHWA, but the Team learned early in the data collection effort that the Department of Finance’s
oversight requirements were actually more of a concern to many ITS project managers. To better
understand the DOF’s requirements and their applicability to ITS projects, the Department of
Finance was interviewed as part of the data collection effort for this project. The notes from this
interview are included in Appendix C. The key DOF requirements as they apply to ITS projects
are briefly described here.
Feasibility Study Report ( FSR)
The State Acquisition Manual ( SAM) requires a
feasibility study to be performed for every IT
project. For projects that exceed Caltrans
departmental budget authority ($ 500k), a FSR must
be prepared and submitted to DOF for approval. An
FSR is required whether the project is funded with
local, state, or federal funds since it is spending
authority, not budgets, which are approved.
The FSR must contain the following information:
1. A description of the business problem or
opportunity the project is intended to
address.
2. The project objectives, i. e., the significant
results that must be achieved for an
alternative to be an effective response to the
problem or opportunity being addressed.
3. A thorough description of the selected
alternative, including the hardware,
software and personnel that will be used.
4. A discussion and economic analysis of each
of the alternatives considered in the
feasibility study that meets the established
objectives and functional requirements, and
the reasons for rejecting the alternatives that
were not selected.
5. A complete description of the information
technology capabilities and the conditions
8
that must exist in order to satisfy each defined objective.