Operations Research at Yellowstone National Park

by Sam Alessi, Catherine Plowman, William Burkhardt, Larry Nolan and Bob Jones

Yellowstone National Park had expressed a need for improved project management and worker safety capability due to high incident rates and increasingly complex project environment. The operations research study goals were to identify the underlying problem from both a procedural and a cultural perspective. A method entitled Safety Capability Maturity Modelling was created to appraise both management and safety practices as well as gain an understanding of the underlying work culture. Results identified specific work practice areas to be improved including safety planning and procedures, employee involvement, and corrective actions procedures. A significant cultural issue, reprisal for stopping unsafe work, was also identified. An intervention was introduced to the park to supplant the negative workplace reprisal stories with examples of supervisor praise for putting safety first. This operations research is on-going and hopefully will help the park improve its work efficiency and safety performance.

-oo0oo-

Introduction

Yellowstone National Park (YNP or ‘the park’) is close neighbour to the Idaho National Engineering and Environmental Laboratory (Figure 1) and both have found it beneficial to partner in areas of mutual interest. Through this collaborative relationship, the park became interested in learning more about the INEEL’s capability to manage multiple, complex systems and infrastructure (SE, 2002). A focus for our collaboration became evident during our initial discussions with park leadership about needs. The park leadership indicated that safety and organisational improvement were key issues.

Figure 1: Location and proximity of Yellowstone National Park (YNP) and the Idaho National
Engineering and Environmental Laboratory (INEEL)

Safety was the number one issue since fatalities, high incident rates, and Occupational Safety and Health Administration (OSHA) citations have drawn attention to employee safety from a national level. Recent efforts to improve safety were useful, though improvement was not as quick as desired and not uniform among divisions and work groups. Nationally, the Risk Management Division of the US National Park Service commissioned the development of a social science agenda to help address the National Park Service safety issues. The plan (Machlis and Tuler, 1998) highlighted the need for a science-based understanding of the underlying cultural or 'social factors' that affect safety within the National Park Service (NPS). It listed contributing factors of higher than normal incident rates, increasingly complex work tasks, evolving regulations, organisational changes and limited budgets, resources, and staff. 'NPS accidents occur in activities that result from the unique combination of organisational culture, work environment, and job requirements. Unlike many employees of other bureaus and agencies, NPS employees have more interactions with the public, and must balance preservation of natural resources with visitor enjoyment. Such interaction can create unique work requirements.' (Machlis and Tuler, 1998, page 3). Furthermore, the report states that the 'total system' must be included so that the interacting influences noted above can be assessed.

Other needs also were considered highly important, such as highway construction, make/buy decisions, construction management, snow ploughing, rehabilitation of Mammoth buildings/logistics etc. Overall, 47 major issues were identified, many of which are interrelated. YNP is indeed composed of many complex interrelated systems that present a formidable systems management and operations research challenge.

The objective of this work was to ultimately provide systemic interventions at the park that would stimulate an improved safety climate. First, we needed to understand the problem, especially the attitudes of employees toward safety. From this understanding we can bring about interventions that act to improve the work climate. Additionally, we wanted to look at safety and management practices simultaneously and formulate interventions that would likewise improve both safety and project management capability (Alessi et al, 2000; Plowman and Alessi, 2000).

Strategy

A strategy was formed, in association with Mr Nelson Siler the YNP safety officer, to address the need for enhancing both the park's systems management capability and its safety performance. The main objectives of this operations research were:

1. Understanding employee attitudes and beliefs about safety
2. Understand how safety is integrated with the general management practices
3. Uncover barriers to improving safety
4. Propose approaches to removing barriers

We chose to modify an existing approach to organisational improvement that could baseline both management and safety practices. The approach is entitled Capability Maturity Modelling (CMM) (CMMI, 2000; EIA, 1998; FAA, 1997). To address objective 2, we chose to create a new approach that we have entitled Safety Capability Maturity Modelling (SCMM).

The term 'capability' refers to the ability to perform work on a scale of increasing organisational 'maturity'. The approach compares a consensus standard 'model' of work practices against local work practices. This is done initially by using an employee questionnaire. Analysis of the responses pinpoints problematic areas and issues within the organisation. Interviews with leadership and focused discussions with staff then augment questionnaire results by probing deeper into these issues. The method provides information on the quality of management practices in addition to employee attitudes and beliefs.

Methodology

Capability Maturity Modelling (CMM) and appraisal is used internationally by Defence contractors to evaluate and improve their organisational, management, and technical performance. The approach uses an extensive normative questionnaire to establish the organisation's opinion of the 'maturity' of its existing work practices. Next, employee interviews validate the information obtained from the questionnaire and reveal employees' underlying beliefs and attitudes. From this analysis, an organisation is scored relative to its current 'capability' and 'maturity' to perform work systematically, efficiently, and effectively. This process is efficient since it is generally carried out within one week and involves only 10 hrs of individual staff time.

Based on any revealed areas of need, interventions are developed and implemented to improve the organisation’s maturity. This evaluation and improvement process is conducted every 2 to 4 years so that improvement can be measured and continuous improvement institutionalised.

We selected the Systems Engineering Capability Model and the associated Appraisal Method (EIA, 1998) for use at YNP because the park was interested in how the systems engineering approach could help them manage complex park systems in a coordinated and sustainable manner. This maturity model assesses the most pertinent work areas important to project management and technical integration. The model is also easily tailored to new work environments and is publicly available.

We adapted this model for park safety management by using the original three general categories: Management, Technical, and Environment, and adding the following safety-related focus areas: Ensure Safety, Demonstrate Leadership, and Communicate Effectively. Additionally, a new category entitled Park Safety was added. The safety information was obtained from Mr Nelson Siler and was based on his accumulated experience as a National Park Service Safety Officer. Mr Nelson Siler also advised us on tailoring the checklist to the park environment. He indicated that some project management focus areas would be unfamiliar to park staff and management, since they are not practiced. We decided that rather than remove these questions we would include them to provide a baseline, and learning experience, of standard project management practices.

An organisation can be thought to perform work at varying levels of ‘capability’ and ‘maturity’ in each of the theme areas. This concept of maturity level is defined as follows:

Level 5 Optimising --- Process improvements are planned and implemented
Level 4 Measured --- Process is measured
Level 3 Defined --- Processes are documented and articulated
Level 2 Managed --- Projects are managed to a plan
Level 1 Performed --- Work performed informally

Achieving a given level requires achieving maturity at the next lower level. As a rule, reaching level 5 maturity takes time and commitment such that an organisation should plan to take approximately 2 years to increase one level or 10 years to progress from level 1 to level 5 (SEI, 2000).

The EIA-731 appraisal involves three phases: preassessment, on-site, post-assessment. Pre-assessment consists mainly of eliciting management support and on-site logistics. The on-site phase has eight main steps entitled:

1. Administer questionnaire
2. Analyse questionnaire
3. Develop exploratory interview questions
4. Conduct interviews and focus groups
5. Analyze exploratory data
6. Formulate findings
7. Member check findings
8. Develop intervention report

The on-site phase is conducted within one week to provide rapid feedback and minimise impact to the project. Post-assessment involves management briefings and preparation of an intervention plan.

On-site, an appraisal team consisting of three to five people work full time to facilitate the appraisal process and analyse the information. Approximately five to ten hours of individual participant time is required during the week. A questionnaire consisting of over 600 questions is administered to a sample obtained from three personnel levels within the organisation or project -- department managers, project leaders and workforce practitioners. We improved this delivery by simplifying question wording (Dustin, 2000) and automating the questionnaire delivery (Graden and Nipper, 2000). The questionnaire is organised in a nested hierarchical arrangement consisting of general categories, focus areas, and themes. Maturity levels are included in all themes in a crossed type of classification.

Also, the questions or items are posed differently than a standard sample survey approach (Hinkin, 1995). The standard approach presents a statement like 'My project is exceptionally adept at requirements management' and is scored on a 1-n scale, such as the popular n=5 Likert scale. CMM items instead ask if a certain practice is being preformed within the organisation or project. Respondents are given choices of, yes, no, don't know, or not applicable. The general form of the maturity questions and their number in EIA-731 is described in Table 1.

Table 1: EIA-731 Maturity framework, question general form and number of questions

The team then formulates interview and focus group questions that target the problematic areas identified by the questionnaire. Using these questions as a starting point, the team probes into the claims, concerns, and issues made by employees during interviews. These workplace 'stories' obtained from employees are rich in information about the culture of the organisation. We have also improved this qualitative portion of the CMM by integrating more advanced human systems inquiry approaches (Alessi and Mayhew, 1995; Mayhew and Alessi, 1994).

Team members take notes during these interviews and transcribe these onto note cards. The note cards are then combined based on their similarity and each stack is given a descriptive name. The named stacks of employee statements, both positive and negative, become the findings of the appraisal. The findings are discussed with the employees to check for agreement and passed along to an employee team that will build the action plan. The approach is designed to provide feedback quickly to the organisation and to keep employee involvement to a minimum.

The appraisal is conducted within one week and determines main organisational needs. Based on an initial questionnaire, problematic work practices are identified and interview questions are formulated. Employee comments made during interviews and focus groups are organised into findings that make up the results of the appraisal.

Table 2: Number of employees participating in the safety appraisal by Division

Results

The capability maturity appraisal was conducted the week of February 26th through March 2nd 2001 at YNP. Participants included 7 Division Chiefs, 13 Supervisors, and 23 Staff. Here the designation ‘supervisor’ included employees who performed both supervisory duties and staff functions. Table 2 shows the number of participants and their respective percentages within the appraisal group and relative to all employees within the work Division. We attempted to sample in proportions similar to the actual division population percents but logistics and the small size of some divisions complicated this effort.

Seventy nine percent of the participants work in the Mammoth District within the park, while the remaining 21% work among the Canyon Village, Grant/West Thumb, Lake/Fishing Bridge, Madison, and Tower/Roosevelt Districts. Thirty percent of the participants were female and 26% are involved in a formal YNP safety group.

Seventy two percent of employees participating said that they had been involved in a near miss during their career at YNP. Fifty six percent claim to have been involved in an accident of which 37% had personal injury and 7% had lost more than 3 days due to injury. These rates are consistent with OSHA reportable rate for 1991 to 1996 of 7.5 injuries per 100 permanent employees.

Figure 2 shows the percent ‘yes’ and ‘no’ responses from the 527 item questionnaire delivered to the participants on the first day of the appraisal. Curves are also presented for each supervisory level by maturity level. ‘Yes’ responses decrease linearly for each supervisory level. Approximately, a 25% difference in response was observed between the chiefs and staff and a 12% difference with supervisors. This pattern is identical from level 1 to 4. The pattern shows two characteristics of capability maturity questionnaire data that are normally observed. First, the decreasing ‘yes’ response suggests that the higher maturity questions represent a more refined work environment that is increasingly more difficult to achieve. Second, the greater proportion of ‘yes’ responses are positively correlated with supervisory levels since managers know more and tend to be more generous than staff about the quality of the organisation. At level 5, the chiefs increase their ‘yes’ responses while supervisors and staff remain constant. This can likely be interpreted as indicating the interest YNP senior management has in improving its efficiency and effectiveness, so they answer ‘yes’ to more questions that are about process improvement efforts within the organisation.

Figure 2: Percent ‘yes’ and ‘no’ responses by maturity level and supervisory level from the 527 item
CMM questionnaire.

‘No’ responses (Figure 2) ranged from 20 to 30% and were similar among supervisory levels. The greatest difference was seen between the level 1 and level 5 questions. Staff said ‘no’ more often at the lower level and chiefs had a higher ‘no’ response at the higher maturity levels.

‘Not applicable’ responses were low (1 to 4%) for all supervisory levels, which indicates that employees felt that the work practice areas are relevant for YNP. ‘Don’t know’ responses increase with maturity level, are highest for staff, and ranged from 12 to 56%. Since the ‘Don’t know’ response essentially makes up the difference between 100% and the sum of ‘yes’ and ‘no’ responses, Figure 2 can be used to gain an understanding of the patterns among ‘Don't Know’ responses.

Typically, after an appraisal is completed an organisation is given a maturity level score. This score is a symbol within an industry group that the organisation has achieved systematic, efficient, effective, and in this case, safe work practices. In this way, CMM maturity achievement is similar to the more familiar Baldrige Quality Award or ISO-9000 certification. To give YNP an overall score, we have set an arbitrary proportion of ‘yes,’ ‘no,’ ‘don't know’, and ‘not applicable’ for achieving any maturity level and use a new approach to determining CMM level statistically (Alessi, 2002). The approach uses a simple proportions test where the goal proportions are predetermined. For the park we chose null hypothesis test proportions of< 50% ‘yes’, > 20% ‘no’, and > 15% ‘don't know’ and ‘not applicable’ at all levels within the organisation. Maturity is achieved by rejecting the null. Based on these criteria, the park, overall, has not achieved any of the capability maturity levels since the proportion of staff ‘yes’ responses are too low and did not produce a significant proportion test result. The highest proportion for staff responses was 44% ‘yes’ at level 1 (Table 3).

Table 3: Safety capability maturity appraisal questionnaire percent responses for three supervisory
levels, four response levels, and five maturity levels. Percent response is presented above the
number of responses.

Work Practice Employee Disagreement

A very useful analysis of the CMM questionnaire data is to look at differences in opinions on individual questions that occurred between supervisory levels, ie chiefs, supervisors and staff. In an organisation that is mature, capable, and working efficiently, we would expect that people throughout the organisation would reasonably agree when asked if specific tasks are being performed. Disagreement indicates potential problem areas that should be investigated further.

The following discussion looks at SCMM questions that showed maximal disagreement across the largest distance (ie chiefs to staff) among supervisory level. Fifteen questions (Table 4) received 100% ‘yes’ responses from chiefs and less than 40% ‘yes’ from staff. The areas of safety planning, safety procedures, and employee involvement received ~ 10% ‘yes’ from staff and supervisors. Other areas of strong disagreement included planning solutions, training, and correcting unsafe work. It is informative to simply read the questions displayed in Table 4 an note the strong degree of disagreement between chiefs and other employees. The table is ranked by greatest disagreement between chiefs and staff. Columns of % less values when subtracted from 100% will give the percent ‘yes’ response that was obtained by supervisors and staff.

Table 4: Safety capability appraisal themes and question responses where chiefs were in total
agreement (100% yes) and staff had the largest disagreement with chiefs (chief-staff % difference).

The comparison of individual items can be performed in numerous ways. In fact, it is very useful for the appraisal teams to pose questions to the questionnaire data, view sorted lists like that presented in Table 4 and discuss the implications of these responses. For example, areas where chiefs responded in 100% agreement and supervisors disagreed most strongly included: employee involvement, planning and documentation, corrective action and training commensurate with responsibility. Other areas that chiefs said are not practiced include positive employee interaction and communication. Chiefs also state that project integration does not occur among park systems and projects. Finally, chiefs, supervisors, and staff all agree that there is no incentive for employees to report near misses.

Another interesting finding from this information is that while chiefs state that planning is not consistent with funds and there are not adequate resources for safety, both supervisors and staff responded that funds and resources were adequate. This is interesting since it is well known that the park does not have adequate resources. We interpreted this response by staff to indicate how effective chiefs have been at managing the scarce resources that are available.

Interview and Focus Group Results

The areas of agreement and disagreement within the questionnaire results are used to help the appraisal team create interview and focus group questions. It was decided to focus the questioning in the areas of employee communications, employee perception of safety and stop work, and how YNP manages work risk.

Employees told us that most risks are managed by experience and often in a reactionary manner. They felt that there has been improvement in the past few years but they are concerned that the recent loss of safety leadership will result in losing the progress that has been gained.

In the areas of communications and employee involvement, employees felt that information flow is sporadic or not occurring, though they feel that communication on safety issues has been effective and useful. We also found evidence of ‘pay back’ for stopping unsafe work. This reprisal for acting to prevent an accident is unfortunate since many safety programs are based on open communication. This issue should become the highest priority if YNP is serious about improving its safety performance.

Intervention

The appraisal findings were briefed to the appraisal participants and to others within the park. Volunteer employee teams also began to work on ways to solve some of the issues identified. Over the next year, progress continued, but was slowed due to a number of senior management changes. This section presents one particular intervention directed specifically at the cultural issue of reprisal. The intervention also demonstrates how active, dynamic involvement is as important to change as are the formal plans and procedures in which we typically place our greatest emphasis for organisational improvement.

The principle author was invited to present the findings to different groups within the park. As part of one particular meeting the division chief got up and asked his division whether they had a problem with reprisal. I then facilitated a discussion about this issue that resulted in specific ideas for change, but also acted as a intervention in itself, through the types of questions asked and opportunity to discuss this sensitive issue in an employee meeting.

Interestingly, the employees stated that, though they had never seen any evidence of reprisal in their division, that they had heard ‘stories’ of it occurring. Furthermore, this tended to make them concerned and timid about raising a concern or stopping unsafe work. Even without personal evidence, these employees were influenced by workplace ‘stories’ that contributed to defining the work culture of Yellowstone.

A useful inquiry technique, especially when a group is mired in a negative, problem saturated dialogue, is to ask about the opposite. I tried this and asked whether there was any time then someone stopped work or raised a serious safety concern and not only was there not reprisal but praise. After a moment, a woman spoke up and described a situation where her work group had a concern about possible avalanche but since it was early in the season, and there was little snow, they thought they might get teased that they were making an excuse to not do the job. Nonetheless, they decided to stop the job and were subsequently praised by their supervisor for putting safety first.

Here we have an example of the opposite of reprisal. To make this an organisational wide intervention these type of personal human stories need to be spread throughout the park. I next introduced an approach used at the INEEL to communicate safety information and suggested that more of these type of positive safe behavior stories be included on a regular basis. The approach is entitled ‘The Daily Constitutional’ and is a one-page flyer that is posted in bathroom stalls. Personal safety stories, recall notices, and other safety information are effectively communicated to employees in this manner.

Summary

Operations research was conducted at Yellowstone National Park to improve management and safety capability. An approach entitled Safety Capability Maturity Modelling (SCMM) was created by adding safety categories and questions to the existing Electronics Industries Alliance 731 interim standard.

The approach identified a number of organizational weaknesses which have allowed the park to target its interventions in specific areas. Additionally, cultural issues were identified the most significant of which there is evidence of reprisal for concerns or stopping unsafe work.

A specific intervention targeted at reprisal was introduced. The intervention attempt supplanted the reprisal ‘stories’ with stop work stories that resulted in positive management feedback.

For the interested reader

• Alessi R S, J Johnesee, C Plowman, and N Siler. 2000. Extending EIA-731 Systems Engineering Capability Model Appraisal Methods for Safety and Tailoring the Method to Yellowstone National Park. In E Arnold, K Blanshan, L Brezinski, J Jakoubek, C Jones, P Kar, C Kowalski, K Rawlinson, D Walden and S Wolf (eds), A Decade of Progress… A New Century of Opportunity. Tenth Annual International Symposium Proceedings of the International Council on Systems Engineering (INCOSE). July 16-20. Minneapolis, MN.
• Alessi Sam, 2002, Simple Statistics for use with Capability Maturity Models, Systems Engineering, 5(3).
• Alessi R S and Mayhew M E, 1995, Teaming techniques derived from human systems theory: Neutrality, hypothesizing and circularity, p377-384. In B. Schoening and B. Wittig (eds.) Systems engineering in the global market place, Fifth annual international symposium proceedings of the National Council on Systems Engineering (NCOSE). July 23-26, St. Louis, MO.
• CMMI 2000, Capability Maturity Model Integration, Carnegie Mellon University URL: http://www.sei.cmu.edu/cmmi/cmmi.html. Last modified: 15 September 2000.
• Dustin A L and C Graden, 2000, Tailoring the EIA/IS-731.2 Questionnaire, p623-628. In E Arnold, K Blanshan, L Brezinski, J Jakoubek, C Jones, P Kar, C Kowalski, K Rawlinson, D Walden and S Wolf (eds). A Decade of Progress…A New Century of Opportunity. Tenth Annual International Symposium Proceedings of the International Council on Systems Engineering (INCOSE), July 16-20, Minneapolis, MN.
• EIA, 1998, Systems Engineering Capability Model (SECM), EIA/IS 731, Electronic Industries Alliance (EIA). Government Electronics and Information Technology Association. URL:http://www.geia.org/eoc/G47/. Last modified 31 March 1999.
• FAA, 1997, The FAA - Integrated Capability Maturity Model SM. Federal Aviation Administration. URL:http://www.faa.gov/aio/iCMM/FAA-iCMMx.htm. Last modified: June 1, 1999.
• Graden C and D Nipper, 2000, An Innovative Adaptation of the EA/IS 731.2 Systems Engineering Capability Model Appraisal Method, p629-634. In E Arnold, K Blanshan, L Brezinski, J Jakoubek, C Jones, P Kar, C Kowalski, K Rawlinson, D Walden and S Wolf (eds). A Decade of Progress…A New Century of Opportunity. Tenth Annual International Symposium Proceedings of the International Council on Systems Engineering (INCOSE), July 16-20, Minneapolis, MN.
• Hinkin T R, 1995, A review of scale development practices in the study of organizations, Journal of Management, 21(5), 967-988.
• Machlis Gary E and Seth P Tuler, 1998, A Social Science Plan for Employee Safety in the National Park Service.
• Mayhew, Mick and R Sam Alessi, 1994, Responsive Constructivism and Systems Engineering Education. Systems Engineering: A Competitive Edge in a Changing World. Proc. Fourth Ann. Int. Sym. of The National Council On Systems Engineering (NCOSE). August 10-12, San Jose, CA.
• Plowman, Catherine and Sam Alessi, 2002, Safety Systems Intervention Plan for Yellowstone National Park. INCOSE. SEI. 2000, Capability Maturity Models, Carnegie Mellon University. URL:http://www.sei.cmu.edu/cmm/cmms/cmms.html. Last Modified: 10 August 2000.

SAM ALESSI is a senior engineering with the Idaho National Engineering and Environmental Laboratory in Idaho, USA. He also directs the University of Idaho Systems Engineering Graduate program and teaches classes in systems requirements and design, ethics, and human inquiry and intervention. He is past director of the International Council On Systems Engineering and has participated in the development of early capability maturity models, code of ethics for systems engineering, and certification of systems engineers. Sam's technical work has included non-linear modelling of porous media fate and transport, information systems development, stakeholder involvement and organisational improvement. Dr R Sam Alessi: alesrs@inel.gov

First published to members of the Operational Research Society in OR Insight January- March 2003