ENGG404 Rubric3 Final Report 2023F v20231121

ENGG404 Final Report Marking Rubric: 1 of 8 2023F Notes: 1) Members of the Instruction Team will refer to the corresponding section in the Team Project Instruction Manual for additional details about the required content of the report. 2) In some sections and chapters, the students may not have exactly complied with the requirements as specified in the Team Project Instruction Manual. This is not necessarily wrong and should not result in deductions. Please review the differences and if an explanation is provided, then assess accordingly. For example, the Business Case Analysis may indic ate to use the “top 2 recommendations” or “1/30 th the value of the PEAP losses”, but the students used only the second recommendation or the total value of the PEAP losses; if an explanation is given as to what they did and/or how they deviated and/or why they deviated from the expected, then this is acceptable. 3) We are looking for rational evaluations and content, and not formatting. Use your consistent judgement in assessing the team project. 4) Recall that a thorough review may not have been done for the Progress Report, that feedback may not be complete, and that some deficiencies may not have been noted. It is the Team’s responsibility to ensure that all requi rements for the entire Final Report are met. 5) For TA’s: In all cases of marking the Final Report, the TA shall strictly adhere to the requirements, the marks awarded, and the deductions assessed as described here, unless there is an approved exception. 6) For Teams: Please direct any questions about the marking on your RCA Chart to the professor that is assigned to your team, and not to the Teaching Assistant. Final Report Focus Areas and Details on Marks to be Awarded: Marks File Name (5 marks): Submit the RCA Chart in PDF file type by uploading to eClass with file name in this format: ENGG404 202yt Team nnn company-location Final Report.pdf where:  “y” completes the year of the academic term  “t” is the term e.g. F or W or S  “nnn” is your team number ; it can be in 1 or 2, or 3 digits e.g. Team 6 , Team 06 , or Team 006 are equally acceptable  “company - location” is a two -word descriptor of your loss incident e.g. Sunrise-Toronto e.g. ENGG404 2020F Team 455 Sunrise-Toronto Final Report.pdf Award 5 marks for a correct file-name, 0 marks for any deviation from this standard. 5 Cover Page and Table of Contents (2 marks total)  The expectation is for a full, complete, and error-free CP and TOC:  Award 2 marks for a CP & TOC in full and complete detail.  Deduct 2 marks if the CP and TOC are incomplete in any manner. 2 Executive Summary: The Executive Summary must NOT exceed 6 pages. Make allowance for a poorly formatted table that causes text to carry-over onto page 7. Each of the following sections should be in the order of ½ to 1 page each, although students may add more content in one section e.g. context and purpose, and less in another section e.g. list of latent causes. The sections in the Executive Summary must be a summary of and consistent with the corresponding appendix. The Marker / Grader should read this sec tion as would an “executive”: ask yourself “does it convey the essential information such that the executive summary makes overall sense and provides some direction to me (the e xecutive)?” ES: Incident Description and Losses (3 marks): Award 3 marks if:  The reader understands what the loss incident was (where, when, major events in the overall loss incident e.g. a gas release followed by an explosion), and  There is a full accounting of the losses i.e. impacts on PEAP.  Deduct 3 marks for any gap in above. 3 ES: Context and Purpose (2 marks): This section must convey the industry overview, and the context and purpose of this team and the incident investigation report. Award marks for details as noted below. Item: Mark Industry:  nature of activity i.e. what they do i.e. production or service, (1 mark)  how they do it (1 mark) Purpose:  Work for ABC, (1 mark);  Asked to investigate to determine latent causes (1 mark);  Will make / have made recommendations (1 mark). 3 Total: 5 Multiply score by 2/5 to obtain score out of 2: 2 2 ES: List of the Latent Causes (2 marks): A table or arranged bullet list of the latent causes aligned to the risk management system elements.  At least eighteen specific latent causes (3 marks):  Organized in the table as shown below or comparable bullet list (2 marks): Deduct 1 mark for the first deficiency; 2 or more marks for more significant deficiencies e.g. the latent causes are expressed in general or broad terms. Award only 2 marks if latent causes are listed and specific, but not categorized. Award only 1 mark if latent causes are listed but not categorized and not specific. 2

ENGG404 Final Report Marking Rubric: 2 of 8 2023F The elements must be in the left column, with at least 18 latent causes listed appropriately in the right column for each of the elements: Element: Latent Causes 2) Risk Assessment and Management of Risks A brief but specific statement of Latent Cause #3 3) Community Awareness and Emergency Preparedness. “No latent causes found” or similar phrasing, or the row is not listed if there are no latent causes. 4) Management of Change. A brief but specific statement of Latent Cause #4 5) Etc. Etc. Then multiply score by 2/5 to obtain score out of 2: ES: Key Recommendations and Leading Indicators: (5 marks): Award 5 marks for the insertion of the table from Chapter 5. Deduct 2 marks for only a word summary or a list of the recommendations and rank. Deduct 1 mark for each missing column, to a maximum of 5 marks. Rank Recommendation Gain Index Effort Index Total Score & Bucket Latent Causes Addressed & RME# Leading Indicators 1 2 Etc. 5 ES: Business Case Analysis (5 marks): Award 3 marks for the insertion of the table from Chapter 6. Minimum required is the partial table as shown here. Full table is acceptable. Top Two Recommendations Total Cost of Improvements Annual Risk Exposure without Improvements Annual Risk Exposure with Improvements Gross Benefit Net Benefit Deduct 1 mark if the content of the table is in a list form (bullets and sub-bullets). Award 2 marks for reasonable and logical discussion on conclusions drawn from this business case analysis. 5 ES: Issues with Implementation and Next Steps (5 marks total) The rationale for the selection of a topical area to explore is not needed, nor is the determination of the drawbacks and pitfalls. This section is a summary of Chapter 7 and Chapter 8.  Is there a discu ssion of the one topical area (e.g. “ long- term culture” ) (1 mark) and its two main challenges for implementation ( e.g. “lack of experience with organizational change, lack of discipline in the organization to use documented procedures” ) of one recommendation? (2 marks)  Is there a discussion about what you want Management to do now (1 mark) and what might happen if management does NOT implement your set of recommendations i.e. what is a probable outcome at ABC’s Operations / Facility? (1 mark) This paragraph should convey a sense of the importance of your recommendations and how confident you are in your recommendations. 5 ES: Deductions  If any content in the Executive Summary describes what the tool or process does or why it was chosen, deduct 1 mark for each occurrence.  For example, if the content described that “A root cause analysis (was selected) as the tool to identify latent causes of the incident) in order to make recommendations (to prevent a similar incident) …”, then deduct 1 mark for this one occurrence.  Apply this deduction in each section of the executive sections that you are marking. varies Chapter 1: Incident Description and Losses: (2 marks total)  A detailed narrative of the sequence of events leading up to the loss incident AND any pertinent subsequent events. (1 marks):  The specific details of the loss incident i.e. when, where, brief explanation as to what happened. (1 marks):  State the losses for PEAP, bullet list or table. All four must be stated, even if “No impact”. (2 marks):  A & P must be stated in quantified terms e.g. dollar or productivity value. (1 marks): Then multiply score by 2/5 to obtain score out of 2: 2 Chapter 2: Industry Overview; Context & Purpose; Scope & Boundaries (5 marks): This chapter consists of 3 sections. 2.1 Industry Overview (score of of 5 * 1/5 = 1 mark total): Does this sub-section answer these questions?  What is the industry? What does it do i.e. production / manufacturing, logistics, services? National, continental, or global reach? (3 marks):  What is the nature of the company’s operation, facility, or activity? (2 marks): 2.2 Context and Purpose (score of 5 * 2/5 = 2 marks total): 5

ENGG404 Final Report Marking Rubric: 4 of 8 2023F second highest, etc. > if not numbered or not properly ranked in order and no explanation, deduct 2 marks. Assess 40 marks:  For each of the 4 key recommendations (10 marks per recommendation, 4 recommendations, 36 marks total), check for the following items:  2 marks: A short description or “title” of this key recommendation; it must be numbered and it must be in actionable terms; for example: “ 1) Develop and implement a Maintenance work program for the overall management of inspections, preventative maintenance, and planned and unplanned repairs for the levy system. ”  1 mark: Longer description of the recommendation with specific deliverable actions or executable actions as sub-bullets. o Deliverable #1 o Deliverable #2 o Etc.  1 mark: Gain Index and rationale for this GI value  1 mark: Effort Index and rationale for this EI value, including scores and descriptions for each of the six sub-factors: o Sub-factor #1 o Sub-factor #2 o Etc.  1 mark: Total Score and “bucket” i.e. “Low Hanging Fruit”:  1 mark: Nature of Fix:  1 mark: List of latent causes addressed  1 mark: Alignment to Element(s):  1 mark: List a Leading Indicator for each recommendation to monitor the effectiveness of your recommendation You will note that the first key recommendation has a total score of 10 marks. A team will have a set of 4 key recommendations, each scored in exactly the same way. If all sets look like this, award 10 marks for each set. If most have the same deficiency (deficiencies), then score each the same way. Note: If a Team ranks a recommendation of a lower score higher than a recommendation with a higher score, then look for an explanation or rationale. Similarly, for recommendations with the same score, look for an explanation or rationale as to why one was ranked higher than the other. No penalty should be assessed if the rationale is given.  If the set of key recommendations (e.g. the short actionable description and the deliverables/executables) sound rational and are clearly aligned to latent causes, then make no further adjustments.  If the set of key recommendations lacks in rationale or are not clearly aligned with latent causes / system elements or predominantly address basic and/or immediuate causes, then deduct 3 marks for each recommendation to a maximum of 12 marks deduction. Assess 5 marks: Award 5 marks for a complete and correct table; no part marks. The Key Recommendations must be presented in a Summary Table. Check that the components for each line of recommendation in the summary table are listed as follows:  Rank # and Score MUST match highest to lowest;  Total Score MUST equal Gain Index times Effort Index;  Recommendations MUST be stated in actionable terms;  Recommendation MUST be aligned to the “latent cause addressed” and the “element”;  Leading Indicator for that recommendation Rank Recommendation Gain Index Effort Index Total Score & Bucket Latent Causes Addressed & RME # Leading Indicator 1 2 3 4 Chapter 5: Business Case Analysis: (30 marks total) Award 15 marks as follows. Then scale-up by 2X factor: Item: Marks Table: Award 6 marks total, 1 mark for the table and 5 marks for content as shown: Top Two Recommendations Total Cost of Improvements 1 mark Annual Risk Exposure without Improvements 1 mark Annual Risk Exposure with Improvements 1 mark Gross Benefit 1 mark Net Benefit 1 mark 6 Award 3 marks for a response to this question: “Referring to the Complex Effort vs. Gain Tool, in which quadrant does this case fall? Explain.” Response includes: Quadrant (1 mark), a discussion point about the effort (1 mark), a discussion point about the gain (1 mark) 3 30

ENGG404 Final Report Marking Rubric: 5 of 8 2023F Award 2 marks for a response to this question: “What conclusion do you draw about this Case? Explain.” Response includes: “ The business case analysis supports / does not support the decision to move forward / proceed with implementing the recommendations (1 mark) because the net benefit is significantly more than the costs / is not significant or is negative .” (1 mark) 2 Award 4 marks for a response to this question: “What other factors must you consider? Recall, that this approach is a preliminary assessment of the Case under study, and is definitely not a sound basis on which to make a final decision.” Look for 4 unique points of discussion matching these seven or other considerations: “ Further work must be done to:  increase the degree of certainty on likelihoods of the loss incident;  improve certainty / accuracy of the cost estimates;  improve certainty / accuracy of the savings estimates;  improve certainty / accuracy of the annual risk exposure values;  consider and quantify other factors which may provide additional benefits;  improve accuracy of the life expectancy of the project; and finally,  consider the time-value of money. ” 4 Then multiply the score by 30/15 = score out of 30: 30 Some teams may note a point “4)”; this point is for explanatory purposes only to inform the marker that the team has deviated from the model above of “top two recommendations”. Examples of acceptable deviation from the above model: The Business Case Analy sis indicates to use the “top 2 recommendations” or “1/30th the value of the PEAP losses”, but the students used only the second recommendation or the total value of the PEAP losses. If an explanation is given as to what they did and/or how they deviated and/or why they deviated from the expected, then this is acceptable and do not deduct marks. No marks awarded for including this point. Chapter 6: Issues with Implementation: (10 marks total) This section is wide open to the discretion of the team. The rationale for the selection of a topical area to explore is not needed, nor is the determination of the drawbacks and pitfalls. Item: Marks Has one of the top recommendations been stated? Check that the recommendation selected is identified in the previous table of recommendations. 2 Has a topical area from Chapter 7 Leadership, Motivation, etc. (e.g. corporate culture, leadership commitment) been stated and briefly described? Note that “cost or expense or logistics” or other non -cultural / non-leadership topic is NOT acceptable. 2 For the topical area selected, has the first of two challenges (hurdles, issues, drawbacks, or pitfalls) been stated (1 mark) and explored (1mark)? Is this challenge relevant (1 mark) – does it make sense in the context of the recommendation? 3 For the topical area selected, has the second of two challenges (hurdles, issues, drawbacks, or pitfalls) been stated (1 mark) and explored (1 mark)? Is this challenge relevant (1 mark) – does it make sense in the context of the recommendation? 3 Total: 10 Example:  a statement of a recommendation (2 marks): ;  a statement and description of the topical area e.g. “long - term culture” (2 marks): ;  first of two challenges, statement, exploration question, and relevancy (3 marks): : “lack of experience with organizational change”;  second of two challenges, statement, exploration question, and relevancy (3 marks): : “lack of discipline in the organization to use documented pr ocedures”. 10 Chapter7: Next Steps: (5 marks total) This section is NOT a summary of the report process, the report findings, the latent causes, or the recommendations i.e. if it summarizes the recommendations and/or describes how the report was generated, the how’s and why’s, then assess 0 marks. This brief discussion must answer these questions: What is it that you want Management to do now? Look for a discussion with these statements or similar:  “ We want management to support the key recommendations as listed in this report. (1 mark)” and continuing with a discussion using any one of the top four recommendations as an example. “Consider this (combined) recommendation as an example: … (1 mark for stating one or a combination of recommen dations)”  How will this recommendation prevent an event similar to the actual loss event (1 mark)?  “What if I don’t?” What might happen if management does NOT implement your set of recommendations i.e. what is a probable outcome at ABC’s Operations / Faci lity? This paragraph should convey a sense of the importance of your recommendations and how confident you are in your recommendations. Look for this statement or similar: “If this set of recommendations is not implemented, then it is only a matter of tim e before a significant loss incident occurs due to these latent causes (1 mark) because … (1 mark for stating the rationale).” 5 References: (3 marks): The expectation is for a set of references. Acceptable reference styles are: IEEE or ASCE, APA, or MLA, with in- text or in-line citation. Award 3 marks outright; for minor deficiencies or incorrect / mixed referencing, deduct 1 mark; deduct two marks for gross deficiencies. 3 Overall: Spelling, Grammar; Rationale and Logic; Format and Organization; Overall Flow (10 marks total) The expectation is for a professional report. Award 10 marks outright Apply deductions as noted:  Spelling and Grammar – deduct 1 mark for each error to a maximum of 10 marks 10

ENGG404 Final Report Marking Rubric: 7 of 8 2023F Appendix D: Bow-Tie Diagram (25 marks total) Award marks as follows for each of the items in the Bow-tie Diagram: Item Description Marks Diagram Layout The general layout of the diagram conforms to the standard layout as shown in the examples. Label the boxes and shapes. Colors and special shapes are not necessary 2 Top Event Statement A simple, concise phrase that conveys the nature of the Top Event. (It should answer: What happens if the hazard becomes uncontrolled? 1 Hazard Statement A simple, concise phrase that conveys the nature of the Hazard. (It should answer: What energy source are you trying to control?) 1 THREATS Each of the threats should independently cause the top event. Deductions will be made if any two of the three threats are correlated to each other. (2 marks deduction). Identification of Threat 1, 2, 3 The phrase should convey the nature of the threat. It should answer: What scenario, condition, or practice could cause the top event?) 1 x 3 = 3 Preventative Control Measures (PCMs) (3 PCMs are REQUIRED for each threat) The phrases should convey the nature of the PCMs identified for each Threat. (Each PCM will be awarded 1 point)  It should answer: What steps, initiators, or actions will prevent the Top Event from happening?  Could include one or more Engineering Controls, Administrative Controls, or a recommendation from any of the latent causes identified in the RCA chart.  The control must be specific to the threat identified. Deduct 1 mark for each PCM that is not specific to the threat identified 3 x 3 = 9 CONSEQUENCES Answers what could happen to PEAP. All three consequences must be identified from the PEAP impacts from Chapter 1 of the team project report. Deduct 2 marks if all the three consequences relate to just one of the PEAP impacts. Identification of Consequence 1, 2, 3  The phrase should convey the nature of the first consequence. (It should answer the question: What are the impacts on PEAP if the top event occurs?) 1 x 3 = 3 Mitigative Control Measures (MCMs) (2 MCMs are REQUIRED for each consequence) The phrases should convey the nature of the MCMs identified for each Consequence. (Each MCM will be awarded 1 point)  It should answer: What steps, initiatives, or action will mitigate the consequences of the top event?  Could include one or more Engineering Controls, Administrative Controls, or a recommendation from any of the latent causes identified in the RCA chart.  The control must be specific to the threat identified. Deduct 1 mark for each PCM that is not specific to the consequence identified. 2 x 3 = 6 Total 25 25 Appendix E: Root Cause Analysis – Chart (70 marks total) Mark the RCA Chart on these components: Marks Award five (5) marks for a complete “Chart Title” box: Team #; Company Name; Name of Loss Incident; Location; date of loss incident; date of RCA Chart. Do NOT list the team members. Example: Chart Title: Team #11: DPC Enterprises; Chlorine Transfer Hose Rupture and Release; Festus, Missouri; 14-Aug-2002; date of RCA Chart: 5-July-2023 5 Award six (6) marks for a table that lists the “RME #’s, their counts, and the total”, and a complete “Legend Box” (for any symbols or abbreviations used on the chart). RME# Count Symbols Abbreviations #2 RAMR 3 EDF = Engineering and Design Factor #4 MOC 2 AND JF = Job Factor #6 PECI 4 PF = Personal Factor Etc. OR Etc. Total 21 Etc. 6 Use different colours to identify the boxes for immediate, basic, and latent causes. 5 Incident Description: Start the RCA Chart with a clearly and succinctly stated Incident Description. A well-stated Incident Description will include: the date, the location (city and province/state/country), the core of the incident (e.g. explosion, or fire, or collapse or breach), and the impacts on PEAP in quantified terms. Imagine you are working, that you have done an extensive investigation, and now you are presenting your findings within your organization. A well-developed RCA should be a standalone document that tells the why & how of the incident, and this starts with the incident description box at the top of the chart. Here are two equally good examples of a well-stated Incident Description (the bold italics text indicates key points of your incident description – you do NOT need to include the bold, italics in your statement): 10 70

ENGG404 Final Report Marking Rubric: 8 of 8 2023F On October 12, 2019 (when) in New Orleans, LA, (where) at the Hard Rock Hotel (company name) , the hotel structure buckled and collapsed (overall nature of the loss incident) . (Impacts on PEAP follow) (P): Three (3) workers were killed and 19 were injured. (E): Immediately following the collapse, a large amount of debris and dust were released into the air impacting the surrounding area. (A): This collapse resulted in the full demolition of the hotel, and 3 of the surrounding buildings, a loss of $450 million in assets and demolition and clean-up costs of $10 million. (P): The hotel was never rebuilt at a loss of $26 million in annual sales revenues. On October 12, 2019 (when) in New Orleans, LA, (where) at the Hard Rock Hotel (company name) , a key structural column within the hotel structure buckled and collapsed (an event leading up to the loss incident) , resulting in the collapse of the entire hotel structure (overall nature of the loss incident) . (Impacts on P): Three (3) workers were killed and 19 were injured. (E): Immediately following the collapse, a large amount of debris and dust were released into the air and contaminated the surrounding area. (A): This collapse resulted in the full demolition of the hotel, and 3 of the surrounding buildings, a loss of $450 million in assets and demolition and clean-up costs of $10 million. (P): The hotel was never rebuilt at a loss of $26 million in annual sales revenues. State, number, and label immediate causes in the RCA chart, “I1 - SP”, “I2 - SC”, “I3 - SP” 5 State, number, and label basic causes in the RCA chart with category, sub-category, and sub-sub- category e.g. “B1 -EDF inadequate technical design”, “B2 - JF inadequate training”, “B3 - PF improper motivation” labels; identify at least 24 basic causes, with the first basic cause in each root pathway being identified and the last basic cause in each root pathway being identified; 24 x 2 marks each = 48 marks Note: If a team submits more than 24 labelled basic causes, prorate the marks for this part of the RCA Chart accordingly: e.g. 30 BC’s with score of 54 out of 60 (=30x2) prorates to 43 out of 48. 48 The use of AND and OR gates must make sense and be consistent 4 The RCA Chart must have sufficient breadth and depth to identify at least 18 latent causes, and no more than 25 with a maximum of 4 in any one RME element, #2 through to #11. The latent causes must be specific and unique to the root. The latent causes must be clearly identified with “LC” code and category (P, S, C), and numbered. (18x0.5 = 9 marks, deduct 0.5 marks for each “LC” if not clearly identified, labelled, numbered, or numbered from left -to-right.) Examples: LC1-S, LC2-S, etc. 9 Follow ALL roots from the incident description through to the latent cause; the rationale and logic must be clearly evident. (nominal 9 roots x 3 marks per root = 27 marks) Examine all roots x maximum 3 marks deducted per root examined = 27 marks as follows: one logic leap in a root = 1 mark deducted; two or more logic leaps or slightly confusing in a root = 2 marks deducted; a root does not make sense at all and necessitates significant feedback to correct = 3 marks deducted. 27 The RCA Chart must have NO latent causes relating to RME #1 MLCA. Deduct 3 marks for each Latent Cause that relates to #1 MLCA. Deduct 1 mark each for any cause statement in a root that indirectly relates to or implicitly refers to #1 MLCA e.g. “productivity was prioritized over safety”. -3 Each latent cause is logically and clearly linked to a Risk Management System Element; (18 x 0.5 mark each = 9 marks): 9 Each of the latent causes (at least 18 unique) must be explicitly stated in the RCA chart and in terms relevant to the loss incident unde r study; Example Latent Cause statement: “ LC1-S: The risk assessment program did not include a standard to assess the risks of working in the exclusion zone.” This latent cause statement would be marked as follows with bold italics text showing where mark s are awarded: “ LC1-S: (see above for marks on the labelling and link to RME) The risk assessment program (1mark for the name of the program that aligns with the RME) did not include a standard (1 mark for this standard phrase) to assess (1 mark for an active verb relevant to the program name) the risks of working in the exclusion zone (1 mark for a statement specific to the root). #2 RAMR”; 18 LC’s x 4 marks each = 72 marks Note: If a team submits more than 18 labelled latent causes, prorate the marks for this part of the RCA Chart accordingly: e.g. 20 LC’s, score of 70 out of 80 (=20x4) prorates to 63 out of 72. 72 Spelling in the RCA Chart: The expectation is for a professional report. Ten (10) marks are awarded outright. Deduct 2 marks for each word that is spelled incorrectly, to a maximum of 10 marks. 10 Total 210 Deductions: Deduct 35 marks (of 210 marks) if the RCA Chart is NOT legible in the uploaded report. -35 Adjustment Factor = 0.33; take the score for this section and divide by 3 e.g. a score out of 210 x 210 / 3 = score out of 70 marks 70 Deduction on the RCA Chart for Lack of Improvements per Feedback on the Progress Report Feedback was provided on the Progress Report with the intent that the teams improve their RCA Chart for the Final Report. This adjustment is intended to account for incorporating the feedback. Adjust the final score on the Final RCA Chart based on the relative amount of feedback points that have been addressed: Extent of Improvements Assessment More than 90% No deduction. Less than 90% of the feedback points from the Progress Report have been addressed Take total score of the RCA Chart and deduct 50% No changes apparent 100% deduction Varies TOTAL: 310