From Compliance to Performance - PowerPoint Presentation

From Compliance to Performance Introduction of Performance Based Regulation (PBR) into POA Oversight Mark Barker 2 nd March 2016

Introduction Mark Barker Principal Airworthiness Surveyor UK Civil Aviation AuthorityAirworthiness DivisionCivil and Military Approvals SectorEASA Production Organisation Approvals (POA) ManagerCAA Focal for MAA Design Approved Organisation Scheme (DAOS) 2

Subjects Introduction of Performance Based Regulation into UK POA Developments in Supplier Oversight Work on identified risks with the Prime and SME Manufacturing GroupsFindings update …How do I get less audits ? 3

Prior to CAA 4

CAA … 5

Common Themes in that time All of these audits were based on verifying compliance Increasing commonality in requirements within Aviation Increasing commonality of approach expected from all RegulatorsDirection from Compliance to Performance … 6

Better Regulation Principles Legislative and Regulatory Reform Act 2006 Regulators’ Code 2014 Any regulation should be:transparentaccountableproportionateconsistenttargeted 7

8

Leaflet C-180 9

Amendment 2 to Leaflet C-180 will require the addition of the following paragraph into your Production Exposition:- This is then used to generate the rating to determine the level of CAA Witness:- Risk Based Approach:- Suppliers 10Approximate Percentage of Production Contracted Approximate Number of Units produced by Subcontractors Approximate Number of UK Subcontractors Number Undertaking Critical Activity (UK) Approximate Number of Overseas Subcontractors Number Undertaking Critical Activity (O’seas) FTE Personnel Undertaking Supplier QA Activity Number of on-site audits undertaken annually Result Score 0 0 25% 1 50% 2 75%+ 3 Result Score 0 0 100 1 200-400 2 400+ 3 Result Score 0 0 100 1 200-400 2 400+ 3 Result Score 0 0 2-5 1 10 2 50 3 Result Score 0 0 10 1 50 2 100+ 3 Result Score 0 0 2-5 1 10 2 50 3 Result Score 0 3 2-5 2 10 1 50+ 0 Result Score 0 3 2-5 2 10 1 50+ 0 Green < 11 - Desktop review once per 2-year cycle. Amber (12-16 ) - Desktop + 1 Witnessed Visit per 2-year cycle. Red 17 + = Top 10 - Desktop + 2 Visits per 2-year cycle, prioritise on 1 Critical and 1 Overseas Additional visits may be necessary in case of very large/complex organisations

Change Notification (Para 2.4 in Leaflet C-180) 21.A.147 normally requires application to CAA for a change in significant subcontractor or subcontractors undertaking critical parts - Variation. For those organisations where we have established regular interface meetings to review performance, update to C-180 introduces an alternative:- But can we recognise Performance ? 11

12 Surveyor will be able to agree a process where intended product transfers are discussed during regular Interface meetings. Variation then only needed when CAA decides to undertake direct sampling. Process noted in Exposition, Meeting Minutes will act as record of notification and the decision whether formal Variation is needed.2.4.3 Interface Meetings

Subjects Introduction of Performance Based Regulation into UK POA Developments in Supplier Oversight Leaflet C180 and IN-2016/008 (Regulatory risk for the sub-contractor or supply chain to a POA course)Work on identified risks with the Prime and SME Manufacturing GroupsFindings update … How do I get less audits ? 13

Sub-Contractor / Supply Chain Course 14

Aviation Manufacturing Groups (AMG) Expression of Interest from previous POA Seminar Response was such that we adopted two Groups:- Prime AMG – Taken from the top Ten Supply Chain Involvement organisations identified under Leaflet C-180. Rolls Royce Airbus Bombardier Agusta Westland GE (Dowty Props) Britten-Norman Aircraft BAE Systems Safran (Messier Dowty) GKN Aerospace Spirit Aerospace Eaton Aerospace 15

PAMG High Top 10 - Action 16 Risk Industry+ CAAHigh Response9. Tier 2/Tier 3 Supply77 (1st)106 (1st) 9 (1st) 23. Lack of Risk Management for Supply70 (2nd)95 (2nd)7 (2 nd =) 7. Emergent Technology incl. ALM 67 (3 rd ) 94 (3 rd ) 6 (3 rd ) 18. Incoming Competency levels 63 (4 th ) 86 (4 th ) 7 (2 nd =) 8. Special processes, NADCAP, Crimp .. 62 (5 th ) 83 (5 th ) 4 (5 th =) 19. NDT Skills, competence/retirement 59 (6 th ) 80 (7 th ) 5 (4 th =) 12. Common approach on non-EU release 59 (6 th)70 (13th) 5 (4th =)13.Policing of intent of design data/IPR59 (6th)77 (8th)4 (5th =)14. PMA +NAA training in acceptance59 (6th)72 (12th)4 (5th =)26. Lack of Lateral View of Escapes58 (7th)81 (6th)4 (5th =)

Aviation Manufacturing Groups (AMG) Small and Medium Enterprises (SME) AMG 31 initial expressions of interest but inaugural meeting was less well attended:- Britten Norman Saywell Barnbrook Systems ACRO Seating Hybrid Air Vehicles DHSL Airbus Helicopter 17

SME Top 10 Areas for Action 18 Risk ScoreRaw Material Supply74Subcontractor Control63Capacity/Availability of Subcontractors46Production Competence42Supplier N/C to Purchase Order41Procurement not Involving Engineering 38 Flow down of treatment requirements 32 Legacy Support/Obsolescence 28 Single Source Supply 26 Competency Incoming 25 Proliferation of prime requirements 24

SME Aviation Manufacturing Groups We are looking to agree common progress areas and hear the voice of all levels of our industry. Should your name be here as well ? 19

Production Overview 20

CAA Focus Areas

So what are we finding ? 22

2007 23

2013/14 24

25

There’s not a lot of difference in the picture …. Traditional regulation model “more audits” is not delivering a good safety return All Regulators are addressing a steadily growing Aviation sector with either static or reducing resource. We need to evolve, and we need organisations to evolve.What did you notice ?26

27 2015

The majority of recent production quality escapes requiring safety action have involved non-conformances that were already known within the organisation via concessions, incomplete FAIRs, internal audits etc. But the root cause was never addressed …Repeated concessions against design data can contribute to a culture of tolerance of “low level” non-conformance.Compliance ►► ►► PerformanceCommon Factors28

Direct involvement of Accountable Manager Regular review based on tailored Metrics appropriate to each major Production Organisation based on a common core:- MORs (Now Mandatory and Voluntary)Quality Escapes (not just those leading to a reportable occurrence)Right First Time/Reject RateSupplier Performance Concessions/RejectionsEngineering Concessions/Query NotesInternal Audit PerformanceAdditional topics relevant to Organisation/Product:-Significant Changes (rapidly expanding organisations)First Time Pass Rate (rig testing output) Those specific to organisation improvements/issues (confidential). CAA Performance Conversation 29

Key importance of a ‘word picture’ supporting each KPI to provide meaning to the raw numbers. One of the action Tasks for discussion with members of the Prime Aviation Manufacturing Group (PAMG). CAA is preparing a list of KPIs from core topics and current models for agreement with the Group. This will include agreeing meaning of terms and providing room for the narrative.Key Performance Indicator Discussion30

Movement of all areas of the regulations (both EASA and Military) is towards extension of review periods where it is justified by the extent of risk of the product being manufactured and the performance of the organisation. But Rule change is slow … So CAA has recently agreed with EASA that from 2016 onwards, those production organisations which are rated as “Green” in terms of organisational complexity and product risk can receive an interim Desktop Audit instead of an on-site audit based on a performance submission. So how can the CAA recognise Performance ?31

Will not be applied where Organisation:- Is newly approved (first 2-year audit cycle) Manufactures Product Manufactures Critical PartsHas received a Level 1 finding or Level 2 finding of High Safety Significance within the preceding cycleHas an identified Supply Chain complexity rating of ‘Amber’ or ‘Red’Is audited in support of a co-operative arrangement with another NAA or a bilateral partner or under direct contract to EASA Limitations 32

Organisation legal entity registration is unchanged Organisation structure is either unchanged or all changes previously advised to CAA QMS on which the approval is based remains in place and has not been significantly changed Evidence of conduct of internal audit programme (audits planned, audits undertaken, summary of key findings and status)Notification of any external quality escapes where parts have been released and found to be outside design data KPI Criteria 33

Where Surveyor is satisfied with overall Organisation performance and the content of the response, the Desktop Audit can replace the normally scheduled Annual on-site Surveillance as recorded on Q-Pulse. So to get less audits … Effect 34

Thank you for listening Questions…