1 Use of PEAR’s Auditing Application of AS9101 - PowerPoint Presentation

Slide1

1

Use of PEAR’s Auditing Application of AS9101

Boston, MAJuly 21-22, 2011Henry Gamboa UL DQS Aerospace Program ManagerMichelle BartonUL DQS Senior Lead Assessor/Reviewer

Auditor Workshop

Boston, MA

July 21-22, 2011Slide2

PEAR DefinitionA PEAR is a document stating results and providing evidence of determination on the effectiveness of a process.Slide3

PEAR Intent/PurposeThe PEAR was introduced to provide a tool that allows the auditor to systematically evaluate the effectiveness of a process based on the organizations method for monitoring and measuring their processes.Slide4

What is Effectiveness?It is the extent to which planned activities are realized and planned results are achieved

(Reference ISO 9000 Quality management systems – Fundamentals and Vocabulary)Slide5

What is Effectiveness?

In other words, is the process effective in:Fulfilling customer requirements?Delivering quality products, on time?

Improving continually?Meeting established goals and objectives?More than just a conformity assessment!Slide6

What will this Mean to the Auditor?

Review and examine an organization’s “Effectiveness” from:The customer viewpoint; satisfaction and performance.Organizational leadership goals and objectives.Alignment of process performance measures with quality objectives and customer requirements.

Criteria used for deciding what data to analyze for determining process effectiveness.Slide7

General rules for use/application of a PEAR:

At a minimum, the results of each audited product realization process, as determined by the organization (reference 9100-series clauses 4.1 and 7.1), shall be recorded on a PEAR.Each assessed process shall be recorded on a separate PEAR.

There is no preset minimal number of PEAR’s required from an auditSlide8

To what level of detail does a process have to be to require a PEAR?

Consider what defines a process – inputs, outputs, process owners, etc.If they are the same, then a higher level PEAR might be appropriate.

If they are not the same, then 2 separate PEARs should be created.In determining the appropriate level reference the Form Instructions for the the QMS Process Matrix Report (Appendix D) – it shows that Product Realization includes: Manufacturing-Machining, Manufacturing-Paint, Manufacturing – Assembly, etc.Slide9

When does a PEAR have to be done?

Reference quality manual & other QMS documentation for description and sequence of interaction of processes

At a minimum, the results of each audited product realization process, as determined by the organization (reference 9100-series clauses 4.1 and 7.1), shall be recorded on a PEARWill discuss more during group activitySlide10

New Focus on Audit Planning in AS9101D

The AS9101D now requires substantial review of information/data in preparing for audits. Included in this data is the identification of processes including their interactions and sequencing. The identification of these processes will then be used in preparing the audit plan/agenda and to identify the PEAR processes. Slide11

Process Based Audit Tool is an Input into the PEAR

The information that is recorded on the process based tool will then be an input for the completing the PEAR later.Slide12

Process Based Audit Tool is an Input into the PEAR (cont.)

Collection of Data (4.3.2.2 of AS9101D) -NOTE 1: The processes can be depicted in various ways [e.g., process maps, turtle diagrams, SIPOC method (breakdown of supplier, inputs, process steps/tasks, outputs, customer), octopus].AATT training -

Whatever audit tool is used, it should include process control methods for monitoring and measuring process effectiveness.Slide13

Electronic Assembly TurtleSlide14

Process Details/InterfacesSlide15

Process Details/InterfacesForm Completion Instructions from AS9101D - Summarize the process activities, inputs, and outputs; including the identification of associated process interfaces.Slide16

Process Details/Interfaces –

Example for Electronics Assembly ProcessProcess Details/Interfaces –

Inputs into the process include traveler, BOM, pick list, work instructions, kitted assembly & MRP need date. Process flow is as follows: “Assemble per drawing – Tune & align unit – pre-cap inspection – laser weld lid – environmental test – final test – paint & label – final inspect.” The outputs include completed assemblies and completed assembly records. Process interfaces include production control, planning and engineering. Question: Is there something that could be added to this description of process details/interfaces?Answer: The specific details of the process flow sequence alone does not highlight potential risk factors in meeting customer requirements and measureable objectives of the process.Slide17

Same Example, Different Write up

Process Details/Interfaces – Inputs into the process include kitted assembly and build paperwork & MRP need date. Process flow is as follows: “Assemble per drawing – Tune & align unit – pre-cap inspection – laser weld lid – environmental test – final test – paint & label – final inspect.” The nature of the product developed and manufactured in the organization requires highly specialized and state of the art microwave/hybrid processes. The product itself can be very complex and on occasion requires frequent design and development changes. The outputs includes completed assembly and assembly records. Process interfaces include production control, planning and engineering.

Question: Why is this process description better than the previous?Answer: Because it highlights potential risk factors (product complexity, frequent design changes) in meeting customer requirements and measurable objectives of the process. Slide18

Organizations method for determining

process effectivenessSlide19

Organizations method for determining process effectiveness

Form Completion Instructions from AS9101D - Describe the method used by the organization to determine process effectiveness [e.g., identification of Key Performance Indicators (KPIs) and associated targets, process capability data].Slide20

Organizations method for determining process effectiveness –

Example for Electronic Assembly ProcessOrganizations method for determining process effectiveness -

Quality and on time delivery is tracked periodically.Question: What is wrong with this description?Answer: Description is too brief and does not tell how quality is tracked (e.g., 1st pass yield, RMA levels, scrap, etc.). It also does not tell how often quality and on time delivery is tracked and by whom. Slide21

Organizations method for determining process effectiveness

Same Example, Different Write up1

st pass yields and on-time delivery is tracked daily and discussed during daily production meetings with cross functional teams. This information is summarized and reported monthly by the quality manager to the general manager. The general manager will go over this information on a quarterly basis with all employees during state of the company meetings.Question – Why is this example better than the previous?Answer – Because description is more thorough and describes how quality is tracked (1st pass yields). It also mentions how often quality and on time delivery is tracked (daily production meetings, monthly reporting to GM and quarterly state of the company meetings)Slide22

Auditor observations & comments supporting

process effectiveness determinationSlide23

Auditor observations & comments supporting process effectiveness determination

Form Completion Instructions from AS9101D - Annotate relevant objective evidence, observed conditions, data, information, comments, etc. to support the auditor's statement of effectiveness or ineffectivenessSlide24

Auditor observations & comments supporting process effectiveness determination

Electronic Assembly Process Example

Assessed 5 items through the assembly process. Work instructions were available for the operator at stations in each of cells. Talked with 6 operators all who knew the quality policy. Records were available for review. All gauges had calibration stickers on them. Verified the training records for each of the employees interviewed and all had records of competence for activity observed performing.Question: What is wrong with this description?Answer: This description gives the evidence of compliance that was assessed. It does not address the effectiveness of the process. Based on this information the reader of the PEAR would not be able to determine what Level the process is performing to.Slide25

Auditor observations & comments supporting process effectiveness determination

Same Example, Different Write upOn time delivery target of 95% has not been met for the past 12 months. Average for past 3 months has been slightly below 75%. Factory is running at capacity according to Manufacturing Manager. Actions taken include hiring 2 new swing shift technicians and 7 new day shift technicians. Also, training of technicians of highly specialized processes is on-going. In addition, action is planned to automate some of the existing tuning and testing processes which should help improve delivery. Also, weekly material status meetings are having a positive impact to schedule according to area manager.

Question: Why is this better than the previous response?Answer: This description tells of the performance of the process that was observed during the audit rather than identifying the evidence of conformance.Slide26

Determining the appropriate

level of effectivenessSlide27

New Levels of Effectiveness introduced in PEAR documents.

1) Process not implemented; planned results are not achieved?2) Process implemented; planned results are not achieved and appropriate actions not taken?

3) Process implemented; planned results are not achieved, but appropriate actions being taken?4) Process implemented; planned results are achieved?Slide28

Determining the appropriate level of effectiveness

Form Instructions – Indicate to what extent the audited process was determined effective.

Effectiveness: extent to which planned activities are realized and planned results achieved (reference ISO 9000).NOTE 1: A major (ma) NCR shall be issued when the effectiveness level of the process is rated a "1".NOTE 2: A major (ma) or minor (mi) NCR shall be issued when the effectiveness level of the process is rated a "2" (see nonconformity definitions in clause 3).Slide29

Determining the appropriate level of effectivenessWhat is written in the previous 2 sections should support this determination.

In essence, the reader of the PEAR should already know what level the process is performing at before looking at the checked box.Slide30

Example of Determining Effectiveness Level

Electronic Assembly ProcessOn time delivery target for electronic assembly process of 95% has not been met for the past 12 months. Average for past 3 months has been slightly below 75%. According to the Assembly Manager, the factory is running at capacity but he has not been able to find new qualified technicians for hire required for some of its highly specialized processes. Performance data posted in the assembly area shows that electronic assembly has been running on over time for the past 9 months and 1

st pass yield targets have not been met. RMA levels during this time frame have exceeded the goal of 1% or less for product shipped and is currently at 3%. Slide31

What level of effectiveness would you assign?

1) Process not implemented; planned results are not achieved?2) Process implemented; planned results are not achieved and appropriate actions not taken?

3) Process implemented; planned results are not achieved, but appropriate actions being taken?4) Process implemented; planned results are achieved?Slide32

Example of Determining Effectiveness Level

Electronic Assembly ProcessOn time delivery target of 95% has not been met for the past 12 months. Average for past 3 months has been slightly below 75%. Factory is running at capacity according to Manufacturing Manager. Actions taken include hiring 2 new swing shift technicians and 7 new day shift technicians. Also, training of technicians of highly specialized processes is on-going. In addition, action is planned to automate some of the existing tuning and testing processes which should help improve delivery. Also, weekly material status meetings are having a positive impact to schedule according to area manager. Slide33

What level of effectiveness would you assign?

1) Process not implemented; planned results are not achieved?2) Process implemented; planned results are not achieved and appropriate actions not taken?

3) Process implemented; planned results are not achieved, but appropriate actions being taken?4) Process implemented; planned results are achieved?Slide34

Example of Determining Effectiveness Level

Electronic Assembly ProcessOn time delivery target of 95% has not been met for the past 12 months. Average for past 3 months has been slightly below 55%. Several key personnel have left the organization since the last surveillance due to retirement and other reasons. Replacements have not been trained and there are no plans to have them trained until the next big contract is awarded. Furthermore, many of the assembly processes have not been documented and there seems to be a dependence on tribal knowledge on how to get the job done. Several production travelers sampled contained operations that were not signed off and had inspection operations that were by-passed. Customer complaints and RMA’s have tripled in the past 9 months.Slide35

What level of effectiveness would you assign?

1) Process not implemented; planned results are not achieved?2) Process implemented; planned results are not achieved and appropriate actions not taken?

3) Process implemented; planned results are not achieved, but appropriate actions being taken?4) Process implemented; planned results are achieved?Slide36

Group ActivitySlide37

Credits & AcknowledgementsIAQG AS9100 TeamSlide38

Questions?