What Are KPI’s Not Telling Us? - PowerPoint Presentation

Slide1

What Are KPI’s Not Telling Us?

July 22-23, 2015A. Michael McRandallNSF-ISR

Slide2

Introduction

Mike McRandall Business Unit Manager, Aerospace NSF-ISR Over 30 years experience in the aerospace industry in both quality and operations

2Slide3

Workshop Agenda

Process Identification (can’t measure ‘em if you don’t know ‘em)Process EffectivenessWhat are we measuring?Does it get us where we need to be?

Why aren’t OTD & Quality adequate measures for all?

Options When Organization Is Process Challenged

Case Studies/Examples

Open Discussion

3Slide4

How Effective Are We In Applying The “Process Approach” To Auditing?

Not Very…We have been touting a “process-based approach” to auditing for over 15 years and we still have:Auditees with no idea of what a process actually isAuditors in the same boat

Industry “experts” just as confused and exacerbating the problem further

Our Methods are Not Overly Effective…WHY?

4Slide5

Process Identification

Are processes properly identified?Is there a difference between an activity and a process?Are processes and procedures the same thing?Is every standard clause/element a process?Who identifies these processes?5Slide6

Process Definition

Defined as*: “a set of interrelated or interacting activities which transforms inputs into outputs.” These activities require allocation of resources (people, materials, equipment, etc.)A process must have defined:ObjectivesInputsOutputsActivitiesResources

If you can’t describe what you’re doing as a process, you don’t know

what you’re doing…

6Slide7

Activity or Process???

An Activity is a unit of work to be completed within the execution of a processActivities can be performed in isolation - but a process requires a group of activities in conjunction with resources to actually be, a process9001:2008 Section 0.2 confirms: “…An activity or set of activities using resources, and managed in order to enable the transformation of inputs into outputs, can be considered a process.”

7Slide8

Procedure or Process…Really??

These are not interchangeable terms and do not mean the same thingProcedure*: A specified way to carry out an activity or a processThough not as common place as in the recent past this is still seen too often within an organization’s documentation.

8Slide9

Who Identifies These Processes?

The organization, and only the organization,identifies (determines/defines) their processes (see 4.1a of the standard) It is not the place of the CB

It is not the place of the auditor

The organization must also describe the

interaction between their processes - commonly

referred to as the IOP

(4.2.2c of the standard)

9Slide10

Are All Clauses/Elements of The Standard, Processes?

Simple Answer: No Remember what a process is and what it takes to fit the definition. Also remember that an organization is required to monitor, measure and analyze their processes. So if they choose to call each element of the standard a process, then they need to be able to demonstrate how each element qualifies as a process and how those processes are monitored, measured and analyzed.

10Slide11

Assessing Process Effectiveness

What is a KPI (Key Performance Indicator)? Well, it is a metric, not a target The target would be the objective. The KPIs show us how we are performing toward our objective(s)

or targets.

The objective should be directly related to the process being assessed

It can, and should, tie into a more global business objective (OTD, quality, customer satisfaction, etc.)

The KPIs should allow stakeholders (management, owners, auditors, etc) to assess the effectiveness of the process.

11Slide12

Will These Measures Get Us There?

If the targets are either misplaced or disjointed from the overall company goals and objectives, the results might be, less than satisfactory. It is important to be able to show the tie-in between the process objectives (along with the KPIs used to track them) and the overall company objectives because…things may not always be as they seem.Or as a wise man once queried: Do outputs really matter?

12Slide13

The Proper KPIs?

Let’s try a sports analogy. We’ll use the 2014 Colorado Rockies of MLB) as a shining example13Slide14

2014 MLB Final - Team On-base Percentage (OBP)

14Slide15

2014 MLB Final - Team Total Runs Scored

15Slide16

2014 MLB Final - Team Batting Average

16Slide17

2014 MLB Final - Team Slugging Pct

17Slide18

As Individuals…

18Slide19

Some of Them Could Pick Too

19Slide20

So They Tore It Up…Right?

Fourth overall in MLB, on base percentage Third overall in MLB, runs scored Second overall in MLB, batting average First overall in MLB, slugging percentage

20Slide21

Maybe Not So Much…

21Slide22

What Happened??

22Slide23

Best Starting Pitcher…72nd in MLB/ERA

23Slide24

Are We Looking At The Right Things?

24What’s In It For Me??Slide25

Are We Looking At The Right Things?

What’s In It For Us??25Slide26

How About OTD and Quality?

They are required to be measured and addressed (5.2) – they must be good KPIs for every process, right? They are a means of having the organization maintain focus on the customer In most cases, they do not represent impactful data with regard to individual process performance KPIs point to symptoms, whether good or bad

OTD and Quality are not symptoms – they are results

We’ll touch more on this during our open forum/case studies session

26Slide27

OTD or W/L

The Rockies supply us, again, with ammunition for this battle Company Goal – Make Playoffs (10 teams total) Objective – 88 Wins (think OTD/95%) Actual – 66 Wins (think OTD/70%)Now making “Wins” your OTD equivalent and using Offense as one of your product realization processes, how accurate and adequate would it be to use Wins as your KPI for that process (Offense)?

27Slide28

Offense As a Process

Remember, The Rockies were: 4th in MLB for OBP 3rd in MLB for runs scored 2nd

in MLB for batting average

1

st

in MLB for slugging percentage

But their KPI for this process (Wins), says that they failed, big time. So…what is the CA for this process?

28Slide29

What if The Organization is Process- Challenged?

They showed me their IOP and it doesn’t correspond with what they do… Start at 4.1a and work out from there Have they determined the processes? (Y/N)

Have they determined the sequence and interaction? (Y/N)

If yes to both, but still no match – 4.2.2c*

They don’t have a process map or chart…

First of all, is this a problem?

˚

If a narrative exists but does not meet the intent of the requirements then a discussion needs to take place to determine what element of the standard the finding will be written against.

29Slide30

Case Study / Open Discussion

You are auditing the purchasing process of Ellenem Industries and they have identified Customer OTD as their method for determining process effectiveness. Their target is 97% and their actual OTD was 75%. They also use OTD as the only KPI for this process. During the course of your process audit you find that they monitor and measure many things relative to purchasing such as: supplier quality, ppv, supplier delivery to contract requirements, time to receipt of quote, inventory turns and flow down escapes – all of which are performing to internal expectations. You found no nonconformances to the standard during your audit of this process.

Seeing as how they missed their target of 97% (by a considerable margin), what type of actions would you expect to see they have taken to address this ineffective process?

What actions, if any, would you as the auditor take?

30Slide31

Case Study / Open Discussion

You are auditing the manufacturing process of Alacrity Amalgamated, a mid-size company that does short run jig grinding to customer specifications. They have identified product scrap as their effectiveness measure, with a target of < 2% of total volume. Their actual scrap rate is 4.2%. They have identified 2 KPIs for this process and they are: 1) the number of findings generated during internal audits (less than 3 / meeting target) and 2) the capacity scheduling of each work center (to be at 80% / meeting target). There were three findings against conformity to the standard identified during this audit:1) Failed to validate tools prior to release to production (7.5.1.3), 2) Undocumented changes to production processes (7.5.1.2), 3) No evidence that operations completed as planned (7.5.1h). Their response to missing their scrap target was to more closely analyze MRB minutes.

As an auditor, what, if any, actions would you take?

31Slide32

In Summation

Process Identification is Key KPIs Should Point to Symptoms OTD and Quality Are Not Great Individual Process KPIs Challenge Organizations to Understand Their Processes And to Identify KPIs to Monitor The Process

Thank you all for your time and patience

32