Evaluation Criteria - PowerPoint Presentation

Slide1

Evaluation Criteria

S519Slide2

What is evaluation?What are the steps involved?

What are step1 and step2?What are absolute or relative merit?

Review

S519Slide3

To build a criterion list, consider the following procedures:A needs assessment

Logic model of linking the evaluand to the needsAn assesment of other relevant values

, such as process, outcomes, and cost

A strategy to organize your criterion checklist

Step3: Defining evaluative criteria

Make sure that you go into the evaluation with a well-thought-out plan so that you know what you need to know, where to get that information, and how you are going to put it together when you write up your report.

S519Slide4

Needs that we identify become the outcome criteria we use for the evaluationThe data collected during the needs assessment phase can be used as baseline data for comparison if we wish to track change in certain outcome variables.

Needs assessment

S519Slide5

Understand the true needs of your evaluation end users (consumers or impactees)

Who are your end users?They are the person or entity who buys or uses

a product or service, enroll in a training program,etc.

Upstream stakeholder (i.e. People on upper level of the structure – manager, designer)

Immediate recipients (i.e. People who directly consume your product or service – consumer, trainee)

Downstream consumers (i.e. People who indirectly involved in your evaluation)

Needs assessment

S519Slide6

Rules to follow during the evaluationSystematic (step-by-step, thorough)

Objective (free from bias)Transparent (easy to repeat and follow)

Needs assessment

S519Slide7

Needs vs. Wants

Difference and why

A need is something without which unsatisfactory functioning occurs.

Different kind of needs

Context dependence

Conscious needs vs. Unconsious needs

Needs we know and needs we do not know

Met needs vs. Unmet needs

Building a factory (increase job, but create pollution)

Performance needs vs. Instrumental needs„need to do“ something for satisfactory functioning (actual problems) vs. Proposed solutions Access email vs. Lightweight laptopMost of the case, performance needs is considered, but not the instrumental needsUnderstanding needs

S519Slide8

Two phases:Identifying and documenting performance needs

Investigating the underlying causes of performance needs

Needs assessment method

S519Slide9

Steps

Examples

Document the extent

of the presenting need by gathering evidence

- Ask police for teen drug arrests

Find

out more about individuals

- Is the

drug use more prevalent among boys or girls

Look for unnoticed performance needs - e.g. By combining of open-ended inquiryHave there also been problems with violence, or other crimes asking parents, teachers or community membersCome out the clear picture of the needs- Lay out the evidenceIdentifying performance needs

Identify the performance needs for an increase in drug abuse by teenagers

S519Slide10

Using logic model

Investigating the cause of the needs

Training program

Improved skills

Improved performance

If we implement this

We will address this underlying need

Which should solve our performance problem

S519Slide11

If we implement this

We will address this underlying need

Which should solve our performance

problem if that was the true underlying need

Training program

Unskilled

employee

Education and communication

Improved

performancePerformance incentivesUnmotivated staffJob enrichmentUnchallenged job

Provision of resources

Lack of resources

Flexible scheduling

Home and work conflict

Organizational climate intervention

Bad working enviornment

Exercise

S519Slide12

Asking potential participants about their needs is just the tip of the icebergMany important information has to be gathered in some other way

Working with experienced upstream impactees

Working with project officer for grant writing training program

Identifying correct cause by asking open-end inquiry to downstream and upstream impactees

Poor involvement of grant writing

Lack of motivation – increase incentive

Lack of confidence – build up self confidence

More to do

S519Slide13

Think about other criteria based onProcess evaluation

Outcome evaluationComparative cost-effectiveness

Exportability

Like: legal, ethical, authenticity, scentific, economic, historical requirements, etc.

Being diagnostic

Collecting facts

Other relevant criteria

S519Slide14

Needs assessment

Identify consumers or impactees (e.g. Table3.2)

Identify different needs (e.g. Table3.3)

Logic model (e.g. Exhibit3.6 and Exhibit3.7)

An assessment of other relavent values with the consideration of process, outcome and cost (e.g. Table3.4)

Organizing your criteria

see step4 output report

Step3: output report

S519Slide15

Rolling designunanticipated criteria may add to the list

Existing criteria may be modifiedNever draw a conclusion based on a single piece of evidenc

e

Try to view the same issue from different angles through:

Different types of data (both qualitative and quantitative)

Multiple sources of information (e.g., existing document, observations, input from more than one group of stakeholders)

Step4: Organizing criteria and indentifying sources of evidence

Triangulation

: using different ways or data to verify the conclusion

Step-by-Step

: start the small scope of data collecting, extend it slowly later on.

S519Slide16

When organizing your criteria, always keep the followings in mind:Process

How good are the evaluand‘s content and implementationOutcomes

How good are the impacts on immediate recipients and other impactees

Comparative Cost-Effectives

How costly is it? Excessive, quite high, acceptable or reasonable

Exportability

How can we extend this to other settings?

Checkpoints

S519Slide17

Process evaluation

Content

What the evaluand consists of, i.e., basic components or design)

Implementation

How well or efficiently the evaluand was implemented or delivered to the consumers who needed it

Other features

Any other features that make the program good or bad which are not covered by the first two and are not outcomes or cost-related criteria

The process evaluation checkpoint

S519Slide18

Now feed into these three categories with all the process-relevant ingredients we generated from:

the needs assessment and other relevant needs (e.g. Table3.3, Table 3.4, Exhibit3.6)

Add additional consideration if necessary

The process evaluation checkpoint

category

Subcategories

and criteria

Source of evdience

Content evaluation

Implementation evlauationOthers (if necessary)

S519Slide19

What is outcomeThings that happen as a result of the program

Outcomes can affect anyone listed as consumersHow to do

Based on logic model in step3 (e.g. Exhibit3.6 and Exhibit3.7)

Organize them into subcategories

See Table4.3 (D-p60)

The outcome evaluation checkpoint

category

Subcategories

and criteria

Source of evdienceKnowledge, skill and attitude gainApplication of knowledge, skill and attitudes

S519Slide20

Any evaluation has to take cost into accountWhat are costs?

MoneyTimeEffort

Space

Opportunity costs

The comparative cost-effectiveness checkpoint

S519Slide21

Type of costs

Money, resources, time, opportunities

Costs When

Preparation, implementation, maintenance, evaluation

Costs to whom

Participants, community organizations, workshop provider, others

D-p61

Cost cube (Scriven, 1991)

S519Slide22

Preparation

Implementation

Maintenance

Evaluation

Money

Resources

Time

Opportunity

Costs

to Whom

Comparison

with competitors

Cost cube table

Find out the most important costs for your evaluation

Compare your cost with other competitors

S519Slide23

What elements of the evaluand (i.e., innovative design or approach) might make it potentially valuable or a significant contribution or advance in another setting

Think about examples

The exportability checkpoint

S519Slide24

Checkpoints forProcess

Outcomes (e.g., Table4.3)Comparative Cost-Effectives (e.g., cost cube table)Exportability

Short summary of potential areas for exportability

Step4: Output report

S519