Rater Reliability - PowerPoint Presentation

Rater Reliability
Rater Reliability

Rater Reliability - Description


Science and Practice Nate Israel PhD Chapin Hall for Children at the University of Chicago Goals Understand what Reliability means to participants Review scientific literature on interrater reliability of common behavioral health concerns ID: 380073 Download Presentation

Tags

cans reliability rater rating reliability cans rating rater ansa disagreement professional reliable common science health behavioral collaborative inter cross

Download Section

Please download the presentation from below link :


Download Presentation - The PPT/PDF document "Rater Reliability" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Embed / Share - Rater Reliability


Presentation on theme: "Rater Reliability"— Presentation transcript


Slide1

Rater Reliability:Science and Practice

Nate Israel, PhD

Chapin Hall for Children

at the

University of ChicagoSlide2

Goals

Understand what ‘Reliability’ means to participants

Review scientific literature on inter-rater reliability of common behavioral health concerns

Describe the difference between pre- and post- rating triangulation

Walk through scenarios for making sure our CANS / ANSA ratings are reliable

Slide3

Reliability

When someone in our profession says that something is ‘reliable’ or not, what does that mean?

When we are using the CANS / ANSA, what kinds of reliability matter the most to us?

Are there times when we actually

expect

our data to be unreliable in some way?Slide4

Science, Psychometrics, and Reliability

Common types of reliability:

Cross-time

Cross-item

Cross-rater

What does each of these tell us, clinically?

Slide5

Science, Psychometrics, and Reliability

Research on inter-rater reliability

of common behavioral health needs

Raters commonly include:

Professional (often a psychiatrist if diagnoses are involved)

Parent

Youth

…sometimes an additional party, such as a teacher

Slide6

Reliability: The Data

Research findings are (generally) consistent across a variety of studies

What would you guess that inter-rater reliability of common behavioral health concerns would be?

.9 is exceptionally reliable

.8 is highly reliable

.7 is often the minimum threshold for an instrument’s reliability (for research purposes)

.6 is seen as somewhat

un

reliable

Slide7

Reliability: Variation by Rater

Slide8

Reliability: Variation by Rater

Slide9

Reliability: Variation by Rater

Slide10

Reliability: AAAAAAAAhhhhhhhhhhh

!!!!

Science agrees:

raters disagree

Disagreement is substantial.

Disagreement is consistent

What do we do with disagreement??

Slide11

Reliability: A Pragmatic Approach

This disagreement is discovered with most behavioral health measures

after the measure is filled out

The CANS and ANSA are designed to be completed differently

The goal of the collaborative process underlying the CANS and ANSA is to

surface and address disagreement before a rating is made

Slide12

Reliability: Building Collaboration

This approach is sometimes referred to as

pre-rating triangulation

This means that you get important information from all relevant / available sources before making a rating

You make the rating in consultation with the client

Slide13

Reliability: Moments of Truth

In making a rating this way you can surface differences of opinion and address them

This is the heart of therapy: building and acting on a common understanding

It’s probably part of why

collaborative assessment processes are associated with both better engagement and a small TREATMENT effect

Slide14

Reliability: The Tough Parts

True Collaboration often means dealing with initially strong differences of opinion about needs and strengths

The CANS is designed to help people get through the toughest, most stigmatizing part of disagreement: the Why

It does this by

allowing you to build a sense of Why something is happening together with the client

Slide15

Example: Different Perspectives

Johnny is a 16-year old male youth

His

Spanish teacher

reports

that he has been skipping class and thinks that he is hanging out with ‘the wrong crowd’ and ‘probably getting into trouble’ and wants these issues addressed

Johnny reports that he sometimes misses Spanish class ‘because it’s right after lunch’ and denies being involved in any delinquent behavior

Slide16

Example: Different Perspectives

Jenna is a 36-year old female adult

Her CPS caseworker reports that she recently failed a drug screen (heroin), and that as long as she uses, her child’s permanency plan will not include her as the preferred placement outcome for the child

Jenna strongly denies using heroin and loudly protests being rated as needing to address Substance Use concerns. She wants her child back in her custody.

Slide17

Example: Different Perspectives

How can you make this about the What?

What kind of treatment goal(s) would be consistent with Jenna’s perspective and desire, and the case worker’s needs?

Slide18

Reliability: A CANS/ANSA Perspective

Reliability is

not separate

from our process of relating to, and working with our child, family, and adult clients

The more we make the ratings about their action implications, the more useful, and reliable, they become

Slide19

Professional-on-Professional Doubt

We may have questions about how a professional arrived at a CANS / ANSA rating

The CANS / ANSA are designed to be

transparent

When there is a question about a rating,

it should be askedSlide20

Professional-on-Professional Doubt

We have a dirty little secret in some places of our profession; we sometimes institutionalize unreliability

This may seem protective of clients, the system,

etc

Yet with the CANS and ANSA

accuracy is advocacy

For us to get better we have to be honest and collaborative with our clients, selves, supervisors, and managersSlide21

Additional Thoughts / Questions

Thanks for spending this morning together.

If there’s any other way I can be helpful, please let me know.

I hope to see you at the CANS conference in November!

Shom More....