Science and Practice Nate Israel PhD Chapin Hall for Children at the University of Chicago Goals Understand what Reliability means to participants Review scientific literature on interrater reliability of common behavioral health concerns ID: 380073
Download Presentation The PPT/PDF document "Rater Reliability" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Rater Reliability:Science and Practice
Nate Israel, PhD
Chapin Hall for Children
at the
University of ChicagoSlide2
Goals
Understand what ‘Reliability’ means to participants
Review scientific literature on inter-rater reliability of common behavioral health concerns
Describe the difference between pre- and post- rating triangulation
Walk through scenarios for making sure our CANS / ANSA ratings are reliable
Slide3
Reliability
When someone in our profession says that something is ‘reliable’ or not, what does that mean?
When we are using the CANS / ANSA, what kinds of reliability matter the most to us?
Are there times when we actually
expect
our data to be unreliable in some way?Slide4
Science, Psychometrics, and Reliability
Common types of reliability:
Cross-time
Cross-item
Cross-rater
What does each of these tell us, clinically?
Slide5
Science, Psychometrics, and Reliability
Research on inter-rater reliability
of common behavioral health needs
Raters commonly include:
Professional (often a psychiatrist if diagnoses are involved)
Parent
Youth
…sometimes an additional party, such as a teacher
Slide6
Reliability: The Data
Research findings are (generally) consistent across a variety of studies
What would you guess that inter-rater reliability of common behavioral health concerns would be?
.9 is exceptionally reliable
.8 is highly reliable
.7 is often the minimum threshold for an instrument’s reliability (for research purposes)
.6 is seen as somewhat
un
reliable
Slide7
Reliability: Variation by Rater
Slide8
Reliability: Variation by Rater
Slide9
Reliability: Variation by Rater
Slide10
Reliability: AAAAAAAAhhhhhhhhhhh
!!!!
Science agrees:
raters disagree
Disagreement is substantial.
Disagreement is consistent
What do we do with disagreement??
Slide11
Reliability: A Pragmatic Approach
This disagreement is discovered with most behavioral health measures
after the measure is filled out
The CANS and ANSA are designed to be completed differently
The goal of the collaborative process underlying the CANS and ANSA is to
surface and address disagreement before a rating is made
Slide12
Reliability: Building Collaboration
This approach is sometimes referred to as
pre-rating triangulation
This means that you get important information from all relevant / available sources before making a rating
You make the rating in consultation with the client
Slide13
Reliability: Moments of Truth
In making a rating this way you can surface differences of opinion and address them
This is the heart of therapy: building and acting on a common understanding
It’s probably part of why
collaborative assessment processes are associated with both better engagement and a small TREATMENT effect
Slide14
Reliability: The Tough Parts
True Collaboration often means dealing with initially strong differences of opinion about needs and strengths
The CANS is designed to help people get through the toughest, most stigmatizing part of disagreement: the Why
It does this by
allowing you to build a sense of Why something is happening together with the client
Slide15
Example: Different Perspectives
Johnny is a 16-year old male youth
His
Spanish teacher
reports
that he has been skipping class and thinks that he is hanging out with ‘the wrong crowd’ and ‘probably getting into trouble’ and wants these issues addressed
Johnny reports that he sometimes misses Spanish class ‘because it’s right after lunch’ and denies being involved in any delinquent behavior
Slide16
Example: Different Perspectives
Jenna is a 36-year old female adult
Her CPS caseworker reports that she recently failed a drug screen (heroin), and that as long as she uses, her child’s permanency plan will not include her as the preferred placement outcome for the child
Jenna strongly denies using heroin and loudly protests being rated as needing to address Substance Use concerns. She wants her child back in her custody.
Slide17
Example: Different Perspectives
How can you make this about the What?
What kind of treatment goal(s) would be consistent with Jenna’s perspective and desire, and the case worker’s needs?
Slide18
Reliability: A CANS/ANSA Perspective
Reliability is
not separate
from our process of relating to, and working with our child, family, and adult clients
The more we make the ratings about their action implications, the more useful, and reliable, they become
Slide19
Professional-on-Professional Doubt
We may have questions about how a professional arrived at a CANS / ANSA rating
The CANS / ANSA are designed to be
transparent
When there is a question about a rating,
it should be askedSlide20
Professional-on-Professional Doubt
We have a dirty little secret in some places of our profession; we sometimes institutionalize unreliability
This may seem protective of clients, the system,
etc
Yet with the CANS and ANSA
accuracy is advocacy
For us to get better we have to be honest and collaborative with our clients, selves, supervisors, and managersSlide21
Additional Thoughts / Questions
Thanks for spending this morning together.
If there’s any other way I can be helpful, please let me know.
I hope to see you at the CANS conference in November!