How the Social Security Administration Has Used Data Analysis to Improve the Disability Program A Brief Overview of the Disability Process 2 Disability is the gateway to retirement from the workforce for many unfortunate people ID: 499884
Download Presentation The PPT/PDF document "Social Security Disability Adjudication" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Social Security Disability Adjudication
How the Social Security Administration Has Used Data Analysis to Improve the Disability ProgramSlide2
A Brief Overview of the Disability Process
2
Disability is the gateway to retirement from the workforce for many unfortunate people
There are up to four levels of administrative review. Claims are typically filed in local field offices or online
Disability claims are considered at the initial and reconsideration stages by 54 State agencies known as the Disability Determination Services (DDS)
Hearings and appeals are handled by federal employees in SSA’s Office of Disability Adjudication and ReviewSlide3
Appeals Council Review
3
The Appeals Council processes appeals of hearing decisions and dismissals
The Appeals Council also has authority to randomly or selectively sample hearing decisions and dismissalsSlide4
Civil Actions
4
The Appeals Council’s action is the last step of the administrative review process unless the Council issues an order of remand
An individual may file a civil action in federal court to seek judicial review of SSA’s final decision, usually an ALJ’s decision
The Appeals Council is responsible for
producing
a certified copy of the administrative recordSlide5
Comparative Data FY 2014
5Slide6
Social Security Disability Adjudication
Defining and Improving Quality in Disability Adjudication Slide7
Improving the Quality of Service Delivery
We
continuously look for ways to improve the quality and consistency of our case adjudication, while reducing processing times and the cost of the services we provide
Our goal has always been to deliver fair and accurate decisions as quickly as we can
In the last
7 - 10
years, we
have moved
toward electronic processing of
claims, opening
up new opportunities to improve our business processes and service
deliverySlide8
A Measurable Definition of Quality for Decisions
We developed a
clear, understandable, and
measurable
definition of quality from a customer standpoint
A Quality Decision Is…
factually accurate,
procedurally adequate,
policy compliant
timely issued, and
supported by the evidence
8Slide9
Mapping the Adjudication Process
We built a decision-tree mapping the regulatory requirements regarding the issues that need to be addressed in each case
We determined the
pathing
to each of the 2000 possible outcomes in disability cases
We developed analytical tools to guide adjudicators through appropriate pathing
We developed analytical tools to deconstruct and capture data about the decisions that are madeSlide10
Mapping the Service Delivery Requirements
10Slide11
Policy Compliant Decisional Tools
We developed two types of tools to guide adjudicators through policy compliant
pathing
:
The electronic cases analysis tool (
eCAT
) is used at the initial and reconsideration levels to guide adjudicators in reaching appropriate conclusions
The electronic Bench Book (
eBB
) is used at the hearing level in a similar manner
The Appeals Council Analysis Tool is used at the Council to guide reviewers in pinpointing errors in hearing level adjudicationSlide12
Gathering Actionable Data
Slide13
Gathering Actionable DataSlide14
Social Security Disability Adjudication
Analyzing, Visualizing and Utilizing
Data
to Improve PerformanceSlide15
Analyzing and Visualizing the Data
The capture of structured data in the analysis tools provided a wealth of information that we can mine to improve the quality and consistency of disability adjudication.
The Agency has more than 14 petabytes of data housed in more than 200 separate databases
We use a variety techniques to mine the data, including:
Regression analysis
Clustering analysis
Pattern mining
Computational linguistics
We
also employ a variety of data visualization techniques
Slide16
The Focused Review Process
Typically we identify outlier behaviors in the data and address the most significant ones
first
Outliers might be doing things better or worse than other
employees
We conduct focused reviews of outliers to learn what may account for their differences in service delivery
We conduct focused reviews of issues or of the work of individuals Slide17
Common Findings of Focused Reviews
Common findings
of
focused reviews
of
hearing decisions include:
Inadequate development of the record
Lack of supporting rationale
RFC problems/ opinions not properly evaluated
Lack of adherence to business processes
We had seen these problems for years and remands were not particularly effective. We wanted to know:
What is causing these errors and what can we do to change behaviors to reduce these errors?
17Slide18
Improving Heuristics in Case Adjudication
What seemed clear was that some employees had developed heuristics that do not always match policy requirements
Heuristics are typically used in problem solving, particularly complex problem solving
Heuristics are a mental framework that people have relied on in the past – they allow people to simplify consideration of the issues
Thus, although employees generally try to do the right thing, they did not always have a clear understanding of what that is
18Slide19
Improving Heuristics in Case Adjudication
We take the view that generally employees are self-motivated and will seek out responsibility in achieving objectives to which they are committed
Thus we needed to consider why people were developing heuristic models that were inconsistent with the policiesSlide20
Improving Heuristics In Case Adjudication
Noble Laureate and Professor Daniel
Kahneman
suggests these requirements for the development of proper heuristics:
A stable world in which problems are solved
Immersion and experience in that world
Immediate and recurring feedback regarding the heuristics developed
20Slide21
Social Security Disability Adjudication
Pushing Feedback and Training to Adjudicators - How MI Doing - Slide22
Providing Feedback - Training Design
We re-designed our training to focus on the end of training and what an employee had to be able to do (skills) beginning on day one
By changing our focus to the end result, we created a curriculum designed to be inter-active & skill based
We
created new training materials that blend research, analysis and casework into a comprehensive interactive
experience
In 2011, and again in 2015, the Council won the prestigious Deming Award from the Graduate School USA for outstanding federal government trainingSlide23
Improving Heuristics in Case
Adjudication
We also developed a tool called “How MI Doing” to provide direct feedback and push training to ALJs when in-person training is not feasible
How MI Doing includes:
Comparative information regarding dispositions, timeliness, productivity
Comparative information about the quality of work produced
Multi-level training modules tied directly to identified errors in casework (reasons for remand)
23Slide24
This next view shows the agree rates for decisions (blue) and dismissals (orange) by region. Click on a column to drill down to the next level.
24Slide25
If you want training information pertaining to a remand reason, click on the reason.
Remand Reason TrainingSlide26
How MI Doing – Desk Guides
26
The Tier 1 document always contains Appeals Council feedback, SSA policy
, why this issue matters, situations to watch out for, and what do to resolve the issue.Slide27
How MI Doing – Desk Guides
27Slide28
Social Security Disability Adjudication
Results of Feedback and Training EffortsSlide29
Change in High/Low Allowance Rate ALJs
Slide30
Changes in Remand & Appeal Rates
2009
2010
2011
2012
2013
2014
Appeals to AC as
a Ratio
of Appealable Cases
36.67%
39.49%
45.17%
38.85%
37.59%
40.76%
AC Remands as
a Percentage
of All Dispositions
22.12%
21.77%
21.19%
18.62%
17.11%
14.34%
New Court Cases Processed at AC
as
a Ratio
of Appealable
AC Dispositions
19.03%
16.23%
15.07%
12.99%
13.67%
14.44%
Federal Court Remands as a Ratio of Court Dispositions
47.5%
46.9%
46.12%
44.8%
42.35%
42.57%
30Slide31
Social Security Disability Adjudication
Improving Operational EfficiencySlide32
Differential Case Management
Adjudicators handle hundreds of claims per year, but there are about 2000 different types of decisions that can be issued in disability claims
We used k-means clustering techniques to do this, running quadrillions of calculations using hundreds of case characteristics and dozens of pivotal issues to sort cases by similarities
Sorting
and assigning similar cases improves the speed of processing, as adjudicators can apply the same policy compliant
pathing
to a series of similar
cases
Preliminary data shows errors and returns for pre-decisional rework have been cut in halfSlide33Slide34
How OAO Improved Timeliness – Differential Case ProcessingSlide35
Growth in RRs, Dispositions vs. Staff Growth
FY 2009-FY 2013Slide36
Results – Increased DispositionsSlide37
Social Security Disability Adjudication
Computational LinguisticsSlide38
New Technologies
Optical Character Recognition
Converts images of text into editable/searchable text.
Natural Language Processing
A field of computer science focused on creating systems that can ‘understand’ the meaning of human (natural) language, such as text.
Machine Learning
Systems that can study a set of data and
independently
find commonalities in the data or make ‘informed choices’ in the face of new data.
5. The
claimant has the residual
functional capacity to perform
sedentary
work
except: he can occasionally
stoop and
kneel but must never crawl
.
residual
functional capacity
to perform
sedentary
he
can
occasionally
stoop
and
kneel
must
never
crawl
Example
: A hearing decision, including its RFC, are converted from an image into text:
.. The computer analyzes the text’s meaning - dividing sentences, forming associations, etc..
… finally, the computer outputs what it’s learned as data we can use, such as Excel data.Slide39
Natural Language Processing Algorithms: An Example of These New Technologies In Action
Feature 1: Extracts New Data
Our Natural Language Process algorithms extract a range of previously unavailable/incomplete data from the text of hearing decisions. Some highlights:
Data Item
SSA
Data Now
Our NLP
Step 3 Listing Numbers Met/Equaled
None
✓
Step 5 Medical-Vocational
Rules
None
✓
Residual Functional Capacity
Limitations
(‘lift 20 pounds total’; ‘no stooping’, etc.)
None
✓
Source Names Noted
by the Adjudicator
(
‘
Robert Stevens, M.D.
,
opined
…’)
None
✓
Step
2 Severe/Non-Severe Impairments
Limited (Max: 2)
✓
(All)
An example: Our natural language processing captured ~284 cases where fibromyalgia was a severe MDI in the decision text but that information was not captured by CPMS:Slide40
The NLP Algorithms: An Example of These New Technologies In Action
Feature 2: Real-Time Hearing Decision Quality Checks
Our Natural Language Processing algorithms can automatically detect a variety of errors/deficiencies present in hearing decisions. This data can be used to alert decision writers to the errors in real-time, enabling them to correct them
before
decisions are issued:
Example of what an alert could look like:Slide41
Social Security Disability Adjudication
Improving Policies and ProceduresSlide42
CLUSTERING ANALYSIS HELPED VISUALIZE THE INCONNECTIVITY OF PIVOTAL POLICY ISSUESSlide43
Why We Don’t Change Our Policies More Frequently
Our current practice is to mobilize and analyze data about a problem, determine if we can address it with training, and then progress through changes in sub-regulatory guidance, before moving on to regulatory or legislative proposals
We publish changes to HALLEX and POMS, the operational instructions for disability adjudicators on an almost daily basis
Multiple Social Security Rulings are issued annually
Other types of changes require more extensive
consideration by people external to SSASlide44
ACUS Projects
Treating Source Rule and Role of the Appeals Council
Duty of Candor and Submission of All Evidence
Closing the Record at the Hearing Level
Symptom Evaluation
Federal
Court Variances
ACUS studied the following:
The impact of SSA’s treating physician rules on the role of courts in reviewing our decisions and consider measures that SSA could take to reduce the number of cases remanded to it by courts; and
The role of the Appeals Council in reviewing cases to reduce any observed variances.
ACUS studied the following:
The Act, any amendments to the Act, and SSA’s current regulations regarding the duty of candor and submission of all evidence in disability claims; and
The requirements from other administrative tribunals as well as the Federal Rules of Evidence, Federal Rules of Civil Procedure, and other authority regarding the duty of candor and the submission of evidence.
ACUS studied the following:
SSA’s current regulations, past regulatory initiatives, relating to closure of the record after ALJs issue decisions;
The impact of SSA’s current pilot program in the Boston region that requires closure of the record after ALJs issue decisions with one exception; and
Any other possible alternatives and exceptions to a closed record at the hearings level.
ACUS is studying the following:
The Social Security
Act, SSA’s current regulations, and SSA’s sub-regulatory policy and development and documentation practices regarding how SSA adjudicators at all levels evaluate claimants’ symptoms, including pain, in the adjudication of social security disability claims.
ACUS is studying the following:
ACUS will survey and analyze federal court interpretations and applications of SSA’s rules and regulations. ACUS also will note patterns that show consistencies or inconsistencies in these interpretations or applications by specific federal courts, as well as varied judicial practices and procedures in federal cases involving social security disability insurance and supplemental
security income.Slide45
Social Security Disability Adjudication
Issues Addressed by District Courts in the 9
th
CircuitSlide46
2014 New Court Cases and Court Remands
Slide47
Evaluation of Subjective Complaints
Slide48
Evaluation of Medical Opinion Evidence and Residual Functional Capacity
Slide49
“Credit As True” Doctrine
In a series of cases dating back to 1988, the 9
th
Circuit has indicated it will credit as true both medical opinions from treating sources and testimony of witnesses that has been either improperly rejected or not adequately addressed in the hearing decision
The 9
th
Circuit requires that the agency must provide “clear and convincing reasons” when rejecting claimant testimony or treating source opinionsSlide50
“Credit As True” and the Evaluation of Treating Source Opinion Evidence
A 2013 ACUS
sponsored research report
noted
that Federal court preoccupation with the weight that ALJs assign to treating sources often has diverted court focus from the pivotal question of a claimant’s disability
(
see “
Assessing
the Efficacy of the Treating Physician
Rule”
– available at
www.acus.gov
)
The agency is actively engaged in the process of drafting new regulations regarding the evaluation of medical opinion evidence Slide51
Social Security Disability Adjudication
How the Social Security Administration Has Used Data Analysis to Improve the Disability Program