2 Assessment of Competence More than One Approach Traditional primarily external scrutiny Emerging more internal selfdirected Research educationallydriven 3 Tests standardized where possible to maximize reliability eg simulations MCQ virtual SP ID: 680774
Download Presentation The PPT/PDF document "1 Portfolios for Lifelong Competence Ass..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
1
Portfolios for Lifelong Competence AssessmentSlide2
2
Assessment of Competence - More than One Approach
Traditional (primarily external scrutiny)
Emerging (more internal, self-directed)
Research (educationally-driven)Slide3
3
Tests standardized where possible to maximize reliability (e.g. simulations - MCQ/ virtual, SP)
Narrowly focused on knowledge and reasoning
High stakes, summative
May focus on individuals (“assessment”) or programs (“evaluation”)
All about minimal competence
Preferred approach of regulators (viz. societal protection)
Assessment of Competence - The Traditional Approach Slide4
4
Assessment of Competence - The Emerging Approach
Tests tailored to the actual practice and learning needs of the individual
Assessment beyond knowledge (e.g. includes skills and behaviors)
Low stakes, formative, abundant feedback
Usually focuses on the individual
All about driving continuous improvement
Impelled by innate professionalism (internal locus of accountability), not just external scrutinySlide5
5
Assessment of Competence - The Research Approach
Seeking better pedagogies, best practices
Broadening base of assessment, shaping learning
Finding effective balance between traditional and emerging approaches
Focus may be on individual, but more often programmatic
Seeking balance between baseline competence and continuing improvement
Primarily the realm of professional
educators
Driven
by need to define effective educational approaches Slide6
6
Availability of Data Sources and
Methodologies
For the Traditional approach - Good
For Emerging & Research approaches - PoorSlide7
7
Needed: Access to Many New Data
Sets - Some Examples
Work learning experience - scope of clinical practice and changes over time
Clinical performance - processes of care, outcomes, management
Behaviors and skills
Achievement of competency milestones
Longitudinal research data set Slide8
8
Comes the Portfolio
aka eFolio - Definition
An mechanism to collect electronic data sets in near real time (i.e. A Process)
A particular collection of data and data sets specific to purpose - “Assessment of competence” (i.e. A Thing)
Both a Process and a Thing (i.e. A “Proing”)Slide9
9
The Good News: Bad News Debate
Healthcare professionals leave a massive contrail of electronic data in their daily work
BUT relevant data are very fragmented, scattered across a galaxy of repositories
HIT now allows the potential for sourcing and compilation of data in near real time
BUT the relevant repositories have scant tradition nor appetite for data-sharingSlide10
10
The eFolio Connector Vision
Data Commons
A collection of Portfolios and Repositories that can provide data for many different applications
eFolio Connector
Suite of apps compiling data reports
for the learner, and for other viewers authorized by the learner
Data SourcingSlide11
11
Data Commons
A collection of Portfolios and Repositories that can provide data for many different applications
eFolio Connector
Suite of apps compiling data reports
for the learner, and for other viewers authorized by the learner
eFolio Connector: Initial Data Sets Planned
eFolio Connector
Suite of apps compiling data reports
for a variety of viewers authorized by the learner identified
Data Commons
Bridging Many Portfolios and Repositories, and providing data for many different applications
Data SourcingSlide12
12
Data Commons
A collection of Portfolios and Repositories that can provide data for many different applications
eFolio Connector
Suite of apps compiling data reports
for the learner, and for other viewers authorized by the learner
eFolio Connector
Suite of apps compiling data reports
for a variety of viewers authorized by the learner identified
Some Other Important Issues
Data Sourcing
“Virtual
Repository”
Reporting out to authorized viewers specifications
Trusted Agent Hub Slide13
13
The “Virtual Repository”: Affiliated Repositories
Sign standard data-sharing agreements (Commons likely to become separate entity)
Generically willing to share data, but decide on case by case basis
Follow data custodian best practices
Expose data in XML, comply with technical specifications for each data set
May consume data, with authorization of the learner identified
Might charge fee to Apps for data provided Slide14
14
More on Trusted Agent Hub
Provides a backbone of connectivity across Data Commons infrastructure
Includes array of System Tools (Apps) for
ID matching
De-identification
Authentication
Business rules
Permissions
Transaction tracking and auditingSlide15
Channels of Distribution
Viewers (authorization by learner or de-identification for identified data)
Data Commons
Apps and Data Compilations
More on Report-Out to Viewers
Report OutSlide16
16
Compiling Data Sets: How Sausage Really Gets Made
Clearly define purpose (e.g. Understanding my practice)
Devise necessary schema (e.g. to codify narrative)
Do use cases to define content, context, use of data set
Find reliable sources of data required
Execute agreement with repositories to share data
Build technical specification to facilitate interoperability
Source data from repositories using web services
Pay any agreed-upon fees to repositories providing data Slide17
Data Commons
eFolio
Examples of New Data Reports In Development
Data SourcingSlide18
18
What If?
There is no current source for data required?
Add needed data to existing repository(ies)
Build a new repository, affiliate with Commons
Permanent access to data required, but repository(ies) unwilling or unable to maintain
Arrange with repository to provide archiving
Build central archiving function outside Commons
Learner says that data in a report is wrong?
Build central process for reconciliation of data
Slide19
19
What if? -2
Legal discovery is attempted on one of the data sets generated?
Compiled data sets are expunged by policy immediately after report-out (cf. Mandalas) and no data values are stored in the system
A request is received for identified data from someone other than the relevant learner
No date are moved, compiled or reported out without
authorization
of the learner Slide20
20
Evolution of the eFolio Connector - Feasibility Work
2007 Invitational Conference (sponsored by AAMC, ACGME, FSMB and
NBME)
2008
Pilot Organizational Meeting; User advisory group established
2010 Feasibility Pilot - built ANSI-conformant international technical specification for “Educational Trajectory” data setSlide21
21
Evolution of the eFolio Connector: Pre-Implementation
2011
At least two more specifications (likely “Assessment of Milestones” and “My Practice”)
Marketing, pricing, and business model
Prototype featuring 3-4 reports for feedbackSlide22
22
Conclusions
The eFolio Connector can source data from a diverse galaxy of repositories affiliated with the Data Commons and provide potentially useful data reports that
bridge the continuum
provide feedback on competence and
inform lifelong learning
Reports are available in near real time to learner and any other viewers authorized