Outcome Measures and Informatics research NURS 737: Nursing Informatics Concepts and Practice - PowerPoint Presentation

Outcome Measures and Informatics research NURS 737: Nursing Informatics Concepts and Practice in System AdoptionModule 8 This document is intended solely for the use of N737. Not for distribution

Contents IntroductionBackgroundImportance of outcome measurement in CIS implementationChallenges and strategiesOutcome dimensions and measurementSelected Evaluation Dimensions and Their AssessmentHealthcare AnalyticsInformatics Continuous Quality Improvement (CQI) Strategies Nursing Informatics Research

Introduction 3

Background Hospital Information System (HIS)“An information system used to collect, store, process, retrieve, and communicate patient care and administrative information for all hospital-affiliated activities and to satisfy the functional requirements of all authorized users.”Clinical Information Systems (CISs)“A patient oriented hospital information system” (Handbook of Medical Informatics) 4

Background Types of CISsComputerized provider order entry (CPOE)Clinical Decision Support Systems (CDSS)Lab and radiology systemsClinical documentation systems, etc. Clinicians’ role in CIS implementation Participate in system evaluation and deployment Nurses and physicians are major users of CISs Must live with them!5

Importance of Outcome Measurement A critical component in delivering safe and quality care Substantial costs for implementing hospital-wide CISs If you do not assess outcomes of the system implementation? What are the consequences?

Importance of Outcome Measurement If you do not assess outcomes of the system implementation? Cannot know whether the system is working as it is supposed to Cannot claim the implementation was a success Cannot justify the costs

Challenges AttributesUsed by many different user groups Interconnected with many different programs (e.g., different areas)For real patient care needs Usually tailored to each organization8 Evaluation of CIS impact is complex. Measurement Difficulty Different aspects of outcomes. Interpretation of the findings is difficult. Controlling confounding variables is difficult. Generalization of the findings is difficult.

Strategies What can we do? Used by many different user groupsSelect evaluation measures relevant to each group (i.e., one single measure will not work)Interconnected with many different programs Assign a weight to each area during the planning phaseControlling confounding variables is difficultSelect appropriate research design Usually tailored to each organization Consider variations in the design Take them into account when you interpret the findings 9

Outcome Dimensions and Measurement What to measure (Dimensions) (Meijden, Tange, Troost, & Hasman, 2003)(1) system quality(2) information quality(3) usage(4) user satisfaction(5) individual impact(6) organizational impactHow to measure (Methods) Objective or quantitative methods Subjective or qualitative methods Other specific methods 10

1. System quality 2. Information quality 3. Usage Usability; Time savings; Intrinsic features creating extra work; Response time; Security; Data accuracy, etc. Completeness; Accuracy of data; Timeliness; Comprehensiveness; Consistency; Format, etc. Number of entries; Frequency of use; Duration of use; Self-reported usage; Frequency of use of specific functions 11 Measurement Dimensions 4. User satisfaction 5. Individual impact 6. Organizational impact User satisfaction; Attitude; Usability; Expectations; Competence Improved patient safety; clinical effectiveness; Changed clinical work patterns; Job satisfaction Communication and collaboration; Cost (time savings; reduction of staff; number of procedures reduced)

12 Selected Evaluation Dimensions and Their Assessment

User Satisfaction 13

User Satisfaction User Satisfaction (in health care settings) A measure of system performance as defined by customersThe most commonly used surrogate measure to assess the success of system implementationClarification of terms Usefulness: whether the system can be used to achieve desired goals 1. Utility – whether the functionality of the system in principle can do what is needed 2. Usability – how well users can use that functionality (user satisfaction is a dimension of Usability) 14

15 MeasureDimension Reliability / Validity User 1. Questionnaire For User Interaction Satisfaction (QUIS) A measure of overall system satisfaction with 6 scales measuring 9 specific interface factors. Assess users' subjective satisfaction with specific aspects of the human-computer interface; Includes demographic questionnaire Alpha: .94 - .95 (Web version) EFA General users 2. PU / PEU 12 items; Perceived Usefulness Scale [PU]; Perceived Ease of Use Scale [PEU] Alpha: .98 (PU); .94 (PEU) Physicians, nurses 3. Questionnaire on Computer Systems and Decision Making 40 items; Employee morale, reductions in employees, goals being met and overall satisfaction with the systems; 40 items Selected User Satisfaction Measures

16 MeasureDimension Reliability / Validity User 4. Physician Order Entry User Satisfaction and Usage Survey   Items 1-16: Assess general satisfaction, reliability, speed, ease of use, training, and impact on productivity and patient care.   Items 17-25: Assess specific features of POE -- respondents were asked to indicate whether they use each feature, and if so, to rate its usefulness. Alpha: .86 Providers (Physicians, Dentists, Nurse practitioners, Pharmacists, etc.) 5. End-User Computing Satisfaction A 12-item; Assess five components of end-user satisfaction: content, accuracy, format, ease of use, and timeliness. Alpha: .92 Factor analysis; Criterion validity ; discriminant validity Selected User Satisfaction Measures (cont.)

References 1. Chin, J. P., Diehl, V. A. and Norman, K. L. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. Proceedings of SIGCHI '88, (pp. 213-218), New York: ACM/SIGCHI.  2. Mazzoleni, M. C., Baiardi, P., Giorgi, I., Franchi, G., Marconi, R., & Cortesi, M. (1996). Assessing users' satisfaction through perception of usefulness and ease of use in the daily interaction with a hospital information system. Proceedings/AMIA Annual Fall Symposium. p. 752-756 3. Hatcher, M. (1998). Impact of information systems on acute care hospitals: results from a survey in the United States. Journal of Medical Systems, 22(6), 379-387. (Tool is in “Hatcher, M. (1997). Survey of acute care hospitals in the United States relative to technology usage and technology transfer. Journal of Medical Systems, 21(5), 323-337.) 4. Lee, F., Teich, J. M., Spurr, C. D., & Bates, D. W. (1996). Implementation of physician order entry: user satisfaction and self- reported usage patterns. Journal of the American Medical Informatics Association, 13 , 42-55. (this tool also used in Wilson, J. P., Bulatao, P. T., & Rascati, K. L. (2000). Satisfaction with a computerized practitioner order-entry system at two military health care facilities. American Journal of Health-System Pharmacy, 57 (23), 2188-2195.)  5. Doll, W. J., & Torkzadeh, G. (1988). The Measurement of End-User Computing Satisfaction. MIS Quarterly, 12 (2), 259-274. 17

18 AHRQ Health IT Survey Compendium (Agency for Health care Research and Policy : https://healthit.ahrq.gov/health-it-tools-and-resources/health-it-survey-compendium)

User Satisfaction: Measurement Challenges Various confounding variablesPersonal factors: Experience with computer use, attitude, education levels, etc. Organization factor: Staffing, organizational infrastructure, culture, etc. Must take them into account in study designs and interpretation of findings. Reliability and validity of the measuresFew measures have evidence of reliability and validity.  Unreliable and invalid outcomes 19

Organizational Impact 20

Dimensions for the Organizational Decision-Makers Technical(Relates to system function, performance, and reliability)Maintenance, downtime Upgradeability, compatibility Professional (Direct impacts on first-line users and patients) Information access Ease of use, time savings Organizational Impact on organization as a whole Policy change Organizational strategy Economic ROI (Return on Investment) Risk management Setup and training costs Ethical (Relating to patient privacy and data access) Accountability Data-driven autonomy Legal Legality issues surrounding electronic patient data records 21 DOMAIN

Organizational Impact: Measurement Challenges Complex evaluation dimensions (e.g., dimensions for decision-makers)Triangulation: Use of multiple data sources and methods to investigate the same phenomenon.1) Validate findings with the help of others2) Complementary results increase ‘completeness’22

Types of Triangulation DataVarious data sources: EHR, patients, etc. Nurses surveyed from different sites Investigator Computer scientist and clinicians jointly analyze focus group interviews Theory Data are analyzed based on various perspectives, hypotheses or theories Methods Applying group interviews along with questionnaires Two different questionnaires to assess user attitudes 23

Individual Impact (Clinicians & Patients)24

Major Areas of AssessmentPatient outcomesprimary outcomesclinically relevant outcomesStandard of care outcomesProcess/performance improvement in practice 25

Clinical Outcome Examples: Framework Intervention Outcomes Specific CIS solution Patient outcomes Standard of care Process improvement Example: BMA* system Reduced Implementation Removal of error rates of EBP** redundant charting 26 (*BMA = Barcode Medication Administration; **EBP = Evidence-Based Practice )

Clinical Outcome ExamplesCDSS and ADE Why are adverse drug events (ADEs) important?ADEs are #1 Medical ErrorThe main justification of CPOE, Barcode Medication Administration and other HIT systems Intervention: Computerized ADE Surveillance27

Clinical Outcome Examples 28

Clinical Outcome ExamplesCDSS and ADE Patient outcomes: Standard of care: Process improvement: 29

Clinical Outcome Examples CDSS and ADE Patient outcomes Reduced medication errorsStandard of care ADE reporting Process improvement Reduction of time for monitoring and reporting 30

Clinical Outcome Examples CDSS and CAUTIsWhy are adverse Catheter associated urinary tract infections (CAUTIs) important?CAUTIs are one of the most common hospital acquired infections (HAIs).Intervention: CDSS (alert system)

Clinical Outcome Examples CDSS and CAUTIs Patient outcomesReduction of hospital acquired infection Standard of care: Improve the quality of care by offering evidence-based proactive (EBP) guidelines to nursesProcess improvement:Alerts and reminders can help fill the gap between current practice and EBP. 32

Clinical Outcomes: Measurement Approaches Example: CAUTIsA hospital can embed nurse driven EBP protocol for timely removal of the urinary catheter within the EHR and assess the outcomes.Excellent example of how nursing-driven practice can improve safety and reduce medical costs.

Clinical Outcomes: Measurement Approaches 34

Measurement: Standard Methods Randomized Controlled Trials (RCT)Requires manipulation, random assignment, and control groupCan have Objective outcome measures, or Subjective measuresQuasi-randomized designMissing random assignment, control group or manipulationCan still provide some protection from validity threats 35

Measurement: Standard MethodsSurveys, questionnairesStandardized: Cheap, reliable and valid, but non-specific (generic)Custom-developed: Either untested or expensive to validate, but more specific, relevantInterviews: Provide subjective dataStructured or unstructured36

Measurement: Standard Methods Observations Structured data collection based on observable behaviorsReal-time “live” coding or video-based analysisProvides objective measures of performanceDon’t trust what people say, check what they do37

Measurement : Other MethodsCritical incident technique (CIT)Recall and describe specific events that influenced outcomes, positive and negative; what led up to the events; what influenced the outcomes?Provides access to rare eventsInsight into user-perception of system and surrounding issuesGrounded-theory approach Useful for understanding the broader impact of an implementation, identifying issuesCan be used to develop questionnaire, or other measurement techniques 38

Measurement: Considerations Many outcomes are possible.e.g. Main effects or secondary effectsThere are many more methodologies and techniques available. Various methods can be used - Triangulate!39

System Usage 40

System Usage Analysis 41Usage is a complex outcomeUsage analysis: examining work environment to study patterns of useData log analysis, server logs;Is help-function being used?Keystroke captureAre people using the system the way it was intended?Which keys/functions are used most often

Healthcare Analytics

Healthcare Analytics Analytics helps use data and to make healthcare and business decisions by using various analytical methods. Analytic programs apply statistics, formulas, modeling, and present findings with visual analytical tools such as score cards and dashboards. Analytics can categorized into the decision support systems (DSS).Analytics may be descriptive, predictive, or prescriptive. 3

Healthcare Analytics An enterprise data warehouses (EDW) takes data from multiple systems, extracts and transforms the data into a standardized and normalized format. The meta-data (the data about the data) contains data descriptions (e.g., data source, formats, descriptions)4

Healthcare Analytics (cont.) Users can access the data or use the exported data to analyze and create reports using “middleware.” Common reporting/analytics tools used in hospitals: Crystal reports, Business Objects, etcExcel, Access Database, SQL Standard Query LanguageExample data visualization tools/dashboards are: Qlikview, Tableau, Spotfire, Microsoft Visual Stack, etc.5

Informatics Continuous Quality Improvement (CQI) Strategies (HealthIT.gov. (2013). Continuous Quality Improvement (CQI) strategies to Optimize your Practice)6

Continuous Quality Improvement (CQI) Strategies Selection of specific CQI strategies based on several factors. This class will provide a brief overview of frequently used strategies: The Institute for Healthcare Improvement Model for ImprovementSix SigmaLeanBaldrige Quality Award Criteria JC IM StandardsISO 9000 Series7

The Institute for Healthcare Improvement (IHI) Model Set aimsEstablish measures Select changesTest changesImplement changes Spread changes8

Informatics CQI Strategies Six Sigma (for more detailed information, refer to the readings)A measure of quality based on Z-distributionA data-driven methodology to eliminate defects in any process, from manufacturing to service levels. (Defect is defined as anything outside of customer specifications.)Represents six standard deviations between the mean and the nearest specification limitTo achieve Six Sigma, a process must not produce more than 3.4 defects per million opportunities. 9

Informatics CQI StrategiesSix Sigma (cont.)Two Six Sigma sub-methodologies: DMAIC and DMADV. 1. DMAIC Methodology (define, measure, analyze, improve, control): Used to improve existing processes falling below user specification or not performing adequately2. DMADV Methodology (define, measure, analyze, design, verify): Used when a product or process is not in existence at the company and one needs to be developed10

Measure Analyze Improve Control Verify Design Analyze Measure Define DMADV when building a process from scratch (e.g. Implementing a new CIS) DMAIC when improving upon an existing process (e.g. Process improvement projects) Deepak Pillai MD, MBA Define Six Sigma (cont .) 11

Lean Kaizen Method Marisa Wilson, DNSc., MHSc., RN-BC12

Lean Kaizen A key to following the Framework of Implementation Success FactorsA tool to realizing most of the Components of Success FactorsAn activity to applying a Sociotechnical Approach to organizational information technology planning and analysis13

Kaizen Japanese word for “continuous improvement”Typically organized in a 3 to 5 day workshopLean Kaizen is a workshop using lean tools to improve a process14

Lean Lean – Focus on eliminating waste The 7 categories of wasteWaiting Rework MotionUnderutilizationOverproductionOver-processingInventory15

Applied to Technology Interface The 7 wastes Waiting (slow systems)Rework (re-entering lost data)Motion (moving from screen to screen)Underutilization (partial implementation)Overproduction (reports with no action)Over-processing (redundant data entry)Inventory (storage of unnecessary data)16

Kaizen Continuous ImprovementWorkshop approach Front line staff over a 3 – 5 day periodEmphasis on action17

Keys to success Involve everyone who touches the processKeep the workgroup manageableInclude “resource” people to be available to answer questionsEmpower the team to implement change18

Tools VSM Value Stream Map documents the high level steps and information flow in a processSOE Sequence of events documents each step Identifies non-value added steps ( waste)Spaghetti DiagramDocuments flow to capture waste of motion 19

Technology Interface What value does the technology add?How does the introduction of technology change the way work gets done?What are the CTQ’s? (Critical to Quality)What are the CCR’s? (Critical Customer Requirements)20

Provider Order Entry Example Signaling of new orders much different in electronic format than paper systemSystem should fit seamlessly with process for care deliveryTheory vs. Practice:Computer on wheels with wireless connectivity poses data loss issues when signal is lost21

Concepts Visual managementFlags (new order notification)Color codingKanban Signal to do work (signature required)Queueing (chemotherapy orders) Takt timePace at which the process must operate to meet demand22

Summary Kaizen ApproachOffers the opportunity for immediate feedback from front-line users of the technology Encourages prompt actionUtilizes a team approachContributes to a culture of continuous improvementCan be useful at any phase of the Information Systems Lifecycle23

Baldrige Quality Award Criteria The Malcolm Baldrige Quality Award stems from an approach and methodology that enables continued excellence through self-assessment.This QI strategy is focused on total organizational improvement and instituting a culture of CQI.24

Baldrige Quality Award Criteria 25

Leading Strategies for CQI (adapted from: http://www.healthit.gov/sites/default/files/continuousqualityimprovementprimer_feb2014.pdf)Strategy Brief Description Type of CQI Initiative IHI Model for Improvement Emphasizes the use of the Plan-Do-Study-Act methodology to establish aims, define the problem, identify measures of success, and systematically test them in short, rapid cycles.   Emphasis on process and outcome. Best for specific problems whose solutions can be refined over time. Is a gradual, incremental approach to CQI. Ideal for achieving small, quick wins; applying lessons learned to new cycles and identifying best practices. Lean Alleviates overburden and inconsistency in processes by eliminating waste, redundancy, and unnecessary effort.   Emphasis is on developing efficient systems that involve whole groups or clusters of related processes.   Is focused on groups or whole categories of related processes. Emphasis on process. Simplifies overcomplicated processes and considers interdependencies. Best for known problems with known system change solution. Integrated throughout the organization or practice. Ideal for large complex health care organizations and practice networks that want to standardize operations across multiple units or practice sites. 26

Leading Strategies for CQI(adapted from: http://www.healthit.gov/sites/default/files/continuousqualityimprovementprimer_feb2014.pdf)Strategy Brief Description Type of CQI Initiative Six Sigma Emphasizes identifying and removing the causes of defects (errors) and minimizing variability in business processes. Uses statistical methods and a hierarchy of users within the organization (Black Belts, Green Belts, etc.) who are experts in these methods. Emphasis on processes and outcomes. Best for processes plagued by wide Is a heavily quantitative approach to CQI. Can be adapted for targeted changes to specific processes. Typically combined with Lean when the focus is on efficiency and quality. Ideal for practices that want to rigorously quantify improvements in safety, quality, and cost effectiveness.   Baldrige Award Criteria Emphasizes identifying problems and setting up teams to take ownership of those problems. Promotes culture of continued excellence via team-based self-assessment in seven criteria areas. Is less focused on the specific steps to achieve improvement. Emphasis on structure and outcomes. Best for practice-wide problem assessment and goal setting. A broad, holistic approach to CQI initiated at strategic times. Ideal for practices that want to establish a new CQI system or overhaul an existing one. 27

Informatics PI Strategies Joint Commission Information Management StandardsStandard IM.01.01.01 – Information Management Planning Standard IM.01.01.03 – Continuity of Information Management Standard IM.02.01.01 – Protective Privacy of Health Information Standard IM.02.01.03 – Security & Integrity of Information 28

Informatics PI Strategies Joint Commission Information Management Standards (cont.)Standard IM.02.02.01 – Collection of Health Information Standard IM.02.02.03 – Retrieval, Dissemination and Transmission of Health Information Standard IM.03.01.01 – Knowledge-based Resources 29

Informatics PI Strategies International Organization for Standardization (ISO)International Organization for Standardization (ISO) is a non-governmental organization that develops and publishes International Standards. The purpose of ISO is to facilitate international trade by providing a set of standards that are recognized and respected worldwide. ISO standards are constantly developed and revised. 30

Informatics PI Strategies ISO (cont.)Various ISO standards for healthcare information systems ISO 9000 series of standards is a set of standards for quality management Examples:ISO 9000:2005 – Covers the basic concepts and vocabularyISO 9001:2008 – Criteria for a quality management system (can be certified) ISO 9004:2009 – Focuses on how to make a quality management system efficient and effective ISO 19011:2011 - Guidance on internal and external audits of quality management systems71

72 Research in Nursing and Healthcare Informatics

Changes in Informatics Research Focus In 1990, Greenes and Shortliffe: …the focus of healthcare informatics is moving from supporting the infrastructure of medicine to meeting the needs of healthcare professionals through supporting education, decision making, communication, and many other aspects of professional activity.... 73

74 Changes in Informatics Research Focus In the 21st century, increasing availability of interactive information accessible to consumers via the Internet Emphasis on prevention, self-managed care, and family members’ involvement. Consumers can use information technology to gain access to information and control their own health care, thereby utilizing healthcare resources more efficiently. Most recent focus, “Meaningful use of EHR!” eHealth and mHealth Big Data

Areas of Interest Standardized terminology Health care information systemsConsumer health informatics (i.e., eHealth / mHealth)InteroperabilityHuman computer interactionPatient safetyEHR/ PHR / CPOE / Bar-coding, etc. Big data Health information exchange 75

Changes in the Context for Nursing Informatics Research for 2008-2018 (Bakken, Stone, & Larson, 2008, 2012)Genomic health careShifting research paradigmsNIH Roadmap for Medical ResearchInterdisciplinary and translational research Beyond comparative effectiveness research, there is a need to build the science of dissemination and implementation.Social (Web 2.0) technologies blogs, wikis, podcasts, Really Simple Syndication (RSS) feeds, social software (e.g., Facebook and MySpace, social networking sites), etc. 76

Changes in the Context for Nursing Informatics Research for 2008-2018(Bakken, Stone, & Larson, 2008, 2012) “ …a nursing informatics agenda for 2008-18 must expand users of interest to include interdisciplinary researchers; build upon the knowledge gained in nursing concept representation to address genomic and environmental data; guide the reengineering of nursing practice; harness new technologies to empower patients and their caregivers for collaborative knowledge development; develop user-configurable software approaches that support complex data visualization, analysis, and predictive modeling; facilitate the development of middle-range nursing informatics theories; and encourage innovative evaluation methodologies that attend to human-computer interface factors and organizational context.” 77

Frameworks and Theories in Research 78

Why Frameworks and Theories? A model or theory can guide the entire research process. It can help: to develop an interventionto identify specific variables needed to be measurede.g., independent and dependent variables, outcome variables, or confounding variables needed to be controlled for or taken into consideration to identify the timing to measure themapplication of findingsBottom line – it helps you stay on target and achieve the research goal based on an established scientific approach. 79

Frameworks and Theories A few selected theories and frameworks applicable in NI researchSystems TheoryLewin’s Basic Stages for Planned ChangeDonobedian’s ModelTechnology Acceptance Model (TAM)Social Cognitive Theory 80

Selected Theories Used in Technology-based Interventions 81

Technology Acceptance Model (TAM) Davis, F. D., Bagozzi, R. P., and Warshaw, P. R. "User Acceptance of Computer Technology: A Comparison of Two Theoretical Models," Management Science, 35, 1989, 982-1003. Developed by F. D. Davis (1989)A model explains how users accept and use a technology.42

Technology Acceptance Model (TAM) Perceived usefulness (PU): "the degree to which a person believes that using a particular system would enhance his or her job performance"Perceived ease of use (PEOU): “the degree to which a person believes that using a particular system would be free from effort" Extended models - TAM 2, TAM 3, Unified Theory of Acceptance and Use of Technology (UTAUT)43

Venkatesh , V., Morris, M.G., Davis, F.D., and Davis, G.B. “User Acceptance of Information Technology: Toward a Unified View,” MIS Quarterly, 27, 2003, 425-478. 44

Diffusion of Innovation Theory Developed by E.M. Rogers in 1962Explains “adoption”A change gains momentum and diffuses through society.Change does not happen simultaneously.Something is done differently than previously.The user perceives the idea, behavior, or product as new and innovative.45

Diffusion of Innovation TheoryAdopter categories – understanding behavior helps to strategize a successful adoptionInnovators – venturesome, take risksEarly adopters – enjoy being leaders, embrace changeEarly majority – willing to adopt, adopt before the average personLate majority – skeptical of changeLaggards – bound by tradition, conservative46

Diffusion of Innovation TheoryFactors that Influence Adoption of an InnovationRelative Advantage CompatibilityComplexity Triability Observability47