Part 2 of 3 By: Danielle Davidov, PhD & - PowerPoint Presentation

Slide1

Part 2 of 3 By: Danielle Davidov, PhD &Steve Davis, MSW, MPA

Introduction to Research:

Measurement

Slide2

1) Threats to research studies 2) Steps in the research design process 3) Identifying and defining variables 4) Validity and reliability of measurement

OUTLINE

Slide3

Starts After the research question has been developed and refined The who, what, when, where, and how of

research

It comprises the

Materials and Methods

and Limitations sections of publications

Research Design

Slide4

The goal of research design is to provide the most valid and correct answer to the questioni.e., we want to make sure we are “doing it right”This is done by minimizing the threats

to the soundness of your study’s conclusion(s):

CHANCE

BIAS

CONFOUNDING

Why is design important?

Slide5

The threat that the study’s findings are merely the result of random processes (chance)i.e., the findings are a “fluke”

We can’t do much to control random error

Also referred to as:

Type 1 Error

Random Error Unsystematic Error

Study Threats: Chance

Slide6

The threat that the study’s results are due to an unfair preference given to one group or a set of outcomes in a studyWe can try to control bias in our study design and subject recruitment Also referred to as:

Systematic Error

Study Threats: Bias

Slide7

The threat that the association or relationship observed in the study is influenced by or related to another variableWe can control for this in our study design, subject recruitment, and data analysis techniques

Study Threats: Confounding

Slide8

We try to minimize the three main threats during all stages of design process, which are:1) Identifying and Defining Variables*2) Selecting Measurement Methods*3) Selecting (Sample) Subjects

4) Selecting a Research Design

5) Establishing an Analysis Plan

*We

will be talking about steps 1 and 2 in this presentation

Steps in Research Design

Slide9

What do you want to measure? (Identify) Ex) Patient satisfaction levels with ultrasound vs. history

and physical

exam only

How do you want to operationalize “patient satisfaction?” (Define)

Ex)

Answers of “Good”, “Very Good”, or “Excellent” on a survey given to patients about the care they received in the emergency department

Identify & Define variables

Slide10

Classifying Variables: Independent VariableThe variable that has an effect on or influences the dependent variable. This is the FACTOR/INTERVENTIONi.e.)

History and Physical Exam or Ultrasound + H & P

Dependent

Variable

The variable that is affected

by, or dependent upon, the independent variable. This is the OUTCOME

i.e.)

Patient Satisfaction

Identification & Definition of Variables

Slide11

Classifying Variables (continued)Confounding Variables –a variable that is related to both the independent and the dependent variableCONFOUNDER or CONTROL variable Common confounders/controls in medical research:

Age

Gender

Race

Severity of Illness

Identification & Definition of Variables

Slide12

Controlling for Confounding Variables Not adequately controlling for confounding variables can have disastrous consequences on your research Identify and define as many as possibleFrom previous literatureFrom clinical observations

From theory

Identification & Definition of Variables

What if we didn’t consider these important variables when examining the relationship between the Independent and Dependent variables???

Slide13

Operationalizing variablesThe process of defining variables in a measurable way.Identification & Definition of Variables

Slide14

Levels of Measurement (NOIR)Nominal

Ordinal

Interval

Ratio

Identification & Definition of Variables

Lower

Higher

Slide15

Characteristic data that cannot be rank ordered This data is “categorical” – made up of “categories”, not “levels” or “increments” Ex) Ice cream flavors—vanilla is not “better” or “more” than chocolate

Examples

:

Gender

, Race, Student, Marital Status, State or Country of Residence,

Insurance Status, Discharge Status, etc.

Yes/No Responses are Nominal

This type of data is usually “descriptive”

Used to describe a population or sample

Nominal Level Data

Slide16

Data that can be rank ordered but that do not have measurable distances between each level of rank Likert Scales - Strongly Disagree to Strongly AgreeClass rank - Freshman, Sophomore, Junior, Senior

; PGY-I

, PGY-II, PGY-III Degree of

illness: None

, Mild, Moderate, Severe

Senior is a higher rank

than Freshman

, but there

is

no way to

quantify

how

much

higher Senior

is vs. “

Freshman” or how much “more” illness

those

with a severe illness have compared to

those

with a mild illness

Ordinal Level Data

Slide17

Data that can be ordered and that have a measurable distance between each levelThe Interval Scale - Distances between positions are equal, but "0" is an arbitrary assignment. For example, with temperature, each degree is equally distant from another, but "0" does not mean that there is no temperature. It is simply a reference point on the

scale.

The Ratio Scale

- All positions are equally distant and "0" means that the value is truly "0". If you have "0" money, you have none. But if you have $200, you have twice as much as a friend who has $100.

Examples of Interval/Ratio Data: AgeHeight

Weight

Many

Clinical Serum

Levels

Blood Pressure

Interval/Ratio Data

Slide18

Defining variables at higher levels of measurement allows the use of statistical tests that have more PowerPower = the probability of finding a true relationship of difference if it genuinely existsIt is usually better to collect data at higher levels of measurement and then collapse into categories later

Ex) Age

What is your age? ____

(best)

vs. What is your age? vs. 18 – 25

26 – 35 36 – 45 etc.

vs.

Under

40 & over 40

Levels of Measurement and Power

Slide19

Once you have defined and operationalized your research question’s variables, you must decide how to measure them and/or what measurement tool you will use. There are two forms of error that we must minimize when selecting measurement methods and/or tools:Random error (CHANCE)Nonrandom error (BIAS AND CONFOUNDING)

Selecting a Measurement Method

Slide20

To minimize random error we choose a tool or method that is RELIABLE Reliability – The extent to which a measurement method or tool produces the same results over several measurements AKA

precision

Threats to Reliability

Observer error

: different measurements from the same or different observers (i.e., blood pressure readings) Instrument error: different measurements from the instrument itself due to extraneous environmental factors

Subject error: different

measurements from the natural biological variability among humans

RELIABILITY

Slide21

How to assess Reliability:Repeat measurements on the same subject.Give a survey at two different time pointsTake blood pressure at two different time points

Use more than one observer.

Assess inter-rater agreement

Have two different people take blood pressure

How to maximize Reliability: Standardize the measurement methods

Choose surveys and instruments that have been proven to be reliable

Train observers

Refine & update

instruments

Repetition

Averaging the measures can cancel out error.

Assessing & Maximizing reliability

Slide22

To minimize nonrandom error we choose a measurement method and/or tool that is VALIDValidity – The extent to which a measurement method and/or tool measures what is sets out to measure AKA Accuracy

Threats to Validity

Observer bias

:

conscious or unconscious distortion in the perception and/or reporting of the measurementSubject bias

: bias-distortion of self-reported measurements due to subjects beliefs and biases

Hawthorne Effects

and Social

Desirability

Instrument bias:

consistently

biased or

inaccurate

measurements due to such things as worn parts or mechanical

malfunction

Lack

of a clear gold

standard

: No “best” instrument out there

Abstract/behavioral variables

: These things are difficult to measure

Patient

satisfaction, pain, quality of life, intelligence

validity

Slide23

Strategies for maximizing ValidityBlinding Ex) Do not allow physician who is taking blood pressure readings to know which subjects are receiving blood pressure medication

Deception

Ex) Do not allow subjects to know which “group” they are in

Give placebos

Instrument

Calibration

Make sure instruments are working properly

Use standardized/validated surveys and assessment

tools

Find these from literature searches

Usually better to use “pre-made” surveys or instruments than creating one from scratch

Maximizing Validity

Slide24

Identify and define your variables at the VERY beginning of your study Don’t forget your control or confounding variables!Using higher levels of measurement is better! Choose instruments and data collection tools that are: RELIABLE – produce the same results over time (precise)

VALID

– produce results that represent “the truth” (accurate)

In Summary

Slide25

Go through “Introduction to Research Part 3: Sampling and Design” Next steps

Slide26

Hulley SB, Cummings SR, Browner WS, Grady D, Hearst N, Newman TB. Designing Clinical Research. 2nd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2001:37-49

Spector

PE. Research Designs

. Newbury Park, CA: SAGE Publications, Inc.; 1981. ISBN: 0-8039-1709-0

http://www.research-assessment-adviser.com/levels-of-measurement.html

references