Biostatistical Design Considerations for community engaged research - PowerPoint Presentation

Slide1

Biostatistical Design Considerations for community engaged research

Chris Wichman, PhD

Department of Biostatistics

College of Public Health

University of Nebraska Medical Center

Slide2

Topic Outline

Traditional Research Design Concepts

Design Changes to Accommodate Community Based Research

Outcomes and Interpretation

Slide3

Traditional Research design concepts

Slide4

Definitions

Study / Experimental Unit – smallest unit to which study conditions are independently applied.

Observational Unit – smallest unit on which measurements are taken

Condition / Treatment Design – how are study conditions organized

Study / Experimental Design – how are experimental units organized

Slide5

Goal

Utilize a design that:

addresses study intent;

minimizes bias;

addresses appropriate questions about treatments / conditions of interest

Slide6

Study intent

Must be tied directly to research question and resulting hypotheses.

Comparative Effectiveness – two-sided hypotheses

Superiority (inferiority) – one-sided hypotheses

Non-inferiority – equivalence tests

Quality Improvement – can use any type of hypothesis, but not required

Feasibility – focus on ability to conduct the study / implement the process

Sustainability – longitudinal – can gains made be maintained

Slide7

Condition / treatment designs

Single factor, two or more levels

Multi-factor with main effects

Multi-factor with interactions

Ex. 4 factor levels, interested in mean

of each level

Ex. Factor 1: 2 levels; factor 2: 3 levels

interested in means of factor 1 levels

Without regard to factor 2

and vice versa

Ex. Factor 1: 2 levels; factor 2: 3 levels

interested in the effect of factor 1 &

factor 2, allowing for a different effect

of one factor considering the other.

Slide8

Study designs

How the subjects are arranged relative to factor-level combinations.

Researcher specifies the design

Examples:

Completely Randomized

Blocked

Pre – Post

Repeated Measures

Nested Design

Split / Crossover Design

Slide9

Minimizing bias

Confounding

Randomization

Blocking

Stratification

Balance

Slide10

Confounding variables

Definition 1: “Two variables are confounded if they appear in such a pattern that their separate effects cannot be distinguished”.[Moses, 2009].

Definition 2: – Confounding variables are those that may compete with the exposure of interest (

eg

, treatment) in explaining the outcome of a study”. [Skelly, et al, 2012].

“Confounding factors may mask an actual association or, more commonly, falsely demonstrate an apparent association between the treatment and outcome when no real association between them exists”[Skelly et al, 2012].

Slide11

Randomization

Randomly assign study conditions (treatments) to subjects

May need / want to do this by strata to balance confounders

1, 2, 3, 4, 5, 6, 7, 8,9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20

Subjects

1, 2, 3, 5, 7, 9, 10, 12, 15, 19

4, 6, 8,11, 13, 14, 16, 17, 18, 20

Condition 1

Condition 2

Slide12

Blocking

…

Block 1

Block B

Block – Groupings of subjects that are alike

Ex. Domestic Violence CBOs – Blocking would be done on service type: crisis, shelter, support group, etc.

Replicate the base study within each block at least once (all conditions present at least once)

Block can be modeled as a factor with B levels

Slide13

Stratification & balance

Stratification

Natural groupings in which we want to ensure sufficient representation

When building a sample of participants, want to make sure sex, age, education, race / ethnicity of the target population is represented.

Analytically, subgroup analyses – additional way of dealing with confounders

Balance

“Equal” representation under each study condition

Slide14

design changes to accommodate Community based research

Slide15

Design Changes

Larger in scale – more than one site

Sites often become the study / experimental unit

Individuals at a given site are units of observation

Site outcomes are an aggregation of measures on OUs.

Slide16

Sample size Determining factors

Depends on the statistical hypothesis to be tested, study design, type of primary outcome, number of groups, type of analysis that will be used

Need preliminary/pilot data to estimate effect size, a measure of the expected differences between the groups.

Balance between type I and type II errors

Justice System

- Trial

Innocent

Guilty

Guilty Verdict

Type I Error

Correct

Not Guilty Verdict

Correct

Type II Error

Statistics

– Hypothesis Test

Null True

Null

False

Reject Null

Type I Error

(False positive)

Correct (Power)

Fail to RejectCorrectType II Error(False negative)

𝛼

𝛽

Slide17

Power and sample size

Given the effect to be measured, the uncertainty in the measurement, and the allowable type I error rate:

what sample size do I need to achieve 80% power to reject the null hypothesis when the null hypothesis is wrong?

Slide18

Sample size analysis

Less straight forward than in traditional studies

Must account for clustering

Process balances number of clusters (EUs) and number of OUs

Requires researchers to have a good understanding of numbers per cluster

Seasonal / time effects (random effect of time)

Cluster type effects

Local issues

Slide19

Biggest Issue to Understand

Power cannot be increased by an increase in observational units alone

At some point, you must increase the number of clusters

Example: Dichotomous outcome, 9 clusters available, each cluster can support 320 to 360 observational units. Effect size to be detected is moderately large (0.72)

Slide20

Desired Design

9 clusters

320 – 360 subjects per cluster

Dichotomous outcome effect size = 0.72

Minimum power = 0.80

Slide21

Optional Design

Use original cluster as a design block

Cluster on “practitioner” w/in block

Average of 4 clusters per block

Average of 80-90 subjects per cluster

Slide22

Cluster vs block

Cluster

Blocking

CBO1

CBO9

CBO9

CBO1

Slide23

Failing to account for clustering

Unexplained variance is underestimated

Type I error is inflated (false importance)

Possible Simpson’s Paradox Issues

Conclusions change when the clustering is accounted for compared to when it is ignored

Slide24

Model Based analyses of clustered studies

Requires mixed models (fixed and random effects)

Accounts for correlation, random time and cluster effects

Interpretation of results are less straight forward

Especially for observational unit level effects

Slide25

Typical Outcomes FOR CBR

Slide26

Outcome and measurement level

Reach – measured on the cluster level

Effectiveness – measured on the cluster level

Implementation – measured on the cluster level

Improvement – measured on the cluster and / or observational levels

Maintenance – measured on the cluster and / or observational levels

Slide27

Reach

Definition of eligible

All comers?

New clients only (naïve to the program)

Previous clients meeting specific criteria

Based on type of interaction (subset of all comers)

Definition for number referred

Need to define a failure

Certain number of encounters before failure

Extenuating Circumstances

Acute / crisis vs normal encounter

 

Slide28

Effectiveness

Definition of participation

Show up once

75% or more

etc

 

Slide29

improvement

At the cluster level – proportion

At the observational level – proportion or numerical

Improvement could be defined as:

% change in a targeted outcome

Difference scores

Specific time point comparison between groups

 

Slide30

maintenance

Utilizes the same outcome measure as improvement

Requires a longitudinal study to address

PI / research team need to define

apriori

what maintenance means (equivalence window)

By hypothesis, this should really be a designed equivalence test

Issue, typically comes about after a comparison period (primary outcome for power)

Solution – use post-hoc (1-2

α

) confidence interval

Want the interval to contain 0

Slide31

implementation

Adherence to the Protocol =

Dosing

Automatic reminders

Automatic reminders + small group sessions

Automatic reminders + small group sessions + incentive

Consistency

Do process and / or procedures change based on time, day, or contact type?

Adherence to core principles

Are the expectations of the process / protocol adhered to

 

Slide32

Conclusion

Traditional Study Design Principles carry over to Community Based Research

Study units are typically at least one level above the observational unit

Sample size for clustered trials must strike a balance between number of clusters and number of observational units

Studies are typically designed to get at cluster level outcomes

Observational unit level performance can be captured – just harder to implement and interpret

Slide33

Questions?

Slide34

references

Moses, LE. Statistical Concepts Fundamental to Investigations. Chapter 1: Medical Uses of Statistics, 3

rd

Ed. Editors:

Bailar

III, JC and

Hoaglin

, DC. Wiley & Sons, Hoboken, NJ 2009.

Skelly AC,

Dettori

JR, Brodt ED. Assessing bias: the importance of considering confounding.

Evid Based Spine Care J

. 2012;3(1):9–12. doi:10.1055/s-0031-1298595