Question Writing Guide ABNM SAM Questions - PowerPoint Presentation

Slide1

Question Writing GuideABNM SAM Questions

J. Anthony Parker, MD PhDAssociate Executive Director, ABNMBeth Israel Deaconess Medical CenterBoston, MassachusettsTony_Parker@BIDMC.Harvard.edu

Caveat Lector

Slide2

PsychometricsOrigin for Questions Writing RulesPsychometrics is the field of study concerned with the theory and technique of psychological measurement, which includes the measurement of knowledge, abilities, attitudes, personality traits, and educational measurement. The field is primarily concerned with the construction and validation of measurement instruments such as questionnaires, tests, and personality assessments

. – Wikipedia

Slide3

Test Goal is to Measure AbilityLogistic regression where the probability of a correctanswer is a function of F

n, the ability of person, n bi, the difficulty of question, iProbability

Ability of person , n,

F

n

Difficulty of question, i, bi

Ability 0.3

Wikipedia

Slide4

Can Also Measure Question DifficultyLogistic regression where the probability of a correctanswer is a function of F

n, the ability of person, n bi, the difficulty of question, iProbabilityDifficulty of question, i, bi

Ability of person , n,

F

n

Difficulty 0.3

Wikipedia

Slide5

Question DiscriminationThe slope determines how well a questiondiscriminates between candidate abilities ai

, ability discriminationlog-odds correct answer = ai • (Fn - bi)Wikipedia

Probability

Difficulty of question, i, bi

Ability of person , n,

F

n

= 0.3

Difficulty 0.3

Slope, a

i

Discrimination

Slide6

ABNM ProcedureAssume questions are good and correctly keyedMeasure candidate tentative abilityUsing tentative abilities measure question difficulty Discard questions that are too hard or too easyAlso measure question-ability

discrimination Correct wrong keys Discard questions that not correlated with abilityUsing improved test, measure final candidate abilityAdjust to standard independent of candidates and test

Slide7

Writing Good Questions:Questions that Correlate with AbilityPsychometricians have developed rules for writing good questionsRules are based on question performanceBad questions test abilities unrelated to the test goal

Slide8

Some Rules for Good QuestionsNo negatives (not, except, none)No absolutes (always, never, all, every, only, must)No overlapping rangesNo “all of the above” or “none of the above”No mutually exclusive answers (one is correct)Logical

order (numerical, alphabetical, category)Single concept per question No multiple true / false questions

Slide9

Which of the following is truein Nuclear Medicine?MIBG is used to image somatostatin receptorsA photon is completely absorbed during a

Compton interaction18F-FDG is transported into cells by GLUTLymphoma is never FDG-avidMultiple true / false “which is true?”Multiple unrelated concepts“Never” is an absolute

Slide10

With regard to the thyroid which is correct?

Pentetreotide is taken up in the parafollicular cellsFollicular cells are non-iodine-avid

Stromal cells take up iodide

Pentetreotide is

not

taken by the thyroid

Multiple true /

false “which is correct?”

First and last answers are mutually exclusive

One of them must be correct

“Non” and “not” are negatives

Slide11

In the thyroid, pentetreotide istaken up MOST avidly in:

parafollicular cells (C cells)follicular cellscolloidstroma

Slide12

In the thyroid, pentetreotide istaken up MOST avidly in:

parafollicular cells (C cells)follicular cellscolloidstroma

Slide13

Which of the following is

not correct?11C-acetate is metabolized in the the TCA cycle

11

C-

choline can be used for prostate cancer

18

F-fluorodexoyglucose (FDG

) is

not

phosphorylated

in the cell

18

F-fluorothymidine (FLT) is correlated with cellular

proliferation

Not in stem and in 3rd answerMultiple true / false “which is correct?”Multiple unrelated concepts

Slide14

MOST closely correlatedwith proliferation?

11C-acetate11C-choline18F-fluorodexoyglucose (FDG)18F-fluorothymidine (FLT)

Slide15

MOST closely correlatedwith proliferation?

Good distractors are often partially true, but not the best answer

11

C-acetate

11

C-choline

18

F-fluorodexoyglucose (FDG)

18

F-fluorothymidine (FLT)

Slide16

What is the BEST administered dose to use for a 18F-FDG PET pediatric scan?

≤ 1 MBq/kg (≤ 0.03 mCi/kg)1 - 4 MBq/kg (0.03 - 0.11 mCi/kg)4 - 16 MBq/kg (0.11 - 0.43 mCi/kg)10 - 64 MBq/kg (0.27 - 1.73 mCi/kg)

Overlapping ranges

4-16 and 10-64 both include 10-16

1-4 and 4-16 both include 4

Slide17

What is the BEST administered dose to use for a 18F-FDG PET pediatric scan?

1 MBq/kg (0.03 mCi/kg)4 MBq/kg (0.11 mCi/kg)16 MBq/kg (0.43 mCi/kg)64 MBq/kg (1.73 mCi/kg)

Slide18

What is the BEST administered dose to use for a 18F-FDG PET pediatric scan?

1 MBq/kg (0.03 mCi/kg)4 MBq/kg (0.11 mCi/kg)16 MBq/kg (0.43 mCi/kg)64 MBq/kg (1.73 mCi/kg)

Gelfand

: Pediatric

radiopharmaceutical administered

Doses: 2010 north american consensus guidelines.

J Nucl Med 2011;52:318

Slide19

Parallel Answers / Avoid TeachingAnswers should be parallel in construction and lengthGood test takers know that “teachers like to teach” The correct answer is often the longestTendency to make correct answer exactly correct

For testing, avoid teaching

Slide20

Urinary excretion of FDG is related to:

FDG concentration overwhelms GLUTrenal GLUT has low affinity for FDGboth of the proximal and distal convoluted tubule sodium glucose transporters (SGLT1 & SGLT2

respectively

)

have

low affinity for FDG

there is little hexokinase in the kidney

Asymmetric answers

T

eaching

Slide21

Urinary excretion of FDG is related to:

FDG concentration overwhelms GLUTrenal GLUT has low affinity for FDGSGLT1/2 have low affinity for FDGthere is little hexokinase in the kidney

Slide22

Urinary excretion of FDG is related to:

FDG concentration overwhelms GLUTrenal GLUT has low affinity for FDGSGLT1/2 have low affinity for FDGthere is little hexokinase in the kidney

Slide23

Glucose Reuptake in the Kidney

Slide24

Stem (Question)1. Ask a question Given …, what is the MOST likely diagnosis?2. Give and incomplete statement Given …, the BEST next step is:3. Describe a situation followed #1 or #2

A one-year-old girl had jaundice and a 5 x 6 cm mass in the porta. Total bilirubin is 7 mg/dL. …

Slide25

Stem (Question)Put most of the content in stem Answers shortMove common words in answers to stemRemove unnecessary words or sentences

Slide26

In a 73-year-old man with metastaticmelanoma, what is the MOST likely cause

of the FDG uptake shown in the figure?

prostate cancer

melanoma

normal bladder

normal urethra

Slide27

In a 73-year-old man with metastaticmelanoma, what is the MOST

likely causeof the FDG uptake shown in the figure?

prostate cancer

melanoma

normal bladder

normal urethra

Slide28

Distractors (Incorrect Answers)Should be plausibleLogical misconceptionsCommon misunderstandingsReasonable not the BEST answer not the

MOST likely answer

Slide29

Testing vs. Teaching

Wright Mapcandidatesitemsmore able

l

ess able

hard

easy

Testing

distinguish candidates

difficult items

judgment required

low passing %

Teaching

help learning

easier items

emphasize key points

Slide30

Question TaxonomyRecall (fact) The energy of 99mTc is:Interpretive (diagnosis) A patient with … had an FDG-PET/CT shown

in the figure. The MOST likely diagnosis is:Problem solving (management, judgment) A patient with … had an FDG-PET/CT shown in the figure. The BEST next step is:

Slide31

Higher Taxonomy PreferredOften involve more advanced reasoning More complex medical task Still single clinical conceptDistractors may be easier to writeSome topics require lower taxonomy questions

Slide32

Stable iodine (SSKI or Lugol’s solution) should be administered prior to:

pentetreotide (Octreoscan®)iobenguane (MIBG)a contrast FDG-PET/CTibritumomab tiuxetan (Zevalin®)

Slide33

Stable iodine (SSKI or Lugol’s solution) should be administered prior to:

pentetreotide (Octreoscan®)iobenguane (MIBG)a contrast FDG-PET/CTibritumomab tiuxetan (Zevalin®)

Recall question

p

entetreotide confused with MIBG

i

britumomab tiuxetan confused with tositumomab

Slide34

Based on the electrocardiogram, whatis the BEST diagnosis?

Difficult to come up with 4 plausible diagnoses

Slide35

A 45 y/o diabetic man with weakness and shortness of breath for 3 days presents for exercise MPI. The pre-exercise ECG is shown.

Slide36

What should be done next?

Obtain right precordial leads.Use dipyridamole stress instead.Accompany the patient to the EW.Proceed as planned.

Slide37

Slide38

What should be done next?

Obtain right precordial leads.Use dipyridamole stress instead.Accompany the patient to the EW.Proceed as planned.

Slide39

Writing DistractorsDifficulty writing reasonable distractors Change taxonomyManagement distractors often easier to write

Slide40

The BEST diagnosis is:

Slide41

The BEST diagnosis is:

normal thymus

mediastinal metastases

?

?

Diagnostic question

Slide42

The uptake in the chest is related to:

Slide43

The uptake in the chest is related to:

the Na / I symporter

somatostatin receptor, type 2

brown fat

metastases

Diagnosis & biodistribution

Slide44

A syringe of 90Y-ibritumomab tiuxetan (Zevalin ®) measuring 1025 MBq (27.7 mCi) needs to be discarded due to QC failure. After transfer to a vial, measurements are vial: 815 MBq (22.0 mCi), syringe: 15 MBq (0.4 mCi). What should be done next?

Declare a medical eventSurvey the areaCall the radiation safety officer (RSO)Store for decay

Slide45

A syringe of 90Y-ibritumomab tiuxetan (Zevalin ®) measuring 1025 MBq (27.7 mCi) needs to be discarded due to QC failure. After transfer to a vial, measurements are vial: 815 MBq (22.0 mCi), syringe: 15 MBq (0.4 mCi). What should be done next?

Declare a medical eventSurvey the areaCall the radiation safety officer (RSO)Store for decay

Slide46

Which Isotope is MOST Difficult toAccurately Measure in a Dose Calibrator?18F90Y99mTc111In

Slide47

Which Isotope is MOST Difficult toAccurately Measure in a Dose Calibrator?18F90Y99mTc

111In

Slide48

What radiopharmaceutical was administered?

99m

Tc-pertechnetate

67

Ga-citrate

111

In-pentetreotide

123

I-MIBG

Slide49

What radiopharmaceutical was administered?

99m

Tc-pertechnetate

67

Ga-citrate

111

In-pentetreotide

123

I-MIBG

Slide50

What’s the Radiopharmaceutical?

Stupid Questions, but I Use ThemShould know normal and common variant biodistribution at organ, cell, and receptor levelTo be able to identify abnormal biodistribution

identify radiopharmaceutical QC

understand unusual findings

Slide51

After injection of 111In-DTPA in a patient

with normal renal function?the half-time of excretion is 16 minutesthe half-time of excretion

8

minutes

the half-time of excretion

1.7

hours

the half-time of excretion

36 minutes

Repeated words in answers

Slide52

How long after injection of 111In-DTPA is

1/2 excreted with normal renal function?16 minutes8 minutes1.7 hours

36 minutes

Answers not in logical order

Slide53

How long after injection of 111In-DTPA is

1/2 excreted with normal renal function?8 minutes16 minutes36 minutes

1.7 hours

Slide54

How long after injection of 111In-DTPA is

1/2 excreted with normal renal function?8 minutes16 minutes

36 minutes

1.7 hours

http://www.doseinfo-radar.com/

Renogram

Slide55

Some Rules for Good Questions (1/2)No negatives (not, except, none)No absolutes (always, never, all, every, only, must)No overlapping rangesNo “all of the above” or “none of the above”No mutually exclusive answers (one is correct)

Logical order (numerical, alphabetical, category)Symmetric answers

Slide56

Some Rules for Good Questions (2/2)Put concepts and common words in stemDistractors should be reasonableTry to use higher taxonomy Often easier to write distractors Some fact questions are appropriate

Single concept per question No multiple true / false questions Medically, not question, complexity

Slide57

Reference Guidelines3-page guide ABNM_Well_Formatted_SAM_Question.docx1-page list of does and don’ts ABNM_QuestionWritingDoesAndDonts.docx26-page guide (ABNM psychometric

consultants) http://www.measurementresearch.com/24-page guide ABR_Item_Writers_Guide.pdf

Slide58

Applause