a s measures of text structure and of reading comprehension May 14 2012 BSI Nijmegen Nederland Roy Clariana RClarianapsuedu Clariana RB 2010 Multidecision approaches for eliciting knowledge structure In D Ifenthaler P PirnayDummer amp NM Seel Eds ID: 725854
Download Presentation The PPT/PDF document "1 Alternative measures of knowledge str..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
1
Alternative measures of knowledge structure: as measures of text structureand of reading comprehensionMay 14, 2012BSI Nijmegen, NederlandRoy ClarianaRClariana@psu.edu
Clariana, R.B. (2010). Multi-decision approaches for eliciting knowledge structure. In D. Ifenthaler, P. Pirnay-Dummer, & N.M. Seel (Eds.),
Computer-Based Diagnostics and Systematic Analysis of Knowledge
(Chapter 4, pp. 41-59). New York, NY: Springer.
linkSlide2
Overview
IntroductionI am an instructional designer and a connectionist, so my language may be a little different, also slow me down if my accent is difficultMy intent today is to describe my research on several approaches for measuring Knowledge Structure (KS) and along the way, describe tools, and maybe show extra ways of thinking about text, knowledge, comprehension, and learning2Slide3
KS: Encompassing theoretical positions
Cognitive structures (de Jong & Ferguson-Hessler, 1986; Fenker, 1975; Korz & Schulz, 2010; Naveh-Benjamin, McKeachie, Lin, & Tucker, 1986; Shavelson, 1972)Conceptual networks (Goldsmith et al., 1991)Conceptual representations (Geeslin & Shavelson, 1975; Novick & Hmelo, 1994); (McKeithen, Reitman, Rueter, & Hirtle, 1981)C
onceptual
structures
(
Geeslin
&
Shavelson, 1975; Novick & Hmelo, 1994) Knowledge organization and knowledge structures (McKeithen et al., 1981)Semantic structures (Gentner, 1983; Riddoch & Humphreys, 1999).
3Slide4
KS: Encompassing theoretical positions
Spatial knowledge (de Jong & Ferguson-Hessler, 1996; Dunbar & Joffe, 1997; Jee, Gentner, Forbus, Sageman, & Uttal, 2009; Korz & Schulz, 2010; Schuldes, Boland, Roth, Strube, Krömker, & Frank, 2011) Categorical knowledge (Candidi, Vicario, Abreu, & Aglioti, 2010; Matsuka, Yamauchi, Hanson, & Hanson, 2005; Stone & Valentine, 2007; Wang, Rong, & Yu, 2008) Conceptual knowledge
(de
Jong & Ferguson-
Hessler
, 1996; Edwards, 1993;
Gallese
& Lakoff, 2005; Hallett, Nunes, & Bryant, 2010; Rittle-Johnson & Star, 2009) 4Slide5
KS: My sandbox model
Our symbolic connectionist view:Knowledge structure (or structural knowledge) refers to how information elements are organized, in people and in artifactsA departure from most theories, we propose that knowledge structure is pre-propositional, but that KS is the precursor of meaningful expression and the underpinning of thoughtSaid differently, knowledge structure is the mental lexicon that consists of weighted associations (that can be represented as vectors) between knowledge elements5Slide6
KS is worth measuring
Measures of content knowledge structure have been empirically and theoretically related to memory, classroom learning, insight, category judgment, rhyme, novice-to-expert transition (Nash, Bravaco, & Simonson, 2006) and reading comprehension (Britton & Gulgoz, 1991; Guthrie, Wigfield, Barbosa, Perencevich, Taboada, Davis, Scafiddi, & Tonks, 2004; Ozgungor & Guthrie, 2004), and
And findings
for combining individual
knowledge structures
to form
group mental models
(Cureeu, P.L., Schalk, R., & Schruijer, S., 2010; DeChurch & Mesmer-Magnus, 2010; Johnson & O’Connor, 2008; Mohammed, Ferzandi, & Hamilton, 2010; Pirnay-Dummer, Ifenthaler, & Spector, 2010). 6Slide7
Applied to reading comprehension, KS as a measure of the situation model
Ferstl & Kintsch (1999)Textbase (the text’s semantic content and structure, van Dijk & Kintsch, 1983)Situation model (the integration of the ‘episodic’ text memory with prior domain knowledge, van Dijk & Kintsch, 1983); also called mental model of the text, the text model, the discourse model7Ferstl, E.C., & Kintsch, W. (1999). Learning from text: structural knowledge assessment in the study of discourse comprehension. In van Oostendorp and Goldman (eds.),
The construction of mental representations during reading
. Mahwah, NJ: Lawrence
Earlbaum
.Slide8
Visually
8Ferstl, E.C., & Kintsch, W. (1999). Learning from text: structural knowledge assessment in the study of discourse comprehension. In van Oostendorp and Goldman (eds.), The construction of mental representations during reading. Mahwah, NJ: Lawrence Earlbaum.
Text base
Updated situation model
(post list recall)
Situation model
(pre list recall)Slide9
A KS measure of the
situation modelFerstl & Kintsch (1999) used pre-and-post-reading list-cued partially-free recall to elicit KS of the birthday story (which obtains asymmetric matrices)Participants – 42 undergraduate students (CU Boulder)Pre-reading cued-association KS task: Students were presented by computer a 60 word list of birthday-related terms to view one at a time (randomized), and then were given the list on paper with 3 blanks beside each list term and were asked to write in the 3 terms from the list that come to mindReading: Students then read the 600-word long birthday storyPost-reading cued-association KS task: i.e., same as pre-task, fill in the list9Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage.
International Journal of Instructional Media, 35
(2), 229-236
.Slide10
Results
Established that the KS cued association paradigm was appropriate for assessing background knowledge and text memoryThis KS approach facilitated interpretation, depicting how the text ‘added to’ the post reading situation model (see their figure 10.4, p.260); provided a different or other way to think about reading comprehension (p.268)Test-retest reliability may be a problem for this KS approach10Ferstl, E.C., & Kintsch, W. (1999). Learning from text: structural knowledge assessment in the study of discourse comprehension. In van Oostendorp and Goldman (eds.), The construction of mental representations during reading. Mahwah, NJ: Lawrence Earlbaum.Slide11
Another KS measure of the
text base (or situation model?)Clariana & Koul (2008), we asked students to draw concept maps (KS) of a textParticipants – 16 graduate students in a science instructional methods course (Penn State GV)First, students discussed concept maps in classThen working in dyads (8 pairs), students were given a 255 word passage on the heart and circulatory system and were asked to create a concept map of itKS data sources 8 dyad concept maps of the text1 expert concept map of the textA Pathfinder network (PFNet) map of the text automatically formed by ALA-Reader software11
Clariana
, R.B., &
Koul
, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage.
International Journal of Instructional Media, 35
(2), 229-236.Slide12
Data
26 terms identified across all of the maps and text(Text concept map), dyads’ concept map link lines entered into a 26 x 26 half matrixMatrix analyzed using Pathfinder Knot12
Link Array
a
b
c
d
e
f
g
a left atrium
-
b lungs
0
-
c oxygenate
0
1
-
d pulmonary artery
0
1
0
-
e pulmonary vein
1
1
0
0
-
f deoxgenate
0
1
0
0
0
-
g right ventricle
0
0
0
1
0
0
-
(n
2
-n)/2 pair-wise comparisons
Clariana
, R.B., &
Koul
, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage.
International Journal of Instructional Media, 35
(2), 229-236
.Slide13
Data as percent overlap
Percent overlap was calculated as links in common divided by the average total links132
5
4
4
% overlap =
4
/ ((6+8)/2) % overlap =
4
/
7
%
overlap =
57%
e.g., Dyad
PFNet
e.g., Expert
PFNet
Clariana
, R.B., &
Koul
, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage.
International Journal of Instructional Media, 35
(2), 229-236
.Slide14
Data as percent overlap
14In the epigraph to Educational Psychology: A Cognitive View, Ausubel (1968) says, “The most important single factor influencing learning is what the learner already knows.”An aspect of measurement reliability and validity
low
good
ALA-Reader
Clariana
, R.B., &
Koul
, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage.
International Journal of Instructional Media, 35
(2), 229-236
.Slide15
The strong influence of
prior domain knowledgeFigure 3. The relationship between the number of propositions in the dyad concept maps and the average percent agreement with the 255-word text passage (* shows dyads with a science major).15Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35
(2), 229-236
.
Only those with prior domain knowledge could adequately ‘capture’ the textSlide16
ALA-Reader papers
ALA-Reader converts text KSClariana, R.B., Wallace, P.E., & Godshalk, V.M. (2009). Deriving and measuring group knowledge structure from essays: The effects of anaphoric reference. Educational Technology Research and Development, 57, 725-737. Clariana, R.B., & Wallace, P. E. (2007). A computer-based approach for deriving and measuring individual and team knowledge structure from essay questions. Journal of Educational Computing Research, 37 (3), 209-225.Koul, R., Clariana, R.B., & Salehi, R. (2005). Comparing several human and computer-based methods for scoring concept maps and essays. Journal of Educational Computing Research, 32 (3), 261-273. Clariana, R.B. (2010). Deriving group knowledge structure from semantic maps and from essays. In D. Ifenthaler, P. Pirnay-Dummer, & N.M. Seel (Eds.),
Computer-Based Diagnostics and Systematic Analysis of Knowledge
(Chapter 7, pp. 117-130). New York, NY: Springer
.
16
Also see
HIMAT/DEEP software and Hamlet softwareSlide17
KS for influencing learning
e.g., Trumpower et al. (2010) used knowledge structure of computer programming represented as network graphs to pinpoint knowledge gaps17KS elicited as pair-wise comparisons and data-reduced to networks using Pathfinder KNOTLearners’ networks then compared to an expert referent network
Trumpower
, D.L.,
Sharara
, H., & Goldsmith, T.E. (2010). Specificity of
Structural Assessment
of Knowledge. Journal of Technology, Learning, and Assessment, 8(5). Retrieved from http://www.jtla.org.Slide18
KS for influencing learning
The problems were intended to be complex enough so that the solution depended on integration of several interrelated concepts (relational)18The presence of subsets of links in participants’ PFnets differentially predicted performance on two types of problems, thereby providing evidence of the specificity of knowledge structure
Trumpower
, D.L.,
Sharara
, H., & Goldsmith, T.E. (2010). Specificity of
Structural Assessment
of Knowledge. Journal of Technology, Learning, and Assessment, 8(5). Retrieved from http://www.jtla.org.Slide19
Protein structure as an analogy
of knowledge structure in reading comprehensionChristian Anfinsen received the Nobel Prize in Chemistry in 1972: Linear sequence of amino acids enzyme structure enzyme functionIs like:Linear sequence of words in a text knowledge structure retrieval function19Slide20
AA Linear sequence
enzyme structure function
20
APRKFFVGGNWKMNGKRKSLGELIHTLDGAKLSADTEVVCGAPSIYLDFARQKLDAKIGVAAQNCYKVPKGAFTGEISPAMIKDIGAAWVILGHSERRHVFGESDELIGQKVAHALAEGLGVIACIGEKLDEREAGITEKVVFQETKAIADNVKDWSKVVLAYEPVWAIGTGKTATPQQAQEVHEKLRGWLKTHVSDAVAVQSRIIYGGSVTGGNCKELASQHDVDGFLVGGASLKPEFVDIINAKH
Triose Phosphate
Isomerase
:
http://www.cs.wustl.edu/~taoju/research/shapematch-final.pdfSlide21
Read linear sequence of words in text
21Hyona, J., & Lorch, R.F. (2004). Effects of topic headings on text processing: evidence from adult readers’ eye fixation patterns.
Learning
and
Instruction, 14
, 131–152.
Figure 1, p.136Slide22
Knowledge structure
22
Retrieval function
A
B (propositional knowledge):
Where do pandas live?
In the wild
A
B,C,D (relational knowledge):
What do we know about pandas today
? Pandas are heading towards extinction in the wild due to climate change
Retrieval structure
linearSlide23
Read
KS Retrieval function23
Relational
Retrieval structure
Retrieval function
A
B (propositional knowledge):
Where do pandas live?
In the wild
A
B,C,D (relational knowledge):
What do we know about pandas today
? Pandas are heading towards extinction in the wild due to climate changeSlide24
Summary of the introduction
KS cuts across theories, we support connectionist viewsKS is worth measuring, it correlates with many kinds of performanceKS can be measured in different waysKS has been used to visually represent the reading comprehension situation modelKS has been used to visually represent the text structureSpecific KS structure leads to specific cognitive performanceEnzyme Analogy: linear chain structure function24Slide25
Measuring knowledge structure
My foundation and trajectory for measuring KS:Vygotsky (in Luria, 1979); Miller (1969) card-sorting approaches Deese’s (1965) ideas on the structure of association in language and thought Kintsch and Landauer’s ideas on representing text structure, and latent semantic analysisRecent neural network representations (e.g., Elman, 1995)Jonassen, Beissner, and Yacci (1993) 25Slide26
written text
similarity
ratings
free
recall
concept maps
Clariana
&
Koul
, 2008
Ferstl
&
Kintsch
,
1999
Trumpower
,
Sharara
,
& Goldsmith,
2010
Dave
Jonassen’s
summary
o
f KS measures…
Knowledge
representation
Knowledge
comparison
Knowledge
elicitation
Jonassen, Beissner, & Yacci (1993), page 22
26
Elicit responses
represent responses compare responseSlide27
Dave
Jonassen’s summary …
graph
building
similarity
ratings
semantic
proximity
word
associations
card
sort
ordered
recall
free
recall
additive
trees
hierarchical
clustering
ordered
trees
minimum
spanning
trees
link
weighted
Pathfinder
nets
Networks
Dimensional
principal
components
MDS – multidimensional scaling
cluster
analysis
expert/
novice
qualitative
graph
comparisons
quantitative
graph
comparisons
relatedness
coefficients
scaling
solutions
C of
PFNets
Trees
Knowledge
representation
Knowledge
comparison
Knowledge
elicitation
Jonassen
,
Beissner
, &
Yacci
(1993), page 22
27
To show different KR
l
et’s do an example …
concept maps
written textSlide28
Knowledge Representation (KR)
Multidimensional scaling (MDS) - Family of distance and scalar-product (factor) models. Re-scales a set of dis/similarity data into distances and produces the low-dimensional configuration that generated them(e.g., see: http://www.tonycoxon.com/EssexSummerSchool/MDS-whynot.pdf)Pathfinder Knowledge Network Organizing Tool (KNOT) algorithms take estimates of the proximities between pairs of items as input and define a network representation of the items. The network (a PFNET) consists of the items as nodes and a set of links (which may be either directed or undirected for symmetrical or non-symmetrical proximity estimates) connecting pairs of the nodes. (See: http://interlinkinc.net/KNOT.html)28Slide29
Pathfinder Network (PFNet
) analysisPathfinder seeks the least weighted path to connect all terms, shoots for n-1 links if possiblePathfinder is a mathematical approach for representing and comparing networks, see: http://interlinkinc.net/index.htmlPathfinder data reduction is based on the least weighted path between nodes (terms), so for example, Deese’s 171 data points become 18 data points. Only the salient or important data is retained.Pathfinder PFNet uses, for example:Library reference analysisUse google to search to see many more examples of how Pathfinder can be used
29
Note that
Ferstl
&
Kintsch
(1999) used PathfinderSlide30
Deese (1965), free recall data (p.56)
30
Deese, J. (1965).
The structure of associations in language and thought
. Baltimore, MD: John Hopkins Press, page 56
Full array (n * n): 19 x 19 = 361
Half array ((n
2
– n)/2): ((19 x 19) –19 )/2 = 171
100 participants are shown a list of related words, one at a time, and asked to free recall a related termSlide31
Deese (1965), free recall data (p.56)
Deese, J. (1965).
The structure of associations in language and thought
. Baltimore, MD: John Hopkins Press, page 56
Full array (n * n): 19 x 19 = 361
Half array ((n
2
– n)/2): ((19 x 19) –19 )/2 = 171
31Slide32
Using MDS in SPSS
Start SPSS and open this Deese data fileUnder Analyze, select Scale, then select Multidimensional Scaling (ALSCAL)… 1. Move Variable from left to right 2. Create distances from data 3. Model 4. Options
How to - next
page
32Slide33
Select all of these
33Slide34
MDS of the Deese
data34Slide35
Both are “
correct solutions”.WARNING!!
The Hague
Amsterdam
Utrecht
Eindhoven
Nijmegen
Side issue, the MDS obtains alternate
visual representations (
e.g., enantiomorphism)
Like geographic
data, for example,
MDS may
be oriented in different
ways
(describe Ellen
Taricani’s
2002 dissertation, handing out teacher maps post-reading is a bad idea)
35
The Hague
Amsterdam
Utrecht
Eindhoven
NijmegenSlide36
How good is the MDS representation for displaying the relationship raw data?
Many dimensions (in this case 19) reduced to 2 dimensionsCheck the “stress” value to estimate how strained the results are
MDS is an
algorithmic, power, approach rather than based on a
distribution model, so
no assumptions about data structure are required…
36Slide37
PFNet of
Deese data37
summer
spring
sunshine
yellow
color
blue
sky
flower
garden
nature
butterfly
cocoon
moth
wing
bees
bird
fly
bug
insectSlide38
MDS and PFNet
of the exact same data from Deese
Pathfinder KNOT
PFNet
(i.e., local structure, verbatim,
proposition specific)
SPSS
MDS
(i.e., global structure,
relational, fuzzy, gist)
38Slide39
MDS and
PFNet of the exact same data from Deese
Pathfinder KNOT
PFNet
(i.e., local structure, verbatim,
proposition specific)
SPSS
MDS
(i.e., global structure,
relational, fuzzy, gist)
39
Blue lines reproduce the
PFNet
linksSlide40
MDS and PFNet data reduction
MDS uses all of the raw data to reduce the dimensions in the representation; if the stress is not too large, global clustering is likely to be good but local clustering less so, and the MDS distances between terms within a tight cluster of terms are more likely to misrepresent the relatedness raw data.Pathfinder uses only the strongest relationship data (typically 80% of the raw data is discarded). Pathfinder analysis provides “a fuller representation of the salient semantic structures than minimal spanning trees, but also a more accurate representation of local structures than multidimensional scaling techniques.” (Chen, 1999, p. 408)40Slide41
Dave
Jonassen’s summary …
graph
building
similarity
ratings
semantic
proximity
word
associations
card
sort
ordered
recall
free
recall
additive
trees
hierarchical
clustering
ordered
trees
minimum
spanning
trees
link
weighted
Pathfinder
nets
Networks
Dimensional
principal
components
MDS – multidimensional scaling
cluster
analysis
expert/
novice
qualitative
graph
comparisons
quantitative
graph
comparisons
relatedness
coefficients
scaling
solutions
C of
PFNets
Trees
Knowledge
representation
Knowledge
comparison
Knowledge
elicitation
Jonassen
,
Beissner
, &
Yacci
(1993), page 22
41
concept maps
written text
Sabine
Klois
used …
distance
dataSlide42
Poindexter and Clariana
Participants – undergraduate students in an intro Educational Psychology course (Penn State Erie)Setup – complete a demographic survey and how to make a concept map lessonText based lesson interventions – instructional text on the “human heart” with either proposition specific or relational lesson approachKS measured as ‘distances’ between terms in a concept map (a form of card sorting) and also concept map link data, but analyzed with Pathfinder KNOTPoindexter, M. T., & Clariana
, R. B.
(2006).
The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes.
International Journal of Instructional Media, 33
(2),
177-184.42Slide43
Treatments
Relational condition, participants were required to “unscramble” sentences (following Einstein, McDaniel, Bowers, & Stevens, 1984) in one paragraph in each of the five sections or about 20% of the total text contentProposition-specific condition (following Hamilton, 1985), participants answered three or four adjunct constructed response questions (taken nearly verbatim from the text) provided at the end of each of the five sections, for a total of 17 questions covering about 20% of the total text content (no feedback was provided).43Poindexter, M. T., & Clariana, R. B.
(2006).
The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes.
International Journal of Instructional Media, 33
(2),
177-184.Slide44
DK and KS Posttests
DK - Declarative Knowledge (Dwyer, 1976)Identification drawing test (20)Terminology multiple-choice items (20), declarative knowledge, e.g., the lesson text states A B, the posttest asks A ?(B, x, y, z) (explicitly stated)Comprehension multiple-choice items (20), inference required, e.g., given A B and B C in the lesson text, posttest asks A ?(C, x, y, z) (implicit, not stated)KS - Knowledge structure
Concept map link-based common scores
Concept map d
istance-based common scores
44
Poindexter, M. T., &
Clariana
, R. B.
(2006).
The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes.
International Journal of Instructional Media, 33
(2),
177-184.Slide45
45
Note that declarative knowledge multiple-choice posttest items are sensitive to the linear order of the lesson textIf the lesson text is A B, paraphrasing the stem (A’) and/or transposing stem and response (B A) to create posttest questions influences performance. When MC posttest is:Identical to lesson (A B): 77% Transposed from lesson (B A): 71
%
Paraphrased from
lesson (A’
B)
: 69% Both T & P from lesson (B A’): 67%
posttest
Bormuth
, J. R., Manning, J., Carr, J., & Pearson, D. (1970). Children’s comprehension of
between and
within sentence syntactic structure.
Journal of Educational Psychology, 61
, 349–357
.
Clariana
, R.B. &
Koul
, R. (2006). The effects of different forms of feedback on fuzzy and verbatim memory of science principles.
British Journal of Educational Psychology, 76
(2), 259-270
.Slide46
Recording link and distance data in a concept map
46
lungs
oxygenated
deoxygenated
pulmonary artery
pulmonary vein
left atrium
right ventricle
Link Array
a
b
c
d
e
f
g
a left atrium
-
b lungs
0
-
c oxygenate
0
1
-
d pulmonary artery
0
1
0
-
e pulmonary vein
1
1
0
0
-
f deoxgenate
0
1
0
0
0
-
g right ventricle
0
0
0
1
0
0
-
Distance Array
a
b
c
d
e
f
g
a left atrium
-
b lungs
120
-
c oxygenate
150
36
-
d pulmonary artery
108
84
120
-
e pulmonary vein
73
102
114
138
-
f deoxgenate
156
42
54
84
144
-
g right ventricle
66
102
138
42
114
120
-
moves through
to the
passes into
to the
Student’s concept map
(n
2
-n)/2 pair-wise comparisonsSlide47
Distance raw data reduction by Pathfinder KNOT
47Pathfinder Network
a
b
c
d
e
f
g
a left atrium
-
b lungs
0
-
c oxygenate
0
1
-
d pulmonary artery
0
1
0
-
e pulmonary vein
1
1
0
0
-
f deoxgenate
0
1
0
0
0
-
g right ventricle
0
0
0
1
0
0
-
Distance Array
a
b
c
d
e
f
g
a left atrium
-
b lungs
120
-
c oxygenate
150
36
-
d pulmonary artery
108
84
120
-
e pulmonary vein
73
102
114
138
-
f deoxgenate
156
42
54
84
144
-
g right ventricle
66
102
138
42
114
120
-
lungs
oxygenated
deoxygenated
pulmonary artery
pulmonary vein
left atrium
right ventricle
Pathfinder network
(based on distances)
(21 distance data points reduced to 6 link data points) Slide48
Example of link and distance PFNets
for the same concept map48
lungs
oxygenated
deoxygenated
pulmonary artery
pulmonary vein
left atrium
right ventricle
Pathfinder network
(from distance data)
lungs
oxygenated
deoxygenated
pulmonary artery
pulmonary vein
left atrium
right ventricle
moves through
to the
passes into
to the
Student’s concept map
(i.e., link data)Slide49
Means and sd
49
Map-link
Map-dist
Poindexter, M. T., &
Clariana
, R. B.
(2006).
The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes.
International Journal of Instructional Media, 33
(2),
177-184.Slide50
Analysis
MANOVA (relational, proposition-specific, and control) and five dependent variables including ID, TERM, COMP, Map-prop, and Map-assoc. COMP was significance, F = 5.25, MSe = 17.836, p = 0.015, none of the other dependent variables were significance. Follow-up Scheffé tests revealed that the proposition-specific group’s COMP mean was significantly greater than the control group’s COMP mean (see previous Table). 50Poindexter, M. T., & Clariana, R. B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes.
International Journal of Instructional Media, 33
(2),
177-184.Slide51
Correlations
All sig. at p<.05
Compare to Taricani & Clariana
next
Map-link
Map-link
Map-distance
51
Poindexter, M. T., &
Clariana
, R. B.
(2006).
The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes.
International Journal of Instructional Media, 33
(2),
177-184.
(drawing)
(MC)
(MC)
Verbatim
A
B
Inference
A
CSlide52
Compare the correlation results to a related follow-up investigation
Taricani, E. M. & Clariana, R. B. (2006). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 53 (4), 61-78.
Taricani
&
Clariana
(2006)
Term Comp
Link data
0.78 0.54
Distance data
0.48 0.61
52
Poindexter &
Clariana
(2006)
Term
Comp
Link data
0.77 0.53
Distance data
0.69 0.71Slide53
Clariana and Marker (2007)
Participants – 68 graduate students in INSYS intro ISD courseComputer-based lesson – text, graphics, and questions on instructional design, either asked to generate headings for each section or notSeven sections referred to as A through G, each cover a component of the Dick and Carey modelKS as a sorting task and a new listwise taskClariana, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge. Journal of Educational Computing Research, 37 (2), 173-191. link53Slide54
Posttests
Declarative Knowledge – 30-item constructed response terminology test, 15 items from lesson sections B, D, and F (called “used”) and 15 items from A, C, E, and G (called “not used”)Knowledge structure – Posttest focuses on 15 terms used in sections B, D, and FListwise rating task agreement scores (compared to linear and cluster referent)Sorting task agreement scores (compared to linear and cluster referent)
List and sorting used by Sabine
Klois
, note:
sorting
not the same as card sorting
Clariana
, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge.
Journal of Educational Computing Research, 37
(2), 173-191.
link
54Slide55
Listwise rating task …
(available at: www.personal.psu.edu/rbc4)Clariana, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge. Journal of Educational Computing Research, 37
(2), 173-191.
link
55Slide56
Sorting task …
Clariana, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge. Journal of Educational Computing Research, 37 (2), 173-191. link56Slide57
An example student PFNet
Show how to count linear and nonlinear here …Clariana, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge. Journal of Educational Computing Research, 37
(2), 173-191.
link
57Slide58
Means and standard deviations
58Clariana, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge. Journal of Educational Computing Research, 37 (2), 173-191. linkSlide59
Analysis
The cued recall and sorting task posttest data were analyzed by a 2 (Treatment: Headings vs. No Headings) × 2 (Posttest: cued recall and sorting task) mixed ANOVA. The first is a between-subjects factor and the second is the within subjects factor. The Treatment main effect was not significant, F(1, 61) = 0.220, MSE = 0.045, p = .94. The Posttest repeated measure was significant, F(1, 61) = 18.874, MSE = 0.022, p < .001, showing that the mean cued recall test score (M = 0.59) was greater than the mean sorting task score (M = 0.47). Finally, the anticipated disordinal interaction of Treatment and Posttest factors was significant, F(1, 61) = 5.119,
MSE
= 0.022,
p
= .027
59
Clariana, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge. Journal of Educational Computing Research, 37 (2), 173-191. linkSlide60
Generate headings when reading: better ‘structure’ but worse ‘recall’
60Clariana, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge. Journal of Educational Computing Research, 37 (2), 173-191. linkDeclarative knowledge
Knowledge structure (KS)Slide61
61 of 54
Comparison of listwise and sorting KS
2.5
2.7
2.9
3.1
3.3
3.5
3.7
3.9
4.1
Linear
Non-linear
no Head
Headings
Linear
Non-linear
Sorting
task
(more relational)
i.e., A1
A3 or A4 or A5
Listwise
task
(more linear)
i.e., A1
A2
Clariana
, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge.
Journal of Educational Computing Research, 37
(2), 173-191.
linkSlide62
Correlations of interest
62Clariana, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge. Journal of Educational Computing Research, 37 (2), 173-191. linkSlide63
Brain scans – in proficient readers, text with no headings requires right hemisphere activity to achieve coherence (more work), some students will not be able to form coherence
63http://brain.oxfordjournals.org/cgi/reprint/122/7/1317
headings
no headings
RH
LH
RH
LH
“Consistent with previous studies…the right middle temporal regions may be especially important for integrative processes needed to achieve global coherence during discourse processing.” (p.1317 St. George,
Kutas
, Martinez, &
Sereno
, 1999)Slide64
Review - Generate headings when reading: better ‘structure’ but worse ‘recall’
64Clariana, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge. Journal of Educational Computing Research, 37 (2), 173-191. linkDeclarative knowledge
Knowledge structure (KS)Slide65
Comments
The better structured knowledge of the Headings group (i.e., more like the author’s text schema) should allow the learners to more flexibly use that knowledge (Jonassen & Wang, 1993) which should influence the reader’s ability to form inferences and comprehend the lesson text, but this apparently comes at the expense of text details. These results are consistent with and help explain previous investigations that have reported that learners who generate headings score lower than no-headings control groups on lower-order outcomes but score higher on inference and comprehension tests (Dee-Lucas & DiVesta, 1980; Jonassen et al., 1985; Wittrock & Kelly, 1984). (These papers are listed on the next screen)65Slide66
Generative learning (relational lesson tasks) DK KS reversal r
eference listDee-Lucas, D. & DiVesta, F. F. (1980). Learner-generated organizational aids: Effects on learning from text. Journal of Educational Psychology, 72(3), 304-311.Jonassen, D. H., Hartley, J., & Trueman, M. (1985, April). The effects of learner generated versus experimenter-provided headings on immediate and delayed recall and comprehension. Chicago: American Educational Research Association (ERIC ED 254 567).Wittrock, M. C., & Kelly, R. (1984). Teaching reading comprehension to adults in basic skills courses. Final Report, Project No. MDA 903-82-C-0169). University of California, Los Angeles.66Slide67
MDS explanation:Read with terms A
® I67
Link Array
(no color)
I
H
G
F
D
C
B
A
E
MDS
Connectivity Matrix (
Kintsch
, 1998)Slide68
Same reading with terms A
® I, but with section headings68
I
H
G
F
E
D
C
B
A
blue
red
green
blue
red
green
Link Array
(with headings)
MDS
Headings (i.e., color names)Slide69
MDS of connectivity matrices
69?…. Context (like topic headings) may alter memory structure in a regular way, and we can think about it visually.
I
H
G
F
E
D
C
B
A
blue
red
green
I
H
G
F
D
C
B
A
E
No color names MDS
C
olor names MDS
tighter
clustersSlide70
Explanation using Lawrence Frase’s
matrix multiplication to explain inference70Frase, L.T. (1969). Structural analysis of the knowledge that results from thinking about text. Journal of Educational Psychology, 60 (6, monograph, part 2), 1-16. Read A B and B C, model of the effects of context (as headings) on verbatim and inference activation(also notice B-A, C-A, and B-C activations)Slide71
71 of 54
Clariana and Prestera (2009)Background color as a weak context variableParticipants – 80 graduate students in INSYS intro instructional design courseComputer-based lesson – text, graphics, and questions with feedback on ISD, presented in 5 sections, each section covered a component of the Dick and Carey model (items with feedback should present STRONG AB effects)Intervention – lesson presented either with or without a color band on the left margin (this use of color should have WEAK relational effects)
Clariana
, R.B., &
Prestera
, G.E. (2009). The effects of lesson screen background color on declarative and structural knowledge.
Journal of Educational Computing Research, 40
(3), 281 -293. linkSlide72
Example lesson screen
72Clariana, R.B., & Prestera, G.E. (2009). The effects of lesson screen background color on declarative and structural knowledge. Journal of Educational Computing Research, 40 (3), 281 -293. linkColororNo colorSlide73
73 of 54
PosttestsDeclarative Knowledge vocabulary posttest – 18 constructed response items (fill in the blank) and 18 multiple choice items terminology test (strong AB)Knowledge structure posttest – sort the 36 vocabulary terms (same sorting task as Clariana & Marker (2006) above)Results: The anticipated disordinal interaction of Subtest and Lesson Color was significant, F(1, 71) = 5.008, MSe = 0.618, p = .028,
with lesson
color enhancing structural knowledge scores and inhibiting declarative
knowledge scores.
Clariana
, R.B., &
Prestera
, G.E. (2009). The effects of lesson screen background color on declarative and structural knowledge.
Journal of Educational Computing Research, 40
(3), 281 -293.
linkSlide74
Lesson and posttest means
74Clariana, R.B., & Prestera, G.E. (2009). The effects of lesson screen background color on declarative and structural knowledge. Journal of Educational Computing Research, 40 (3), 281 -293. linkSlide75
Another disordinal
interaction of declarative and structural knowledge75Clariana, R.B., & Prestera, G.E. (2009). The effects of lesson screen background color on declarative and structural knowledge. Journal of Educational Computing Research, 40 (3), 281 -293. linkSlide76
Section summary
Different measurement approaches are better for prompting memory for linear or cluster KSLinear lesson tasks establish linear KS and relational (generative) lesson tasks establish relational KSModels can account for verbatim and inference outcomesNext section - Alternative measures of KS76Slide77
For KS, more terms may be better
77Goldsmith et al. (1991) the relationship between the number of terms included in Pathfinder network analysis (elicited as pair-wise) and the predictive ability of the resulting PFNets to predict end-of-course grades.But only if these are really IMPORTANT terms (Clariana & Taricani, 2010)
Goldsmith, T.E., Johnson, P.J., & Acton, W.H. (1991). Assessing structural knowledge.
Journal of Educational Psychology, 83
(1), 88-96
.
Clariana, R.B., & Taricani, E. M. (2010). The consequences of increasing the number of sterms used to score open-ended concept maps. International Journal of Instructional Media, 37 (2), 163-173. linkSlide78
Raw data
reductionby Pathfinder KNOT78Terms = 10Raw data = 45
PFNet
= 9
PFNet
as % of raw data = 20%
Terms = 20
Raw data = 190PFNet = 19PFNet as % of raw data = 10%
Terms = 30
Raw data = 435
PFNet
= 29
PFNet
as % of raw data = 7%
Number of terms (n)
Raw data (half array,
(n
2
-n
)/2
)
Methods that elicit pairwise association fatigue with more then 20 to 30 terms)
KNOT tries to form a path with n-1 linksSlide79
KS measurement
More terms are better but the problem with eliciting KS using pairwise comparisons (more than 20!)So, we need a valid and efficient measure of KS … recall from above that:Recall that Ferstl & Kintsch (1999) used a more efficient cued-recall list approach (3 recalls for each term)Clariana & Marker (2007) added a ‘listwise’ approach, with one recognition retrieval for each term and a ‘sorting’ approach (dragging all terms around on the screen at the same time)Do ‘listwise’ and ‘sorting’ results compare with the more traditional and accepted ‘pairwise’ approach? If yes, then these two can handle large lists of terms.79Slide80
80 of 54
Clariana and Wallace (2009)Compared pairwise, listwise, and sortingParticipants – 84 undergraduate students in businessAll students completed 3 computer-delivered KS measures – listwise, pairwise, and sorting (randomized) using the 15 major concepts of the courseStudents grouped for analysis into high and low groups based on a media split of their end-of-course multiple-choice examClariana, R.B., & Wallace, P. E. (2009). A comparison of pairwise, listwise, and clustering approaches for eliciting structural knowledge in information systems courses. International Journal of Instructional Media, 36,
139–143.
Slide81
The three approaches
81
pairwise
l
istwise
sorting
Clariana
, R.B., & Wallace, P. E. (2009). A comparison of pairwise,
listwise
, and clustering approaches for eliciting structural knowledge in information systems courses.
International Journal of Instructional Media, 36
,
139–143.
Slide82
Sorting and listing are faster than pairwise
Time to complete the tasks:pair-wise approach, X = 447.4 s (sd = 140.6)list-wise approach, X = 193.3 s (sd = 79.6)Sorting approach, X = 115.5 s (sd = 62.7)Concurrent / convergent validity: Do the 3 elicitation tasks obtain similar raw data and PFNet data?82
Clariana
, R.B., & Wallace, P. E. (2009). A comparison of pairwise,
listwise
, and clustering approaches for eliciting structural knowledge in information systems courses.
International Journal of Instructional Media, 36
, 139–143. Slide83
Individuals’ raw data
arrayswere not similar (correlations)Therefore, the 3 approaches do not elicit the same raw data associations, individuals’ raw data seems to be idiosyncratic or flaky or noisy; however the group average raw data are much more alike (averaging within a group ‘smooths out’ idiosyncrasy)83
Clariana
, R.B., & Wallace, P. E. (2009). A comparison of pairwise,
listwise
, and clustering approaches for eliciting structural knowledge in information systems courses.
International Journal of Instructional Media, 36
, 139–143. Slide84
% overlap based on
‘group average’ PFNet common scores (intersection)84Clariana, R.B., & Wallace, P. E. (2009). A comparison of pairwise, listwise, and clustering approaches for eliciting structural knowledge in information systems courses. International Journal of Instructional Media, 36, 139–143.
pairwise
listwise
sorting
referentSlide85
Sabine’s experts
Pairwise
Listwise
Sorting
% overlap
Expert_A
Expert_B
Expert_C
Expert_D
Expert_ave
Expert_A
Expert_B
Expert_C
Expert_D
Expert_ave
Expert_A
Expert_B
Expert_C
Expert_D
Expert_ave
Expert_A
one
one
one
Expert_B
0.46
one
0.60
one
0.43
one
Expert_C
0.27
0.44
one
0.67
0.73
one
0.29
0.43
one
Expert_D
0.43
0.57
0.42
one
0.53
0.53
0.53
one
0.43
0.21
0.36
one
Expert_ave
0.71
0.56
0.52
0.54
one
0.79
0.79
0.79
0.58
one
0.43
0.57
0.64
0.36
one
each
avg.
=
0.47
0.51
0.41
0.49
0.58
0.65
0.66
0.68
0.54
0.74
0.39
0.41
0.43
0.34
0.50All avg. =0.49
0.65
0.41
85Slide86
Next directions for KS research?
Continue to find valid and efficient KS approachesAnd close with a few provocative comments …86Slide87
1st
year undergraduate textbook in ISTan obvious ‘collage’87Slide88
Web reading F-pattern?
Heatmaps from user eyetracking studies of three websites. The areas where users looked the most are colored red; the yellow areas indicate fewer views, followed by the least-viewed blue areas. Gray areas didn't attract any fixations.http://www.useit.com/alertbox/reading_pattern.html88Slide89
Gaze plot of the 4 main classes of web search reading behavior
search-dominant navigation-dominant tool-dominant successful
http://
www.useit.com/alertbox/fancy-formatting.html
89Slide90
Sources of eye-tracking
http://www.miratech.com/blog/eye-tracking-lecture-web.htmlhttp://www.youtube.com/watch?v=X60VPJDLAeM&feature=player_embedded90Slide91
Altered reading due to web experience?
If students are not reading linearly, or are using (or not using) headings and other text signals (color, underline, highlights) differently, then the KS will be differentSpecific KS can accomplish specific kinds of mental ‘work’ and other KS other work (the protein analogy)So determining how today’s students read hypertext and web materials, and whether this transfer back to paper-based text is an important questionKS is one tool that can complement existing measures and help explain this91Slide92
Term activation across sentences
92