Discussing study limitations in Applied Linguistics research: - PowerPoint Presentation

Slide1

Discussing study limitations in Applied Linguistics research: A corpus-based comparison of Masters dissertations and published research articles

Philip Nathan

English Language Centre

Durham

University

BAAL Corpus SIG Durham 2017Slide2

Limitation components identified in RAs in a range of disciplines.Forming elements of Discussion/Conclusion sections.Limited research on these limitations components beyond identification.

Limitations Components in Research Writing Slide3

Bunton, (2005) examined 45 doctoral theses conclusions sections Limitations moves were identified in more than 25% of conclusions in ‘thesis-oriented science and technology theses and theses in social sciences and also in field-oriented theses

.

Three

explicitly titled sections labelled ‘limitations’ were identified.

Prevalence/Frequency of OccurrenceSlide4

Dudley-Evans (1994) identified in the context of masters dissertations in the biological sciences, the presence of 3 broad moves in discussions sections, namely Introduction, Evaluation of results and Conclusions and future work. [But according to Peacock (2002)

–

empirical

data

is

not presented to support his structure]

The evaluation of results move comprised in itself 9 moves one of which was identified as a limitations move. Moves occurred in a cyclical pattern within the evaluation move

Prevalence/Frequency of OccurrenceSlide5

Yang and Allison (1993) identified a limitations move in 1 results section within an evaluating the study move.They identify stating limitations as part of an ‘evaluating the study’ move within RA discussion sections in applied linguistics research articles. This occurred in 25% of the 8 discussion sections they looked at

.

In

conclusions sections, 7 of 13 conclusion sections contained limitations elements within an Evaluating the study move

but

also separate sections called pedagogical implications

within which two limitations moves (referred to as steps) were identified within the Evaluating the study move.Overall 40% of the texts examined in this research contained limitations components

Prevalence/FrequencySlide6

Limitations components also identified in for example, Hill. Sopsela and West (1982) in experimental RAsTessuto (2014)

– in 40% of law RAs

Kanoksalipatham

(2005) – in 80% of biochemistry RAs

Peacock (2002) in 43% of RA Conclusion sections across 7 disciplines 9 - from 31.5% in Environmental Science RAs to 62.5% in Language and Linguistics/

Prevalence/FrequencySlide7

Basturkmen (2012) identifies no limitations sections in Dentistry RAsStoller and Robinson (2013) identify no limitations components in Chemistry RAs

Prealence

/FrequencySlide8

A limitation is an element usually tied to research methodology which limits or reduces the level of generalisation that can be applied to research conducted and/or which

represents

an element of the research which provides a

potential

threat to the validity of the findings and conclusions.

What are limitations?Slide9

Thompson (2005) identifies limitations as one element of a conclusions move titled ‘consolidating the research’ with one purpose of the conclusions section identified as being ‘to forestall criticisms by stating the limitations of the research’.

Purpose of limitations components:Slide10

Limitations is either placed within an ‘Evaluation of results/research’ move found in the Discussion/Conclusion section of RAs. (Dudley-Evans, 1994) This move is found in a 3-move structure: Introduction, Evaluation of results and

Conclusions

and future

work

Swales and Feak (1994) suggest 3 conclusions moves Consolidate the research space, limitations, further research

Rhetorical positioningSlide11

Bitchener (2010:207) uses the following 3 moves within the limitations component: limitation

– significance – further

research

Within the limitations moveSlide12

On our MA TESOL/Applied Linguistics for TESOL, limitations sections written by MA dissertation seemed rather weak.Pedagogical aim of improving the writing of these components by these MA students.A poor limitations section can fatally undermine research writing.While recognising different contexts and purposes, set out to compare published limitations components with MA limitations components.

Motivation for the StudySlide13

Primarily, the sample size is not large enough to generalize the results. It may not reflect the different situations regarding to different teachers’ perception, practice and students’ opinion and preference that across diverse contexts. Thus, it can be only [used] as a reference of WCF studies in Chinese middle school context. In addition, the sample size of the interview is too small, which

means

some results present huge individual differences and

[are] difficult

to

generalize…

Last but not least, there are some design flaws in the questionnaire

design. Even though we made an effort

to

distinguish ‘opinions’ and ‘facts’ and use the

percentages

to present all the statistics.

[sample 0061415]

Sample of a limitations text which could be (significantly) improvedSlide14

Firstly, in this study, there is a small-size population, which may make the result inaccurate because the data is not enough: only 45 participants [were] involved this study. …Secondly, another limitation of this study is the short span of the speaking course. Since the study is conducted in a speaking class in university, we [didn’t]

have the additional time and the peer feedback sessions are limited.

[sample 0151415]

Sample of a limitations text which could be (significantly) improvedSlide15

18 [30] MA Applied Linguistics Limitation Components [6416 words] approx. 90% NNS30 RA Limitations Components – balanced from ESP J, JEAP, JSW [ 6802 words]Dissertation average word count = 356

Journal average word count = 227

CorporaSlide16

Initial hypotheses:RAs - “to forestall criticisms of the research supporting research validity such that the RA will be considered acceptable by reviewers and readers and therefore be published”

Dissertations - “to demonstrate knowledge and understanding of research principles to examiners and to forestall criticisms of the research such that examiners consider the work to be valid”

Communicative purposes (following

Askehave

and Swales, 2001)Slide17

Significantly different rhetorical moves and move frequencies would occur in RAs and MA dissertationsMA dissertations would have lots of negative language compared to published RAs.MA dissertations would contain long lists of limitations compared to published RAs.

Hypotheses – Explanations for perceived weaknessSlide18

LimitationsLimitations and pedagogical applicationsLimitations and future researchLimitations of the studyIn RAs limitations most frequently appeared as elements of conclusion sections

Signalling of LimitationsSlide19

See handout Limitations Sections: Overall Move Structure(s)Slide20

See handout [text to move] Prototypical RA Limitations StructureSlide21

Moves

RA frequency (26)

Diss

Freq

(16)

Signalling consideration of limitations52%

67%

Strengths/achievements (all)

56%

73%

Rationale

48%

87%

Impact

48%

53%

Remedy (all)

72%

80%

Move count comparison (selected moves) Slide22

Dissertations contained many of the different moves more frequently than in RAs.Longer texts (15000 v 7000 for journals) so there’s more to write and this might suggest a greater variety of moves are needed.Consideration Slide23

Average 2.12 cycles per sample in RAsAverage 3 cycles per sample in dissertationsNot necessarily of significance in terms of rhetorical move patterns, as dissertations are longer texts.

Frequency of Limitation CyclesSlide24

Keyword analysis

can/ may/could frequencies, might indicate more hedging in dissertations.Slide25

But also notably:

Lemma

RA

Diss

Limiters

only

0.29

0.46

too

0.03

0.08

Concessives

/

Contrast

markers

although

0.13

0.08

though

0.06

0.02

despite

0.10

0.03

even

though

0.01

0.00

nevertheless

0.06

0.00

however

0.29

0.45

but

0.09

0.23

Lexis

Explore

0.19

0.06Slide26

The findings of this study should be interpreted with caution, not only because of the relatively small size of the study but also because of the nature of the writing tasks used. [RA LIM 2]The limitation of the study may result from the relatively small

database of only 25 medical RAs

,…[RA Lim 8]

Nevertheless, there are some limitations associated with the study, specifically the fact that the validating

corpora are

relatively small

. [RA Lim 4]Use of ‘Relatively’ Slide27

Although the larger national study provided a wealth of rich data across a broad spectrum of students in the U.S.,… [ RA LIM1]Finally, even though this study makes valuable contributions to advance L2/FL writing research, …

[RAJSW1]

Although

the present study has yielded a number of significant implications concerning a link between writing-to-learn and learning-to-write, … [RAJSW8]

Despite

the inherent limitation of a single case, however, we believe that the description of the teaching context of the teacher’s genre work implementation and the

detailed SFL analyses of Ji Soo’s texts before and after the genre work provide rich evidence for…… [RAJSW9]

Expressing strengths: RAsSlide28

Therefore, the results and conclusion derived from this triangulation method are more valid. [sample 0041314 ]This research has achieved the purpose and answered the research questions, however…

[sample 0081314]

By employing semi-structured interviews, 8 Chinese students’ perspectives on peer feedback were exposed to this study, which largely satisfies research questions. Simultaneously, some new findings were also revealed through interviewing target learners.

However.. [sample 013]

Expressing strengths: DissertationsSlide29

In addition, the sample size of the interview is too small, which makes some results present huge individual differences and difficult to generalize. [sample 0061415] In fact, some respondents commented that the questionnaire was too long for them to finish patiently. [sample 011]

‘too’ – in dissertationsSlide30

only 45 participants [were] involved this studyas the ones who could respond the interview were only two , due to the lack of free time. [sample 0151415]

However, I only managed to collect data from three universities,

[sample 011]

All feel rather negative!

‘Only’ in dissertationsSlide31

The limitation of the study may result from the relatively small database of only 25 medical RAs, … [ESPJ LIM8] [It’s not small, it’s ‘relatively small’]Given the labour intensive nature of the analysis only a limited number

of writing

samples could be examined in this exploratory study

. [JEAP LIM4]

[this is an exploratory study so the number of writing

samples doesn’t matter so much/and we did a lot of hard

work!]

One of them is that we conducted only one interview with some of the participants

. [JEAP LIM1]

[but we conducted more than one interview with

most of them]

‘Only’ in RAs: Positive spin?Slide32

1. MA dissertations would have lots of negative language compared to published RAs. - some greater use of ‘only’ ‘too’ but

perhaps more prevalent is lack of

mitigation/spin to reduce the impact of negative

statements and reduce the sense of negativity. 2. MA dissertations would contain long lists of limitations compared to published RAs

.

There are on average more limitations mentioned in MA dissertation texts, but this is considered due to these texts being longer and more needing to be written.

ConclusionsSlide33

No significant differences in frequency of deployment of rhetorical moves were identified in this work.There appear to be differences in the use of concessives between MA and RA writers. Professional/published RA writers

have a range of strategies for mitigating

limitations and for making limitations/problems more palatable.

There’s evidence that students understand the formal rhetorical moves required, but they need linguistic support in realising moves and spinning their statements.

ConclusionsSlide34

Teach/familiarise students with rhetorical structure/elements of limitations sections.Provide full, complete and accurate communicative purposes of these limitations elements.Teach students tactics strategies for mitigating their limitations.Teach students about the realities of research writing (persuasion, selectivity, integrity

etc

)

Pedagogical ImplicationsSlide35

Refining of the rhetorical move model Co-rater analysis regarding (a) the rhetorical move structure (b) quality of limitation samplesInterviews with student dissertation writersInterviews with RA writers

Fine-grained linguistic analysis of both RA and Dissertation samples to identify rhetorical tactics and

mis

-steps

Analyse doctoral limitations texts

Future work Slide36

Dudley-Evans, T. (1994). Genre analysis: An approach to text analysis for ESP. In M. Coulthard (Ed.), Advances in written text analysis (pp. 219–228). London: Routledge.Hewings, M. (1993). The end! How to conclude a dissertation. In G. M. Blue (Ed.), Language, learning and success: Studying through English. London: Modern English Publications and The British Council, Macmillan.

Hill, S. S.,

Soppelsa

, B. F., & West, G. K. (1982). Teaching ESL students to read and write experimental research papers. TESOL Quarterly, 16(3), 333–347

Holmes, R. (1997) Genre analysis and the social sciences: An investigation of research article discussion sections in three disciplines English for Specific Purposes 16 (4)

321-337

Peacock (2002) Communicative moves in the discussion sections of research articles System 30 479-497

ReferencesSlide37

Stoller and Robinson (2013) Chemistry journal articles; An interdisciplinary approach to move analysis with pedagogical aims English for Specific Purposes 32 (2013) 45-57Tessuto, G. (2014) Generic structure

and rhetorical moves in English language empirical law research articles: Sites of interdisciplinary and interdiscursive crossover.

Thompson, P. (2005) Points of focus and position: intertextual reference in PhD theses’ Journal of English for Academic Purposes 4:3017-323

Yang, R., & Allison, D. (2003). Research articles in applied linguistics: Moving from results to conclusions. English for Specific Purposes, 22, 365–385.

References