ISTE Workshop - PowerPoint Presentation

Slide1

ISTE Workshop Research Methods in Educational Technology

IIT BombayFebruary 2-9, 2013Slide2

Status: Have you done your Homework?

Pre-workshopAssignmentSubmission

Feb.2 Feb.3-8‘Idea proposal’ submission

Feb.9In-workshopFeb.10-20‘Study-planning submission

Feb.20 ~

T4E 2013 paper draft

Registered

participants

6357

4934

2054+

?

?

4934

2054*

Not

doing the assignments and directly submitting something

on

Feb 20

will

not

earn a

certificate

*Submissions as

on Feb. 9, 6

am

Path

to course certificate & T4E

paper

 Slide3

Recap

(Revisiting what we have done so far)Slide4

What is this workshop about?

This workshop is about ET research.

This workshop is about how you solved the teaching-learning "problem" in your class.Slide5

What is Educational Technology?

Tech For Education

Creation and use of technologies for teaching-learning.Creation and use of technology tools to facilitate teaching-learning.

Tech Of Education

Creation and use of strategies for teaching-learning.

Focus on what to do with the technology, rather than the technology itself.Slide6

This Workshop is a journey

To

ET Researcherconduct systematic studies to get data about whether our ideas are working.

provide evidence to support our conclusions.

From

ET Practitioner

teach students;

facilitate their

learning.

come up with ideas for doing the above.Slide7

What route are we taking

Recall: There are no fast-lanes; no

short-cutsWe cannot simply *listen* to 'lectures' and expect to absorb the required knowledge.We have to *DO* the activities sincerely

.

We have to challenge ourselves to go beyond the obvious ideas

.

Recall:

The theories underlying this workshop

Spiral Curriculum

Active LearningSlide8

Our Journey at a glance

Your idea of:

a teaching-learning problem what teacher (you) & the students will do

 How to prove your idea works?Evaluate pre-w/s assignment for strong paper features

In-workshop activities

Guidelines

Moodle

resources/ Self-study

Moodle

queries

Mentoring by TAs

Fill up & submit ‘Idea proposal’

form

Fill up ‘Study planning template’ for your idea

In-workshop activities

Guidelines

Moodle

resources/ Self-study

Moodle

queries

Mentoring by TAs

Submit your ‘Study planning’

proposal

Moodle

queries

TA mentor

Submit paper draft

Pre-workshop

Feb. 2

Feb. 3-8

Feb. 9

Feb. 10-20

Feb. 20 ~Slide9

In-workshop

activities

Guidelines

Research Methodology Moodle resources for Self-study

What qualifies as a research paper

What points to address in abstract

How to evaluate a research paper

Features of a strong research paper

What referees look for in a paper

Peer discussion on pre-workshop assignment

Clicker questions on acceptable research paper

TPS on : given scenario to research study

TPS on : your idea to ET research study

Evaluation of research papers

Workshop slides

Videos: reading research papers

+

doing literature survey

iii. Mid-Workshop assignment

Overview of different research paper sections

Overview on how to plan a research study (Novelty, positioning, Procedure soundness, Evidence to collect)

Feb. 2Slide10

In-workshop

activities

Guidelines

Research Methodology Moodle resources for Self-study

i

. T4E checklist

ii. Citing research papers

iii. Paraphrase to avoid plagiarism

What are types of ET research studies

What is soundness of procedure

Clicker question on what evidence to collect

Clicker question on research ethics

How to

paraphrase

other research paper

findings

vi. Addressing

queries and clarifications

Workshop slides +

What are types of ET research problems

How to plan and execute ET

research

studies

What are research ethics

Feb. 9Slide11

What are referees looking for

Referees look for

Your paper must have

Novelty

Analysis of prior work to show that your idea is unique

Positioning

Analysis to show that your work is required, how your work advances the state of the art

Soundness of procedure

Steps to show that you have implemented solution carefully

Evidence to support claim

Data to show that your solution works as claimed

Overall coherence

Consistency between parts of your paper – treatment should address problem, results should give answer to problem

2

nd

Feb 2013Slide12

What exactly is meant by ‘Novelty’?

Dictionary: “The quality of being new, unique, original, innovative, or unusual”.

What has to be novel?  At least one

of the below

:

Your Problem

– Research Question(s).

Your Solution – Strategy to solve a known problem.

Your Domain – Adapt a known solution

to your context

One of your

Moodle

queries:

Can

a non-innovative strategy be developed into a strong research

paper?

Yes,

provided it is positioned well (See next slide).

Strong to WeakSlide13

What exactly is meant

by ‘Positioning’?

Dictionary: “situation/relation with respect to others”.

How to do positioning?

 Do

both

of the below:

1) Have you shown analysis

of

related

prior work to bring out the gaps?

papers that have addressed a problem similar to yours

papers

that have a solution approach similar to

yours

2

) Does your solution address any of the gaps

above?

As the novelty of your problem or solution decreases, the accuracy of your positioning must increase!Slide14

Explain the relation to other work clearly

Awful

The galumphing problem has attracted much attention [3,8,10,18,26,32,37]

Bad

Smith [36] and Jones [27] worked on galumphing.

Poor

Smith [36] addressed galumphing by blitzing, whereas Jones [27] took a flitzing approach

Good

Smith's blitzing approach to galumphing[36] achieved 60% coverage [39]. Jones [27] achieved 80% by flitzing, but only for pointer-free cases [16].

Better

(Good Above) + We modified the blitzing approach to use the kernel representation of flitzing and achieved 90% coverage while relaxing the restriction so that only cyclic data structures are prohibited.

Source: Mary Shaw, Writing good Software Engineering Research Papers, ICSE 2003Slide15

One-line summary –

What is the referee’s job

?Referee's job is NOT

to find

reasons to accept your paper, but to find reasons to

reject

your paper!

So, not

doing all the

parts

required for a paper – Novelty, Positioning, Soundness, Evidence =>

Wasted

time and effort for you and for

referee

The order is important. Even if your work is sound, it can get rejected if it is not positioned properly.Slide16

Submitted

paper

Positioning

wrt

related work

Soundness of procedure

Paper recommended for Acceptance

Soundness of evaluation

Novelty of problem/solution

If

it does not pass the filter

Again:

What

is the

referee’s

job

?

Typical Acceptance Ratio: 15-20%

80%Slide17

Moving

Ahead(Today and Beyond)

:Towards paper acceptanceSlide18

First Activity for today!

Pair Activity: Peer-review of Idea Proposal assignment

.10 Minutes1. Form pairs and exchange your idea proposal assignment submission with your partner2. Read your partner’s answer to Q3.

Are you able to understand the idea?

Yes/ No

Does the idea sound exciting to you?

Yes/No

3. Read your partner’s answer to Q5.

Is the gap in prior work evident to you?

Yes/No

4. Explain to your partner what aspects of his/her answers are not coming out clearly.Slide19

Poll

How many of you felt that:

You have explained well but your partner still did not understand either your idea or your gap analysis?Co-ordinators quickly poll and send response by chat.Slide20

Value of peer-review

Never Forget:

It is YOUR responsibility to write your paper in such a way that the reader can understand and follow it easily.If your friend/colleague has difficulty following your paper, what do you think the referee is going to do?Value to writers:

Every peer-review comment tells you what part of your paper you need to improve upon.

Value to readers:

Every paper you peer-review improves your analytical skills which will improve your writing!Slide21

Continue with the activity

Pair Activity:

Peer-review of Idea Proposal assignment.20 Minutes5. Read your partner’s answer to Q6. Suppose you have to replicate the study,

Do you think that the procedure is described in sufficient detail?

Yes/ No

Do you think that the data being collected can give evidence that the idea works?

Yes/No

6. Explain to your partner what additional information you require to replicate the procedure of his/her study.

7. Explain to your partner what additional data is required.Slide22

Poll

How many of you observed that:

Your partner pointed out something that you had overlooked? Something that will improve your study.Co-ordinators do local poll. We don’t need this answer.

Check if our prediction is true:

At least

80%

of the participants would have found that their partner’s comments help to improve their study.Slide23

Moral of the story (activity)

Don’t wait to complete your study and write your paper before you seek feedback from peers/colleagues.

Don’t wait to send your work to an ‘expert’ for getting feedback. First get it from your immediate peers.The more we seek feedback, the more our work will improve, provided we act on the feedback. Slide24

Peer-review ethics

As reviewers, we have to follow three main norms:

Non-disclosure: Since the work that we are reviewing is unpublished, we should maintain its confidentiality. We should not disclose its details to anyone without the author’s consent.Non-plagiarism: We should not ‘steal’ the author’s ideas and directly use it in our study. More on this later.Conflict-of-Interest: If we are already working on the same topic as the author, then we should not be the reviewer, to avoid any conflict-of-interest.Slide25

Your

queries

We have got your queries in three ways: Log of chat session from Feb 2nd workshop. Previous video Q&A session. Queries posted on Moodle

The core queries that are relevant to all participants will be addressed in today’s sessions; Some will be answered through

Moodle

.

You can also send queries as chat messages, or

Moodle

posts, during the tea and lunch breaks.Slide26

Some

queries to be taken up todayWhat to measure for learning, other than marks?

How to setup study and measure systematically?  Next sessionHow to find prior work in a systematic manner?If there is no related work yet, of the present work, what is to write down in the research paper?

Journal

or

conference,

which is good for research

purposes?

 Last sessionSlide27

Next

session: Delving into RMET

Referees look for

Your paper must have

Novelty

Analysis of prior work to show that your idea is unique

Positioning

Analysis to show that your work is required, how your work advances the state of the art

Soundness of procedure

Steps to show that you have implemented solution carefully

Evidence to support claim

Data to show that your solution works as claimed

Overall coherence

Consistency between parts of your paper – treatment should address problem, results should give answer to problem

9

th

Feb 2013