Ingrid Nielsen Department of Management Faculty of Business amp Economics Monash University 1503 amp 1599 19 June 2014 ABDC Journal Quality List Review 2013 2 How were panel members chosen ID: 553179
Download Presentation The PPT/PDF document "ABDC Journal Quality List Review 2013" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
ABDC Journal Quality List Review 2013
Ingrid NielsenDepartment of ManagementFaculty of Business & EconomicsMonash University
1503 & 1599 Slide2
19 June 2014
ABDC Journal Quality List Review 2013
2
How were panel members chosen?
A call for self-nomination to panels 1501, 1502, 1503, 1504, 1505, 1506 and 1507 was made on 8
th
April by Stephen Taylor, Chair of BARDsNet
Applications closed on 19
th
April
Applications were reviewed by the BARDsNet Executive, the overall Review Chair, Robert Faff, and the already appointed Panel Chairs
Due consideration was given to size, disciplinary representation, gender balance, availability and demonstrated expertise to contribute to the process
I was advised on 3
rd
May that my application to be a panel member had been successful
Initially, the panel comprised Neal Ashkanasy, Bob Cavana, Gavin Jack, Ingrid Nielsen and Vikas Kumar, Slide3
19 June 2014
ABDC Journal Quality List Review 2013
3
What happened specifically with 1503?
For personal reasons, the appointed 1503 Panel Chair, Neal Ashkanasy, decided to step aside from the Chair role
I was appointed to assume the Chair role on 21
st
May
Following feedback about a perceived expertise gap on the 1503 panel in the HR/ER areas, Adrian Wilkinson was added to the panel, also on 21
st
May
The 1503 panel was larger than any other panel, because of the size and disciplinary breadth of 1503 (741 journals in 2010; 821 in 2013)
There may have been people in the community who remained unhappy with the panel’s gender balance, but I was not unhappy with it
The particular mix of panel members worked really well together over the coming months on what was an enormously complex taskSlide4
19 June 2013
ABDC Journal Quality List Review 2013
4
How did the process unfold: submissions
Calls for submissions were made on the ABDC website on May 1, 2013, with a deadline of May 31, 2013
Submissions could be made in four (4) separate categories:
Request to add a new journal (Form A)
Request to downgrade a journal’s ranking (Form B)
Request to upgrade a journal’s ranking (Form C)
Request to transfer a journal’s FOR code (Form D)
All submissions were managed through the ABDC Office, where each was tagged to reflect the panel (MAN), the Form and the journal – eg., MAN_FC_F_096
A ‘short form’ of each submission was also created in the ABDC Office, tagged with an ‘S’ – eg., MAN_FC_S_096Slide5
19 June 2013
ABDC Journal Quality List Review 2013
5
How did the process unfold: submissions
Dropbox files were created in the ABDC Office for each Panel containing all submissions for a given Panel
An Excel spreadsheet summarizing each Panel’s full complement of submissions was also contained within the Dropbox and each submission number corresponded to the Excel line number
Links to the completed Dropbox files were sent to Panel Chairs on 24
th
June
Submissions with missing information and/or made on an incorrect form were initially highlighted in the Excel sheet in yellow until all submitters had been contacted and asked to update missing information and/or use the correct formSlide6
19 June 2013
ABDC Journal Quality List Review 2013
6
How did the process unfold: framework of the panel deliberations
In respect of logistics, the ABDC allowed each panel to devise its own internal processes as there was much heterogeneity in terms of numbers and locations of panel members as well as submission numbers
In respect of referents, panels were guided to refer to a reasonably homogenous set of indicators, however there was appropriate leeway given to differences in the ways certain disciplines viewed particular indicators
It would be reasonable to say that common referents across the panels included recourse to other well-established rankings, SSCI impact factors and immediacy indices, citation metrics, rejection rates, editorial board composition and in some cases, actual peer review was undertaken and further views were sought from the communitySlide7
19 June 2013
ABDC Journal Quality List Review 2013
7
How did the process unfold: 1503 panel deliberation principles
A decision was taken early on that no panel member would be eligible to deliberate on a submission for any journal for which he or she was a past, current or incoming editor or for which he or she was currently on the editorial board or had held editorial board membership within the past five years
In respect of the above, a decision was taken that in the event that less than four panel members remained to deliberate any given submission, the Chair would to seek the view(s) of other community members to take the total number of decision points to at least five
A decision was taken early on that all panel members must disclose during deliberation on any given journal that he or she had published in that journal during the past five yearsSlide8
19 June 2013
ABDC Journal Quality List Review 2013
8
How did the process unfold: submissions to 1503
A total of 205 submissions were made to 1503 as follows:
Form A – 68
Form B – 1
Form C – 136
The panel endorsed removal of forty journals, where the content was deemed out of scope, the journal had insufficient English language content or the journal was a duplicate in error on the original list
Panels were required to arrive at ranking distributions in the following approximate ranges:
A*: 5 – 7%
A: 15 – 25%
B: 35 – 40%
C: remainderSlide9
19 June 2013
ABDC Journal Quality List Review 2013
9
How did the process unfold: 1503 panel deliberation process
A first cut was achieved via responses to a multiple choice Google doc, eg.,
I agree that journal X should be upgraded from B to A
I disagree. Journal X should remain a B
I disagree. Journal X should be double upgraded to A*
I disagree. Journal X should be downgraded to C
Where all members were in agreement, the submission outcome was
pencilled
as resolved
Where majority plus 1 (5) was achieved, the submission outcome was pencilled as resolved
except for
cases where the dissenter was considered the primary expert in that areaSlide10
19 June 2013
ABDC Journal Quality List Review 2013
10
How did the process unfold: 1503 panel deliberation process
A teleconference was held to attempt to resolve the ‘expert dissenter’ cases
Each unresolved submission was assigned to two panel members whose tasks were to make a written case for or against the submission
These cases were submitted in writing to the Chair and pulled together into a second round Google doc for voting
A face to face meeting of the panel followed, where all unresolved submissions were discussed and resolved, with two exceptions…Slide11
19 June 2013
ABDC Journal Quality List Review 2013
11
How did the process unfold: APJHR & JMO
Almost as much time and consideration was given to these two decisions as to the rest of the review task combined
For each journal, multiple confidential views were sought in light of the ‘local relevance’ factor
Several people declined to provide a view because they could not arrive at a decision themselves
We were required by the ABDC to retain the status quo unless the case for change was clearcut and in neither case could we achieve a majority view supporting change
In both cases though, the panel and its ad hoc members strongly encouraged resubmission to the next reviewSlide12
19 June 2013
ABDC Journal Quality List Review 2013
12
How did the process unfold: between panel relationships
Where submissions existed that proposed FOR code change, these discussions took place between the implicated Panel Chairs
Where erroneous double coding existed, decisions as to final FOR codes took place between implicated Panel Chairs and the higher ranking was adopted
Discussions about any other between panel issues took place at two face-to-face meetings of the Panel Chairs, held at UTS
The major between panel issue arose as a result of the Economics panel taking a decision to wholesale delete 236 journals from their portion of the list, without advising any of the other Panel Chairs
A request came to me on
September 17 t
o review the titles (not the rankings) of the deleted Economics journals with a view to developing a solution to reinstating them in the list
My review of these titles precipitated the creation of a 1599 coded sectionSlide13
19 June 2013
ABDC Journal Quality List Review 2013
13
How did the process unfold: Economics delisted journals in 1503 and 1599
Given time and availability constraints, the tasks of the 1599 panel were undertaken only by myself and Gavin Jack
Of the delisted Economics journals, 65 were brought into 1599 and 82 we brought into 1503
Taken together, the journals brought into 1599 formed a fairly internally consistent group of journals covering environmental science, political science and international relations
The journals taken in to 1503 were those where a disciplinary presence of the journal type already existed in 1503 (eg., public administration)Slide14
19 June 2013
ABDC Journal Quality List Review 2013
14
How did the process unfold: feedback
The draft list was released for feedback on 11 September, 2013 with a deadline for responses of 30 September, 2013
The criteria for
feedback were the identification of errors in the draft list (ranking, spelling,
FoR
code, duplication
etc
); and did
not
extend to reiteration of previous arguments
406 responses were received, 142 of these related to 80 journals in 1503
Responses of the following type were acted upon:
Duplications
Reports of predatory journals
Errors in the title or ISSN/ISBN
Responses of the following type were not acted upon:
Reiteration of a previous submission with no further evidenceSlide15
19 June 2013
ABDC Journal Quality List Review 2013
15
Final outcome
The final rankings distributions for 1503 and 1599 were:
1503
A*: 6.8% (56)
A: 23.4% (192)
B: 24.7% (203)
C: 45.1% (370)
1599
A*: 10.6% (7)
A: 28.8% (19)
B: 33.3% (22)
C: 27.3% (18)