ISTICNR Pisa Italy Happy 10 th Birthday CLEF 2009 Workshop 30 September 2 October Corfu Greece CLEF Objectives Stimulate the development of multilingual IR systems for European languages ID: 790398
Download The PPT/PDF document "Welcome to CLEF 2009 Carol Peters" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Welcome to CLEF 2009
Carol Peters
ISTI-CNR, Pisa, Italy
Slide2Happy 10th Birthday
CLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide3CLEF Objectives
Stimulate the development of multilingual IR systems (for European languages !)
Create a CLIR/MLIA community
Construct publicly available test-suites
BY
Conducting annual evaluation campaigns
Designing tracks/tasks to meet emerging needs and to stimulate research in the”right” direction
Objective: truly multilingual/multimedia systems
CLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide4Evolution of CLEF
CLEF 2000 Tracks
mono-, bi- & multilingual text doc retrieval (Ad Hoc)
mono- and cross-language information on structured scientific data (Domain-Specific)
CLEF 2001
New
interactive cross-language retrieval (iCLEF)
CLEF 2002
New
cross-language spoken document retrieval (CL-SR)
CLEF 2003 New multiple language question answering (QA@CLEF) cross-language retrieval in image collections (ImageCLEF)CLEF 2005 New multilingual retrieval of Web documents (WebCLEF) cross-language geographical retrieval (GeoCLEF)CLEF 2008New cross-language video retrieval (VideoCLEF) multilingual information filtering (INFILE@CLEF) CLEF 2009New intellectual property (CLEF-IP) log file analysis (LogCLEF) large-scale grid experiments (Grid@CLEF)
Slide5CLEF Tracks: 2000 - 2009
Slide6Multilingual textual document retrieval (Ad Hoc)
Interactive cross-language retrieval (iCLEF)
Multiple language question answering (QA@CLEF)
Cross-language retrieval in image collections
(ImageCLEF)
Multilingual information filtering (INFILE@CLEF)
Cross-language video retrieval (VideoCLEF)
Intellectual property (CLEF-IP) – New this year
Log file analysis (LogCLEF) – New this year MorphoChallenge (in collaboration with Pascal NoE)
CLEF 2009 Tracks
Pilot:
Grid Experiments (Grid@CLEF)CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Slide7CLEF2009 Coordination
Adaptive Informatics Research Centre, Helsinki University of Technology, Finland
Berlin School of Library and Information Science, Humboldt-Universität zu Berlin
Business Information Systems, U. Applied Sciences Western Switzerland, Sierre, Switzerland
CEA LIST, France
Centre for Evaluation of Human Language & Multimodal Communication Technologies (CELCT), Italy
Centruum vor Wiskunde en Informatica, Amsterdam, NL
Computer Science Department, U. Basque Country, Spain
Computer Vision and Multimedia Lab, U. Geneva, CHData Base Research Group, U. Tehran, IranDept. of Computer Science and Information Systems, U. Limerick, Ireland Dept. of Information Engineering, U. Padua, Italy
Dept. of Information Science, U. Hildesheim, GermanyDept. of Information Studies, U. Sheffield, UK
Dept. of Medical Informatics and Clinical Epidemiology, Oregon Health and Science U., USA
Dept. Medical Informatics, U. Hospitals and University of Geneva, Switzerland
Department of Medical Informatics, Aachen University of Technology (RWTH), GermanyEvaluations and Language Resources Distribution Agency, Paris, FranceGERiiCO, Université de Lille, FranceIdiap Research Institute, SwitzerlandInformation Retrieval Facility (IRF), AustriaLaboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur (LIMSI), Orsay, FranceU. Nacional de Educación a Distancia, Madrid, SpainLinguateca, Sintef ICT, NorwayLinguistic Modelling Lab., Bulgarian Acad SciMatrixware Information Services, AustriaMediamatics, Delft University of Technology, The NetherlandsMitre Corporation, USANIST, USANLE Lab., Universidad Politènica de Valencia, SpainResearch Institute for Artificial Intelligence, Romanian Academy, RomaniaRomanian Institute for Computer Science, RomaniaRoyal Institute of Technology (KTH), Stockholm, SwedenSchool of Computing, DCU, IrelandSwedish Institute of Computer Science, Sweden TALP Research Center, Universitat Politécnica de Catalunya, Barcelona, Spain. CLEF is coordinated by the Istituto di Scienza e Tecnologie dell'Informazione, Consiglio Nazionale delle Ricerche, PisaThe following Institutions are contributing to the organisation of the different tracks of the CLEF 2008 campaign:CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Slide8CLEF Steering Committee
Maristella Agosti, U.Padove, Italy
Martin Braschler, Zurich, Switzerland
Amedeo Cappelli, ISTI-CNR & CELCT, Italy
Hsin-Hsi Chen, National Taiwan U., Taipei, Taiwan
Khalid Choukri, ELRA/ELDA, Paris, France
Paul Clough, University of Sheffield, UK
Thomas Deselaers, RWTH Aachen University, Germany
Giorgio Di Nunzio, U. Padova, ItalyDavid A. Evans, Clairvoyance Corporation, USA Nicola Ferro, U. Padova, Italy
Christian Fluhr, CEA-LIST, Fontenay-aux-Roses, France Norbert Fuhr, University of Duisburg, Germany
Frederic C. Gey, U.C. Berkeley, USA
Julio Gonzalo, LSI-UNED, Madrid, Spain
Donna Harman, NIST, USA Gareth Jones, Dublin City University, Ireland Franciska de Jong, University of Twente, Netherlands Noriko Kando, NII, Tokyo, Japan Jussi Karlgren, SICS, Sweden Michael Kluck, German Institute for International and Security Affairs, Berlin, Germany Natalia Loukachevitch, Moscow State University, Russia Bernardo Magnini, ITC-irst, Trento, Italy Thomas Mandl, U. Hildesheim, GermanyPaul McNamee, Johns Hopkins University, USA Henning Müller, University & University Hospitals of Geneva, SwitzerlandDouglas W. Oard, University of Maryland, USA Anselmo Peňas, LSI-UNED, Madrid, SpainVivien Petras, Humboldt University, Berlin, GermanyMaarten de Rijke, University of Amsterdam, Netherlands Diana Santos, Linguateca, Sintef, Oslo, Norway Jacques Savoy, University of Neuchatel, Switzerland Peter Schäuble, Eurospider Information Technologies, Switzerland Richard Sutcliffe, University of Limerick, Ireland Hans Uszkoreit, German Research Center for Artificial Intelligence (DFKI), Germany Felisa Verdejo, LSI-UNED, Madrid, SpainJosé Luis Vicedo, University of Alicante, Spain Ellen Voorhees, NIST, USA Christa Womser-Hacker, University of Hildesheim, Germany CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Slide9CLEF2009:
Track Coordinators
Ad Hoc
:
Abolfazl AleAhmad, Hadi Amiri, Eneko Agirre, Giorgio Di Nunzio, Nicola Ferro, Nicolas Moreau,
Arantxa
Otegi, Vivien PetrasiCLEF:
Paul Clough, Julio Gonzalo, Jussi Karlgren QA@CLEF: Iñaki Alegria, Davide Buscaldi, Luís Miguel Cabral, Pere
R. Comas, Corina Forascu, Pamela Forner, Olivier
Galibert
, Danilo Giampiccolo, Nicolas Moreau, Djamel Mostefa, Petya Osenova, Anselmo Peñas,
Álvaro Rodrigo, Sophie Rosset, Paolo Rosso, Diana Santos, Richard Sutcliff and Jordi TurmoImageCLEF: Brian Bakke, Steven Bedrick, Barbara Caputo, Paul Clough, Peter Dunker, Thomas Deselaers, Thomas Deserno, Ivan Eggel, Mark Oliver Güld, William Hersh, Patric Jensfelt, Charles E. Kahn Jr., Jana Kludas, Jayashree Kalpathy–Cramer, Henning Müller, Stefanie Nowak, Monica Lestari Paramita, Andrzej Pronobis, Saïd Radhouani, Mark Sanderson, Tatiana Tommasi, Theodora Tsikrika and Petra Welter VideoCLEF: Gareth J.F. Jones, Martha Larson and Eamonn Newman INFILE: Romaric Besançon, Stéphane Chaudiron, Khalid Choukri, Meriama Laïb, Djamel Mostefa and Ismaïl TimimiCLEF-IP: Florina Piroi, Giovanna Roda, John Tait and Veronika Zenz LogCLEF: Maristella Agosti, Giorgio Di Nunzio, Christine Doran, Inderjeet Mani, Thomas Mandl, Julia Maria Schulz and Alexander Yeh Grid@CLEF: Nicola Ferro and Donna HarmanMorphoChallenge: Graeme W. Blackwood, William Byrne Mikko Kurimo, Ville T. Turunen and Sami Virpioja CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Slide10CLEF 2009:
Participating
Groups
CLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide11CLEF: Trend in Participation
CLEF2009:
Europe
= 81(69); N. America = 18(12); Asia = 16(15), S. America = 1(3), Africa = 1(
1
)
CLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide12CLEF2009:Participants per Track
Ad Hoc: 28(26)
iCLEF: 6(6)
QA@CLEF: 25(29)
ImageCLEF: 62 (42)
INFILE: 5(1)
VideoCLEF: 7(5)
MorphoChallenge: 9(6)
CLEF-1P: 15 (new)
LogCLEF: 6 (new)Grid@CLEF: 2 (pilot)
CLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide13Participation by Track
CLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide14CLEF 2000 – 2009
Participation per Track
CLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide15CLEF Achievements
Creation of very active multidisciplinary international
research community
Investigation of core issues in MLIA
development of multiple language processing tools
creation of linguistic resources
implementation of appropriate cross-language retrieval
models and algorithms for different tasks and languages;Creation of important reusable test collections and resources in diverse media for a large number of European languagesSignificant and quantifiable improvements in the performance of MLIA systems
CLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide16Test Collections
2000
News documents in 4 languages
GIRT German Social Science database
2009
A subset of CLEF multilingual corpus of news documents (Ad Hoc WSD-Robust)
Hamshahri Persian newspaper corpus (Ad Hoc Persian task)
TEL Library catalog records plus log files (Ad Hoc TEL task and LogCLEF
)
Flickr web-based image database (iCLEF) ResPubliQA document collection, (QAatCLEF: ResPubliQA) Transcripts of European parliamentary sessions & French news broadcasts (QAST) BELGAPICTURE image collection (ImageCLEFPhoto) Multilingual collections of Wikipedia documents and images (ImageCLEFwiki) Articles and images from Radiology and Radiography; IRMA collection for medical image annotation (ImageCLEFmed and medAnnotation) Dutch and English documentary television programs (VideoCLEF) Agence France Press (AFPnewswire stories in Arabic, French and English (INFILE) Patent documents from the European Patent Office (CLEF-IP)
Slide17Outline of Talk
Why, How, What
DIRECT@work in CLEF
CLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide18CLEF
Publications
9 CLEF
Post-workshop
Proceedings
,
Springer
LNCSNearly 1000 Working Notes Reports on-lineGoogle
Scholar > 15,000 publications
with
CLEF – cross-language information retrievalCLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Slide19CLEF 2008 Proceedings
Evaluating
Systems
for
Multilingual and Multimodal Information Access 9th Workshop
of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised
Selected Papers
Series
:
Lecture Notes in Computer Science , Vol. 5706Peters, C.; Deselaers, T., Ferro, N., Gonzalo, J., Jones, G.J.F., Kurimo, M., Mandl, Th.; Peñas, A.; Petras, V (Eds.) CLEF 2009 Workshop30 September – 2 October, Corfu, GreeceMany, many thanks to Danilo Giampiccolo, CELCT
Slide20CLEF & TrebleCLEF
CLEF 2008 & 2009 is an activity of the TrebleCLEF Coordination Action under the Seventh Framework Programme of the European Commission.
TrebleCLEF organises a set of dissemination activities in the multilingual information access field.
CLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide21Treble-CLEF
The CLEF research results have led to development of a new generation of multilingual retrieval system prototypes
BUT
lack of technology transfer
Treble-CLEF
extends the CLEF activity by:continuing to promote MLIA R&D via evaluation campaigns;providing a consistent training activity: tutorials, workshops, summer school;
producing best practice guidelines for system implementation;providing resources to encourage the multilingual system development.www.trebleclef.euCLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide22Approach
Evaluation
test collections and laboratory evaluation
user evaluation
log analysis
Best Practices & Guidelines
system-oriented aspects of MLIA applications
collaborative user studies
user-oriented aspects of MLIA interfaces
Dissemination and Training
tutorials
workshops
summer schoolCLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Slide23Treble-CLEF Events
Workshop on Novel Methodologies for Evaluation in Information Retrieval, ECIR’08, Glasgow, Scotland
Workshop on Best Practices for the Development of Multilingual Information Access Systems, Segovia, Spain, June 08
Workshop on Best Practices for System Developers: Bringing Multilingual Information Access to Operational Systems, Winterthur, Switzerland, October 2008
Workshop on Best Practices in Query Log Analysis, London, May 2009
Summer School on Multilingual information Access, Pisa, June 2009
MLIA Technology Day – Dissemination of Results of Best Practices Workshops, 8 December 2009
CLEF 2009 Workshop
30 September – 2 October, Corfu, Greece
Slide24CLEF 2010
Can CLEF continue
without
external funding
for
the
central coordination???????
Participate in the discussion on Friday afternoon
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Slide25Before I Forget
Posters
:
Put up
between
8.00 & 13.30
Thursday
morningPool Room 2:
ImageCLEF PostersPool Room 1: All other Posters & Demos
USB StickECDL Conference
Proceedings
All Workshop ProceedingsSocial Dinner - TonightHotel restaurant – 8 pm (Badge)
Slide26CLEF 2002
Rome
CLEF 2006
Alicante
CLEF 2005
Vienna
CLEF 2004
Bath
CLEF 2003
Trondheim
CLEF 2008
Aarhus
CLEF 2001DarmstadtCLEF 2007Budapest
CLEF 2000
Lisbon
CLEF2009
Corfu
CLEF2010
?????
Come to CLEF – and see Europe!
Slide27CLEF 2009
Thank you for your attention
and
ENJOY THE Workshop !
CLEF 2008 Workshop, Aarhus, Denmark
17-19 September 2008