Il progetto EGI- Engage e la - PowerPoint Presentation

Slide1

Il progetto EGI-

Engage

e la

grid

di produzione italiana

Riunione

CCR, 16 settembre 2015

Luciano Gaido

Slide2

EGI-Engage: stato del progetto

EGI-Engage

è la prosecuzione di EGI-InSPIRECoordinatore progetto: EGI.euINFN partecipa come lead beneficiary di una JRU con INAF e INGVIl Grant Agreement e il Consortium Agreement sono stati firmatiRichiesto un amendment perchè un partner tailandese (unfunded) è uscito dal progetto per problemi sulla procedura di arbitratoAttività iniziate il 1 marzoNO kick-off meeting  EGI Conference a Lisbona (18-22 maggio)

2

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide3

EGI-Engage: Informazioni generali

Durata

: 30 mesiPartner coinvolti: 47 (di cui 43 funded)3TotaleEGI.eu INFN Effort (PM)1.161276,523,8 %948,0%Budget (€)8.000.0002.827.136 35,3 %599.5007,5 %Budget totale JRU italiana (INFN+INAF+INGV): 721.5 K€ (9%)

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide4

Distribuzione effort

n

el progetto4Work Packageattivitàpartner coinvolti% effort (PM)NA1Project management48.0%NA2Strategy, policy and communication2017.0%

JRA1-JRA2-SA1

E-infrastructure Commons, Platform for the Data Commons, Operations

12

35.0%

SA2

Knowledge

Commons (Distributed Competence Centres)

47

39.0%

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide5

EGI-Engage: Attività INFN

Due

ruoli di coordinamento generale (EGI), assegnati all’INFN dopo una call pubblica (persone senior scelte in base al CV): Profilo “Operations Officer”: (1 FTE) per Vincenzo Spinoso (30 PM in SA1.1) [Bari]Profilo “Technical Outreach Expert”: (1 FTE) per Diego Scardaci In particolare: 7.5 PM in NA1.2, 1 in NA1.3, 6.5 in JRA1.2 e 15 in JRA2.3 [Catania] Altre attività:DCC Mobrain (SA2.5): 2 PM [Padova] DCC DARIAH (SA2.6): 13.5 PM [Catania]DCC Lifewatch (SA2.7): 3 PM [Bari]JRA2.4 (GPU): 6 PM [

Padova*] NA2.2 (Marketplace & Pay4Use): 5 PM

[Bari]

NA2.3 (SME engagement): 2.5 PM

[CNAF]

SA1.3 (Platform for long tail): 2 PM

[Catania]

*

Contributo

da Milano per modifiche a BLAH5EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide6

Risultati del primo semestre: NA2.1

6

NA2.1 “Communication and Dissemination”: worked  on  the  organisation  of  the  EGI  flagship  events, the EGI Conference in Lisbon (May 2015) and started preparation for the EGI Community Forum in Bari (9-13 Nov 2015).supported dissemination and outreach with its communication channels such as: the EGI Inspired newsletter case studies and external articles launched a revised EGI Champions programme. EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide7

Risultati del primo semestre: NA2.2

7

NA2.2 “Strategy, Business Development and Exploitation”: exploring cross-­border procurement opportunities, understanding how to move the pay-­for‐use prototype into production creating the conceptual model of the EGI marketplaceimproving the management of EGI services through better defined processes and procedures according to the FitSM IT service management standard. The Open Science Commons, discussed in Lisbon among policy makers, national research ministry representatives and e-­Infrastructure representatives, has been adopted at a European level in the conclusions of the European Competitiveness Council (May 2015)The implementation plans advanced in the area of the Data Commons: a white paper on how the novel initiative “Open Science Cloud” could be an instrument to implement the data commons in all scientific disciplines has been produced and it will

be further advanced with the contribution of user communities.

the

EGI strategy

has been finalized through

a broad consultation process

started

in

2014

An

activity to

revise and update the EGI service portfolio and solutions has started at PM05.EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide8

Risultati del primo semestre: NA2.3

8

NA2.3 “SME/Industry Engagement and Big Data Value Chain”:spent the first period setting up the business engagement programme for outreach to industry started discussions with organizations to formalize potential collaborations including the Big Data Value Association. an initial market analysis was conducted in selected sections (marine, agriculture)EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide9

Risultati del primo semestre: JRA1.1

9

JRA1.1 “Authentication and Authorization Infrastructure”:has been working on establishing contacts with other projects and infrastructures (AARC, GN4, EUDAT2020 and PRACE) in order to work together towards an interoperable AAI A list of new AAI services including identity and service provider proxy services, have been added to the AAI service roadmap of EGIsupported user communities – most importantly the EPOS Research Infrastructure – in defining its AAI infrastructure the analysis of the interoperability of the EGI and EUDAT AAI services has started pilot activities to test a non‐X.509 based AAI model for the EGI Cloud federation continued (https://wiki.egi.eu/wiki/VT_AAI)an operational plan to roll out such services into production will be produced at the beginning of the second part of the first reporting period

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide10

Risultati del primo semestre: JRA1.2, JRA1.3, JRA1.4 e JRA1.5

10

JRA1.2 “Service Registry and Marketplace”: focused on requirements gathering and analysis to prepare a concept for the marketplace and identify interfaces with other EGI tools many surveys and interviews involving different stakeholders have been organized to collect input from service providers and user communitiesJRA1.3 “Accounting”:some developments around Data, Cloud and Storage accounting collection of requirements for the new Accounting PortalJRA1.4 “Operations tools”: a number of developments to improve existing EGI tools according to EGI needs

JRA1.5 “Resource Allocation – e-

­GRANT”:

continued implementation of improvements needed for EGI Resource Allocation and EGI Pay-

­for-

­‐Use

process

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide11

Risultati del primo semestre: JRA2.1

11

JRA2.1 “Federated Open Data”:gathered open data use cases and requirements from different data providers and research disciplines during the specific session of the EGI conference in Lisbonstarted collecting communities’ requirements to capture plans and need for data managementa milestone report (M4.1 "Open Data Platform: requirements and implementation plans“) is being preparedan initial architecture for the open data platform has been designed. a Virtual Team involving EGI technology providers and resource centres was kicked off aiming at the implementation of a distributed testbed for testing of data and storage federation solutions. The current two primary use cases are related to the Human Brain Project and ELIXIR

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide12

Risultati del primo semestre: JRA2.2

12

JRA2.2 “Federated Cloud”:efforts focused in setting up process for tracking and prioritization of the needed developments maintenance of the existing integration tools regarding the support of OCCI support – the VM management open standard of OGF developed the Keystone-­VOMS integration module and new version of the accounting collection tools for OpenStack (https://keystone-voms.readthedocs.org/en/stable-juno)initial contact with some of Product Teams for UMD integration has been established and the integration is now an ongoing processthe EGI Cloud federation model was revised and discussed with the aim of accommodating the needs of community clouds only supporting cloud middleware framework native interfaces.EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide13

Risultati del primo semestre: JRA2.3

13

JRA2.3 “e-­Infrastructures Integration”:started integration of D4Science by porting of two selected use cases to the EGI Federated Cloud defined integration plan of the EGI Federated Cloud and the Canadian distributed cloud infrastructure (CANFAR) to realize the astronomy “Data Commons” integration of CANFAR will be a pathfinder to support the LOFAR and SKA research infrastructures defined integration plans between the Australian research cloud (NeCTAR) and EGI. collaboration activities have been identified to support life science, astronomy and astrophysics and brain research including the possibility to share virtual laboratories and community‐specific tools on the cloud

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide14

Risultati del primo semestre: JRA2.4

14

JRA2.4 “Accelerated Computing”: working on enabling GPGPU and co-­processor support for the EGI HTC and Cloud platform an HTC testbed has been set up and a new version of BLAH has been prototyped for including GPU directives to be passed to the underlying LRMS. the main cloud technologies have been reviewed and GPGPU virtualization in KVM/QEMU has been analyzed and performance tests have been carried out a grid site with GPGPU support has been set up in Florence (cegpu.cerm.unifi.it:8443/cream-pbs-batch)a cloud site with GPGPU support has been set up in Slovacchia (keystone3.ui.savba.sk)EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide15

Risultati del primo semestre: SA1.1

15

SA1.1 “Operations Coordination”:focused on provisioning of reliable production infrastructure by improving and modifying several procedures and manuals supporting operations activities work towards sustainable delivery of EGI Core activities through oversight of the performance  preparing new bidding for period of 20 Months starting from May 2016 significant improvement has been also done in terms of UMD Software provisioning by optimization of verification and releasing the middleware distribution process.Task a cui collabora V. Spinoso, verifica a luglio positiva  confermata

attribuzione attività e budget a INFN

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide16

Risultati del primo semestre: SA1.2

16

SA1.2 “Development of Security Operations”: has been working on security requirements in particular security requirements on endorsed Virtual Machine images new versions of the “EGI-­CSIRT Critical Vulnerability Handling” and “"EGI Software Vulnerability Group – Strategy and Vulnerability Issue Handling" procedures have been produced and work started on four new or revised policies work towards planning and developing a security challenge of the EGI Federated Cloud services coordination of activities with security teams from other e-­Infrastructures has also started.EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide17

Risultati del primo semestre: SA1.3

17

SA1.3 “Integration, Deployment of Grid and Cloud Platforms”: spent effort on prototyping the long tail of science platform integrated with the Catania Science Gateway Framework work towards defining how to integrate the ESA authentication system with the Federated cloud has been performed by EGI and ESA the activities for the D4Science integration focused on the selection of the use cases and services that will run in the EGI Federated cloudEGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide18

Risultati del primo semestre: SA2.1

18

SA2.1 “Training”: worked on the creation of an experts’ network that would collaborate in provisioning and designing training activity provisioning of training events, developing online content in the form of webinars about the EGI HTC, data management and cloud services the task focused on designing and preparing a framework to support trainers and training provisioningEGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide19

Risultati del primo semestre: SA2.2

19

SA2.2 “Technical User Support”: focused efforts in setting up process for gathering and tracking user requirements the engagement plan has been periodically revised and updated contacts have been established with two initiatives: the Human Brain Project and the International Cancer Genome Consortiumall Competence Centres, with the exception of the BBMRI one – who is starting at PM07 –, delivered significant work towards their objectives. Most of the work has been done around requirements gathering which will be the base for further developments. EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide20

EGI-Engage support to the DCC

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide21

Competence Centres in EGI-Engage

ELIXIR

SA2.3BBMRI

SA2.4

MoBrain

SA2.5

DARIAH

SA2.6

LifeWatch

SA2.7

EISCAT_3D

SA2.8

EPOS

SA2.9

Disaster Mitigation

SA2.10

M6

M24

M18

M1

M30

Training

Technical User Support

SA2.1

SA2.2

Life sciences

Humanities

Environmental sciences

Generic

Mar’15

Aug’15

Sep’16

Mar’17

Sep’17

Prep. phase

Prep. phase

Prep. phase

M12

Mar’16

Slide22

Competence

Centers

: ELIXIR e BBMRI ELIXIR (Pan-European RI to build a sustainable infrastructure for biological information supporting life science research) [CSC]CC will select representative use case workflows to develop demonstrators targeting FedClouddevelop best practices materialscreation of VMs for demonstratorsDatasets replicationBBMRI (Biobanking

and BioMolecular resources

RI)

[

SURFsara

]

Cloud

related

objectives

:

Secure data storage infrastructurebiobank interoperability and data discoveryallowing data sharingrespecting protection level required by owners of data and samples

Biobank

Workflow

support

:

Big-data

processing

with

BioBankCloud

(

based

on

Hadoop

)

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

22

Slide23

Competence

Centers

: MoBrain e DARIAHMOBRAIN (Research from Molecule to Brain) [SURFSara]Builds on the experience of WeNMR and NeuGrid4YouCloud related objectives:Bring computing to the datamanaging data, running jobs with ScipionOffer custom virtual machines to research for specific scenarios Enable GPU processing in the EGI infrastructure  GPUs in CloudDARIAH (research from the areas of the Arts and Humanities) [SRCE/RBI]Cloud related objectives:DARIAH eScience Gateway on EGI (based on WS-PGRADE/gUSE) Storing and Accessing DARIAH contents on EGI (based on gLibrary)Multi-Source Distributed Real-Time Search and Information Retrievalbig data stream and batch processing techniques

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

23

Slide24

Competence

Centers

: Lifewatch e EISCAT_3DLIFEWATCH (capture and address the requirements of Biodiversity and Ecosystems research communities) [CSIC]Cloud related objectives:Big data & Ecological Observatories  tools for managing and processing the data from observatories & simulations (R support required)Workflows & Virtual Labs  web gateways, COMPSsCitizen Science:receive

data by

citizens

,

analyze

these

data

open data

access

?

EISCAT_3D (Next generation incoherent scatter radar system) [EISCAT]Cloud related objectives:Develop the EISCAT portal to access

the

data archives

Basic

reanalysis

in

the

portal

processing

of (

big

?) data

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

24

Slide25

Competence

Centers

: EPOS e Disaster MitigationEPOS (Design of the use of grid and cloud for the integrated solid Earth Sciences research as part of the European Plate Observing System) [INFN/INGV]Cloud related objectives:Identify and validate authentication and authorisation services for EPOSTest cloud resources and usage modelscompute/data resources for applicationshosting services Science

gateway (no specific

platform

yet

)

DISASTER MITIGATION

(

Customised

IT services to support the climate and disaster mitigation researchers) [ASGC]Cloud related objectives:web portals for tsunami wave

propagation

simulations

and

for

WRF-

based

weather

simulation

Application

of

the

simulation

portals

for

scientific

scenario

in

disaster

mitigation

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

25

Slide26

La

grid

di produzione italiana26Con la conclusione di EGI-Inspire, e soprattutto la fine di IGI, la struttura di gestione, controllo e supporto della Grid di produzione italiana ha subito un lungo e complesso processo di riorganizzazione.Credo sia doveroso ringraziare tutte le persone che, a vario titolo, hanno contributo alle attività di operations, mantenimento del middleware, user support e formazione.EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015 Vorrei menzionare e ringraziare in modo particolare le persone che hanno contribuito al coordinamento delle attività:Valerio VenturiPaolo VeronesiDaniele CesiniEmidio Giorgio Un ringraziamento speciale va ad Alessandro Paolini, che ha svolto le attività di operations con notevole cura e dedizione.

Slide27

La

grid

di produzione italiana: alcune novità27Con l’avvio di EGI-Engage e l’impegno INFN in alcuni altri progetti importanti, la struttura di coordinamento delle operations è cambiata in modo significativo.Tuttavia questi cambiamenti sono prevalentemente ‘dietro le quinte’, ma non cambiano sostanzialmente procedure e canali di comunicazione (vedi le prossime slide).La nuova struttura di coordinamento e gestione delle operations a livello italiano, che è in fase di attivazione, sarà più snella rispetto alla precedente e prevede il passaggio del coordinamento a Vania Boccia (sezione di Napoli).Vania sarà anche la persona di riferimento per le operations e per la security in EGI.In queste attività sarà coadiuvata da Sergio Traldi (PD) e Diego Michelotto (CNAF).La nuova organizzazione sarà operativa (sperabilmente) a inizio ottobre e Cristina Aiftimiei sta dando un grande contributo per garantire la continuità e facilitare la transizione (grazie!!!)EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide28

Su più di 90 siti registrati nel GOCDB:

34 siti sono in stato ‘closed’

dei 55 siti di produzione con middleware EMI/UMD 3, 2 sono ‘suspended’ e 2 in fase di ri-certificazione e di 4 si sta valutando la chiusuraAttività recenti svolte in ambito operations Censimento dei siti «attivi»Dismissione dei siti con problematiche serie, in accordo con i siteadminAggiornamento dei contatti dei SiteOperationManager e dei SecurityOfficer sul GOCDB Ottimizzazione delle mailing listsAttività utili:Revisione/aggiornamento delle documentazioni utili ai siteadminL’infrastruttura Grid italianaEGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015 28

Slide29

Nulla è cambiato, si fa riferimento alla PROC09 EGI raggiungibile dal link:

https://wiki.italiangrid.it/twiki/bin/view/GridOversight/Info4RODs#Certificazione_dei_siti

Prevede i seguenti passi:Registrazione sul GOCDB del nuovo sito (e relative informazioni di contatto) (ad opera dei siteadmin)Controllo delle informazioni pubblicate dal sitoTest di funzionalità dei servizi GRIDProcedure di certificazione/ri-certificazioneEGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015 29

Slide30

Operations

portal (

https://operations-portal.egi.eu):ROD dashboard (Regional Operators on duty)COD dashboard (Central Operators on duty)  il team COD è stato sostituito dal team EGI Operations SupportSecurity dashboardAvailability/Reliability monitoringAccounting portal EGI: http://accounting.egi.eu/egi.php?Path=1.41Accounting portal INFN: https://faust01.to.infn.it/#/dashboard/script/pledge_sum.jsTicketing systems:Italian GRID helpdesk: https://xgus.ggus.eu/ngi_it/index.php?mode=indexGGUS: https://gus.eu/index.phpL’organizzazione delle operations in EGI è descritta in questa wiki:https://wiki.egi.eu/wiki/EGI_Infrastructure_operations_oversightTool utili/disponibili

EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

30

Slide31

Mailing list attive:

igi-noc@lists.italiangrid.it

: è la lista dei National Operation Officershelpdesk@lists.italiangrid.it: è la email dei site supportersgrid-sec-contacts@lists.italiangrid.it: è la lista dei security officer dei siti della NGI_ITLink utili:Procedure CSIRT:https://www.italiangrid.it/grid_operations/grid-security/operational-procedures Procedure OPERATIONS:https://wiki.egi.eu/wiki/Operations_Procedures Mailing list attive e link utiliEGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015 31

Slide32

Oltre le

operations

…32Middleware:l’INFN ha sviluppato e ha mantenuto un certo numero di componenti Grid che sono usate sulla grid europea (wLCG/EGI): VOMS, CREAM, STORM, WMS, ...L’INFN si è impegnato a garantire a wLCG il supporto e l’aggiornamento (minimale) di queste componenti finchè sarà necessario  non c’è personale dedicato per questo, il supporto viene garantito dal personale INFN (a t.i. o t.d.)User Support (aka attività verso altre comunità di utenti):È svolto in alcune sezioni sulla base di attività/interessi locali o nell’ambito di alcuni progettivedi prossima slideEGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

Slide33

I documenti CCR sulla Cloud: dovrebbero tenere conto delle attività cloud in EGI-Engage e degli altri progetti

p

er sfruttare al meglio le attività e i risultati nei progettiper capitalizzare le attività CCR al di fuori dell’EnteIl prossimo Community Forum di EGI si terrà a Bari dal 10 al 13 novembre. E’ una occasione da non perdere per:rafforzare i nostri legami con altre comunità di utentipresentare i risultati che abbiamo ottenuto e la nostra visione sulla cloud consolidare la nostra posizione a livello nazionale (giornata nazionale il 9 novembre)E’ possibile organizzare altri eventi rilevanti (‘colocated events’), ad es. per INDIGOI prossimi passi?33EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015 Servono però una strategia e una regia per tutte le attività per e con le altre comunità (non solo scientifiche)Dalla mia presentazione al WS CCR di maggio

Da fare!

In corso, ma la giornata nazionale è ancora da definire

Siamo ancora lontani dalla meta