Cloud Processing at ESA [EO Payload Ground Segment] - PowerPoint Presentation

Slide1

Cloud Processing at ESA [EO Payload Ground Segment]

Cristiano Lopes, ESACEOS WGISS-4030/09/2015Slide2

Overview – Cloud Computing Uses

Cloud computing and Earth Observation:Processing,Archiving,Distribution,

Discovery…

Cloud computing uses in ESA EO Ground-Segment

Past activities

On-going activities

Future activities Slide3

Overview – Cloud Computing

Cloud Computing definition:Cloud Computing, Wikipedia article

"

Emerging Technologies/WGISS" Presentation

, Slides 10-15, CEOS Plenary

28

th

Cloud is simply outsourcing in a new form

, Joe

McKendrick, Forbes, Oct 18

th

2014

Traditional cloud service models:

Infrastructure As a Service (IaaS)

Platform As a Service (PaaS)

Software as a Service (SaaS) Slide4

Overview – Cloud

“Cloud services” are simply ICT resources that are hosted and managed by someone else and offered as a service (XaaS).Slide5

Distribution – Content Delivery Network

EO Data Distribution over Content Delivery Network (CDN):CDN provides dynamic geographical load balancing (local caching);GOCE Gravity field data, distributed over CDN with good success (5 releases);

ENVISAT AND ERS

SAR data

,

also distributed with low success (due to data specificities – on-demand data)

Plus:

Very high user satisfaction – download performance,

Transparent caching of data at different

locations,

Peak loads not affecting ESA infrastructure

Minus:

Expensive,

Only for “stable” and consolidated data

Currently

not in use – replaced by a managed hosting service

.Slide6

Infrastructure – Cloud Deployment

EO Data Handling Infrastructure deployment in Cloud Provider:Full deployment of existing infrastructure in a Cloud/Virtualised Environment;“Porting” / Deployment of existing applications as-is (templates): Archive, Processing Management, Distribution, Circulation, Discovery, etc.Initial activities pilots led to full reference and integration platforms deployed in the “Cloud”.

Plus:

Very easy to deploy existing applications;

No noticeable functional issues;

Very Flexible – One environment per need (one mission, one project);

Pay-per-use (when needed) – minimises costs (resources are only used when needed).

Minus:

IPR / License issues;

Performance impacts (when compared with physical infrastructure, 10x in some cases);

Security (Policy, very difficult to demonstrate full compliance for production platforms).

Results led to follow-up activities

.Slide7

Processing – Cloud Deployment

EO Data Processing Pilots/Trials with commercial Cloud Provider:Virtual servers allocated in the cloud – Scheduler and Processing Algorithm. Data transferred off-line via media;ENVISAT MIPAS Reprocessing – 200 VM, 400 CPUs for 5 weeks. 8TiB input, 600GB output.ENVISAT ASAR NRT Processing – 10 VM, 20 CPU; runs with 20 and 50 input products taking ~20 and ~45 minutes.

Plus:

Very easy to deploy existing processing algorithms;

No noticeable data quality issues;

Easily scalable (linked to Provided availability).

Minus:

Relative Expensive

(compared to owning hardware);

Static infrastructure (full environment started and stopped manually – templates used for processing algorithms);

Data transfers cumbersome (Disk In, Network out – costs).

Results led to follow-up activities.Slide8

Infrastructure–Managed Hosted Services

DISSHARM Project and Platform:Managed hosting service for ESA EO Data Distribution;Single-Tenant virtualised environment with dedicated storage;Located within the ESA EO WAN Backbone and close to its data entry;

Currently in Production.

Plus

:

Integrated in ESA EO WAN (data travels locally);

Managed as a service (Built-to requirements; Provider responsible for architecture, security);

Bandwidth (currently 2Gbps, scalable to 10Gbps contractually).

Minus

:

Limited resources (VMs, storage);

No local data redundancy (cost reduction choice);

Results led to follow-up activities.Slide9

Infrastructure–Managed Hosted Services Evolution

ELVIS Platform  DISSHARM Evolution:Managed hosting service for ESA EO Data Distribution & Processing;Single-Tenant virtualised environment (in this case with more processing resources) with dedicated storage, with a multi-tenant cloud

connection (Cloud Bursting);

Access to the Provider’s commercial cloud via internal LAN;

Currently on early stages of production (cloud access still not present).

Example use case - SAR On-the-fly Processing:

Process SAR data (ERS, ENVISAT, ALOS) on download request by user;

Perform processing on the back-end transparently to the user;

Cache output product for next user ;

Cloud bursting as

needed by demand (to be implemented

).Slide10

Processing - Hosted Processing

Super Sites Exploitation Platform and Project (SSEP) :Collaborative environment for end-user’s data exploitation, through coordinated access to data, tools and processing power;Virtual workstation on a cloud platform;Integrated information sharing and support

tools;

Demonstrated on

Helix-Nebula

.

Plus:

Change in paradigm (bring users to data);

Abstract ICT from users (transparently multi-sourced resources);

Minus

:

IPR / License

issues (data, tools);

Cloud-API Nightmare (different cloud providers using different APIs).

Results led to follow-up activities

.Slide11

EO Exploitation Platforms

Earth Observation Exploitation PlatformsFollow to the SSEP conceptsOn-going activities for Thematic Exploitation Platforms (TEP):Geohazards (SSEP follow-up),Coastal, Forestry,

Hydrology, Polar and

Urban

TEP activities are industry driven, with input from ESA on overall architecture:

Open source,

Standards,

Infrastructure Independence

Earth Observation Platform - Test bed (Pilot):

Open Tender: Solution defined by provider and offered as a service;

Hosted processing only (no download of source data);

Hosting Sentinel-2, Landsat and ENVISAT MERIS data;

ESA provide free (pre-paid) access to limited test and science usersSlide12

Lessons learned, challenges, opportunities

Cloud Services/TechnologyLessons learned:New solutions for old problems (e.g. Processing, Distribution):Challenges:Technology maturity

 (N

umerous APIs, different SLAs, reliability of suppliers)

ESA Industrial Policy

 Contract placing;

Security  Compliance with ESA Security Rules.

Opportunities

Neutral Environment with open access to anyone, one place to develop, prototype and run applications;

Potentially unlimited resources allowing for new ways to exploit EO data (big data);

Space to define an EO Cloud Architecture.Slide13

Cloud Processing Framework (1)

Processing of EO Data in the Cloud.Challenge:Multiple data sources;Multiple processing algorithms;Multiple possibilities of joining processing algorithms;

Quasi-Unlimited resources to control.

Opportunity for defining standards and best practices:

From existing multiple solutions;

From existing or new standards;

Tailored to the EO Data Specific Needs.Slide14

Cloud Processing Framework (2)

Processing of EO Data in the Cloud.Potential domains:Data DiscoveryWhere



Opensearch?

Data access

How  OpenDap?, WCS?, HTTP GET [from Object Storage]?

Resource management

How to provision Cloud

Resources

 Cloud Scheduler?, Apache jClouds/libCloud?

How to do VM Management

 Condor?, Torque?, Grid(any)?

Workflow management (Algorithms)

How to “chain” algorithms

 BEPL?, XPDL?Slide15

Cloud Processing – CERN ATLAS.

Learning from others  The ATLAS experiment driven by CERN.

“The

Evolution of Cloud Computing in

ATLAS

”, Taylor et all, ATL-SOFT-PROC-2015-049,

27/05/2015.Slide16

Summary

In ESA Earth Observation, a number of activities related to cloud technology and services have been performed.The Cloud is an enabler that allows ESA to perform is data exploitation activities in a more efficient way.We believe there is room for collaboration and cooperation in the use of the Cloud with other partners.

One such possibility of collaboration is the definition of standards, best-practices and an architecture for EO Data Exploitation.

The Cloud Processing Framework is a first example.Slide17

Thank you!

Cristiano LopesGround Segment System Engineer

Earth Observation Programmes

Phone

:

+39 06 941 80735

Email:

cristiano.lopes@esa.int