Telles Nobrega Big Data Processing as a Service Big Data processing framework provision Ambari Cloudera Vanilla upstream hadoop MapR Spark Storm EDP Elastic Data Processing Running jobs on those frameworks ID: 692468
Download Presentation The PPT/PDF document "Sahara Project Onboarding" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Sahara Project Onboarding
Telles NobregaSlide2
Big Data Processing as a Service
Big Data processing framework provision
AmbariClouderaVanilla (upstream hadoop)MapRSparkStormEDP (Elastic Data Processing)Running jobs on those frameworks
Sahara OverviewSlide3
Sahara Overview
Namings
ServicesSpecific roles of a cluster instanceNode Group TemplatesDescribes a group of nodes within a clusterCluster TemplatesDescribes a group of Node Group templates to form a cluster
Job binary
Job executable (jar, .py)
Job template
Describes a job to be run
Data Sources
Source used pull data into Sahara or store data out of SaharaSlide4
Sahara Overview
Image Generation
Sahara Image Elementstox -e venv -- sahara-image-create -p spark -s [1.3.1|1.6.0|2.1.0|2.2.0] # spark standalonetox -e venv -- sahara-image-create -p vanilla -v 2.7.1 -s [1.6.0|2.1.0|2.2.0] # spark on vanillaSahara Image Packtox -e image -- sahara-image-pack --image CentOS.qcow2 \
--config-file etc/sahara/sahara.conf \
cdh 5.7.0 [cdh 5.7.0 specific arguments, if any]
Slide5
ArchitectureSlide6
ArchitectureSlide7
Sahara Repos
Current
sahara - https://git.openstack.com/openstack/saharasahara-image-elements - https://git.openstack.com/openstack/sahara-image-elementspython-saharaclient - https://git.openstack.com/openstack/python-saharaclient
sahara-dashboard -
https://
git.openstack.com
/openstack/sahara-dashboard
sahara-extra -
https://
git.openstack.com
/openstack/sahara-extra
Planned
sahara-plugins
Slide8
Running Sahara
# first terminal$ sahara-venv/bin/sahara-api --config-file sahara-venv/etc/sahara.conf# second terminal$ sahara-venv/bin/sahara-engine --config-file sahara-venv/etc/sahara.confSlide9
Running Sahara Tests
Unit Tests (from sahara)tox -e py27tox -e py35tox -e pep8...
Scenarios (From sahara-tests)
sahara-scenario {posargs}
Tempests
Install sahara tempest plugin from sahara-tests
Follow the default tempest process (see tempest docs)
Slide10
Sahara Docs
https://docs.openstack.org/sahara/latest/https://docs.openstack.org/sahara-tests/latest/https://docs.openstack.org/sahara/latest/reference/restapi.htmlhttps://developer.openstack.org/api-ref/data-processing/