and Starting HPC Tier 3 Users SESHPC Jan Steiner Zentrum für Informations und Medientechnologie Universität Siegen 41217 SESHPC 2 Outline Motivation Project Structure First Experiences ID: 784367
Download The PPT/PDF document "Services for Experienced" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Services for Experienced and Starting HPC Tier 3 Users (SES-HPC)
Jan Steiner
Zentrum für Informations- und Medientechnologie
Universität Siegen
Slide24.12.17SES-HPC
2
Outline
Motivation
Project Structure
First Experiences
Status
Slide34.12.17SES-HPC
3
Permeability between Tier 3 and higher tiers
Code development on Tier 3
–
productive runs on Tier 1
Less experienced users
Cheaper resources
Start early
Get people onto Tier 3
Facilitate movement to higher tiers
Motivation:
Tier 1
National/ International
Tier 3
Federal/
Local
Tier 2
National/
F
ederal
Slide44.12.17SES-HPC
4
Background: Existing Infrastructure
HorUS
: Current Cluster at
Uni
Siegen
Diverse users
Planned
future cluster
Proposal
underway
Additional
users, e.g. Big Data Good position to support users early
Slide54.12.17SES-HPC
5
Outline
Motivation
Project Structure
First Experiences
Status
Slide64.12.17
SES-HPC
6
Five pillars of support
Teaching and Training
Performance
Anlysis
Third-party Code Support
Tier Change Support
Knowledge Transfer
Slide74.12.17SES-HPC
7
Five pillars of support
Teaching and Training
Beginner and advanced
devs
Hold classes
Advise on external courses
Gauge demand for new courses
Performance Analysis
Third-party Code Support
Tier Change Support
Knowledge Transfer
Slide84.12.17SES-HPC
8
Five pillars of support
Teaching and Training
Beginner and advanced
devs
Hold classes
Advise on external courses
Gauge demand for new courses
Performance Analysis
Experienced code developers
Performance reviews
Performance measurement tools
Third-party Code Support
Tier Change Support
Knowledge Transfer
Slide94.12.17SES-HPC
9
Five pillars of support
Performance Analysis
Experienced code developers
Performance reviews
Performance measurement tools
Third-party Code Support
Users of
comm-ercial
/open-source codes
Support in finding optimal settings
Find most suitable hardware
Tier Change Support
Knowledge Transfer
Teaching and Training
Beginner and advanced
devs
Hold classes
Advise on external courses
Gauge demand for new courses
Slide104.12.17SES-HPC
10
Five pillars of support
Third-party Code Support
Users of
comm-ercial
/open-source codes
Support in finding optimal settings
Find most suitable hardware
Tier Change Support
Dev teams who want to apply for higher tier
hardw
.
Find most suitable hardware
Test and evaluation of software
Knowledge Transfer
Performance Analysis
Experienced code developers
Performance reviews
Performance measurement tools
Teaching and Training
Beginner and advanced
devs
Hold classes
Advise on external courses
Gauge demand for new courses
Slide114.12.17SES-HPC
11
Five pillars of support
Third-party Code Support
Users of
comm-ercial
/open-source codes
Support in finding optimal settings
Find most suitable hardware
Tier Change Support
Dev teams who want to apply for higher tier
hardw
.
Find most suitable hardware
Test and evaluation of software
Knowledge Transfer
All HPC users
Establish and maintain wiki
Organize networking workshops
Performance Analysis
Experienced code developers
Performance reviews
Performance measurement tools
Teaching and Training
Beginner and advanced
devs
Hold classes
Advise on external courses
Gauge demand for new courses
Slide124.12.17SES-HPC
12
Outline
Motivation
Project Structure
First Experiences
Status
Slide134.12.17SES-HPC
13
Example consulting session
Client: PhD student
R script, runs 24 cases in sequence
Script runs 4 weeks, job would often die before that
Paper deadline in 4 weeks
This is not to make fun of him
Slide144.12.17SES-HPC
14
Example consulting session
Act 1
: Troubleshooting
“
W
hat did you set the walltime to?”
“What’s a walltime?”
”
D
efault then, which queue was it in?”
“What’s a queue?”
“OK never mind, lets have a look at your script.”
Slide154.12.17SES-HPC
15
Example consulting session
Act 2:
The Script
„OK, make a shell script with a loop, and set the condition via command line argument for the R script. Then call your script with &“
„What‘s a shell script?“
Script: same calculation done 24 times just within this script
If-blocks with conditions for case 1, 2, …, 24
Slide164.12.17SES-HPC
16
Example consulting session
Act 3: Listen, the Mensa is About
To Close
“Thank you so much! It runs 24 times faster now!”
“Copy your script 24 times, write
condition=1,2,
etc. at the top and name them
script_1.r
,
script_2.r,
…
“
“In the shell script, you write:”
./script_1.r &
./script_2.r &
...
Slide174.12.17SES-HPC
17
Lessons learned
We let this guy on the cluster
Nobody told him anything
Cluster website
At least google “SLURM”
He ran his jobs for months
I met him by sheer coincidence
We actually helped him a lot
Slide184.12.17SES-HPC
18
Lessons learned
Who is the bigger fool?
The fool?
The fool who fails to take him by his hand?
Source: New
Line Cinema
Slide194.12.17SES-HPC
19
Lessons learned
Not
just
user’s job to inform themselves
Don’t know what they don’t know
Not good: “Keep away from cluster”
Help but also educate (sustainability)
Slide204.12.17SES-HPC
20
Outline
Motivation
Project Structure
First Experiences
Status
Slide214.12.17SES-HPC
21
Status
Interviews with all institutes that use cluster
R-script guy is not completely unique
Cluster website review
“Getting started” section
Additional feedback:
Mech.-Eng. Students
Little prior knowledge of Linux
Seminars “Cluster Introduction”, “Linux Introduction”
Slide224.12.17SES-HPC
22
Wanted: second position
Third-party Code Support
Users of
comm-ercial
/open-source codes
Support in finding optimal settings
Find most suitable hardware
Tier Change Support
Dev teams who want to apply for higher tier
hardw
.
Find most suitable hardware
Test and evaluation of software
Knowledge Transfer
All HPC users
Establish and maintain wiki
Organize networking workshops
Performance Analysis
Experienced code developers
Performance reviews
Performance measurement tools
Teaching and Training
Beginner and advanced
devs
Hold classes
Advise on external courses
Gauge demand for new courses
Slide23Thank You For Your K
ind Attention.
4.12.17
SES-HPC
23