Introduction to Linux and Supercomputers - PowerPoint Presentation

Slide1

Introduction to Linux and Supercomputers

Doug Crabill

Senior Academic IT Specialist

Department

of Statistics

Purdue University

dgc@purdue.eduSlide2

What you will learn

How to log into a Linux Supercomputer

Basics of a Linux shell, including moving/editing/creating/deleting files

T

he basic filesystem structure for the data files and scripts you will be using

How to launch programs, see their progress, and terminate them early if necessarySlide3

The rice.rcac.purdue.edu clusterSlide4

The rice.rcac.purdue.edu cluster

Slide5

An individual nodeSlide6

Rice supercomputer stats

660 Nodes, 13,200 total CPU cores

Each nodes has 20 CPU cores, 64GB RAM

1.7 Petabytes of scratch space for this cluster alone

Several Petabytes of long term storage shared among all clusters

Currently #319 on top500.org, Conte is #100.Slide7

Why Linux?Slide8

Why Linux?Slide9

Why Linux?

Can be desktops, but tend to be larger servers in some remote, environmentally controlled data center

Multiple CPU cores per server (~20)

Large amounts of RAM (64GB – 256GB is common)

Multiple users can use the same computer simultaneouslySlide10

Why Linux? (cont.)

Can interact with a graphical interface

More common to interact with a text based interface

Servers tend to stay up for a long time between reboots (months)

Commonly launch programs and walk away for days, weeks, or months as they run

Computations can scale up as servers addedSlide11

But where are the keyboards, mice, and monitors?Slide12

But where are the keyboards, mice, and monitors?Slide13

We use four computer systems

Windows Desktop

: computer in front of you

Thinlinc

: Linux server accessed in a web browser to gain a graphical interface

Rice front-end

: four of them, used to access Rice nodes, submit large jobs to the cluster

Rice node, aka

compute node, back-end

node

: 660 of them, this is where we computeSlide14

Doug’s cute analogy

Thinlinc

: Like the bus you take from the airport to the hotel

Rice front-end

: Like the front desk at the hotel lobby where we make room reservations

Rice node, aka compute node, back-end

node

: Like a private hotel room, where you are free to do your thingSlide15

Thinlinc Linux graphical interface

We will use

Thinlinc

in a web browser (Firefox) to provide a graphical user interface for us to log into a Rice front-end, then a Rice node, where we will do the real computing

Here is what it looks like!Slide16

Logging into ThinlincSlide17

Thinlinc screenshotSlide18

Thinlinc sessions can persist!

Programs/windows that are open and running can persist after closing

browser

Smile patiently while I demonstrate

persistence

If you explicitly click, you will be logged completely out and windows will not persistSlide19

Terminal WindowSlide20

What is a “shell”?

A text-based user interface used to launch programs

Allows arguments to be passed to programs, specifying input/output files, running multiple programs at once

The shell we use is called “bash”

Terminal

is one way of

accessing

a

shellSlide21

Shell promptSlide22

Use

ssh

to connect to Linux

Implementations of

ssh

, available on Windows, Mac, Linux, Android, IOS, etc.

Many implementations of

ssh

exist on each OS, any can be used. Similar to how many web browsers exist, and any can be used

s

sh

alone provides a text-based interface

When used in conjunction with other technologies, graphical interface can be usedSlide23

Connect to Rice front-endSlide24

Rice front-end promptSlide25

Connect to a Rice nodeSlide26

Connect to a Rice nodeSlide27

Slide28

Slide29

Let’s all log in!Slide30

Multiple Terminal windows

Always check your prompt name/color to make sure where your commands will run

To open an additional Terminal window on the same server as an existing Terminal, type:

Terminal &

If you omit the

&

, the first Terminal cannot be used again until the second is closed

Type

exit

to log out of a shellSlide31

Using copy/paste

Using the Windows shortcuts Control-C and Control-V will generally not work, because those keys are special

Either select the text and select Edit/Copy and Edit/Paste

Or select the text which implicitly copies it, and press down on the mouse wheel to paste (don’t roll it, press down like it’s a button)Slide32

Filesystems

Filesystems on Linux similar to network drives on Windows, but without drive letters

Example directories on different filesystems: /home/train01, /depot/

nihomics

, /

scratch/rice/t/train01

Hierarchical. Directory names separated by “/”, not by “\” like with Windows. Avoid spaces in filenames and directory names.Slide33

Filesystems

Filesystems on Linux similar to network drives on Windows, but without drive letters

Example directories on different filesystems: /home/train01, /depot/

nihomics

, /

scratch/rice/t/train01

Hierarchical. Directory names separated by “/”, not by “\” like with Windows. Avoid spaces in filenames and directory names.Slide34

Filesystems on RiceSlide35

Filesystems visible from Windows

You have an O: drive which corresponds to the /depot/

nihomics

course material folder on Rice. This area is read-only. You cannot create/delete/modify files there.

You have an R: drive which corresponds to the scratch directory in your home directory on Rice (really /scratch/rice/t/

trainNN

). This is fully read-write for your accounts.Slide36

Shell features

Shell environment variables used to control settings for how certain things work

Thousands of potential commands can be executed

Commands available varies from one Linux computer to the next, depending on what has been installed, and the value of your PATH environment variableSlide37

Shell features (cont.)

Filename completion (using “Tab” key)

Command completion (using “Tab” key

)

Command line editing using arrow keys (up-arrow key to go to the previous command)Slide38

Let’s get dirty!Slide39

Listing files in Terminal

Type

ls

to list files in the current directory

Type

ls -l

to list files with more detail

Type

ll

to list files with even more detailSlide40

Navigating directories in Terminal

Type

pwd

to see full path to current directory

Type

cd

dirname

to change directories

Type

cd ..

to go to the parent directory, or

cd ../..

to go to the grandparent, etc.

Type

cd ~

to go to your home directory

cd /depot/

nihomics

/data

Absolute paths start with

/

, relative paths are relative to the current directorySlide41

Special directories

/home/

trainNN

– Your home directory

/depot/

nihomics

– Contains data files and scripts used in the workshop (O: drive)

/home/

trainNN

/scratch

– Scratch directory where you will be saving output files, scripts (R: drive)

/scratch/rice/t/

trainNN

– SAME PLACE AS ABOVE!Slide42

Editing, copying, moving filesSlide43

Editing, copying, moving files

g

edit

filename

– Edits

filename

mv

oldname

newname

– Moves a file or directory, possibly to a new directory, possibly renaming the file or directory in the process

cp

oldname

newname

–

Copies files

cp

–r

olddir

newdir

– Copies

olddir

and all files and subdirectories withinSlide44

Create/Remove directories, files

r

m

filename

– removes

filename

m

kdir

dirname

–

creates

dirname

Rmdir

dirname

–

removes

dirname

, but only if

dirname

is empty

Let’s practice, and use filename completion and command line editing while we are at it!Slide45

Can copy/move/delete with GUI

You can always fall back to using the O: drive and R: drive on Windows to move/delete files

On a rice node (green/bold prompt!), open a graphical

file browser to

move/delete files via:

thunar

&

If

you

accidentally omit

the

&

,

your Terminal

cannot be used again until

thunar

is

closedSlide46

Terminating a program

If you are running a program in a terminal window that you would like to terminate, press Control-C

This won’t work if you started that program it with an

&Slide47

See what programs are running

p

s

xuww

-

Show what programs we are running now

PID column shows the

P

rocess ID of each program

Can use

top

to see most CPU intensive programs currently running by everyone on this server. Press

q

or just control-c to exit

topSlide48

Terminate or

kill

or program

Must first know the process id number (PID) using either

ps

xuww

or

top

kill NNNNN

Will kill most programs

k

ill –HUP NNNNN

Use if the previous doesn’t work

kill -9 NNNNN

Use

if the previous doesn’t

workSlide49

Let’s practice starting/killing progs

On a Rice node, type

busy 1000 &

Type it again a few times (use the up-arrow!)

Type

top

to see the PIDs of all the jobs running, press q to quit

Kill all of the busy jobs by typing something like:

kill 24933 24937 24939 24944

Type

top

again to confirm they are goneSlide50

Redirecting input/ouput

Some programs write output to the Terminal/shell screen

We can save it using output redirection

q

stat

-a > out1

Saves results of the command

qstat

-a

to the file

out1

h

ead < out1

See the first 10 lines of

out1

h

ead < out1 > out2

Save to

out2Slide51

Redirecting input/ouput

Can only save the text output that would have normally appeared on the screen. If a program wouldn’t normally generate any text output, nothing will be saved

Terminal > out3

(Nothing is saved!)Slide52

PipesSlide53

Pipes

Pipes are a feature of the shell that permit the output of one program to read as the input of another program, without having to save an intermediate file

All programs involved run simultaneously and process data as they receive itSlide54

Example without pipes

q

stat

-a > out1

a

wk

’{print $2}’ < out1 > out2

s

ort < out2 > out3

u

niq

-c < out3 > out4

s

ort -n < out4Slide55

Same example using pipes

q

stat

-a |

awk

’{print

$2}’ | sort |

uniq

-c | sort –nSlide56

Where to find these slides

From Windows, in:

O:\etc\doug\bigtap-comp1.pdf

From a Rice node, in:

/depot/

nihomics

/

etc

/

doug

/bigtap-comp1.pdfSlide57

Slide58