Introduction to Scripting Workshop - PowerPoint Presentation

Slide1

Introduction to Scripting Workshop

February 23,

2016Slide2

Introduction

George Garrett

&

The HPC Support Team

Research Computing Services

CUITSlide3

Introduction

Please Leave

F

eedbackSlide4

Introduction

Slides will be sent out afterwards.Slide5

Introduction

What is a script?Slide6

Introduction

What is a shell script?Slide7

Introduction

What is a shell script?

A file with shell commands.Slide8

Introduction

Why use scripts?Slide9

Scripting Resources

How

Linux Works

by Brian Ward

Available as an

E-book

from Columbia University Libraries and at Safari Books OnlineSlide10

Scripting Resources

Shell and Scripting Tutorial

http://linuxcommand.org/Slide11

Scripting Resources

Advanced

Bash-Scripting Guide

http://tldp.org/LDP/abs/html/Slide12

Cunix

System: cunix.columbia.edu

User: Your UNISlide13

Access

Windows Instructions

Search for putty on Columbia home page

Select first result

Follow link to Putty download page

Download putty.exe

Run putty.exeSlide14

Access

Mac Instructions

Run terminalSlide15

Access

Mac (Terminal)

$

ssh

UNI@cunix.columbia.edu

Windows (Putty)

Host Name: cunix.columbia.eduSlide16

Access

Does everyone have access?Slide17

Quick Review

shell user interface to the system

$ standard prompt symbol

pwd

print working (current directory)

cd change directory

. current directory

ls list directory contentsSlide18

Quick Review

cat

print a file

sort

sort the lines of a file

grep print lines matching a pattern

echo hi print

hi

sleep 5 wait 5 seconds

man command

manualSlide19

Workshop Setup

$

mkdir

workshop

$ cd workshop

$

cp

/

tmp

/workshop/* .Slide20

Command Line Example

Word Count

from

“Data

Science at the Command

Line”

-

Jeroen

JanssensSlide21

Let’s Download Some Data

Gutenberg Project:

http://

www.gutenberg.org

Which famous novel should we use?

Download plain text version using curl

$

curl [URL] > novel.txt

Trim header and footer, leaving only main textSlide22

Pipes

$ cat

alice.txt | grep rabbit

Pipes

connect output from one command to the input of another commandSlide23

Pipes

$ cat

alice.txt | grep rabbit | sort

You can keep on combining commands with more pipes.Slide24

Pipes

$ cat

alice.txt | grep rabbit |

tr

r w

tr

translate charactersSlide25

wcount

$ cat

wcount

cat

alice.txt |

tr

…Slide26

wcount

$

wcountSlide27

wcount

$

wcount

-bash:

wcount

: command not foundSlide28

wcount

$

./

wcountSlide29

wcount

$

./

wcount

-bash

:./

wcount:Permission

deniedSlide30

wcount

$

ls –l

wcountSlide31

wcount

$

ls –l

wcount

-

rw

-

rw

-

--- [ snip ]Slide32

wcount

$

ls –l

wcount

-

rw

-

rw

-

--- [ snip ]

$

chmod

+x

wcountSlide33

wcount

$

ls –l

wcount

-

rw

-

rw

-

--- [ snip ]

$

chmod

+x

wcount

$ ls –l

wcount

-

rwxrwx

-

-x [ snip ]Slide34

wcount

$

./

wcount

Should work this time.Slide35

file

Determine type of file.

$ file

wcountSlide36

file

Determine type of file.

$ file

wcount

w

count

: ASCII textSlide37

wcount

Choose an editor

nano

 Recommended default

v

i

emacsSlide38

nano

Nano commands are on back of cheat sheet.

^ means “hold down control”Slide39

Edit wcount

$

nano

wcountSlide40

#!

Add “#!” to first line

#!/bin/

sh

cat alice.txt |

tr

…Slide41

#!

$

./

wcount

Still works.Slide42

#!

Some #!

f

irst lines you might see

#!/

bin/

sh

#!/

bin/bash

#!/

usr

/bin/

perl

#!/

usr

/bin/pythonSlide43

file

Has the file type changed?

$ file

wcount

w

count

: POSIX shell script text executableSlide44

Variables

$ file=alice.txt

$ echo $file

a

lice.txtSlide45

Variables

Add

file=alice.txt

to

wcount

.

Replace

cat alice.txt

with

cat

and the variable.Slide46

Variables

#!/

bin/

sh

f

ile=alice.txt

cat $file

|

tr

…Slide47

Variables

You could put $file in double quotes.

Why put quotes around $file?

cat “$file”

|

tr

…Slide48

Command Line Parameters

We’re going to change

wcount

so any file can be specified from the command line.

$ ./

wcount

moby.txtSlide49

Command Line Parameters

Change

wcount

so any file can be specified from the command line.

$ ./

wcount

moby.txt



$1Slide50

Command Line Parameters

Create a new file named “

param

”

Put the #! directive on the first line

On next line type:

echo $1

Save and make executable

Run itSlide51

Command Line Parameters

$ cat

param

#!/bin/

sh

echo $1Slide52

Command Line Parameters

$ ./

param

$ ./

param

alice.txt

$ ./

param

aaa

bbb

cccSlide53

Command Line Parameters

Update

param

to print out $#

e

cho $#

echo $1

Run it with different numbers of parametersSlide54

Command Line Parameters

U

pdate

wcount

to use $1 instead of alice.txt.Slide55

Command Line Parameters

Before:

file=alice.txt

After:

file="$1"Slide56

Command Line Parameters

$ ./

wcount

alice.txtSlide57

if

if [[ condition ]]

t

hen

do something

fiSlide58

[, [[, ((

[ : Standard comparison

[[ : Extended comparison

(( : Mathematical comparisonSlide59

[, [[, ((

[ : Standard comparison

[[ : Extended comparison



(( : Mathematical comparisonSlide60

Comparison with [[

i

f [[ “$count” -

eq

100

]]

-

eq

: equals

-ne : not equal

-

gt

: greater than

…etc.Slide61

if

if [[ $# -ne 1 ]]

t

hen

do something

fiSlide62

if

if [[ $# -ne 1 ]]

t

hen

echo “Usage:

wcount

file”

exit 1

fiSlide63

if

$ ./

wcount

$ ./

wcount

alice.txt

$ ./

wcount

alice.txt junkSlide64

if

if [[ ! –f “$file” ]]

t

hen

echo “Error: File $file not found.”

exit 1

fiSlide65

File Tests

Common tests

-e : File exists

-f : File is “regular”

-d : File is a directory

…many moreSlide66

if

if [[ ! –f “$file” ]]

t

hen

echo “Error: File $file not found.”

exit 1

fiSlide67

Add lines parameter

Show the five most common words:

$ ./

wcount

alice.txt 5Slide68

while

i

=7

while (( $

i

> 4 ))

do

echo $

i

i

=`expr $

i

- 1`

doneSlide69

for

for

i

in

a b c

do

echo

$

i

done

for file in `

ls`

do

ls $file

doneSlide70

for

for

i

in {1

..3}

do

echo

“Repeat $

i

times”

done

Repeat 1

times

Repeat 2

times

Repeat 3

timesSlide71

Adding Comments

Everything the shell encounters after a hash mark on a line is ignored

.

Comments are useful documenting your script.

Or to

make the interpreter ignore

sections of your script

.

#

This is a comment

echo “Hi” # and this is another

commentSlide72

readyourmind

A silly impolite script but it shows a few more things.

read : read input

c

ase : another way to control flowSlide73

Questions

Questions?Slide74

End of Slides

Next week: Intro to High Performance Computing on Yeti