The Shell & You A brief-but-broad Introduction to the Command Line - PowerPoint Presentation

Slide1

The Shell & You

A brief-but-broad Introduction to the Command Line

~ For the

Inquisitive

Mac Sysadmin ~

Part I of ??Slide2

Who is this guy?

Sr. Application Developer @ The Wharton School, UPENN

Run, with an excellent team, a Linux (RHEL) HPC research cluster

Much of my job entails scripting (bash, perl, python) to automate systems deployment and management, as well as application development of user-facing products

WhAT DOES HE DO?

Andrew

Feierabend

English Major

Spent 5 years working in a basement at Springboard Media, Apple Reseller

Been working at an office with a lovely window at the Wharton School ever sinceSlide3

Goal & Structure of this Talk

Super casual

– speak up with questions when and where you experience them welling up inside you. Tears, however, may be held until afterwards.

Accommodating a varied audience – I will do my best to balance introductory information with Intermediate information. There will be no advanced content (ask me afterwards!)Fast – There is a lot to cover, and I don’t do slow very well. Speak up if you have a question!

Information overload! – There will be lots of information. If you don’t understand something, ask. If you don’t care about something (because you’re just too damn smart) wait 5 seconds and I’ll get to something you probably do care about.Slide4

Assumptions and Standards

I will be using

Bash

as my shell in all examplesI will be using an OSX 10.11.3 machineCommands listed in this presentation are in fixed-width font throughoutKey concepts and official jargon are in

bold.All commands and their arguments and parameters are LOWERCASE unless explicitly noted. The command line was born on case-sensitive operating systems, so you should get used to the notion that – when working on the command line – case matters!Slide5

Let’s get started!

What is a shell?

How do I access a shell on my computer?

Why do I care?Slide6

Let’s get started!

(boring definitions)

What is a shell?

A shell is a command language interpreter that executes commands read from the standard input device (keyboard) or from a file, acts on that input, and provides output to the user.How do I access a shell on my computer?

Since audience ~= MacAdmins group, you simply open up /Applications/Utilities/Terminal.app. But I’ll show you later why you want to use iTerm2 insteadWhy do I care?Shell scripting allows for remote management of systems, automated execution of commands, troubleshooting of systems with graphical problems, and – frankly – the ability to do certain things way faster than in the GUI. For many advanced tasks, the command line is the only option.Slide7

The prompt

When you connect to a system (even just launch Terminal) you are presented with a prompt, as above.

A prompt gives you some basic information about your environment

A prompt tells you that the system you are on is ready for your input

A prompt is highly-customizable (through shell-specific config files, more on that later)Slide8

The prompt

I typed

pwd

and hit the Enter key.

pwd stands for ‘print working directory’The shell processed the command, and printed the output on the next line in my terminalThe shell then presented another prompt to tell me that it’s ready for my next commandIf a command (such as pwd) were to exit without generating any output, I would be returned directly to the next prompt without any output being printed to my screen. Slide9

Entering Commands

ls

Command

to list the contents of the directory I’m currently in-la Flags to ls that alters slightly the output of ls| Pipe that passes the output of the first command to the second (or Nth)h

ead

Command

that gets its input from

ls –la

-5

Flag

to head that alters the output

head

providesSlide10

Entering Commands

head

Command

to list the first few lines of a file~/Desktop/text_file.txt Argument to head to provide the file~/Desktop/text\ file.txt

Argument

to

head

to a different file

Note the escape character! Why is that necessary?Slide11

Entering Commands

Command

Predefined command that you can run on the command line

Flag

Changes the default behavior of a command in some wayArgument Provides a file (or socket, user, etc) for the command to act onPipe Passes the output of one command as the input to anotherSlide12

Entering Commands

Flags to commands can usually be appended together in one long string prepended by a hyphen.

Check the man page of your program to see how it prefers to be called, what flags are available, and what they do (more on this in two slides).

(in the leftmost example, I have omitted the usual arguments to each command that would tell the command on what file(s) or directory(s) to act upon)Slide13

Entering Commands – Getting Carried Away

In the above I issue:

5 commands

1 output redirect

3 different methods of command chaining ( |, &&, ; )The above simply:Lists all files in my home directory with a name containing HI, prints only the filename, outputs the results to a file called ‘test’, prints the contents of that file, then lists the full path to that file.

Don’t worry, we’ll come back to the interesting stuff in this example

…Slide14

Wait, then how do I … I’m confused

How do you know what commands are available?

How do you know what flags are available for a given command?

How do you know the right syntax for every command?How do you remember all this crap?Slide15

Wait, then how do I … I’m confused

How do you know what commands are available?

apropos ‘search term’

or man –k ‘search term’ or googleHow do you know what flags are available for a given command?

man command or googleHow do you know the right syntax for every command?man command or googleHow do you remember all this crap? I most certainly do not. See above.Slide16

Navigating the Filesystem

Navigate to / (root) (Mac HD)

Navigate to my Desktop

Print Working Directory

Navigate to /Navigate to my User HomePrint Working DirectoryNavigate to /Navigate to my DesktopPrint Working DirectorySlide17

Navigating the Filesystem – Tab Completion

Shells were invented to make our life on the command line easier, hence:

Tab completion! (demo)

You can tab complete in the Go To dialogue in the Finder too! (CMD-Shift-G), but it will autocomplete to the first match alphabetically, and won’t present a list of possible matches. So, it sucks.You can tab complete anywhere a file or filepath is expected, i.e. with

lsYou can even tab complete names of commands if they’re in your PATH** More on the PATH environment variable laterSlide18

Shell Built-ins – This is why we use Shells

Up / down arrow keys to scroll back through previous commands

Tab completion (previous slide)

h

istory – shows a long list of the commands you’ve ran on this systemctl-r – Search through previous commands entered in this sessionctl-c – Break out of a currently-running process. Essential!ctl-d – Logout from your current user session (shell or system depending)clear – reset your terminal window back to a single prompt. All previous commands and output are still available through search,

history

, or simply scrolling back up.

c

lear

just clears the screen.Slide19

WORKING WITH

TEXT – cat, less

c

at

is simple; it prints out a file, in its entirety, directly to your terminal. You are presented with a prompt again when the file is done being printed to your screen.cat is usually immediately piped to another command. After all, most of the time we blindly print out a whole file, we want to do something more with it than just read through it.less is a pager, meaning that it will print however much of the file to your screen as will fit, and you can page through it with the spacebar or return key at your leisure. It is much easier and pleasant to read through long files using less as opposed to

cat

.

l

ess

is

more

, folksSlide20

WORKING WITH

TEXT

– echo

Echo prints exactly what you type between quotes.Useful in scripts for returning execution state, i.e. “Step 1 complete”Commonly used with redirection in scripts to write log messages:Slide21

HOLD UP THERE – BACKTICKS & variables!

The shell needs to know whether to treat what you type as plaintext or as something that needs to be

expanded

.

Backticks tell the shell to execute everything between them first as if it were a separate command all its own, with the result of that separate command then being used in its place as the rest of the command is parsed.Variables are stored values represented by a handle, or variable name, usually for use later in a program. Variables are expended by the shell when encountered"Slide22

WORKING WITH

TEXT –

sed

s

ed can do a million things, simple substitution – shown above – is probably its most famous utility.sed will match every instance of the thing you want to replace. Make your match highly specific!The s/a/another/g bit is a pattern match. In the third example, I am using what’s known as a regular expression to specify that I want to match only the word ‘more’ IF it starts the line. This way I was able to only replace the first ’more’Slide23

WORKING WITH

TEXT –

sed

Real world example

\r characters are Windows-specific carriage returns. They break the formatting of a text file when that text file is sent via certain Email carriers (Exchange)In the above, I am reading a file into sed, which matches any \r character and replaces it with NOTHING. This is how you delete things using sed.

You can see that I am also using variables, and redirecting my output. Yes the filename of the redirection can be a variable!Slide24

WORKING WITH TEXT – grep

g

rep

is probably one of the best known and most widely used command line tool. It’s CMD-F for your

… everything.You can use the –i flag to make your search case insensitive, and the –c flag to count the number of occurrences of your search pattern instead of printing them:Slide25

WORKING WITH

TEXT – grep

Real world example

p

s –ef is a common way of calling ps, which lists running processes on a system. The –e flag asks for everyone’s processes running (not just my own) and the –f flag requests a full command listing, rather than truncating the command.

Why is this useful? Because this system is running 547 processes. If I know what I’m looking for, grep makes it much faster to find it.Slide26

WORKING WITH

TEXT –

awk

awk

, like sed, can do a million things. In the first example, we rearrange the text.The numbers preceded by dollar signs are called fields. Awk decides where fields start and stop based on its field separator which, by default, is a space.

In the third example, we change the field separator to a comma.Slide27

WORKING WITH

TEXT –

awk

Real world example

Where I work, we have two important networks, and we keep them correlated so that a given machine will always have the same fourth octet in both subnets (above: 172.16.23.100 and 172.16.33.100)Here, grep gives us a single line, awk picks out only the IP (the second field in that line), then awk both prints the first three octets of the IP that I want, and passes in the fourth field it sees (where I have specified a period as the field separator).Slide28

Regular Expressions

f

ind

does just what it sounds like it does. Give it

filepath to start its search from (or give it / to search your whole hard drive), then give it a pattern to look forHere, the pattern I use is a regular expression that matches any filename that ends in .shf

ind

found four files whose filenames end in

.

sh

f

ind

found four files with filenames, finally!Slide29

Regular Expressions – Over the top

This absurd thing parses an IP address fed to it and returns that IP if – and only if – it is a valid IPv4 IP address.

Don’t try to type that out, just copy it from

here

.Don’t worry too much about all that, just get a general sense that the match seems to be 4 sections of complicated number matching separated by periods.Slide30

Regular Expressions – sensible!

Here I

cat

five paragraphs of the famous ’lorem ipsum’ text into

grep.In the first command, I am looking for any instances of the word ‘Sed’ (WITH A CAPITAL S) anywhere in those five paragraphs.In the second, I am looking specifically for instances of the word ‘Sed’ that start a new line

(since grep, by default, uses newlines as a delimiter).Slide31

Regular Expressions – sensible!

Here I

cat

five paragraphs of the famous ’lorem ipsum’ text into

grep, again.In the first command, I am looking for any instances of the word ‘elit’ anywhere in those five paragraphs.In the second, I am looking specifically for instances of the word ‘elit.’ that

end a new line

and have a period after them. Or do they?

What is the difference between the second and the third commands?Slide32

Input and Output Control – Redirection

In the above, I use the redirection command

>

to send everything that would normally be printed to the screen from

echo to the file new_file.txt insteadNote how using the single character > replaces the original line when I redirect my output the second time. > completely erases a file before writing your output. Be careful!Slide33

Input and Output Control – Redirection

In the above, I use the redirection command

>>

to send everything that would normally be printed to the screen from

echo to the file new_file.txt instead, same as with a single >Note how using the double character >> appends to the file, so that both the original line and the new line are now present in the file. >> will always append to the end of the file you are writing to. It will always be the last 1+ lines of your file.Slide34

Users and permissions – POSIX and ACLS

There are two types of permissions, POSIX and ACLs

POSIX is in the format

rwxrwxrwx

and indicates read-write-execute for user-group-otherRead and write are exactly as they sound, execute means:If a file, run it as if it were a program. All shell scripts must be executable.If a directory, allow traversal, that is, allow looking or moving into that directory.ACLs are access control lists and allow for much finer grain control over how a user may access or manipulate a file or directory. ACLs are also far more complicated and could get their own slide deck – we won’t address them further in this one.Slide35

Users and permissions –

chown

and

chmod

You can also set permissions yourself via chown and chmod.chown changes the owner (and/or group)chmod changes the permissions for the existing owner, group, or othersSlide36

Users and permissions – Show Users

User management is beyond the scope of this little deck, but here are some commands to gather information on users. Try ‘

em

out:id – shows general info and group membership for your user

id other_user – shows this info for the specified userlast – history of logins, reboots, and shutdowns for this boxfinger – show all currently-logged in usersp

asswd

– change your password

d

scl

– “Directory Services Command Line”; navigate through various authentication sources, like OD and AD, as if it were a directory structure.Slide37

Users and permissions – root and

sudo

As OSX is based on UNIX (BSD to be precise), it has a root account. This root account is disabled by default, but can be enabled in:

System Preferences > Users & Groups > Login Options > Network Account Server > Join > Open Directory Utility… > Edit menu > Enable Root UserAlternatively, if you are an admin user on your Mac, you can use

sudo to assume the privledges of the root user for only this one invocation.Slide38

The environment

The

environment

is a collection of pre-set variables, values, and aliases that are already in place when you log in.

These values are set by your shell, when it reads a shell-specific config file in your home directory, or on the system level.Slide39

The environment

You can see that the variable $PATH is already set for me when I log in; I did not need to declare it in advance.

The $PATH environment variable tells the shell which directories to search in for programs (which are also sometimes called

binaries

if they are compiled).Slide40

The environment

This is my bash-specific configuration file in my home directory. It is responsible for the format of the prompt, and also where I might put any shortcuts to longer commands, called

aliases

.Slide41

CLI text Editors

For editing files, there are several feature-rich text editors you may use at the command line. These are:

n

ano (sometimes also listed as

pico) – by far the easiest to start with.emacs – unrelated to those soulless white boxes Apple tossed around in the 00’svi (or its enhanced version, vim)Each editor has its own syntax for saving, searching, replacing, even for moving around in the file you’re working on. Be sure to read up on the more advanced editors, or just start with

nano

.

In

nano

, the following key commands should get your started:

ctl

-o

– Write

your file to disk. You will be prompted to name your file

.

ctl

-x

– Exit

nano

. You will be prompted to save your file.

c

tl

-w

– search the entire file you are editing.Slide42

Putting it all together – Shell scripting

Every shell script has the above three components:

The shebang. This is how the shell knows what interpreter to pass the script to.

The COMMENT describing the script’s purpose, expected parameters, etc.

A way to gracefully exitIt is common practice to give your script an extension indicating what programming language it was written for. If you’re not sure, and you’re working on a Mac, use .shEven though the script has an extension, it is the shebang that actually is followed.Slide43

Putting it all together – Shell scripting

This time I’ve adjusted the script to present some rudimentary output, which we’ll take a look at in

a second.Slide44

Putting it all together – Shell scripting

The way to execute a script is by prepending

./

to the script’s filename, as in the above.

However, the script didn’t run, and I got a “permission denied” error. Hmm, what did I miss?Slide45

Putting it all together – Shell scripting

I use

chmod

to add

execute permissions to the owner with chmod u+x. I could also have explicitly set all permissions for that file with chmod 744 instead.Now I should be able to run this!Slide46

Putting it all together – Shell scripting

That’s it! That wasn’t too hard. How about a few more practical examples next, eh?Slide47

Putting it all together – Shell scripting

Real World Example

What did I forget to do in line 2?

Did I finish this script?Slide48

Putting it all together – Shell scripting

Real World Example

Lots

of variables!Slide49

Command Line Standards

24 Hour Time

– Timestamps (like those provided by

ls –l

) are usually on 24 hour time. This means both that 3:15 PM is 15:15 and that 3:15 AM is 03:15.No spaces – Use_underscores-or-hyphens. Unless\ you\ really\ like\ escape\ characters! However, you probably can’t escape this (get it?) on most user machines, since they all likely have their hard drive named “Macintosh HD”. Slide50

Command Line Standards

Lowercase

– This bears saying again! Though OSX is not case-sensitive by default

(but you can set it that way when you format),

the command line will treat everything but files and directories as case-sensitive!Slide51

Real Quick – Bash & Shellshock

Everyone remember this? We’re all safe now, but even shells contract vulnerabilities!Slide52

Tools of the trade – iTerm2

/Applications/Utilities/

Terminal.app

is all well and good. But you’re a bunch of command line junkies now, and you want to use a really powerful tool! iTerm2 provides a host of features above an beyond what Terminal offers. I will touch on a few of the biggest ones here.

iTerm2 is freely downloadable and actively developed by a crack team of people just like us who rely on it to do their job.iTerm2 -- https://www.iterm2.com/ Slide53

Tools of the trade – iTerm2

Logging

– the most important feature of iTerm2! Everything you type or see in your terminal window is logged to a file on your hard drive for later reference. Incomprehensibly handy to have a verbatim log of everything you’ve ever done on the command line on any system you’ve ever connected to. This is NOT ON BY DEFAULT – BE SURE TO SET IT!

Profiles – Set up your iTerm setup however you want – fonts, colors, window transparency, key mapping,

etc – and save profiles of these settings per user, per remote system, or just in general. Profiles are easily switched between with a single key command.Searching – Far better search implementation that helps your search through your entire scrollback history for a specific string or regular expressionSlide54

Tools of the trade – iTerm2

Window Management

– Tile windows, make them transparent, layer them, order them in tabs on one single window, break them out onto multiple monitors, full screen support, the kitchen sink.

Timestamps – Easily turn on and off timestamps for each command you’ve entered. Now you can see – exactly – how long a particular process took to execute, without needing to resort to commands like

time.Built-in Password Manager – handy for admins who connect to many systems.Search across all active sessions – Easy to get right back to what you were doing before the LDAP server went down and you had to bring up 17 windows to troubleshoot it immediately.Restore Accidentally Closed Sessions – default restore time of 5 seconds.Annotate Scrollback History – Useful for providing your history to a client (some like this, some should never ever be given this) or for reminding yourself why you had to use perl to handle this one exception.Slide55

Tools of the trade – Sublime Text

If you find that you spend a lot of time on the command line, especially if you write a lot of scripts, Sublime Text offers a beautiful GUI interface for crafting your text-based scripts.

Sublime Text is ridiculously fully featured, so I’ll let the website speak for itself.

Sublime Text

-- https://www.sublimetext.com/Slide56

WRAPPING UP – Until Next time

This presentation has barely scratched the tip of the surface of the iceberg!

Today we focused mostly on navigating around your computer using the command line, manipulating text, and writing small shell scripts.

I’d like to put together a second presentation that focuses on:Networking (ifconfig,

tcpdump)Diagnostics (top, diskutil)Remote login (ssh, public keys)Scheduling (crontab, Cronnix maybe?)Shell scripts & Casper, shell scripts & ARD

Thanks everyone, I hope this was useful!