CS 294-73 (CCN 27241) - PowerPoint Presentation

Slide1

CS 294-73 (CCN 27241)Software Engineering for Scientific Computinghttp://www.cs.berkeley.edu/~colella/CS294Lecture 4: Development ToolsSlide2

Revision Control SystemsModern scientific computing is no longer a solo effortMost interesting modeling questions that could be simulated by the heroic individual programming scientist have already been investigated“Productivity language” that are meant to alleviate the complexity of programming high performance software have not delivered yetThus, coding is complicated and requires division of roles and responsibilities.Working together on a common code is very error prone without some technology assistance.Two tools to be discussed in this partConcurrent Version System: cvsSubversion: svnVery similar in user interface

2Slide3

Concurrent Version SystemCentral Repository system. There is one master version of the state of the codeUsers have “check outs” or “working copy” of the master respositoryCan access the master respository via several mechanismsrsh connectionssh connectioncvsserver All user interaction is considered a client-side operationTransactional protocol

3Slide4

SSH protocol interaction Setting up keychains (ssh-keygen, ssh-add, ssh-agent)Typing in your password for every single transaction with the central respository is a painssh-keygen

>

ssh-keygen

–

t

dsa

// you will be prompted for passphrase

>ssh anag.lbl.gov>mkdir .ssh:mkdir: .ssh: File exists>exit>cd .ssh>scp id_dsa.pub –l username anag.lbl.gov:.ssh/tempfile>ssh –l username anag.lbl.gov ;cd .ssh>cat authorized_keys2 tempfile >authorized_keys2>exitssh-add>ssh-add // provide passphraseTry ssh again and see if your keychain is working.

4Slide5

Setting up for CVS repository workEnvironmentCVSROOT=:ext:username@anag.lbl.gov:/usr/local/cvsrootCVS_RSH=sshEDITOR=viCommands>cvs checkout Chombo>cvs add [filename|directory]>cvs update

>

cvs

update –

d

// by default update only updates directories that were in this level of the repo when you did your checkout. -

d

says bring in any new directories.

M:

Locally Modified or Merging, apparently without conflicts. This is the essence of concurrent version systemC: merge got confused, Conflicts remain. Your local file has been modified and >>> delimiters have been put in the file telling you where you need to pick a winner, and merge by hand.U or P: Repo version has some updated file that you haven’t touched, your copy replaced.5Slide6

Commands Cont.>cvs diffShow me the difference between my local file and the current repo version as of when I took an update.>cvs logShow a modification history of a file>cvs statShow the repo info of my working copy of this fileTags! >cvs commitCommit my local working copy of file(s) as the new repo version of these filesUser prompted to use EDITOR to create a commit comment

>

cvs

remove // and watch the horror of ‘

remove’ing

directories

You can also use GUI-based tools that invoke these commands for you and help keep things straight

6Slide7

Tags Creates a string to associate with a collection of files and their individual revisions>cvs tag backpoint1 .This is a “working copy” tag. You keep this info in your own local copy>cvs rtag -D <date> <new_tag> [file | directory | module]This is a repository tag. It modifies the repository to associate the new_tag with some revision of the [file|directory|module]-D is a handy way of setting a tag of the repo at some point in time.

HEAD is a special repo tag reserved for the trunk (non-branched part of the repo)

We use this this special tag when working with branches

7Slide8

Branches…. uhm, if nobody objects, I’ll leave this one for now. Branches in CVS are a confusing issueBetter to work in Subversion if you are needing to work with complicated branches8Slide9

Subversion: SVNCentral Repository system. There is one master version of the state of the codeUsers have “check outs” or “working copy” of the master respositoryCan access the master respository via several mechanismsrsh connectionssh connectionsvnserver All user interaction is considered a client-side operationTransactional protocol

9Slide10

Working with SVN >svn checkout svn+ssh://anag.lbl.gov/usr/local/svnroot/Chombo4 MyLocalName>svn update Also a merging/concurrent process, as with CVSNo –d process here. Better, think of it as always applied>

svn

log filename

>

svn

diff [

filename|directory

]

>

svn add [filename|directory]>svn commit [ |filename|directory]>svn delete [filename|directory]10Slide11

SVN Branches and Tags: copyTags and Branches and similar concepts are all handled by the universal copy command>svn copyhttp://svn.example.com/repos/calc/trunk http://svn.example.com/repos/calc/branches/my-calc-branch -m "Creating a private branch of /calc/trunk

.”

Here we are using the http protocol for

svn

. The same command using

ssh

protocol would be similar

svn

copy

svn+ssh://svn.example.com/repos/calc/trunk svn+ssh://svn.example.com/repos/calc/branches/my-calc-branch -m "Creating a private branch of /calc/trunk.”Tags are just the same as branches. Everything is a copySubversion database is clever to not really copy everything. Copies are “shallow” until you modify a file11Slide12

MergingIn svn, everything is just a copy. BranchesTagsOther SVN reposFull Syntax>svn merge left-tree@rev right-tree@rev working-setusers will almost never use fully specified versionBut, it is good to know that the full format exists Typical syntax>

svn

merge svn+ssh://anag.lbl.gov/usr/local/svnroot/Chombo4

Only one argument given here, and no rev specified, so…..

right-tree

and

working-set

is assumed to be the current directory

The

left-rev is assumed to be the most recentYou’ve done an >svn update12Slide13

Branch and Merge >export sroot=svn+ssh://usr/local/anag/svnroot>svn copy $sroot/Chombo4/trunk $sroot/Chombo4/branches/GPU_Experiment>svn checkout $sroot/Chombo4/branches/GPU_Experiment

gpuExperiment

Edit and develop code in

gpuExperiment

Other users can check out your branch and your updates and commits and add etc. operations are seen by each other

>

svn

merge $svnroot/Chombo4/trunk

Keep up to date with edits made to the main development trunk

Is the branch code suitable for the core trunk code ?>cd tmp; svn checkout $svnroot/Chombo4/trunk trunk>cd trunk; svn merge $sroot/Chombo4/branches/GPU_Experiment13Slide14

GNU MakeA tricky bit of script parsing to manipulate files specialized to work well with compiling code lots of features to let you do simple things simply. complicated things without too much work. almost impossible to figure out what is going wrong.Main purpose: turn a set of source code into a library or executable.Only two kinds of objects in a MakefileVariables (lists of strings)RulesOnly a few kinds of flow controlifeq/ifneq/else/endifNo forms or looping available, no jumps, no recursion.

Most difficulties arising from make are related to

Non-trivial variable parsing of the

makefile(s

)

Rules can fire and trigger in non-obvious ways

The mysteries of

regex

14Slide15

The Two type of Variables in GNU MakeRecursively Expanded Variables “=“foo = $(bar)bar = $(ugh) ugh = Huh? all:;echo $(foo)> make allHuh?Variable is executed at the time it is used in a command= means build up a symbol table for this nameNotice $. Like in shell, there is the value ‘bar’ and the variable named ‘bar’

15Slide16

Good points:Order doesn’t matter! Go and try it.Can declare a variable as the composite of many other variables that can filled in by other parts of the MakefileCFLAGS = $(DEBUG_FLAGS) $(OPT_FLAG) $(LIB_FLAGS)Lets a makefile build up sophisticated variables when you don’t know all the suitable inputs, or what parts of the Makefile they will come from>make all DIM=3 Bad points:No appending # error, causes infinite loopCFLAGS = $(CFLAGS) –c Future = declarations can clobber what you specifiedThe last = declaration in the linear parsing of a

Makefile

is the

only

one that matters

16Slide17

Simply Expanded Variables “:=“Immediate mode variable.The variable is assigned it’s value based on the current state of the Makefile parsingNo symbol chain is created.Specific to GNU MakeOften just an easier to understand variable. It acts like variables you know in other languages.can use for appendingCFLAGS := $(CFLAGS) –c –e –mmx

17Slide18

Rulestargets : prerequisites [TAB] recipe[TAB] recipeprerequisites are also called “sources”Simple exampleclobber.o : clobber.cpp clobber.h

config.h

[TAB]

g

++ -

c

–

o

clobber.o clobber.cppclob.ex : clobber.o killerApp.o [TAB] g++ -o clob.ex cobber.o killerApp.o 18Slide19

More powerful rulesPattern Rules%.o : %.cpp$(CC) -c $(CFLAGS) $(CPPFLAGS) $< -o $@#Gives a pattern that can turn a .cpp file into a .o fileMultitarget Rules%.f

%.H : %.

ChF

Suffix Rules

.

c.o

:

$(CC) -

c

$(CFLAGS) $(CPPFLAGS) -o $@ $<19Slide20

Other Makefile commands include$(MAKE)calling a makefile from inside a recipe$(MAKELEVEL) can be looked at to see how deep the call stack isexportsend variables from this level of make to lower makelevelssubstCFLAGs:= $(CFLAGS) $(subst

FALSE,,$(subst

TRUE,-DCH_MPI $(

mpicppflags),$(MPI

)))

foreach

libincludes

= $(

foreach

i,$(LibNames),-I$(CHOMBO_HOME)/src/$i)20Slide21

What the “make” program doesMuch mental confusion about make comes from thinking that the Makefile is the make programEasy to see. It looks like a shell script.Remember: Makefile is only Variables & Rulesmake:parses all of your Makefilebuilds up variable chains (overriding variables defined on command line)builds up rules database

Then looks at what target the user has specified

make then attempts to create a chain of rules from the files that exist to the targets specified.

recursive “=“ variables in source-target expressions are evaluated

Using the date stamp on files discovered in the chain make executes recipes to deliver the target.

“=“ variables are evaluated in recipes.

21Slide22

Demonstration of the pervasive Make ‘error’FooBar = trendyF:= fashionvars:@echo $(FooBar) $(F)ifeq ($(

F),fashion

)

FooBar

=tragic

endif

F:= comedy

>make

varstragic comedy>22Slide23

Chombo Makefiles Variables of interestDIM?=2DEBUG?=TRUE# OPT=FALSEPRECISION?=DOUBLEMPI?=FALSECXX?=g++FC?=g77MPICXX?=mpiCC23

USE_64?=TRUE #64bit pointers

USE_MT?=TRUE #memory tracking

USE_COMPLEX=TRUE #type for

complexnumbers

USE_TIMER?=TRUE #

Chombo::Timer

USE_HDF?=TRUE #HDF5 file i/o

?= special assignment, only if not already set.Slide24

HDF5http://www.hdfgroup.org/ftp/HDF5/current/src/building from the source code is the best option.  download the source filehdf5-1.8.7.tar.gzcd /usr/local

sudo

tar

xzf

hdf5-1.8.7.tar.gz

cp -

r

hdf5-1.8.7 hdf5-1.8.7.parallel

cd

hdf5-1.8.7sudo mkdir buildcd buildsudo ../configure --enable-production --prefix=/usr/local/hdf5-1.8.7 --with-default-api-version=v16sudo make allsudo make testsudo make installcd /usr/local/hdf5-1.8.7.parallelsudo mkdir buildcd buildexport CC=mpic++sudo ../configure --enable-parallel --enable-production --prefix=/usr/local/hdf5-1.8.7.parallel --with-default-api-version=v16sudo make allsudo make testsudo make install your mpi compiler might have a different name on your system.

24Slide25

Writing out an HDF5 file from ChomboChombo/lib/src/AMRIO.HvoidWriteAMRHierarchyHDF5(const string& filename, const Vector<DisjointBoxLayout>& a_vectGrids, const Vector<

LevelData

<

FArrayBox

>* > &

a_vectData

,

const Vector<string>&

a_vectNames

, const Box& a_domain, const Real& a_dx, const Real& a_dt, const Real& a_time, const Vector<int>& a_vectRatio, const int& a_numLevels);There are several variations on this function for special uses.Do not use the write[FAB|Level|etc] functions in your code.Those are for debugging. To be discussed later.25Slide26

Command Line Make>cvs checkout Chombo>cd Chombo/lib/mk>ln –s local/Make.defs.leopard Make.defs.local>cd ..>make –j4 all DIM=3 MPI=TRUE OPT=HIGH

>

ls

libboxtools3d.Darwin.64.mpic++.gfortran.DEBUG.OPTHIGH.MPI.a

libbasetools3d.Darwin.64.mpic++.gfortran.DEBUG.OPTHIGH.MPI.a

.

26