Tutorial MODEL 2009 4 th May 2009 Installation of the AliEn software AliEn and the GRID Authentication File Catalogue Submitting jobs Outline Installation Please do w ID: 588900
Download Presentation The PPT/PDF document "AliEn" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
AliEn Tutorial
MODEL 20094th May, 2009Slide2
Installation of the AliEn
softwareAliEn and the GRID
Authentication
File CatalogueSubmitting jobs
OutlineSlide3
InstallationPlease, do:wget http://alien.cern.ch/alien-torrent-installerchmod +x alien-torrent-installer
./alien-torrent-installerDefault installationFirst two commands fastLet’s leave the 3rd running...Slide4
AliEn, AliEn2ALICE ENvironment on the GRIDGRID middleware
Developed as single entry point to the GRID for ALICE (CERN)All the components necessary to build a GRID and interact with other GRIDsFile System with metadataAuthorization, authentication, job optimization and execution, storage managementAudit, quotas, monitoringInterfaces to various GRID implementationsUsed since:
2002 for centrally managed productions
2006 for user analysisUsed by several Virtual Organizations:ALICE, PANDA, CBM, GPCALMAhttp://alien.cern.ch Slide5
Pablo Saiz5CERNSlide6
Pablo Saiz6LHC
27km particle accelerator100 meters undergroundAbout to start collecting dataSlide7
Pablo Saiz7A Large
Ion Collider ExperimentSlide8
Why do we need a catalogue?Loads of data (several PB per year!!)Distributed communitySame structure from everywhereContains location of the fileIt does not contain the filePossibility of several copies of filesSelect the closest one
Metadata informationWe will see this later on...Slide9
AliEn2 File CatalogueMapping from LFN to SE and PFNUNIX-like file systemMetadata
GUIDUsed by all other services
Tier1
|--./
| |--
cern.ch
/
| | |--user/
| | | |--a/
| | | | |--admin/
| | | | |
| | | | |--
aliprod
/
| | | |
| | | |--f/
| | | | |--
fca
/
| | | |
| | | |--p/
| | | | |--
psaiz
/
| | | | | |--as/
| | | | | |
| | | | | |--dos/
| | | | | |
| | | | | |--local/
|--simulation/
| |--2001-01/
| | |--V3.05/
| | | |--Config.C
| | | |--grun.C
| |--36/
| | |--stderr
| | |--stdin
| | |--stdout
| || |--37/| | |--stderr| | |--stdin| | |--stdout| || |--38/| | |--stderr| | |--stdin| | |--stdout
| | | || | | |--b/| | | | |--barbera/
ALICE USERS
ALICE DATA
ALICE
REMOTESlide10
LFN and GUID catalogues
/
/alice
/alice/user/p/psaiz
/alice/simulation/2006
…
Index
1-JAN-1970
1-JAN-2006
14-FEB-2007
23-AUG-2008
…
Index
GUID
PFN
LFN Catalogue
GUID Catalogue
AliEn File & Metadata Catalogue
LFN
GUID
LFN: Logical File Name (what the user sees)
GUID: Global Unique Identifier (another way of identifying files)
PFN: Physical File NameSlide11
Who uses the catalogue?EverybodyIF A FILE IS NOT IN THE CATALOGUE, IT DOES NOT EXIST ON THE GRIDUsersUser data, user executables ...Job executionExecutable, input, outputPackages
Source code, binary code, post installationSlide12
Back to the exercises...Did the installation finish?Let’s get ready to access the GRIDX509 Certificate authenticationPublic vs Private keyCertificate AuthorityProxy certificatesSlide13
Preparing certificatesIn $HOME/.globus:userkey.pemusercert.pemReadable only by the user (chmod 400)
Make alien use those files:mkdir $HOME/.alienln –s $HOME/.globus $HOME/.alien/globus Slide14
Environment files$ALIEN_ROOT/.Environment$HOME/.alien/EnvironmentBash files sourced before starting aliencat <<EOF > $HOME/.alien/Environment export ALIEN_ORGANISATION=PANDA
export ALIEN_USER=<username> EOFPut alien in the PATHmkdir –p $HOME/binln –s $HOME/alien/bin/alien $HOME/bin/Slide15
alien proxy-init, alien proxy-infoGetting a GRID proxy
*****Slide16
Globus related:
Permissions on $HOME/.globus/userkey.pem are not private to the user – chmod 400 userkey.pemYour certificate authority is exotic and not known to the server.
Your certificate has expired.
Clock skew:Your local computer time is in the future with respect to the server's time.
Your local computer time is more in the past than the certificate life time.
Authentication problemsSlide17
Check the certificates in your machines:
Store them under e.g: $HOME/.globus/
Check that the key file is read
onlySetup the environment$HOME/.alien/EnvironmentPut alien in the path
Get a valid alien token
a
lien proxy-init
Try
it
out !!Slide18
Standard bash shell with grid comands
Main bash features are available.
Not all shell helper programs are available.
Some local commands (like ls, cat etc) are overwritten
with the corresponding GRID commands.
File/path tab completion in the virtual GRID directory.
Accessing
the shellSlide19
Tab completion working!!!
Basic commands ISlide20
Basic commands IISlide21
whereis command
Gives (list of) SE and pfns for a fileSlide22
Viewing the files Slide23
Shell – Copying files (add/get)Slide24
Access the alien shell.
alienCheck your user name:
whoami
List the contents of your home directoryls
Print the working directory
pwd
Create the following directory structure
in your
AliEn
space
:
$HOME/bin (if it is not there)
$HOME/macros and $HOME/
jdl
Get the information of the file:
whereis
/bin/date
Shell – Try it out ISlide25
See the file /bin/date
cat /bin/date
Copy the file to your local machine
cp /bin/date /tmp/my_fileExit the shell and check the file Shell – Try it out IISlide26
Job & Transfer models
Jobs
TaskQueue
Job BrokerJob ManagerCE
JA
Central services
Site services
Splitting
Expired
Priorities
Merging
Zombies
Job optimizers
File catalogue
LFN
GUID
Meta
data
FTD
Packman
MonALISA
JA
JA
CE
SE
Packman
MonALISA
FTD
SE
Site A
Site B
Transfer
Manager
TransferQueue
Transfer
Broker
Transfer optimizers
fts
bbftp
xrdcpSlide27
ALICE sites
Europe
Asia
North AmericaAfrica
See
Costin’s
presentationSlide28
Running jobs
More than 25 CPU/years per daySlide29
AliEn
AliEn user interface
VTD
EDG
LCG /GLITE
CONDOR
ARC/
NORDUGRID
Nice
!
I STILL do
not
have
to
worry
about
ever
changing
GRID
environment
…
Interconnecting GRIDsSlide30
Job statusSlide31
Y
our job is
submitted and
a <JOBID>.is assigned to it.Submitting jobsSlide32
Checking the job statusSlide33
Getting the outputSlide34
Other useful commandsjobListMatchList of CE that can execute the jobSpyOutput of the job while it is runningMasterjobGive information about Slide35
Connect to alien
alien loginSubmit the job
submit <<EOF
Executable=“date”;EOF
Follow the execution
top –id,
ps
trace,
ps
jdl
Get the output
cp /proc/<user>/<id>/job-output/
stdout
/
tmp
/out
Try
it
out!!Slide36
Executable: Compulsory field where we give the lfn of the executable that should be stored in /bin or $V0/bin or $HOME/bin.
Arguments: They will be passed to the executable.
Packages:
Type packages in the shell to see what kind of packages are installed.InputFile: The files that will be transported to the node where the job will run.InputData: It will require that the job will be executed in a site close to the files specified here.
InputDataList:
The filename in which the Job Agent will write the InputData list.
InputDataListFormat:
The format of the InputData list.
JDL fields ISlide37
OutputFile: The files that will be registered in the catalog once the job finishes.
OutputArchive: What files will be archived in a zip file.
Validationcommand:
Specifies the script to be used as a validation script.Email: Receive a mail when the job finishes.TTL: The maximum run time of your job.
Split:
Split the jobs in several sub jobs.
JDL fields IISlide38
Packagespackmangives a list of the packages defined/installedinstalls packagesSlide39
Add:Packages=“pbarprod@pandaroot::nov08“;Try other commands:Executable=“pandaroot”;Multiple submissionSplit=“production:1-5”;Create your own executables
Try it out !!Slide40
ConclusionAliEnInterface to the GRIDDeveloped by ALICEAlso used by PANDA, CBM, GPCALMAIn production since 2002
File CatalogueFile and MetadataJob submission modelData movement and replicationSlide41
Registration – Certificates:
http://alien.cern.ch/twiki/bin/view/Alice/UserRegistration
https://ca.cern.ch/ca/
AliEn: http://alien.cern.ch
Gshell:
http://alien.cern.ch/twiki/bin/view/AliEn/GAPI
User's guide:
http://project-arda-dev.web.cern.ch/project-arda-dev/alice/apiservice/AA-UserGuide-0.0m.pdf
References ISlide42
aliensh Grid Command Online Reference V1.0
http://project-arda-dev.web.cern.ch/project-arda-dev/alice/apiservice/guide/guide-1.0.htm
Previous tutorials:
http://aliceinfo.cern.ch/Offline/Analysis/Tutorial/Event Tag System:
http://pcaliweb02.cern.ch/Offline/Analysis/RunEventTagSystem/EventTags.html#Event%20tag%20system
https://edms.cern.ch/document/788315/1
(INTERNAL NOTE)
References IISlide43
http://alien.cern.ch
Any questions?
Thank you very much!!