erhtjhtyhy DOUG BENJAMIN ANL HEP Doug Benjamin Problem Droids send a REQUESTJOB message to Yoda Yoda would responded with a NEWJOB message to each Droid Sometimes the message wo ID: 853984
Download Pdf The PPT/PDF document "UPDATE ON YODIDROID MPI ISSUES" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
![[Project Name] STEERING COMMITTEE UPDATE](https://thumbs.docslides.com/724393/project-name-steering-committee-update.jpg)
1 UPDATE ON YODI/DROID MPI ISSUES erhtjhty
UPDATE ON YODI/DROID MPI ISSUES erhtjhtyhy DOUG BENJAMIN ANL â HEP Doug Benjamin Problem - ï§ Droids send a REQUEST_JOB message to Yoda ï§ Yoda would responded with a NEW_JOB message to each Droid ï§ Sometimes the message would get through and other times it would not. â Task dependent ï§ This behavior was not seen in all of the 1M event
2 validation jobs sent to Yoda/Droid ï§
validation jobs sent to Yoda/Droid ï§ Until Monday the default message buffer size was 1M bytes. ï§ Messages larger than 1 M bytes were not delivered whether or not they were non - blocking (default) or blocking messages. ï§ As part of debugging sent the message to the log file and discovered the actual message size. ï§ Why are we sending
3 such large messages? 2 Doug Benjamin Cau
such large messages? 2 Doug Benjamin Cause - ï§ Very large NEW_JOB message size 12 MB ï§ Analysis by Tadashi â ï§ Here is the list of attributes and their sizes. 3459039B prodDBlocks 3093179B realDatasetsIn 1696259B dispatchDblock 1230619B GUID 1130839B inFiles 498899B ddmEndPointIn 399119B checksum 365859B scopeIn 355881B inFilePaths
4 332599B fsize 166299B dispatchDBlockT
332599B fsize 166299B dispatchDBlockToken 791B jobPars Note â Yoda needs very little of this information - 3 Doug Benjamin Fixes ï§ Short term patch â in the file yoda_droid.py â https://github.com/PanDAWMS/panda - yoda/blob/master/pandayoda/yoda_droid.py Change Line 92 â from : mpi_default_message_buffer_size = 1000000 To: mpi_d
5 efault_message_buffer_size = 20000000 ï
efault_message_buffer_size = 20000000 ï§ Proper Patch by Tadashi â One problem is that the same information about one file is appended several times, e.g., which happened due to nEventsPerJob nEventsPerFile . I've fixed the panda server. The sizes of prodDBlocks ~ dispatchDBlockToken can be reduced by 1/10. 4 Doug Benjamin Fixes(2) ï§ Harv
6 ester Patch by Tadashi â âI'
ester Patch by Tadashi â âI've added stripJobParams to shared_file_messenger in the git repo. You need in panda_queues.json â "messenger": { ... " stripJobParams ": true ï§ Harvester updated at NERSC and ALCF ï§ Testing has begun at ALCF 5 Doug Benjamin Additional Yoda changes ï§ Need to determine why changes to Yoda config file
7 (default buffersize change) did not t
(default buffersize change) did not take. ï§ Need to accurately report the size of messages in the log files. New code needed to handle the nested nature of the messages. (code to be added today). 6 Doug Benjamin Conclusion ï§ Problem solved ï§ NERSC and ALCF producing events â ï§ Now time to measure the efficiency (events per wall clo