Managing Multi-repository Projects with mepo - PowerPoint Presentation

1. Managing Multi-repository Projects with mepoMatt ThompsonPurnendu ChakrabortyTom Clune

2. OutlineBackgroundmepoOriginsBasic descriptionExamplesStatusQuestions?

3. Background: How to manage multiple subrepos?A strong desire in Git was to keep the GEOS nested folder structure as similar as possible to aid in transition, but still take advantage of separate repositoriesSeveral approaches considered, including combinations ofGit submodulesGit subrepoGit sparse checkoutecbuildUsed by JEDI Does not really allow us to nest the repositories as we wantedNo good way (or any way?) to just checkout the code without building itLearned of NOAA’s document for software managementMultiple nested reposUses manage_externals

4. manage_externals 101Simple Python config script with one block per sub projectURL for repositoryTag/branchDirectory mount pointPros:Easy to understand config syntaxConfigs can be nestedBuilt-in status utility to see changes across multiple reposSupport for git sparse-checkout (added by SI Team) and submodulesCons:Cannot modify the sandboxMost git actions must be done with git in individual repos (checkout, commit, …)Configs can be nested (multiple control points)Locked into either SSH or HTTPS URL[MAPL]required = Truerepo_url = git@github.com:GEOS-ESM/MAPL.gitlocal_path = ./src/Shared/@MAPLtag = v2.3.4protocol = git

5. Deployment under manage_externalsShort story – everything “worked”ProblemsTestingExisting tests almost exclusively system GCM testsEsp. problematic for conventional CI approachesMulti-repository reviews and mergesRequire coordinated PR’s that must be accepted as a whole by multiple gatekeepersAs config files are nested, releases of sub-repos must be done firstPeer-to-peer sharing of codeTagging all repos results in tag pollutionOrdinary git workBranch, commit, push set of modified repos2019-07-19: ~17 Repositories 4 fixtures 10 main repos 3 forksToday: ~44 Repositories 7 fixtures 23 main repos 14 forks

6. mepo originsScientists wanted a broader suite of multi-repo capabilitiesCommit, push, …Specific workflow for managing sub-community feature branches across reposInitial skepticism: “If it were that easy …”Late 2019: Collected use cases and did some storyboardingDebate: extend ME vs. starting over (no claim of wisdom in our choice)Jan. 2020: P. Chakraborty delivered a clean initial implementationPython 3 with YAML config filesMulti-repo “snapshots” (including initial clone) captured in hidden filesMostly a superset of ME capabilities (no support for SVN)Extensible – easy to add additional multi-repo git commandsSupplemented by mepo-cd bash script that allows easy navigation to subprojects

7. mepo: current statusRequirements: Python 3.6+ and PyYAMLmepo currently provides the following commands (italics: commands not in git, * has subcommands)When mirroring git commands, some common flags supported (e.g., checkout -b)clonestatusdifffetchcheckoutbranch*commitpushinitstagetag*stash*listcheckout-if-existsdevelopcomparewhereisunstagesaverestore-statefetch-allpull-all

8. Example mepo YAML configurationGMAO_Shared: local: ./src/Shared/@GMAO_Shared remote: ../GMAO_Shared.git tag: v1.3.3 sparse: ./config/GMAO_Shared.sparse develop: mainMAPL: local: ./src/Shared/@MAPL remote: ../MAPL.git tag: v2.3.4 develop: developFMS: local: ./src/Shared/@FMS remote: ../FMS.git tag: geos/2019.01.02+noaff.1 develop: geos/release/2019.01mepo is controlled by a yaml file with certain allowed keys:local: location to clone the repo within the fixtureremote: URL of remote repository (allows "relative" URLs like git submodule)tag/branch: tag/branch to clone (ends in detached head state)sparse: points to a config file that controls sparse-checkout of repodevelop: records the development branch of repo for use in mepo develop commandrecurse_submodules: clones repo with submodules

9. parallel_build.cshMany/Most developers of GEOS use parallel_build.csh to build and not much changesCurrent State of parallel_build (GCM, LDAS): defaults to manage_externalsRunning parallel_build.csh uses checkout_externals to checkout multi-repo following Externals.cfg-develop flag uses Develop.cfg (develop branch in GEOSgcm_GridComp and GEOSgcm_App)Running parallel_build.csh -mepo uses mepo -develop flag runs mepo develop GEOSgcm_GridComp and GEOSgcm_AppProposed Future Behavior: deprecate manage_externals, use only mepoThe parallel_build.csh -mepo behavior will become the default and the -mepo flag will be (eventually) removedRemove Externals.cfg/Develop.cfgThere might be changes as ADAS workflow developsMight need to support different -develop behaviorPerhaps deprecate -develop flag?

10. git clone, mepo init and mepo cloneA mepo-enabled repository is cloned by git clone, then initialized by using mepo init and then subrepos cloned with mepo clone

11. One Step: mepo cloneBut you can do this all-in-one with a single command: mepo clone

12. mepo statusmepo status provides useful information about the state of your checkout:What tag (t) or branch (b) it's onIf you've changed branches from the original state, it will highlight the repoProvides a useful, verbose output of file status in every repo

13. mepo developThe components.yaml file contains information on the development branches of each repository. With mepo, you can checkout the development branches with mepo develop repo1 repo2

14. mepo checkoutAs in regular git, you now checkout a branch for your changes.You run, mepo checkout –b branch repo1 repo2 which creates the branch a la git checkout –b (aka, git branch followed by git checkout)

15. mepo diffWe can now make our changes on our branches and want to see how they compare.The mepo diff command will provide a git diff of each repository. It also has a similar '--name-only' optionThere is also a '--staged' option that will show staged diffs

16. mepo stageThe process of staging changes is much like that of git, but instead of git add, we use mepo stage repo1 repo2To stage new files, you use mepo stage --untracked.There is also mepo unstage which unstages files

17. mepo commitCommitting with mepo is done with the mepo commit command. Like git, if you pass it -m "message" it uses that message. If you just run mepo commit, then $EDITOR will launch and the message can be entered there. In each case, the message is used for all repositories in the mepo commit command

18. mepo pushPushing changes to GitHub/GitLab/etc. is done with mepo pushmepo push will always do the equivalent to git push -u origin branch so the branch will be tracked

19. mepo saveOne command unique to mepo is mepo save. This command essentially updates the "default" state of the multi-repo. The current branches are considered the "default" ones and a new YAML file is created to reflect this.

20. mepo checkout-if-existsmepo has a unique command that lets you checkout branches on multiple repos: mepo checkout-if-exists. This command will run on all reposIt has a --dry-run flag as well so you can see what repos will be changed before you actually checkout the branches

21. mepo whereis and mepo-cdmepo whereis is a command that tells you where a subrepo is relative to your current possibleWe have also developed a helper function/alias called mepo-cd which can "travel around" the multi-repo using mepo whereis information

22. Deprecating manage_externalsThe SI Team plans to deprecate manage_externals support in GEOS "soon"Removing Externals.cfg files in reposRemoving support in parallel_build.cshWe'll work with GCM, LDAS, and CTM Gatekeepers on removing it The upcoming "5.27.1" ADAS Git tag will be mepo only from the startOur other minor fixtures, e.g., GEOSfvdycore, have been mepo only for a whileNote: This will not affect older tags, just future tags will not have Externals.cfg support

23. Issues and ConcernsConfig files are not nestable (good and bad)No support for SVN merges: should they be handled via mepo? (how to resolve multi-repo conflicts?)Needs better help in wiki and in command lineWhat subcommand flags should be added (probably per request)When/How to support new git commands either on command line (git switch, git restore) or in the backend (git sparse-checkout)Experiment CaptureNeed to figure out how to capture and reconstruct possibly non-committed changes a la Larry's CVS method  mepoFigure out where to actually host these?

24. Questions

25. ReferencesGit flow: https://nvie.com/posts/a-successful-git-branching-model/Manage externals: https://github.com/ESMCI/manage_externalsMepo: https://github.com/GEOS-ESM/mepo